NETWORK CONTROL METHOD AND DATA PROCESSING SYSTEM

Abstract

The present invention relates to a network control method and a data processing system, configured so as to make it possible to effectively perform the process of transferring a large amount of data. In the present invention, a network connection device transmits event data that indicates the extent of change in sensor data at a prescribed time to a data processing device, the sensor data being original data generated by a sensor device, and the event data being transmitted before the original data. The network connection device transmits the original data to the data processing device on the basis of a request for the original data from the data processing device. This technology is applicable to, for example, a network control method for cloud computing.

Description

[Technical field]

[0001] The present disclosure relates to a network control method and a data processing system, and more particularly to a network control method and a data processing system that enable efficient transfer processing of large amounts of data. [Background technology]

[0002] The present applicant has previously proposed an object detection system using a combination of a synchronous image sensor and an asynchronous DVS in Patent Document 1. The synchronous image sensor is a sensor that captures an image in synchronization with a vertical synchronization signal and outputs frame data, which is image data for one frame (screen), at the period of the vertical synchronization signal. DVS is an abbreviation for Dynamic Vision Sensor, and is a sensor that treats a change in pixel luminance as an event and outputs event data indicating the occurrence of an event when the event occurs. Since the DVS outputs event data at the timing when the event occurs without relying on the vertical synchronization signal, it can be said to be an asynchronous (or address control) image sensor.

[0003] The event data generated by the DVS must be available without delay, so the time granularity is extremely fine. If the large amount of data generated by the DVS is indiscriminately injected into the network, the network may collapse if there is a capacity limit, and the truly necessary data may not be processed correctly. On the other hand, if excessive computing resources are always reserved in preparation for large amounts of data, excessive costs will be incurred. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2020 / 195769 Summary of the Invention [Problem to be solved by the invention]

[0005] Therefore, a new method of controlling data transfer to a network is required to ensure that urgent or high-priority data is processed reliably.

[0006] The present disclosure has been made in consideration of such circumstances, and makes it possible to efficiently transfer and process large amounts of data. [Means for solving the problem]

[0007] A network control method according to a first aspect of the present disclosure includes a network connection device that transmits event data indicating a change in sensor data at a predetermined time point, the sensor data being original data generated by a sensor device, to a data processing device prior to the original data, and transmits the original data to the data processing device based on a request for the original data from the data processing device. The event data includes at least observation data identification information for identifying data of an observation target to be notified as the event data, a reference address of the original data, and an event type, the event type being either an update including the data of the change, or a notify notifying only that a change has occurred without including the data of the change, and the reference address of the original data is a reference address of the original data before the change at the specified time point is applied when the event type is the update, and is a reference address of the original data after the change at the specified time point is applied when the event type is the notify. . A data processing system according to a first aspect of the present disclosure includes a network connection device that transmits event data indicating a change in sensor data, which is original data generated by a sensor device, at a predetermined time point to a data processing device before the original data, and transmits the original data to the data processing device based on a request for the original data from the data processing device, wherein the event data includes at least observation data identification information that identifies the observed data to be notified as the event data, a reference address of the original data, and an event type, wherein the event type is either an update that includes the change data, or a notify that does not include the change data and notifies only that a change has occurred, and when the event type is the update, the reference address of the original data is a reference address of the original data before the change in the predetermined time point is applied, and when the event type is notify, it is a reference address of the original data after the change in the predetermined time point has been applied.

[0008] A network control method according to a second aspect of the present disclosure includes a network connection device transmitting event data indicating a change in sensor data at a predetermined point in time, the sensor data being original data generated by a sensor device, to a data processing device prior to the original data, transmitting the original data to the data processing device based on a request for the original data from the data processing device, and, when transmitting the event data, transmitting an event path request requesting the securing of a virtual path with the data processing device. A data processing system according to a second aspect of the present disclosure includes a network connection device that transmits event data indicating a change in sensor data at a predetermined time point, the event data being original data generated by a sensor device, to a data processing device prior to the original data, and transmits the original data to the data processing device based on a request for the original data from the data processing device. When transmitting event data, the network connection device transmits an event path request requesting the securing of a virtual path with the data processing device. .

[0009] In the first and second aspects of the present disclosure, event data indicating a change in sensor data at a predetermined point in time, which is original data generated by a sensor device, is transmitted to a data processing device prior to the original data, and the original data is transmitted to the data processing device based on a request for the original data from the data processing device. Also, in a first aspect, the event data includes at least observation data identification information that identifies the observed data to be notified as the event data, a reference address of the original data, and an event type, and the event type is either an update that includes the data of the change, or a notify that does not include the data of the change and notifies only that a change has occurred, and the reference address of the original data is configured to be the reference address of the original data before the change at the specified time is applied when the event type is the update, and to be the reference address of the original data after the change at the specified time has been applied when the event type is the notify. In the second aspect, when the event data is transmitted, an event pass request is transmitted to request securing of a virtual path with the data processing device.

[0010] In addition, the first aspect of this disclosure and the second How to control the network aspects and The data processing system can be realized by causing a computer to execute a program. The program to be executed by the computer can be provided by transmitting the program via a transmission medium or by recording the program on a recording medium.

[0011] The data processing system may be an independent device or may be an internal block constituting a single device. [Brief description of the drawings]

[0012] [Figure 1] 1 is a block diagram showing a configuration example of a first embodiment of a data processing system to which the present disclosure is applied; [Diagram 2] FIG. 2 is a diagram illustrating event data that an event producer generates and transfers into a network. [Diagram 3] FIG. 11 is a diagram illustrating an example of data adopted as a topic ID. [Figure 4] FIG. 11 is a diagram illustrating an example of data adopted as a topic ID. [Diagram 5] FIG. 11 is a diagram illustrating an example of data adopted as a topic ID. [Figure 6] FIG. 11 is a diagram illustrating an example of data adopted as a topic ID. [Figure 7] 11 is a flowchart illustrating transmission control of event data and original data. [Figure 8] 13 is a flowchart illustrating another example of the transmission control of the event data and the original data. [Figure 9] FIG. 11 is a diagram illustrating a process of summarizing event data. [Figure 10] FIG. 11 is a diagram illustrating a process of summarizing event data. [Figure 11]FIG. 13 is a diagram illustrating a data format of event data when the event type is an update. [Figure 12] 13 is a flowchart of transmission control including a process of notifying an event producer of a topic ID. [Figure 13] FIG. 13 is a diagram illustrating an example of data of a topic registration request. [Figure 14] 11 is a flowchart illustrating transmission control of event data and original data when the event type is "notify." [Figure 15] FIG. 13 is a diagram showing a data format of event data when the event type is "notify." [Figure 16] 13 is a diagram illustrating a process of grouping event data when the event type is "notify." FIG. [Figure 17] 13 is a diagram illustrating a process of grouping event data when the event type is "notify." FIG. [Figure 18] This is an expanded format of the data format in FIG. [Figure 19] 11 is a flowchart illustrating transmission control of event data and original data when individualized data is obtained. [Figure 20] 13 is a flowchart illustrating a process of reserving an event pass. [Figure 21] FIG. 13 is a diagram showing details of an event pass request. [Figure 22] 13 is a flowchart illustrating a process of reserving an event pass. [Diagram 23] 13 is a flowchart illustrating a process of reserving an event pass. [Figure 24] FIG. 13 illustrates a modified example of an event pass request. [Diagram 25] FIG. 1 is a diagram illustrating a transport stack configuration that constitutes an event path. [Figure 26] 11 is a flowchart of event data transmission control including event class negotiation. [Figure 27] 13 is a flowchart of a transmission control of event data including a negotiation for changing a parameter. [Figure 28] 13 is a flowchart of a transmission control of event data including a negotiation for changing a parameter. [Figure 29] FIG. 11 is a block diagram showing a configuration example of a second embodiment of a data processing system to which the present disclosure is applied. [Diagram 30] 30 is a flowchart illustrating transmission control of event data and original data by the data processing system of FIG. 29. [Diagram 31] 13 is a flowchart of an event class change control for negotiating an event class. [Diagram 32] 13 is a flowchart of an event class change control. [Diagram 33] 13 is a flowchart showing an example of control of immediate transfer and delayed transfer. [Diagram 34] FIG. 1 is a block diagram showing an example of the configuration of cloud computing. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, a mode for carrying out the present disclosure (hereinafter, referred to as an embodiment) will be described with reference to the attached drawings. In this specification, the description of "and / or" means that both "and" and "or" are possible. In addition, in this specification and the drawings, components having substantially the same functional configuration are given the same reference numerals to avoid redundant explanation. The description will be given in the following order. 1. First embodiment of data processing system 2. Event data description 3. Transmission control of event data and original data 4. Event data processing 5. Identifying Data Based on Topic ID 6. Transmission control when the event type is Notify 7. Event data processing 8. Event Pass Reservation and Allocation 9. Event Class Negotiation 10. Second embodiment of data processing system 11. Transmission control of event data and original data 12. Event Class Negotiation 13. Immediate and delayed transfers 14. Cloud Computing Configuration Examples

[0014] <1. First embodiment of data processing system> FIG. 1 is a block diagram showing a configuration example of a first embodiment of a data processing system to which the present disclosure is applied.

[0015] The data processing system 1 in FIG.

[0016] The sensor 11 is a sensor device that detects some state and supplies sensor data, which is the detection result, to the event producer 12. Examples of the sensor 11 include an acceleration sensor, a gyro sensor, a magnetic sensor, an odor sensor, an air pressure sensor, a temperature sensor, a humidity sensor, a wind speed sensor, a light sensor (such as an RGB sensor or an IR sensor), a GPS sensor, and the like, which are used as IoT (Internet of Things) sensors.

[0017] The event producer 12 is an application that transfers data acquired by the sensor 11 to the network. The event producer 12 uses the data supplied from the sensor 11 as original data (hereinafter also referred to as original data or simply original) and generates changes to the original data after a certain point in time as event data. Generally, the amount of event data is very small compared to the original data.

[0018] For example, if the sensor 11 is an image sensor (RGB sensor) that receives RGB light, the original data is a captured image obtained by imaging, and the event data can be luminance data indicating a change in luminance value from a previously captured image. Also, for example, if the sensor 11 is a sensor device that detects the state of a product DB that stores the prices of multiple products, the product price itself is the original data, and the change in price is the event data.

[0019] The data processing system 1 also includes an event consumer 14 and an event path manager 15 .

[0020] The event producers 12 transmit the original data and the event data to the event consumers 14 via the event paths 13. The event paths 13 are virtual paths (communication paths) in the network that are assigned by the event path manager 15 for providing the original data and the event data to the event consumers 14.

[0021] The event consumer 14 is an application that uses the data acquired by the sensor 11. The event consumer 14 performs predetermined data processing, such as data analysis processing and recognition processing, using the original data and event data transmitted from the event producer 12 via the event path 13. The event consumer 14 can select the event data and original data required by itself from the data transferred on the network by referring to the topic DB 17.

[0022] In response to a request from the event producer 12 , the event path manager 15 secures a virtual path that connects the event producer 12 and the event consumer 14 from among the routes within the network, and assigns it as the event path 13 .

[0023] The data processing system 1 further includes a topic manager 16 and a topic DB (DataBase) 17.

[0024] The topic manager 16 assigns a topic ID and notifies the event producer 12. A topic represents data to be observed that the event producer 12 notifies as event data, and a topic ID is identification information that identifies the topic. The topic manager 16 also assigns an event class to each topic ID that it assigns, and stores the event class in the topic DB 17. The event class indicates the priority of the virtual path when the event path manager 15 assigns the event path 13. The event class is represented, for example, by a class ID, which is identification information that identifies the class. For example, a priority is predetermined for each class ID, such as class ID 25 being a high priority class and class ID 35 being a low priority class, and the event class is specified by the class ID.

[0025] Before transferring event data, the event producer 12 stores in advance in the topic DB 17 a pair of the topic ID notified by the topic manager 16 and topic description information (Topic Description) that explains the topic. The event producer 12 also requests the event path manager 15 to allocate an event path 13 by specifying an event class.

[0026] The topic DB 17 stores, for each topic, a pair of a topic ID and topic description information registered by the event producer 12. The event class of each topic ID registered by the topic manager 16 is also stored in the topic DB 17 in association with the topic ID. The topic DB 17 is referenced by the event consumer 14 to select the necessary event data.

[0027] The event path 13 connecting the event producer 12 and the event consumer 14 in Fig. 1 is one P2P (Peer to Peer) connection that transmits original data and event data based on sensor data acquired by the sensor 11. Although not shown in the figure, the data processing system 1 has multiple sensors 11, and for each of the multiple sensors 11, an event producer 12 that transfers the sensor data within the network and an event consumer 14 that uses the sensor data are connected by P2P. The event path manager 15 assigns the event path 13 for each P2P connection. The topic manager 16 manages topic IDs for all topics transferred within the network. The topic DB 17 stores information on all topics transferred within the network.

[0028] As described above, various sensor devices can be used as the sensor 11 of the data processing system 1 in Fig. 1, but the following description will be given taking as an example a case where the sensor 11 is an extended DVS consisting of a synchronous image sensor (FIS) and an asynchronous DVS. The synchronous image sensor is a sensor that captures an image in synchronization with a vertical synchronization signal and outputs frame data, which is image data for one frame (screen), at the period of the vertical synchronization signal. The DVS is a sensor that treats a change in pixel luminance as an event and asynchronously outputs event data indicating the occurrence of the event according to the timing of the event occurrence.

[0029] A brief explanation of DVS is given below.

[0030] A DVS is a sensor that has pixels that photoelectrically convert an optical signal to output a pixel signal, and outputs a temporal luminance change of the optical signal as an event signal (event data) based on the pixel signal. Such an event sensor is also called an EVS (event-based vision sensor). A synchronous image sensor captures an image in synchronization with a vertical synchronization signal and outputs frame data, which is image data of one frame (screen), at the period of the vertical synchronization signal, whereas a DVS outputs event data only at the timing when an event occurs, and therefore can be said to be an asynchronous (or address-controlled) camera. In the following, a synchronous image sensor that outputs frame-based image data at a predetermined period (frame rate) is called an FIS to distinguish it from a DVS.

[0031] In the DVS, for example, a voltage signal corresponding to the logarithmic value of the amount of light received by each pixel is detected as a pixel signal. When the logarithmic luminance change value represented by the pixel signal exceeds a predetermined threshold c and changes to brighter, the DVS outputs “+1” which indicates a positive luminance change, and when the logarithmic luminance change value represented by the pixel signal exceeds the predetermined threshold c and changes to darker, the DVS outputs “−1” which indicates a negative luminance change.

[0032] The event data is represented, for example, in the following format called the AER (Address-Event Representation) format. e = (x, y, p, t) ·········(1)

[0033] In equation (1), x, y represent the coordinates of the pixel where a luminance change occurs.

[0034] The time t of an event is a timestamp that indicates the time when the event occurred, and is expressed, for example, by the count value of a counter based on a predetermined clock signal in a sensor. The timestamp corresponding to the timing when an event occurred can be said to be time information that indicates the (relative) time when the event occurred, as long as the interval between events is maintained as it was when the events occurred.

[0035] The polarity p represents the direction of the luminance change when a luminance change (change in light amount) exceeding a predetermined threshold c occurs as an event, and represents whether the luminance change is a plus change (hereinafter also referred to as positive) or a minus change (hereinafter also referred to as negative). For example, the polarity p of an event is represented by "+1" when it is positive and "-1" when it is negative.

[0036] Here, the time series sequence of N events detected by the DVS at a pixel (x, y) in the two-dimensional coordinate space e n = {x, y, p n , t n} (n=1,2,3,...,N (x,y) Let L be a luminance image. Assuming there is no noise, the initial luminance value, that is, the luminance value of pixel (x, y) in the initial state before any events occur, is expressed as L. 0 Let (x, y) be the logarithmic luminance threshold for a positive or negative event to occur, so if p is positive, the logarithmic luminance increases by c, and if p is negative, the logarithmic luminance decreases by c.

[0037] In this case, time t n-1 Luminance image of L n-1 (x, y) and time t n Events n Using this, time t n Luminance image of L n (x, y) can be calculated using the following equation (2). L n (x, y) = L n-1 (x, y) * exp(c) (p n > 0) L n (x, y) = L n-1 (x, y) * exp(-c) (p n < 0) (2)

[0038] However, in practice, as the logarithmic luminance change c becomes smaller, the probability that the DVS picks up noise increases, so some kind of noise correction is required.

[0039] As described above, the DVS outputs only the position coordinates, polarity, and time information of the pixel where a change in brightness was detected. Because only the net change (difference) of position coordinates, polarity, and time information is generated and output, and there is no redundancy in the amount of information in the data, the DVS sensor has a high time resolution on the order of μsec. Because the amount of information is small, it consumes less power than a frame-based image sensor, and when processing data, there is no unnecessary processing load and the processing time can be shortened. Because it is possible to output data at high speed and with low latency, it is possible to obtain the exact time when an event occurred.

[0040] The sensor 11, which is an extended DVS including an FIS that outputs frame data and a DVS that outputs event data, may be in a form in which the FIS and the DVS are adjusted so that they have the same imaging range within one device, or may be configured such that the FIS and the DVS are provided as different devices and arranged adjacent to each other and adjusted so that they have the same imaging range. Also, one sensor in which each pixel can output both the event data of the DVS and the image frame data of the FIS may be used. An example of a sensor in which each pixel can output both the event data of the DVS and the image frame data of the FIS is the DAVIS (Dynamic and Active-pixel Vision Sensor) sensor disclosed in "Brandli et al. A 240x180 130dB 3us latency global shutter spatiotemporal vision sensor, IEEEJSSC, 2014".

[0041] The sensor 11 receives the original data at time t n-1 Luminance image of L n-1 (x, y) and time t n Luminance image of L n Event e corresponding to the luminance change from (x, y) n= { x, y, p n , t n} to the event producer 12.

[0042] <2. Event data description> Next, with reference to FIG. 2, the event data that the event producer 12 transfers into the network will be described.

[0043] As shown in Fig. 2, the event data includes at least a topic ID, an original reference, and an event type, and is transferred to an event path 13 secured in the cloud 21 (network). The event path 13 is assigned by the topic manager 16 according to the topic ID, and is a path according to the event class.

[0044] The topic ID is observation data identification information that identifies the observation target data to be notified as event data.

[0045] The event type indicates the type of event data, and stores either an update, which includes data on changes, or a notify, which does not include data on changes and only notifies that a change has occurred.

[0046] In the case where the event type is update, the original reference (Original Ref) stores a reference address that refers to the original data before the luminance change represented by update is applied. On the other hand, in the case where the event type is notify, the original reference stores a reference address that refers to the updated original data after the change event (luminance change) represented by notify occurs. In the case where the sensor 11 is an extended DVS, the original data is a luminance image L n , i.e., the luminance image L nIt is a set of all pixels, including pixel (x, y) of (x, y). Therefore, when the event type is update, the original data is n The luminance image L before the luminance change is applied. n-1 When the event type is notify, the original data is n The luminance image L after the luminance change n It becomes.

[0047] By default, the event class is set by the topic manager 16 according to the topic, i.e., the observed data to be notified as event data. However, depending on the type of sensor 11, the event class may be set in advance as a fixed one, for example, when a priority specific to the sensor 11 is specified. Alternatively, the event class may be set by some negotiation with the event consumer 14 to which the data is transferred. Regardless of the method by which the event class is set, the event class may be changed according to the traffic conditions within the network, etc.

[0048] 3 to 6 show examples of data that may be used as topic IDs.

[0049] Fig. 3 shows an example in which a sensor ID (sensor identification information) that globally uniquely identifies the sensor 11 is adopted as a topic ID. In the example of Fig. 3, the sensor ID "Sensor-ID-1" is used as the topic ID as is.

[0050] FIG. 4 shows an example in which a combination of a sensor ID that globally uniquely identifies the sensor 11 and an ROI-ID (object identification information) that identifies a specific object region ROI (Region of Interest) within an image that is sensor data is adopted as a topic ID.

[0051] In the example of Fig. 4, the sensor ID is "Sensor-ID-1", and "ROI-ID-1" and "ROI-ID-2" are assigned to identify the object region ROI for each of the two objects, a car and a person, in the image generated by the sensor 11. For the car object, "Sensor-ID-1 / ROI-ID-1" is set as the topic ID, and for the person object, "Sensor-ID-1 / ROI-ID-2" is set as the topic ID. The ROI-ID for identifying the object region ROI only needs to be unique within the sensor.

[0052] FIG. 5 shows an example in which an object ID (global object identification information) that globally uniquely identifies an object detected in an image, which is sensor data, is adopted as a topic ID.

[0053] In Fig. 5, a globally unique "Object-ID-1" is assigned to a car object in an image generated by the sensor 11, and a globally unique "Object-ID-2" is assigned to a person object. This object ID is information that can identify a target object across multiple sensors 11. For a car object, "Object-ID-1" is used as the topic ID, and for a person object, "Object-ID-2" is used as the topic ID.

[0054] FIG. 6 shows an example in which a query ID (query identification information) for identifying a query for the data to be observed is adopted as a topic ID.

[0055] The query ID for identifying a query for the data to be observed is set by the event consumer 14 (or the event subscriber 116 in FIG. 29) which is the party using the data.

[0056] For example, when the event consumer 14 sets a query such as "the trajectory of a man around 60 years old who is behaving suspiciously at an ATM near Osaki Station" as a query to identify an event, a query ID, which is query identification information for identifying the query, is assigned by the topic manager 16 and notified to the event consumer 14 and the sensor 11. The query ID is determined to be globally unique, independent of the sensor 11. In the example of Fig. 6, "Query-Token-ID-1" is assigned as the query ID of the above-mentioned query, and is used as the topic ID.

[0057] <3. Transmission control of event data and original data> Next, the transmission control of the event data and original data acquired by the sensor 11 will be described with reference to the flowchart of FIG.

[0058] In the explanation from Figure 7 onwards, a specific pixel (x, y) among the multiple pixels of the sensor 11 will be described, but all of the controls described below can be performed on the entire luminance image L by applying pixel (x, y) to each of all pixels of the sensor 11.

[0059] First, in step S1, the sensor 11 executes “Capture Original”. Specifically, the sensor 11 captures the original data at time t n-1 Luminance image L n-1 Get (x, y).

[0060] In step S2, the sensor 11 executes “Detect Update & Send Update”. n Luminance image L n (x, y) is detected and time t n-1 Luminance image of L n-1 (x, y) and time t n Luminance image of L n Event data e showing the luminance change with (x, y) n = { x, y, pn , t n} at time t n-1 Luminance image of L n-1 (x, y) to the event producer 12. The event producer 12 receives the event data e n and the luminance image L n-1 Get (x, y).

[0061] In step S3, the event producer 12 executes "Generate Update Event" and in step S4 executes "Send Update Event". Specifically, the event producer 12 generates event data and transmits it to the event consumer 14. In this process, the event e n However, it is sent as is to the event consumer 14 as event data.

[0062] The processes in steps S2 to S4 may be repeatedly executed multiple times as necessary.

[0063] In step S5, the event consumer 14 executes "Request Original." Specifically, the event consumer 14 receives the event data e n The original reference (Original Ref) included in the request is specified to request the original data from the event producer 12.

[0064] When the event producer 12 receives a request for original data from the event consumer 14, it executes “Send Original” in step S6. That is, the event producer 12 sends the original data at time t n-1 Luminance image of L n-1 Send (x, y) to the event consumer 14.

[0065] In step S7, the event consumer 14 receives the time t n-1 Luminance image of L n-1 (x, y) and executes “Update Original & Process Updated Original”. Specifically, the event consumer 14 obtains the updated original at time t n Luminance image of L n If no luminance change occurs at pixel (x, y), the luminance image L is restored with c = 0. n (x, y) is L n (x, y) = L n-1 The result is (x, y) * 1.

[0066] In addition, the event consumer 14 is updated with the luminance image L n A predetermined application process, such as image rendering or analysis, is performed using (x, y).

[0067] The above steps S1 to S7 are repeatedly executed. The event producer 12 erases the original data and event data transmitted in the past at an appropriate timing, such as after a certain time has elapsed since the transmission of the original data.

[0068] According to the above-described control of transmission of event data and original data, by transmitting event data and obtaining original data as necessary, network traffic can be reduced and large amounts of data can be efficiently transferred and processed.

[0069] The transmission control described with reference to Figure 7 is an example in which the event producer 12 transmits original data after receiving a request from the event consumer 14, but there is also a method of unilateral transmission, such as transmitting periodically at a fixed interval.

[0070] FIG. 8 is a flowchart showing another example of the transmission control of the event data and the original data, which is a transmission control in the case where the original data is transmitted periodically.

[0071] In this process, first, in step S21, the sensor 11 executes "Capture Original." That is, the sensor 11 captures the original data at time t n-1 Luminance image L n-1 Get (x, y).

[0072] In step S22, the sensor 11 executes “Detect Update & Send Update”. n Luminance image L n (x, y) is detected and time t n-1 Luminance image of L n-1 (x, y) and time t n Luminance image of L n Event data e showing the luminance change with (x, y) n = { x, y, p n , t n} at time t n-1 Luminance image of L n-1 The event producer 12 notifies the event data e together with (x, y). n and the luminance image L n-1 Get (x, y).

[0073] In step S23, the event producer 12 executes “Send Original.” That is, the event producer 12 sends the original data at time t n-1 Luminance image of L n-1 (x, y) to the event consumer 14. The event consumer 14 receives the luminance image L n-1 Receive (x, y).

[0074] The event producer 12 executes "Generate Update Event" in step S24, and executes "Send Update Event" in step S25. The processes in steps S24 and S25 are similar to the processes in steps S3 and S4 in FIG. 7, and the time t n Events n However, it is sent as is to the event consumer 14 as event data.

[0075] The processes in steps S22 to S25 may be repeatedly executed multiple times as necessary.

[0076] In step S26, the event consumer 14 receives the time t n Event data for n and executes "Update Original & Process Updated Original". That is, the event consumer 14 obtains the n Event data for n and the time t obtained earlier n-1 Luminance image of L n-1 Using (x, y), the time t n Luminance image of L n (x, y). The event consumer 14 then performs a process to recover the updated luminance image L n A predetermined application process, such as image rendering or analysis, is performed using (x, y).

[0077] The event producer 12 erases the original data and event data transmitted in the past at an appropriate timing, such as after a certain time has elapsed since the transmission of the original data.

[0078] The above-mentioned processes in steps S21 to S26 are repeatedly executed. If the event consumer 14 receives duplicate original data, the duplicate original data is discarded.

[0079] <4. Event data processing> In the data transmission control described with reference to FIG. 7, in the "Generate Update Event" and "Send Update Event" executed by the event producer 12, the process of transferring the event data from the sensor 11 to the event consumer 14 as is is executed.

[0080] However, in reality, performing the above-mentioned transmission control for each pixel would increase traffic, so the event producer 12 consolidates a certain amount of update data and transmits it to the event consumer 14, or the event consumer 14 consolidates a certain amount of update data and then requests the original data.

[0081] That is, as shown on the left side of FIG. 9, the "Generate Update Event" process executed by the event producer 12 can include a consolidation process ("Accumulate Update") that consolidates multiple event data for luminance changes, which are update data.

[0082] Alternatively, as shown on the right side of FIG. 9, the "Request Original" process executed by the event consumer 14 can include a consolidation process ("Accumulate Update") for consolidating multiple event data for luminance changes, which are update data.

[0083] There are five possible levels (hereinafter referred to as "summary levels") at which the event producer 12 or the event consumer 14 completes the process of summarizing update data to a certain extent: (1) to (5).

[0084] The summarization level (1) is a level where, in any case, net event data from which noise has been removed is sent as soon as it occurs. According to this summarization level (1), the event producer 12 transfers the acquired event data as is, as shown in FIG.

[0085] The grouping level (2) is a level in which event data for multiple pixels is grouped together until it reaches a value equal to or greater than a predetermined threshold value and then transmitted.

[0086] The synthesis level (3) is a level in which images are synthesized until the area of ​​the object of the subject (rectangular area or edge area) can be extracted and then transmitted.

[0087] The summary level (4) is a level in which the subject objects are summarized until they can be recognized (understood or classified) and determined to be the subject of observation, and then transmitted.

[0088] The synthesis level (5) is a level where even one subject object can be recognized as a block (a group of pixels), and the images are synthesized until the object's trajectory can be tracked (it starts moving), and then transmitted.

[0089] There are various other variations depending on the level and combination of AI processing. These summary levels (1) to (5) can be set for each topic ID.

[0090] The process of consolidating multiple event data can also be divided between the event producer 12 side and the event consumer 14 side.

[0091] (Case 1) in FIG. 10 shows an example of transmission control in which the event producer 12 aggregates one or more event data at any of the aggregation levels (1) to (5) described above, and then performs “Generate Update Event” and “Send Update Event.”

[0092] Specifically, in step S41, the event producer 12 aggregates one or more event data at any of the aggregation levels (1) to (5) described above. Then, in step S42, the event producer 12 executes "Generate Update Event," and in step S43, executes "Send Update Event." In step S44, the event consumer 14 executes "Request Original," requesting the original data from the event producer 12.

[0093] FIG. 10 (Case 2) shows an example of transmission control in which the event consumer 14 aggregates multiple event data at any one of the aggregation levels (2) to (5) described above, and then makes a "Request Original" request.

[0094] Specifically, when the event producer 12 acquires event data from the sensor 11 in step S51, and update data occurs at one pixel, the event producer 12 executes "Generate Update Event" in step S52, and executes "Send Update Event" in step S53. In step S54, the event consumer 14 receives the event data from the event producer 12, and aggregates multiple event data at any of the above-mentioned aggregation levels (2) to (5). Then, in step S55, the event consumer 14 executes "Request Original" to request original data from the event producer 12.

[0095] FIG. 10 (Case 3) shows an example of transmission control in which the event consumer 14 aggregates multiple event data at any one of the aggregation levels (3) to (5) described above, and then makes a "Request Original."

[0096] Specifically, in step S61, the event producer 12 aggregates one or more event data at the above-mentioned aggregation level (1) or (2). Then, in step S62, the event producer 12 executes "Generate Update Event", and in step S63, executes "Send Update Event". In step S64, the event consumer 14 receives the event data from the event producer 12 and aggregates multiple event data at any of the above-mentioned aggregation levels (3) to (5). Then, in step S65, the event consumer 14 executes "Request Original" to request the original data from the event producer 12.

[0097] For example, at time t 1 Luminance image L 1 After (x, y) is acquired by the event producer 12, the event consumer 14 receives the 2 or time t 11 10 event data 2 Or e 11 Then, when the event consumer 14 executes “Request Original” to request the original data from the event producer 12, the event producer 12 sends the original data at time t 1 Luminance image L 1 (x, y) is sent as the original data.

[0098] FIG. 11 shows the data format of the event data when the event type is an update.

[0099] As described with reference to Figure 2, the event data includes a topic ID, an original reference, and an event type, and if the event type is an update, one or more event data are further stored as update data.

[0100] In the topic ID, for example, a sensor ID, a combination of a sensor ID and an ROI-ID, an object ID, a query ID, etc. are stored. In the example of Fig. 11, "Sensor-ID-1" which is a sensor ID that globally uniquely identifies the sensor 11 is stored.

[0101] The event type stores "Update."

[0102] As specific data of the update data, one or more event data e are stored in the AER format of the above-mentioned formula (1).

[0103] Alternatively, the plurality of event data e may be stored in a compressed AER format obtained by compressing the plurality of event data e in the AER format.

[0104] This section explains the Compressed AER format.

[0105] Time t at the same coordinates (x, y) n-k+1 , …, t n-2 , t n-1 , t n k event data e n-k+1 , …, e n-2 , e n-1 , e n Putting it all together, we can express it as (x, y, p n-k+1 , …, p n-2 , p n-1 , p n , t n-k+1 , …, t n-2 , t n-1 , t n ) where the time over these k events is Δt = t n -t n-k+1If k is too large, it increases the delay, so the time Δt and the number k are dynamically determined so as to minimize the delay as much as possible and within the range of the time resolution requirement of the event required by the application (event consumer 14).

[0106] The k event data e in this time range Δt n-k+1 , …, e n-2 , e n-1 , e n If q pieces of data with positive polarity p occur and r pieces of data with negative polarity p occur, the change in logarithmic luminance over the time range Δt is (q - r) = s. Therefore, according to the Compressed AER format, compressed-e(x, y, s, t n ) can be expressed as

[0107] In this way, when it is acceptable to make the time resolution coarse within the range of time Δt, the number of events to be transferred can be reduced by compressing multiple event data e in the Compressed AER format. Note that the range of time Δt is divided into multiple parts, and compressed-e(x, y, s, t n ) compressed data may be stored in multiple pieces.

[0108] The event data also stores information ("Update Format") indicating whether the update data to be stored is in the AER format or the Compressed AER format.

[0109] The original data acquired by the event consumer 14 executing “Request Original” is n-k+1 Luminance image L n-k+1 (x, y), and the compressed event data obtained in the Compressed AER format is compressed-e(x, y, s, t n ), then at time t n Luminance image L nAssuming there is no noise, (x, y) can be calculated using the following formula. L n (x, y) = L n-k+1 (x, y) * exp(c*s)

[0110] Using batching to batch and transmit event data at a predetermined batch level can further reduce network traffic and allow large amounts of data to be transferred efficiently.

[0111] <5. Identifying data based on topic ID> Next, identification of data of the event consumer 14 based on the topic ID will be described.

[0112] As described with reference to Fig. 11, the event data is transferred including a topic ID. As described with reference to Figs. 3 to 6, the topic ID is composed of, for example, a sensor ID, a combination of a sensor ID and an ROI-ID, an object ID, a query ID, etc.

[0113] When the topic ID is configured with a sensor ID, the acquired event data is all the event data emitted by that sensor 11 .

[0114] When the topic ID is a combination of a sensor ID and an ROI ID, or an object ID, the event data emitted by the sensor 11 can be identified in units of an object region ROI or object in an image.

[0115] Therefore, the event consumer 14 can refer to the topic DB 17 and extract and process only the event data required for itself based on the topic ID. That is, the event consumer 14 can look at the event data received by itself, confirm only the event data that it really needs, obtain the original data by "Request Original", apply the updated data, and recover the latest original data. For example, by applying only the event data related to the movement of an object that interests the event consumer 14 in the overall image of a certain sensor 11, it becomes possible to track the latest trajectory of only the target object.

[0116] A pair of a topic ID and topic description information is stored for each topic in the topic DB 17. When the topic ID is configured with a sensor ID, the topic description information stores various attribute information of the sensor 11, for example, location information such as the latitude and longitude of the sensor 11, and information such as the shooting direction (view port).

[0117] When a topic ID is composed of a combination of a sensor ID and an ROI-ID, when the sensor 11 recognizes the object region ROI, information about the object region ROI, such as the region position and size, and the type of object (person, car, etc.), is stored in the topic DB 17 as topic description information.

[0118] When the topic ID is composed of an object ID, the topic manager 16 manages the topic DB 17. Specifically, the topic manager 16 assigns a globally unique ID to the object as an object ID, and stores the topic ID and topic description information in the topic DB 17. The topic manager 16 also notifies the event producer 12 connected to the sensor 11 capturing the object in question to store the object ID assigned by the topic manager 16 in the topic ID of the event data related to the object.

[0119] Even when the topic ID is configured with a query ID linked to a query for the data to be observed, the topic manager 16 manages the topic DB 17. Specifically, the topic manager 16 treats a search query from the event consumer 14 (for example, "the trajectory of a man around 60 years old who is behaving suspiciously at an ATM near Osaki Station") as topic description information, assigns a query ID, and stores it in the topic DB 17. The topic manager 16 also notifies the event producer 12 connected to the sensor 11 capturing the object corresponding to the query to store the query ID assigned by the topic manager 16 in the topic ID of the event data related to the object.

[0120] In this way, by making it possible to assign a topic ID to each sensor 11 or each object to be observed within one sensor 11, for example, the event consumer 14 can determine areas of no interest in the overall brightness image L from the sensor 11 based on topic description information in the topic DB 17, and determine parts that do not need to be updated to the latest state based on the topic ID. This makes it possible to eliminate unnecessary update processing.

[0121] Also, after acquiring all event data emitted by the sensor 11, only the necessary objects may be extracted based on the topic ID, but the event consumer 14 may notify the event producer 12 of the necessary topic IDs in advance, causing the event producer 12 to transfer only the event data related to the necessary objects. This makes it possible to improve the efficiency of the event transfer itself and reduce traffic.

[0122] FIG. 12 shows a flowchart of transmission control including a process in which the event consumer 14 notifies the event producer 12 of a required topic ID in advance.

[0123] In this process, first, in step S81, the event consumer 14 executes a topic registration process ("Register Topic") to notify the event producer 12 of a required topic ID. The event producer 12 receives the topic ID notified from the event consumer 14.

[0124] The processes in steps S82 to S85 are respectively similar to steps S1 to S4 in the flowchart of Fig. 7. However, the difference is that the event producer 12 transmits to the event consumer 14 only the event data of the topic ID notified by the event consumer 14 among the event data acquired from the sensor 11.

[0125] 7, after step S85, when the event producer 12 receives a "Request Original" from the event consumer 14, the event producer 12 transmits the original data to the event consumer 14. The event consumer 14 executes a process to restore the luminance image L, and performs a predetermined application process using the updated luminance image L.

[0126] FIG. 13 shows an example of data of a topic registration request ("Topic Registration Request") sent from the event consumer 14 to the event producer 12 in the topic registration process in step S81.

[0127] 13, the event consumer 14 can notify the level of the summarization process to be performed by the event producer 12, that is, summarization levels (1) to (5), along with the topic ID corresponding to the object, etc., that the event consumer 14 needs, as a topic registration request. When multiple topics are assigned to event data generated by one sensor 11, the summarization level can be specified for each topic.

[0128] <6. Transmission control when the event type is Notify> In the above explanation, the process when the event type is "update" has been explained. Next, the process when the event type is "notify" will be explained.

[0129] For example, when the content of the update change is very large, the event producer 12 transmits event data with an event type of "notify" and notifies the event consumer 14 only of the fact that an update has occurred.

[0130] 14 is a flowchart for explaining the control of transmission of event data and original data when the event type is "notify." This process corresponds to the process in FIG. 7 when the event type is "update."

[0131] First, in step S101, the sensor 11 executes "Capture Original." Specifically, the sensor 11 captures the original data at time t n-1 Luminance image L n-1 Get (x, y).

[0132] In step S102, the sensor 11 executes “Detect Update & Send Update”. n Luminance image L n (x, y) is detected and time t n-1 Luminance image of L n-1 (x, y) and time t n Luminance image of L n Event data e showing the luminance change with (x, y) n = { x, y, p n , t n} at time t n-1 Luminance image of L n-1 The event producer 12 notifies the event data e together with (x, y). n and the luminance image L n-1 Get (x, y).

[0133] In step S103, the event producer 12 executes "Generate Notify Event," and in step S104, executes "Send Notify Event." Specifically, the event producer 12 generates event data whose event type is "notify," and transmits it to the event consumer 14. The event data when the event type is "notify" will be described later.

[0134] The processes in steps S102 to S104 may be repeatedly executed multiple times as necessary.

[0135] In step S105, the event consumer 14 executes "Request Updated Original." Specifically, the event consumer 14 requests the event producer 12 to send the updated original data to which the update changes have been applied.

[0136] When the event producer 12 receives a request for updated original data from the event consumer 14, the event producer 12 executes "Send Updated Original" in step S106. Specifically, the event producer 12 sends the updated original data at time t n Luminance image of L n Send (x, y) to the event consumer 14.

[0137] In step S107, the event consumer 14 receives the updated original data at time t n Luminance image of L n (x, y) and executes “Process Updated Original”. That is, the event consumer 14 obtains the updated time t n Luminance image of L n A predetermined application process, such as image rendering or analysis, is performed using (x, y).

[0138] Fig. 15 shows the data format of the event data when the event type is "notify", which corresponds to the data format of the event data in Fig. 11 when the event type is "update".

[0139] If the event type is "notify", the event data includes a topic ID, an original reference, and an event type.

[0140] In the topic ID, for example, a sensor ID, a combination of a sensor ID and an ROI-ID, an object ID, a query ID, etc. are stored. In the example of Fig. 15, "Sensor-ID-1" which is a sensor ID that globally uniquely identifies the sensor 11 is stored.

[0141] The original reference (Original Ref) contains the reference address of the updated original data. In the example of the processing in Figure 14, the address is the time t n Luminance image of L n The reference address corresponding to (x, y) is stored.

[0142] "Notify" is stored as the event type.

[0143] As described above, when the event type is Notify, the event data of the update change (Event-Type='Update') is not transmitted to the event consumer 14, and only the updated original data is transmitted to the event consumer 14.

[0144] According to the above-mentioned control of sending event data and original data when the event type is "notify", the event producer 12 sends event data indicating the occurrence of a change, and the event consumer 14 retrieves the original data as necessary. This reduces network traffic and enables efficient transfer and processing of large amounts of data.

[0145] <7. Event data processing> The process of grouping event data when the event type is "notify" will be described.

[0146] If the event type is "notify", the event data for the update change is not sent to the event consumer 14, and therefore the event consumer 14 cannot perform the consolidation process. Therefore, when the consolidation process is performed, the process of consolidating one or more event data at any of consolidation levels (1) to (5) is performed only on the event producer 12 side, as shown in Fig. 16. In other words, among (Case 1), (Case 2), and (Case 3) in Fig. 10 when the event type is "update", only the process corresponding to (Case 1) is valid.

[0147] FIG. 17 shows an example of transmission control for performing collective processing when the event type is "notify."

[0148] Specifically, in step S121, the event producer 12 summarizes one or more event data at any of the summary levels (1) to (5) described above. Then, in step S122, the event producer 12 executes "Generate Notify Event" and in step S123 executes "Send Notify Event". In step S124, the event consumer 14 executes "Request Updated Original" to request the original data from the event producer 12.

[0149] Here, when the level of the consolidation process is (3) to (5) among the consolidation levels (1) to (5) described above, that is, when the data is consolidated into individualized data that is individualized according to the requirements of the application by AI processing or the like, it is possible to pass the individualized data directly to the event consumer 14, instead of the updated original data.

[0150] FIG. 18 shows a data format of event data that is an extension of the data format shown in FIG. 15 when the event type is “notify,” and that enables individualized data to be passed to the event consumer 14 in addition to the updated original data.

[0151] In this data format, in addition to the topic ID, original reference, and event type, a personalized data reference (Transformed Original Ref) is stored.

[0152] The individualization data reference stores a reference address of the individualization data. Specifically, when the summary level is level (3), a reference address of the individualization data representing a rectangular area of ​​the subject object or an edge area of ​​the object is stored. When the summary level is level (4), a reference address of the individualization data representing the content recognition or classification of the subject object is stored. When the summary level is level (5), a reference address of the individualization data representing the trajectory of the subject object is stored.

[0153] FIG. 19 is a flowchart for explaining the transmission control of event data and original data when the event consumer 14 specifies individualized data and acquires it from the event producer 12.

[0154] First, in step S141, the sensor 11 executes "Capture Original." That is, the sensor 11 captures the original data at time t n-1 Luminance image Ln-1 Get (x, y).

[0155] In step S142, the sensor 11 executes “Detect Update & Send Update”. n Luminance image L n (x, y) is detected and time t n-1 Luminance image of L n-1 (x, y) and time t n Luminance image of L n Event data e showing the luminance change with (x, y) n = { x, y, p n , t n} at time t n-1 Luminance image of L n-1 The event producer 12 notifies the event data e together with (x, y). n and the luminance image L n-1 Get (x, y).

[0156] In step S143, the event producer 12 executes "Generate Notify Event," and in step S144, executes "Send Notify Event." Specifically, the event producer 12 generates event data with the event type set to "Notify," and transmits it to the event consumer 14. This event data is transmitted in the data format of FIG.

[0157] The processes in steps S142 to S144 may be repeatedly executed multiple times as necessary.

[0158] In step S145, the event consumer 14 executes "Request Transformed Original." Specifically, the event consumer 14 requests the individualized data from the event producer 12 by specifying the reference address of the individualized data reference (Transformed Original Ref).

[0159] When the event producer 12 receives the request for personalized data from the event consumer 14, the event producer 12 executes "Send Transformed Original" in step S146. Specifically, the event producer 12 sends personalized data according to the consolidation level to the event consumer 14.

[0160] In step S147, the event consumer 14 acquires the personalized data transmitted from the event producer 12, and executes "Process Transformed Original." That is, the event consumer 14 performs a predetermined application process using the acquired personalized data.

[0161] As described above, the event consumer 14 can obtain the individualized data itself and perform application processing, or, as in the case where the event type is an update, can obtain the updated original data and perform application processing.

[0162] By batching and transmitting event data at a predetermined batch level using batch processing, network traffic can be further reduced and large amounts of data can be efficiently transferred.

[0163] <8. Event Pass Reservation and Allocation> Next, the reservation and allocation of event passes will be described.

[0164] FIG. 20 is a flowchart showing the process of reserving an event path performed by the event path manager 15. As shown in FIG.

[0165] In this process, first, in step S161, the event producer 12 generates an event path request for requesting the reservation of an event path, and transmits it to the event path manager 15 (Request Event Path).

[0166] In step S162, the event path manager 15 receives the event path request from the event producer 12, and reserves an event path between the event producer 12 and the event consumer 14 based on the event path request.

[0167] FIG. 21 shows details of an event path request sent from an event producer 12 to the event path manager 15.

[0168] The event pass request includes an event class and a parameter.

[0169] In the event class, for example, a class ID that is identification information for identifying a class (priority) is stored.

[0170] The parameters include items that indicate the quality of data transfer, such as QoS Class, Bitrate, and Start / Stop / Duration. QoS Class includes an ID that specifies the (set) transport QoS class, such as delay range, maximum jitter, and maximum error rate. Bitrate includes bandwidth bps, guaranteed bitrate GBR (Guaranteed BitRate), and maximum bitrate MBR (Maximum BitRate). Start / Stop / Duration includes the absolute start time and absolute end time for using the path (including the specification of immediate use, where resources are used as soon as they are allocated), duration, etc. By reserving network resources in advance using this event path request, it becomes possible to predict the maximum delay, etc., before transferring the event data.

[0171] The event path manager 15 dynamically reserves network resources in the network between the event producers 12 and the event consumers 14 to enable the establishment of an event path that meets the conditions indicated in the event path request.

[0172] FIG. 22 is a flowchart for explaining the process of securing an event path, including the timing of securing the event path shown in FIG. 20 and the timing of transmitting the event data in FIG.

[0173] In the example of Fig. 22, the event producer 12 executes "Generate Update Event" in step S181, and then transmits an event path request (Request Event Path) to the event path manager 15 to request securing of an event path in step S182. The process of step S181 is the same as the process of step S3 in Fig. 7, and the process of step S182 is the same as step S161 in Fig. 20.

[0174] Then, in step S183, the event path manager 15 secures an event path between the event producer 12 and the event consumer 14 based on the event path request. After the event path is secured, an ACK is sent from the event path manager 15 to the event producer 12 together with the specified event class.

[0175] When the event producer 12 receives the ACK from the event path manager 15, in step S184, it executes "Send Update Event" to send the event data to the event consumer 14. In "Send Update Event", the event data may be sent with an event class added.

[0176] As described above, the event producer 12 may transmit the event data first, rather than waiting for an ACK from the event path manager 15 before transmitting the event data. In this case, if the event path manager 15 fails to secure an event path, the event producer 12 returns a NAK to the event data.

[0177] The above-mentioned process in FIG. 22 is a process for securing a necessary event path every time a "Generate Update Event" occurs on the event producer 12 side.

[0178] Alternatively, the event path manager 15 may reserve a certain number of event paths at all times before the event producer 12 executes a series of "Generate Update Event" and sequentially assign the necessary event paths from among them. When reserving event paths in advance, they are reserved based on the results of estimating the network traffic between the event producer 12 and the event consumer 14 based on the past statistical data of the sensor 11, etc.

[0179] FIG. 23 is a flowchart illustrating the process of reserving an event path when an event path is reserved in advance and then assigned.

[0180] In the example of FIG. 23, in step S201, the event path manager 15 estimates the network traffic between the event producer 12 and the event consumer 14, and secures an event path between the event producer 12 and the event consumer 14.

[0181] In step S202, the event producer 12 executes "Generate Update Event" and then in step S203 sends an event pass request to the event pass manager 15.

[0182] In step S204, the event path manager 15 reserves an event path between the event producer 12 and the event consumer 14 from among the event paths reserved in advance, and transmits an ACK to the event producer 12 together with the specified event class.

[0183] In step S205, the event producer 12 receives the ACK from the event path manager 15 and executes “Send Update Event” to send the event data to the event consumer 14.

[0184] As described above, if event paths are reserved in advance and allocation is performed from among them, the overhead of establishing network resources that may occur in sequential reservation can be reduced.

[0185] <Event Pass Request Variation> When the event producer 12 sends an event pass request to the event pass manager 15 in step S182 of FIG. 22 or step S203 of FIG. 23, the request may be sent in the format shown on the left side of FIG.

[0186] The event pass request shown on the left side of FIG. 24 is a modified example of the event pass request shown in FIG.

[0187] Comparing the event path request in Figure 24 with the event path request in Figure 21, a topic ID and event type have been added. The event path manager 15 that receives the event path request in Figure 24 shares the topic ID and event type with the event consumer 14 when securing an event path between the event producer 12 and the event consumer 14. As a result, when the event producer 12 executes "Send Update Event" to transmit event data to the event consumer 14, it is only necessary to store and transmit the update data (event data e), as shown on the right side of Figure 24, thereby minimizing the amount of data for each event data item.

[0188] <Transport stack that configures the event path> Next, the transport stack configuration that constitutes the event path will be described.

[0189] The stack configuration when transferring event data through an event path of a user plane on an optical network system is as shown in FIG.

[0190] The transport stack is composed of, from the bottom up, the Fiber Layer (space division multiplexing (SDM) or mode division multiplexing (MDM) within a single fiber), the WDM Layer (wavelength division multiplexing (WDM)), and the TDM Layer (time division multiplexing (TDM)). Above that, as higher-level transport layers, it is composed of an IP packet layer or a non-IP layer.

[0191] An event path is basically established in a connection-oriented manner. That is, when the connection is set up, a virtual path is formed (network resources are secured) between the sender and receiver by GMPLS (Generalized Multi-Protocol Label Switch). In the above example, the event producer 12 is the sender and the event consumer 14 is the receiver, and a virtual path that satisfies each of the requirements is secured in accordance with the event class (delivery requirements such as priority) of the event path request sent by the sender. To secure the virtual path, the Resource reSerVation Protocol (RSVP)-Traffic Engineering (TE) extension for GMPLS, which is exchanged on the control plane, is used.

[0192] Note that while Figure 25 shows an example of a stack configuration on an optical network system, an event path can also be configured by securing a virtual path that satisfies specified transfer requirements in other stack configurations on a wireless network system.

[0193] <9. Event class negotiation> When the event path manager 15 is reserving or allocating an event path, if it is unable to reserve an event path that meets the requirements specified in the event path request from the sending event producer 12, or if the inability to reserve an event path continues, the event path manager 15 negotiates with the event consumer 14 to adjust the event class.

[0194] FIG. 26 shows a flowchart of event data transmission control including event class negotiation.

[0195] In step S221, the event path manager 15 performs an event path reservation or allocation process and determines whether an event path that satisfies the requirements specified in the event path request cannot be reserved or whether such a state continues.

[0196] If it is determined in step S221 that the event path cannot be secured or that this condition will continue, processing proceeds to step S222, and the event path manager 15 negotiates (requests) with the event consumer 14 to change the event class, for example, by asking whether it is acceptable to change the value of the QoS Class parameter of the event path request (Negotiate Delivery Requirements).

[0197] If, in step S222, consent is obtained in negotiation with the event consumer 14 to change the QoS Class value, in step S223 the event path manager 15 sends an event class change instruction (Change Qos Class) to the event producer 12 to relax the delay constraint and change the QoS Class value (Modify Event Class).

[0198] The negotiation between the event path manager 15 and the event consumer 14 may be performed whenever an event class change is required, or may be performed automatically when the event path manager 15 has learned the adaptation requirements to a certain extent. The event consumer 14 side can specify whether the event path manager 15 must perform sequential negotiation or whether it will automatically determine this before or during the transfer of event data.

[0199] The series of processes from step S224 to S228 is the same as step S2 in FIG. 7 and steps S181 to S184 in FIG. 22 or steps S202 to S205 in FIG. 23 described above, and therefore will not be described.

[0200] Also, for example, if an event path that meets the requirements specified in the event path request cannot be secured, or if this condition continues, the output frequency of the event data may be reduced by changing the parameters of the sensor 11.

[0201] FIG. 27 shows a flowchart of the transmission control of event data including negotiation for changing the parameters of the sensor 11.

[0202] In step S241, the event path manager 15 performs an event path reservation or allocation process and determines whether an event path that satisfies the requirements specified in the event path request cannot be reserved or whether such a state continues.

[0203] If it is determined in step S241 that the event path cannot be secured or that such a condition will continue, the process proceeds to step S242, and the event path manager 15 negotiates with the event consumer 14 to request whether the parameters of the sensor 11, for example, the threshold c of the logarithmic luminance change, may be changed (Negotiate Delivery Requirements).

[0204] If consent is obtained in negotiation with the event consumer 14 in step S242 to change the threshold c of the logarithmic luminance change, then in step S243 the event path manager 15 sends an instruction to reduce the bit rate (Reduce Bitrate) to the event producer 12 (Modify Event Class).

[0205] When the event producer 12 receives the instruction to reduce the bit rate, in step S244 it notifies the sensor 11 of an instruction to increase the log-intensity change threshold c (Increment log-intensity threshold c).

[0206] When the sensor 11 receives an instruction from the event producer 12 to increase the threshold c of the logarithmic luminance change, the sensor 11 makes a change to increase the threshold c of the logarithmic luminance change. This reduces the occurrence frequency of events detected in the processing of steps S245 to S249. The processing of steps S245 to S249 is the same as step S2 in Fig. 7 described above, steps S181 to S184 in Fig. 22, or steps S202 to S205 in Fig. 23, and therefore a description thereof will be omitted.

[0207] Conversely, if the event path manager 15 detects that network traffic has improved, it may increase the frequency with which event data is output.

[0208] FIG. 28 shows a flow chart of the event data transmission control including negotiation for changing the parameters of the sensor 11 in a direction to improve the bit rate.

[0209] In step S261, the event path manager 15 determines whether the network traffic has improved.

[0210] If it is determined in step S261 that the network traffic has improved, processing proceeds to step S262, in which the event path manager 15 sends an instruction to increase the parameters of the sensor 11, for example, the bitrate (Increase Bitrate), to the event producer 12 (Modify Event Class).

[0211] When the event producer 12 receives the instruction to increase the bit rate, in step S263 it notifies the sensor 11 of an instruction to decrease the log-intensity change threshold c (Decrement log-intensity threshold c).

[0212] This increases the detection sensitivity, and changes the event class to meet the initial requirements specified in the event pass request. After step S263, the same processes as steps S245 to S249 in Fig. 27 are executed, and the event is detected using the changed logarithmic luminance change threshold c, and the event data is transmitted.

[0213] <10. Second embodiment of data processing system> The data processing system 1 according to the first embodiment described above has a configuration in which the event producer 12 and the event consumer 14 are configured with a P2P (Peer to Peer) connection.

[0214] However, it is also assumed that there are many event consumers 14 and the sensor data generated by the sensor 11 is transferred to many (multiple) event consumers 14. In the second embodiment, the configuration of the data processing system 1 in such a case will be described. Note that in the second embodiment, it is not necessarily required that there be multiple event consumers 14, and it goes without saying that the case where there is only one event consumer 14 is also included.

[0215] FIG. 29 is a block diagram illustrating a configuration example of a second embodiment of a data processing system to which the present disclosure is applied.

[0216] In the data processing system according to the second embodiment, in order to reduce the application execution load on the sensor devices, a broker that mediates data distribution to the cloud is introduced, and a distribution system is configured using a publish / subscribe model.

[0217] Here, the publish / subscribe model is a type of asynchronous messaging paradigm in which the message sender (publisher) is programmed to send a message without assuming a specific recipient (subscriber). Published messages are divided into topics and published. Publishers have no knowledge of subscribers. Subscribers specify topics of interest and receive only messages that belong to those topics. Subscribers have no knowledge of publishers. The publish / subscribe model basically has low coupling between the publisher and subscriber sides, making it highly scalable and adaptable to dynamic network configurations.

[0218] In the second embodiment, a data distributor equivalent to the event producer in the first embodiment is called an event publisher, and a data user equivalent to the event consumer in the first embodiment is called an event subscriber.

[0219] The data processing system 1 in Figure 29 includes a sensor 111, an event publisher 112, a broker 114, an event subscriber 116, and an event path manager 117, with the event publisher 112 and the broker 114 connected by an event path 113, and the broker 114 and the event subscriber 116 connected by an event path 115.

[0220] Although not shown, the data processing system 1 in FIG. 29 also includes a topic manager 16 and a topic DB 17, similar to the first embodiment.

[0221] The sensor 111 may be realized as a standalone sensor device, or may be integrated as part of an edge device in which an event publisher 112 connected to the sensor device is executed. The edge device is realized by, for example, a data processing device such as a server device. Hereinafter, the sensor 111 and the event publisher 112 are collectively referred to as a sensor / edge 211.

[0222] The cloud 221 is composed of a plurality of nodes and a network connecting them. The network is composed of a communication network or a communication path of any communication standard, such as the Internet, a public telephone line network, a wide area communication network for wireless mobile devices such as a so-called 4G line or 5G line, a wide area network (WAN), a local area network (LAN), a wireless communication network that performs communication in accordance with the Bluetooth (registered trademark) standard, a communication path for short-distance wireless communication such as NFC (Near Field Communication), a communication path for infrared communication, a communication network for wired communication in accordance with standards such as HDMI (registered trademark) (High-Definition Multimedia Interface) or USB (Universal Serial Bus). Each node constituting the cloud 221 is composed of a network connection device such as a sensor device, a router, a modem, a hub, a bridge, a switching hub, a base station control device, an exchange, or a server. An application executed on the network connection device as a node functions as a broker 114, an event subscriber 116, an event path manager 117, a topic manager 16, or a topic DB 17.

[0223] The cloud 221 includes an edge cloud 222 arranged on the edge side close to the sensor / edge 211 that injects sensor data into the network, and a center cloud 223 arranged in the rest of the core network. For example, when the network is a mobile phone communication network, the edge cloud 222 is composed of a base station or the like.

[0224] The sensor 111 and the event publisher 112 correspond to the sensor 11 and the event producer 12 in the first embodiment. The sensor 111 is a sensor device that detects some state, and supplies the generated original data to the event publisher 112. The event publisher 112 is an application that acquires the original data from the sensor 111, and transmits the original data and event data to the broker 114 via the event path 113. The event publisher 112 and the broker 114 are connected in a one-to-one relationship.

[0225] The event publisher 112 transfers all acquired event data to the broker 114, and the broker 114 responds to requests from the event subscriber 116 and transmits the event data. In addition to the transmission of event data, original data is also transferred from the event publisher 112 to the broker 114, and the broker 114 responds to requests from the event subscriber 116. The broker 114 is located in the edge cloud 222, and the event subscriber 116 is located in the center cloud 223.

[0226] The broker 114 transmits original data or event data transmitted from the event publisher 112 via the event path 113 to the event subscriber 116 via the event path 115. The broker 114 and the event subscriber 116 are connected in a one-to-many (multiple) relationship.

[0227] The broker 114 acquires and keeps track of the topic ID in advance that the event subscriber 116 is interested in through the topic registration process ("Register Subscription") performed by the event subscriber 116, similar to the topic registration process ("Register Topic") in which the event consumer 14 notifies the event producer 12 of the topic ID required by the event consumer 14 in the first embodiment. Then, when event data in which the event subscriber 116 is interested is sent from the event publisher 112, the broker 114 appropriately secures the event path 115 and transfers it to the event subscriber 116.

[0228] The event path manager 117 is disposed in either the edge cloud 222 or the center cloud 223, and secures and reserves an event path in the same manner as the event path manager 15 in the first embodiment. The event path 113 between the event publisher 112 and the broker 114, and the event path 115 between the broker 114 and the event subscriber 116 are secured and allocated separately. The event path 113 between the event publisher 112 and the broker 114 is secured in response to a request from the event publisher 112, and the event path 115 between the broker 114 and the event subscriber 116 is secured in response to a request from the broker 114.

[0229] <11. Transmission control of event data and original data> Next, the transmission control of the event data and original data by the data processing system 1 in FIG. 29 will be described with reference to the flowchart in FIG.

[0230] First, in step S301, the event subscriber 116 executes a topic registration process ("Register Subscription") to notify the broker 114 of a required topic ID. The broker 114 receives the topic ID notified from the event subscriber 116.

[0231] In step S302, the event path manager 117 secures the event paths 113 and 115. That is, the event path manager 117 estimates the network traffic between the event publisher 112 and the broker 114, and secures the event path 113 between the event publisher 112 and the broker 114. In addition, the event path manager 117 estimates the network traffic between the broker 114 and the event subscriber 116, and secures the event path 115 between the broker 114 and the event subscriber 116.

[0232] In step S311, the sensor 111 executes "Capture Original", and in step S312, executes "Send Original". That is, the sensor 111 transmits the original data at time t n-1 Luminance image L n-1 (x, y) and transmits it to the broker 114. The broker 114 receives the luminance image L n-1 (x, y) is received and stored in the internal memory. Luminance image L n-1 (x, y) may be sent to the broker 114 via the event publisher 112 .

[0233] In step S313, the sensor 111 executes "Detect Update & Send Update." Specifically, the sensor 111 executes "Detect Update & Send Update" at time t n Luminance image L n (x, y) is detected and time t n-1 Luminance image of L n-1 (x, y) and time t n Luminance image of L n Event data e showing the luminance change with (x, y) n = { x, y, p n , t n} to the event publisher 112. The event publisher 112 supplies the event data e n Get the.

[0234] In step S314, the event publisher 112 executes "Generate Update Event" to generate event data. Then, in step S315, the event publisher 112 executes "Request Event Path" to send an event path reservation request to the event path manager 117 and the broker 114. The broker 114 also executes "Request Event Path" to send an event path reservation request to the event path manager 117.

[0235] In step S317, the event path manager 117 individually allocates the event path 113 between the event publisher 112 and the broker 114, and the event path 115 between the broker 114 and the event subscriber 116 from among the event paths secured in advance.

[0236] When the allocation of the event path 113 and the event path 115 is completed, in step S318, the event publisher 112 executes "Send Update Event." Specifically, the event publisher 112 transmits the event data generated in step S314 to the broker 114 via the event path 113. The broker 114 receives the event data from the event publisher 112 and transmits it to a large number of event subscribers 116 via the event path 115.

[0237] The processes in steps S313 to S318 may be repeatedly executed multiple times as necessary.

[0238] In step S319, the event subscriber 116 executes "Request Original." Specifically, the event subscriber 116 receives the event data e nThe event subscriber 116 requests the original data from the broker 114 by specifying the original reference (Original Ref) included in the request. The broker 114 executes “Send Original” based on the “Request Original” from the event subscriber 116, and sends the original data at time t n-1 Luminance image of L n-1 Send (x, y) to the event subscriber 116.

[0239] In step S320, each of the multiple event subscribers 116 receives the time t n-1 Luminance image of L n-1 (x, y) and executes “Update Original & Process Updated Original”. That is, the event subscriber 116 executes “Update Original & Process Updated Original” at time t n Luminance image of L n Execute the process to recover (x, y) death , the updated luminance image L n Perform a predetermined application process using (x, y).

[0240] As described above, when sensor data is distributed to a large number of event subscribers 116, the event publisher 112 transmits the original data and event data to the broker 114, and the broker 114 distributes the original data and event data in response to requests from the large number of event subscribers 116. This makes it possible to reduce the execution load of the application on the sensor / edge 211.

[0241] <12. Event class negotiation> The event classes set in the event path 113 between the event publisher 112 and the broker 114 and the event path 115 between the broker 114 and the event subscriber 116 are basically the same. However, they may be changed depending on the traffic conditions between each broker 114 and the event subscriber 116.

[0242] FIG. 31 shows a flowchart of event class change control in which the event path manager 117 negotiates an event class.

[0243] In step S341, if the event path manager 117 determines that it cannot secure an event path that satisfies the requirements specified in the event path request, or that such a state will continue, it individually negotiates (requests) a change in the event class with the broker 114 or the event subscriber 116 (Negotiate Delivery Requirements). That is, the event path manager 117 negotiates with the broker 114 for the event path 113, and negotiates with the event subscriber 116 for the event path 115.

[0244] If the negotiation in step S341 results in consent to the event class change, in step S342, the event path manager 117 sends an event class change instruction to the event publisher 112 or the broker 114 (Modify Event Class). That is, for a change of the event class of the event path 113, an event class change instruction is sent to the event publisher 112, and for a change of the event class of the event path 115, an event class change instruction is sent to the broker 114.

[0245] Here, the bit rate is adjusted so that the bit rate of the event path 113 between the event publisher 112 and the broker 114 is greater than the maximum value of the bit rate of the (multiple) event paths 115 between the broker 114 and the event subscriber 116. In other words, the requirements between the event publisher 112 and the broker 114 are adjusted to be stricter than the requirements between the broker 114 and the multiple event subscribers 116.

[0246] FIG. 32 shows a flowchart of event class change control in the case where the event class of the event path 115 is reflected in the event class of the event path 113 to achieve overall optimization.

[0247] In some cases, the event class negotiated between the broker 114 and multiple event subscribers 116 may be reflected in the event class between the event publisher 112 and the broker 114, resulting in overall optimization.

[0248] Specifically, in step S361, if the event path manager 117 determines that the event path 115 cannot meet the requirements specified in the event path request, or that such a condition will continue, it negotiates with the event subscriber 116 to request a change in the event class (Negotiate Delivery Requirements).

[0249] If the negotiation in step S361 results in consent to the change of the event class, then in step S362 the event path manager 117 transmits an event class change instruction, for example to lower the bit rate, to the broker 114 (Modify Event Class).

[0250] In step S363, the broker 114 judges whether the maximum bit rate of the multiple event paths 115 between the broker 114 and the event subscriber 116 has stabilized to a predetermined value (judge if stable), and if it is determined that the maximum bit rate has stabilized, sends an event class change instruction to the event path manager 117 to lower the bit rate according to the maximum bit rate (Modify Event Class).

[0251] In step S364, the event path manager 117 transmits an event class modification instruction to the event publisher 112 based on the event class modification instruction from the broker 114 (Modify Event Class).

[0252] For example, if the bit rate between the broker 114 and multiple event subscribers 116 is stable at 5 Mbps, 4 Mbps, and 2 Mbps, the overall traffic can be minimized by changing the bit rate between the event publisher 112 and broker 114 from 10 Mbps to 6 Mbps. In other words, by changing the event class while maintaining the requirements between the event publisher 112 and broker 114 slightly stricter than the requirements between the broker 114 and multiple event subscribers 116, unnecessary consumption of network resources can be suppressed.

[0253] <13. Immediate and delayed transfer> In the above-described control example, one event publisher 112 distributes event data to multiple event subscribers 116 via one broker 114 .

[0254] However, the broker 114 may receive event data with different topic IDs from multiple event publishers 112 (e.g., when one sensor 111 has one topic ID), or may receive event data with multiple different topic IDs from a single event publisher 112 (e.g., when one sensor 111 has topic IDs corresponding to multiple object region ROIs in an image).

[0255] Event data of different topic IDs acquired by the broker 114 has different priorities (event classes). When there is no room in the event path 115 between the broker 114 and each event subscriber 116 and a competition for resources occurs, the broker 114 appropriately selects and executes immediate transfer, which transfers the event data immediately, or delayed transfer (cache transfer), which temporarily caches the event data and then transfers it, according to each priority.

[0256] FIG. 33 is a flowchart showing an example of control of immediate transfer and delayed transfer.

[0257] Steps S381 and S382 in FIG. 33 are an example of control when immediate transfer is performed.

[0258] Specifically, in step S 381 , the event publisher 112 executes “Send Update Event” to send the event data to the broker 114 .

[0259] In step S382, the broker 114 receives the event data from the event publisher 112 and determines the event class of the event data. If the event class of the received event data is an event class with strict delay requirements, in other words, an event class with high priority, the broker 114 immediately transmits the event data to the event subscriber 116.

[0260] Steps S391 to S393 in FIG. 33 are an example of control when performing delayed transfer.

[0261] Specifically, in step S 391 , the event publisher 112 executes “Send Update Event” to send the event data to the broker 114 .

[0262] In step S392, the broker 114 receives the event data from the event publisher 112 and determines the event class of the event data. If the event class of the received event data is an event class with loose delay requirements, in other words, an event class with low priority, the broker 114 temporarily stores (caches) the event data in an internal storage unit.

[0263] Then, in step S393, the broker 114 waits for a certain period of time, and then transmits the temporarily stored event data to the event subscriber 116.

[0264] As described above, the broker 114 can appropriately select and execute immediate or delayed transfer of received event data based on the priority indicated by the event class. It can preferentially allocate network and computation resources to urgent event data, and cache low-priority event data.

[0265] <14. Cloud Computing Configuration Examples> The methods and systems described herein, including the data processing system and network control methods described above, may be implemented using computer programming or engineering techniques, including computer software, firmware, hardware, or any combination or subset thereof.

[0266] FIG. 34 illustrates a block diagram of a computer capable of implementing various embodiments described herein.

[0267] The present disclosure may be embodied as a system, method, and / or computer program product that may include a computer-readable storage medium having computer-readable program instructions recorded thereon that cause one or more processors to perform aspects of the present embodiments.

[0268] A computer-readable storage medium may be a tangible device that can store instructions for use by an instruction execution device (processor). A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of such devices. More specific examples of computer-readable storage media include each of the following (and suitable combinations): a floppy disk, a hard disk, a solid state drive (SSD), a random access memory (RAM), a read only memory (ROM), an erasable and programmable read only memory (EPROM) or Flash (Flash memory), a static random access memory (SRAM), a compact disk (CD or CD-ROM), a digital versatile disc (DVD), a card-type or stick-type memory. A computer-readable storage medium as used in this disclosure is not to be construed as a transitory signal itself, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a wave guide or other transmission medium (e.g., light pulses through a fiber optic cable), or an electric signal transmitted through a wire.

[0269] The computer readable program instructions of the present disclosure may be downloaded from a computer readable storage medium into a suitable computing or processing device or downloaded to an external computer or external storage device via a global network such as the Internet, a local area network, a wide area network, and / or a wireless network, including copper transmission lines, fiber optics, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface within the computing or processing device may receive the computer readable program instructions from the network and transfer the computer readable program instructions for storage in a computer readable storage medium within the computing or processing device.

[0270] The computer readable program instructions for carrying out the processes of the present disclosure include machine language instructions and / or microcode, which may be compiled or interpreted from source code written in any combination of one or more program languages, including assembly language, Basic, Fortran, Java, Python, R, C, C++, C#, or similar programming languages. The computer readable program instructions may be executed entirely on a user's personal computer, notebook computer, tablet, or smartphone, or entirely on a remote computer or computer server, or any combination of these computing devices. The remote computer or computer server may be connected to the user's device or devices via a computer network, such as a local area network, a wide area network, or a global network (e.g., the Internet). In some embodiments, electronic circuitry, including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), programmable logic arrays (PLAs), may execute the computer readable program instructions using information from the computer readable program instructions to configure or customize the electronic circuitry, to implement aspects of the present disclosure.

[0271] Aspects of the present disclosure are described herein with reference to flow diagrams and block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It will be understood by one skilled in the art that each block of the flow diagrams and block diagrams, and combinations of blocks in the flow diagrams and block diagrams, can be implemented by computer readable program instructions.

[0272] Computer-readable program instructions capable of implementing the systems and methods described in this disclosure are utilized in one or more processors (and / or one or more cores within a processor) of a general-purpose computer, special-purpose computer, or other programmable device to produce an apparatus. Execution of the program instructions via the processor of the computer or other programmable device creates a system for implementing the functions described in the flow diagrams and block diagrams of this disclosure. These computer-readable program instructions may also be stored on a computer-readable storage medium that can instruct a computer, programmable device, and / or other device to function in a particular manner. Thus, a computer-readable storage medium having instructions stored thereon is an article of manufacture that includes instructions implementing aspects of the functions identified in the flow diagrams and block diagrams of this disclosure.

[0273] The computer readable program instructions may also be loaded into a computer, other programmable apparatus, or other device to perform a series of operational steps on the computer, other programmable apparatus, or other device to generate a computerized process result. The program instructions, when executed on the computer, other programmable apparatus, or other device, provide the functions identified in the flow diagrams and block diagrams of this disclosure.

[0274] Fig. 34 is a functional block diagram of a network system 800 in which one or more computers, servers, etc. are connected via a network. Note that the hardware and software environment shown in the embodiment of Fig. 34 is shown as an example providing a platform for realizing the software and / or method according to the present disclosure.

[0275] As shown in Figure 34, network system 800 may include, but is not limited to, a computer 805, a network 810, a remote computer 815, a web server 820, a cloud storage server 825, and a computer server 830. In one embodiment, multiple instances of one or more of the functional blocks shown in Figure 34 are used.

[0276] 34 illustrates a more detailed configuration of the computer 805. Note that the functional blocks illustrated in the computer 805 are illustrated to establish exemplary functions, and are not all shown. Also, detailed configurations of the remote computer 815, the web server 820, the cloud storage server 825, and the computer server 830 are not illustrated, but they may include configurations similar to the functional blocks illustrated for the computer 805.

[0277] The computer 805 may be a personal computer (PC), a desktop computer, a laptop computer, a tablet computer, a netbook computer, a personal digital assistant (PDA), a smartphone, or any other programmable electronic device capable of communicating with other devices on the network 810.

[0278] The computer 805 is then configured with a processor 835, a bus 837, memory 840, non-volatile storage 845, a network interface 850, a peripherals interface 855, and a display interface 865. Each of these functions may be implemented in some embodiments as individual electronic subsystems (integrated circuit chips or combinations of chips and related devices), while in other embodiments several functions may be combined and implemented as a single chip (System on Chip or SoC).

[0279] Processor 835 may be one or more single- or multi-chip microprocessors, such as those designed and / or manufactured by Intel Corporation, Advanced Micro Devices, Inc. (AMD), Arm Holdings (Arm), Apple Computer, etc. Examples of microprocessors include Celeron, Pentium, Core i3, Core i5, and Core i7 from Intel Corporation, Opteron, Phenom, Athlon, Turion, and Ryzen from AMD, and Cortex-A, Cortex-R, and Cortex-M from Arm.

[0280] Bus 837 may employ any proprietary or industry standard high speed parallel or serial peripheral interconnect bus, such as, for example, ISA, PCI, PCI Express (PCI-e), AGP, etc.

[0281] The memory 840 and the non-volatile storage 845 are computer readable storage media. The memory 840 may employ any suitable volatile storage device, such as a dynamic random access memory (DRAM) or a static RAM (SRAM). The non-volatile storage 845 may employ one or more of a flexible disk, a hard disk, a solid state drive (SSD), a read only memory (ROM), an erasable and programmable read only memory (EPROM), a flash memory, a compact disk (CD or CD-ROM), a digital versatile disk (DVD), a card type memory, or a stick type memory.

[0282] Additionally, program 848 is a collection of machine-readable instructions and / or data that is stored in non-volatile storage 845 and is used to create, manage, and control certain software functions as detailed herein and illustrated in the drawings. Note that in configurations where memory 840 is significantly faster than non-volatile storage 845, program 848 may be transferred from non-volatile storage 845 to memory 840 before being executed by processor 835.

[0283] Through network interface 850, computer 805 can communicate and interact with other computers over network 810. Network 810 can be, for example, a local area network (LAN), a wide area network (WAN) such as the Internet, or a combination of LANs and WANs, including wired, wireless, or fiber optic connections. In general, network 810 consists of any combination of connections and protocols that support communication between two or more computers and associated devices.

[0284] The peripheral interface 855 can input and output data to and from other devices that may be locally connected to the computer 805. For example, the peripheral interface 855 provides a connection to an external device 860, which may be a keyboard, a mouse, a keypad, a touch screen, and / or other suitable input devices. The external device 860 may also include portable computer-readable storage media, such as thumb drives, portable optical or magnetic disks, and memory cards. Software and data implementing embodiments of the present disclosure, such as the program 848, may be stored on such portable computer-readable storage media. In such an embodiment, the software may be loaded onto the non-volatile storage 845, or alternatively, may be loaded directly onto the memory 840 via the peripheral interface 855. The peripheral interface 855 may use industry standards, such as RS-232 or Universal Serial Bus (USB), to connect to the external device 860.

[0285] Display interface 865 may connect computer 805 to a display 870, which may be used to present a command line or a graphical user interface to a user of computer 805. Display interface 865 may connect to display 870 using one or more of an industry standard or proprietary connection, such as a Video Graphics Array (VGA), Digital Visual Interface (DVI), DisplayPort, or High-Definition Multimedia Interface (HDMI).

[0286] As discussed above, the network interface 850 provides for communication with other computers, storage systems, or devices external to the computer 805. The software programs and data described herein may be downloaded to the non-volatile storage 845 from, for example, the remote computer 815, the web server 820, the cloud storage server 825, and the computer server 830 via the network interface 850 and the network 810. Additionally, the systems and methods of the present disclosure may be performed by one or more computers connected to the computer 805 via the network interface 850 and the network 810. For example, in one embodiment, the systems and methods of the present disclosure may be performed by the remote computer 815, the computer server 830, or a combination of multiple interconnected computers on the network 810.

[0287] Data, datasets, and / or databases employed in embodiments of the systems and methods of the present disclosure may be downloaded from and stored on remote computers 815, web servers 820, cloud storage servers 825, and computer servers 830.

[0288] Here, in this specification, the processing performed by a computer according to a program does not necessarily have to be performed in chronological order according to the order described in the flowchart. In other words, the processing performed by a computer according to a program also includes processing executed in parallel or individually (for example, parallel processing or processing by objects).

[0289] The program may be processed by one computer (processor), or may be distributed among multiple computers. Furthermore, the program may be transferred to a remote computer for execution.

[0290] Furthermore, in this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device in which multiple modules are housed in a single housing, are both systems.

[0291] Also, for example, the configuration described above as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, the configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Also, it is of course possible to add configurations other than those described above to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the system as a whole are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0292] In addition, the present embodiment is not limited to the above-described embodiment, and various modifications are possible within the scope of the gist of the present disclosure. The effects described in this specification are merely examples and are not limiting, and there may be effects other than those described in this specification.

[0293] The present disclosure can have the following configurations. (1) A network connection device, Transmitting event data indicating a change in sensor data at a predetermined time point, which is original data generated by a sensor device, to a data processing device prior to the original data, and transmitting the original data to the data processing device based on a request for the original data from the data processing device. How the network is controlled. (2) The data processing device includes: executing a process of recovering the original data at the predetermined point in time based on the original data acquired by transmitting a request for the original data to the network connection device and the event data previously acquired; A method for controlling a network according to (1) above. (3) The network connection device includes: A grouping process is executed to group a plurality of the event data at a predetermined grouping level, and the grouped event data is transmitted to the data processing device prior to the original data. A method for controlling a network according to (1) or (2) above. (4) The data processing device includes: After acquiring a plurality of the event data at a predetermined aggregation level, a request for the original data is sent to the network connection device to acquire the original data. A method for controlling a network according to any one of (1) to (3). (5) The predetermined summarization level is any one of a unit equal to or greater than a predetermined value determined in advance, a unit in which an area of ​​an object included in the original data can be extracted, a unit in which the object can be recognized, and a unit in which the trajectory of the object can be traced. A method for controlling a network according to (3) or (4). (6) The predetermined summary level is set for each observation data identification information that identifies the data of the observation target to be notified as the event data. A network control method according to any one of (3) to (5). (7) The event data is transmitted including observation data identification information for identifying data of an observation target to be notified as the event data. A method for controlling a network according to any one of (1) to (6). (8) The observation data identification information is any one of sensor identification information for identifying the sensor device, object identification information for identifying an object in unit of the sensor identification information and the sensor device, global object identification information for globally uniquely identifying the object, and query identification information for identifying a query for the data of the observation target. A network control method according to (7) above. (9) The data processing device determines reception of the event data based on the observation data identification information. The network control method according to (8) above. (10) the data processing device notifies the network connection device of the observation data identification information; The network connection device transmits to the data processing device only the event data including the observation data identification information notified from the data processing device. A network control method according to (8) or (9). (11) The event data includes at least observed data identification information for identifying observed data to be notified as the event data, a reference address of the original data, and an event type. A method for controlling a network according to any one of (1) to (10) above. (12) the event type is either an update including data of the change, or a notify notifying only that a change has occurred without including data of the change, The reference address of the original data is a reference address of the original data before the change at the predetermined time point is applied when the event type is the update, and is a reference address of the original data after the change at the predetermined time point is applied when the event type is notify. The network control method according to (11) above. (13) If the event type is the notify, The event data includes a reference address of individualized data obtained by individualizing the plurality of event data at a predetermined aggregation level. The network control method according to (12) above. (14) The individualized data is data in a unit that can extract an area of ​​an object included in the original data, a unit that can recognize the object, or a unit that can trace the trajectory of the object. The network control method according to (13) above. (15) When transmitting the event data, the network connection device transmits an event path request requesting the securing of a virtual path with the data processing device. A network control method according to any one of (1) to (14). (16) The event path request stores identification information for identifying the priority of the virtual path. The network control method according to (15) above. (17) The event pass request includes a parameter indicating the quality of the data transfer. A network control method according to (15) or (16). (18) The event pass request stores observation data identification information for identifying the observation target data to be notified as the event data, and an event type. A network control method according to any one of (15) to (17). (19) The network connection device transmits a plurality of types of the event data to a plurality of the data processing devices, and selects and executes immediate transfer or delayed transfer of the plurality of types of the event data according to the priority of the event data. A network control method according to any one of (1) to (18). (20) A network connection device that transmits event data indicating a change in sensor data at a predetermined time point, which is original data generated by a sensor device, to a data processing device prior to the original data, and transmits the original data to the data processing device based on a request for the original data from the data processing device. A data processing system comprising: [Explanation of symbols]

[0294] 1 Data Processing System, 11 Sensor, 12 Event Producer, 13 Event Path, 14 Event Consumer, 15 Event Path Manager, 16 Topic Manager, 17 Topic DB, 21 Cloud, 111 Sensor, 112 Event Publisher, 113 Event Path, 114 Broker, 115 Event Path, 116 Event Subscriber, 117 Event Path Manager, 211 Sensor / Edge, 221 Cloud, 222 Edge Cloud, 223 Center Cloud, 800 Network System, 805 Computer, 810 Network, 815 Remote Computer, 820 Web Server, 825 Cloud Storage Server, 830 Computer Server, 835 Processor, 840 Memory, 845 Non-volatile Storage, 848 Program, 860 External Device

Claims

1. A network connection device, transmitting event data indicating a change in sensor data at a predetermined time point, which is original data generated by a sensor device, to a data processing device prior to the original data, and transmitting the original data to the data processing device based on a request for the original data from the data processing device; The event data includes at least observation data identification information for identifying observation target data to be notified as the event data, a reference address of the original data, and an event type; the event type is either an update including data of the change, or a notify notifying only that a change has occurred without including data of the change, The reference address of the original data is a reference address of the original data before the change at the predetermined time point is applied when the event type is the update, and is a reference address of the original data after the change at the predetermined time point is applied when the event type is notify. How the network is controlled.

2. A network connection device, transmitting event data indicating a change in sensor data at a predetermined time point, which is original data generated by a sensor device, to a data processing device prior to the original data, and transmitting the original data to the data processing device based on a request for the original data from the data processing device; When transmitting the event data, an event path request is transmitted, which requests securing of a virtual path with the data processing device. How the network is controlled.

3. The data processing device includes: executing a process of recovering the original data at the predetermined point in time based on the original data acquired by transmitting a request for the original data to the network connection device and the event data previously acquired; A method for controlling a network according to claim 1 or 2.

4. The network connection device includes: A grouping process is executed to group a plurality of the event data at a predetermined grouping level, and the grouped event data is transmitted to the data processing device prior to the original data. A method for controlling a network according to claim 1 or 2.

5. The data processing device includes: After acquiring a plurality of the event data at a predetermined aggregation level, a request for the original data is sent to the network connection device to acquire the original data. A method for controlling a network according to claim 1 or 2.

6. The predetermined summarization level is any one of a unit equal to or greater than a predetermined value determined in advance, a unit in which an area of ​​an object included in the original data can be extracted, a unit in which the object can be recognized, and a unit in which the trajectory of the object can be traced. The network control method according to claim 4.

7. The predetermined summary level is set for each observation data identification information that identifies the data of the observation target to be notified as the event data. The network control method according to claim 4.

8. The event data is transmitted including observation data identification information for identifying data of an observation target to be notified as the event data. A method for controlling a network according to claim 1 or 2.

9. The observation data identification information is any one of sensor identification information for identifying the sensor device, object identification information for identifying an object in unit of the sensor identification information and the sensor device, global object identification information for globally uniquely identifying the object, and query identification information for identifying a query for the data of the observation target. The network control method according to claim 8.

10. The data processing device determines reception of the event data based on the observation data identification information. The network control method according to claim 9.

11. the data processing device notifies the network connection device of the observation data identification information; The network connection device transmits to the data processing device only the event data including the observation data identification information notified from the data processing device. The network control method according to claim 9.

12. If the event type is the notify, The event data includes a reference address of individualized data obtained by individualizing the plurality of event data at a predetermined aggregation level. A method for controlling a network according to claim 1.

13. The individualized data is data in a unit that can extract an area of ​​an object included in the original data, a unit that can recognize the object, or a unit that can trace the trajectory of the object. A method for controlling a network according to claim 12.

14. The event path request stores identification information for identifying the priority of the virtual path. The network control method according to claim 2.

15. The event pass request includes a parameter indicating the quality of the data transfer. The network control method according to claim 2.

16. The event pass request stores observation data identification information for identifying the observation target data to be notified as the event data, and an event type. The network control method according to claim 2.

17. The network connection device transmits a plurality of types of the event data to a plurality of the data processing devices, and selects and executes immediate transfer or delayed transfer of the plurality of types of the event data according to the priority of the event data. A method for controlling a network according to claim 1 or 2.

18. A network connection device that transmits event data indicating a change in sensor data at a predetermined time point, which is original data generated by a sensor device, to a data processing device prior to the original data, and transmits the original data to the data processing device based on a request for the original data from the data processing device. Equipped with The event data includes at least observation data identification information for identifying observation target data to be notified as the event data, a reference address of the original data, and an event type; the event type is either an update including data of the change, or a notify notifying only that a change has occurred without including data of the change, The reference address of the original data is a reference address of the original data before the change at the predetermined time point is applied when the event type is the update, and is a reference address of the original data after the change at the predetermined time point is applied when the event type is notify. Data processing system. A network connection device that transmits event data indicating a change in sensor data at a predetermined time point, the sensor data being original data generated by a sensor device, to a data processing device prior to the original data, and transmits the original data to the data processing device based on a request for the original data from the data processing device. Equipped with When transmitting event data, the network connection device transmits an event path request requesting the securing of a virtual path with the data processing device. Data processing system.

Images