Data collection system

Abstract

The purpose of the present disclosure is to make it possible to perform sensing data matching even in environments in which data is delayed or damaged on its way over a network.　The present disclosure is a data collection system that collects sensing data acquired at a plurality of sensor terminals at a management node. The data collection system is characterized by comprising a time-series information source that distributes time-series information that makes it possible to identify a time series. The data collection system is also characterized in that the plurality of sensor terminals use an extension area of a layer 2 communication protocol to transmit the time-series information from the time-series information source.

Description

Technical Field

[0001] The present disclosure relates to sensing data collection in the IoT (Internet of Things).

Background Art

[0002] Using a lightweight communication protocol that is standardized and does not require high performance, obtain network configuration information and device information of terminals and devices. For example, Non-Patent Document 1 reports a method using LLDP (Link Layer Discovery Protocol, see Non-Patent Document 3 for example).

[0003] In the IoT, it is necessary to network-connect a large number of sensor terminals and collect data (sensing data) generated by them. Also, in data utilization in the IoT, not only the sensing data itself generated by sensor terminals but also data related to sensing data, called metadata, has been reported to be important (Non-Patent Document 2, etc.). By obtaining and distributing sensing data and metadata together, it is expected that users can safely and easily utilize sensing data. For example, by using LLDP disclosed in Non-Patent Document 1, metadata (device information) such as manufacturer name and model number related to sensing data can be collected with an economical system configuration.

[0004] By using a plurality of sensor terminals, it becomes possible to create a unmanned store or unmanned cash register system in which store clerks in the store are unmanned. In such an unmanned store or unmanned cash register system, in order to surely grasp the event of "the customer picked up the product on the shelf and put it in the basket", information is verified from both sides such as a camera and a weight sensor installed in the product shelf, and the product that the customer is trying to purchase is identified. The verification of sensing data such as image data and weight data is performed based on the time stamp given to the sensing data.

[0005] However, this timestamp is added by network devices such as IoT-GWs located in the data collection network. Therefore, if delays, data loss, or data reordering occur in the local network from the sensor terminal to the network device, errors will occur in determining the order of the sensing data. Consequently, if delays or data loss occur in the sensing data along the network, it becomes difficult to match sensing data between different sensor terminals. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Yoshiyuki Mihara, Takefumi Yamazaki, Manabu Okamoto, and Atsushi Sato, "Design of the Home Network Map Identification Protocol (HTIP) and its Application to Diagnostic Tools," Transactions of the Information Processing Society of Japan, Consumer Devices & Systems, Vol. 2, No. 3, pp. 34-45, Dec. 2012. [Non-Patent Document 2] Toshihiko Oda, Hiroshi Imai, Takeshi Naito, and Hajime Takebayashi, "A Method for Defining, Generating, and Utilizing Metadata in the Sensing Data Distribution Market," 2018 Annual Conference of the Japanese Society for Artificial Intelligence (32nd), June 2012. [Non-Patent Document 3] IEEE Std 802.1AB-2016, “IEEE Standard for Local and metropolitan area networks-Station and Media Access Control Connectivity Discovery” [Non-Patent Document 4] IEEE Std 802.11TM-2016 P.708(Probe Reqest), P.712(Probe Response) [Overview of the project] [Problems that the invention aims to solve]

[0007] This disclosure aims to enable the matching of sensing data even in environments where delays or data loss occur in sensing data along the network. [Means for solving the problem]

[0008] The data collection system disclosed herein is a data collection system that collects sensing data acquired by multiple sensor terminals into a management node, and includes a time-series information source that distributes time-series information capable of identifying time series. The sensor terminals and management node of the disclosure execute the data collection method of the disclosure.

[0009] Specifically, the sensor terminal transmits the time-series information from a time-series information source that distributes time-series information capable of identifying time series, using an extended area of ​​the Layer 2 communication protocol.

[0010] Specifically, the management node receives time-series information that can identify the time series from the sensor terminal using an extended area of ​​the Layer 2 communication protocol, and uses the received time-series information to determine the order of the sensing data transmitted from the multiple sensor terminals.

[0011] The plurality of sensor terminals may transmit sensing data using the Layer 3 communication protocol domain. In this case, the management node may use the time-series information stored in the extended domain to determine the order of the sensing data transmitted from the plurality of sensor terminals.

[0012] The data collection system of this disclosure includes a network device for transferring the sensing data, and the network device may move the time-series information stored in the extended area to the sensing data storage area. In this case, the management node may use the time-series information stored in the Layer 3 communication protocol area to determine the order of the sensing data transmitted from the plurality of sensor terminals.

[0013] The management node may verify the sensing data transmitted from the plurality of sensor terminals, and based on the result of the verification, control a device different from the plurality of sensor terminals.

[0014] The time series information source may be a beacon that transmits a time signal including time information.

[0015] Note that the above inventions can be combined as much as possible.

Advantages of the Invention

[0016] The present disclosure can enable verification of sensing data even in an environment where data is delayed or lost during network transmission.

Brief Description of the Drawings

[0017] [Figure 1] This is a diagram for explaining the data collection system according to the present disclosure. [Figure 2] This is a diagram for explaining the terminal of the data collection system according to the present disclosure. [Figure 3] This is a diagram for explaining the management node of the data collection system according to the present disclosure. [Figure 4] This is a diagram for explaining the frame transmitted from the terminal to the management node. [Figure 5] An example of a plurality of sensor terminals is shown. [Figure 6] This is a diagram for explaining the data collection system according to the present disclosure.

Embodiments for Carrying Out the Invention

[0018] Embodiments of the present invention will be described with reference to the accompanying drawings. The embodiments described below are examples of the present invention, and the present invention is not limited to the following embodiments. In this specification and the drawings, components with the same reference numerals indicate the same components.

[0019] (Embodiment 1) This embodiment describes the basic configuration of the data collection system. Figure 1 is a diagram illustrating the data acquisition system 301 of this embodiment. The data acquisition system 301 is a data acquisition system that utilizes the extended domain of standardized communication protocols (LLDP, HTIP, IEEE802.11, etc.) to perform communication from terminal 11 to network device 12. Terminal 11 stores the sensing data detected by the sensor device in an area within the frame defined by the communication protocol, different from the area where metadata is stored, and sends it to the network device 12. The network device 12 forwards the frame to the management node 13. The management node 13 associates and stores the sensing data and the metadata based on the information identifying the terminal 11 described in the frame. It is characterized by the following:

[0020] The data collection network 15 is a network that connects terminals 11 located within a specific range to a management node 13. The data collection network 15 can be, for example, a local area network (LAN), a field area network (FAN), or an IoT area network. Within the same data collection network 15, there may be multiple terminals 11 of a single type, or there may be terminals 11 of multiple types.

[0021] Figure 2 is a diagram illustrating terminal 11. Terminal 11 is, for example, an IoT sensor terminal that performs sensing on an object to be observed and generates sensing data. Terminal 11 includes a sensor device 11a, a sensing data storage processing unit 11b, an equipment information storage processing unit 11c, a communication protocol operation unit 11d1, a metadata detection unit (11e1, 11e2, 11e3, ...), and a metadata storage processing unit 11f.

[0022] The sensor device 11a performs sensing on the object being observed and acquires sensing data (main data). Sensing data includes, for example, temperature, image, acceleration, sound, light, CO2, etc. The device information storage processing unit 11c collects information about the device being observed (for example, the manufacturer name, model name, model number, etc.) and stores this information in a predetermined location in the frame (an area that can be used for proprietary purposes, such as the "extended area" or "optional area" defined in the protocol).

[0023] The sensing data storage processing unit 11b stores the sensing data from the sensor device 11a in a predetermined location in the frame (such as the payload portion defined by the protocol). The sensing data storage processing unit 11b may also process the sensing data to fit the format / limitations of the frame's unique extension area, such as by converting it into a certain abbreviated code or by splitting it into multiple frames (fragmentation), before storing it in the frame.

[0024] The sensing data storage processing unit 11b can arbitrarily set the timing for storing sensing data in a frame. For example, the storage timing can be set to each time the sensing data is updated, or the sensing data can be stored after being accumulated for a certain period of time, rather than sequentially. In addition, if the sensing data storage processing unit 11b has accumulated sensing data for a certain period of time, it may also store the record (log) of that data or the results of specific calculations / statistical processing in a frame.

[0025] The type of sensing data stored in the frame and the timing of storage may be fixed or variable. The type of sensing data and the timing of storage may also be dynamically changed based on the terminal 11's own judgment or instructions from the management node 13. Furthermore, the frame transmission period may be fixed or variable. The frame transmission period may also be dynamically changed based on the terminal 11's own judgment or instructions from the management node 13.

[0026] The metadata detection unit 11e acquires information other than device information (metadata). Information other than device information includes, for example, location information, time information, person, object, or event information of the detection target, and other information. However, the present invention is not limited to this information other than device information. To acquire this information, the metadata detection unit 11e has a location information detection unit 11e1, a time detection unit 11e2, a person, object, or event detection unit 11e3, and other detection units.

[0027] The location information detection unit 11e1 is, for example, a GPS, an accelerometer, a gyroscope, or an RSSI receiver such as a Wi-Fi signal or a BLE (Bluetooth Low Energy) beacon signal. The location metadata detected by the location information detection unit 11e1 is location information obtained from GPS signals, BLE beacon signals, radio wave information from wireless communication, non-communication radio wave information (television, radio, radio-controlled clocks, and other noise, etc.), power information, visible light information, sound wave information, vibration information, acceleration information, and other location metadata sources.

[0028] The time detection unit 11e2 is, for example, a receiver of information from GPS, NTP (Network Time Protocol), etc. The time metadata detected by the time detection unit 11e2 is time-related information obtained from GPS signals, NTP information, and other time metadata sources.

[0029] The person, object, and event detection unit 11e3 is a receiver that receives information from sources such as BLE beacons (carried by a person), smartphones carried by a person, and image analysis results. The person, object, or event metadata detected by the person, object, and event detection unit 11e3 is information about people, objects, or events obtained from BLE beacons carried by a person, smartphones carried by a person, image analysis results, and other sources of timely metadata.

[0030] Other metadata detected by the detection unit includes information about the network configuration of the data collection network 15.

[0031] The metadata detection unit 11e may detect all of the multiple detection targets, or it may detect any one of them.

[0032] The metadata storage processing unit 11f stores the data detected by the metadata detection unit 11e as metadata in the extended area or optional area within the frame, as defined by the communication protocol. For example, the metadata storage processing unit 11f can store metadata in the control system frame of an IEEE 804.11 wireless LAN. Specifically, it stores various metadata in the "Vendor Specific" area, which is the extended area of ​​the Probe Request frame. Alternatively, it stores various metadata in the "Vendor Specific" area, which is the extended area of ​​the Probe Response frame.

[0033] The metadata storage processing unit 11f may store metadata in a frame after processing it, such as by converting it into a certain abbreviated code or by splitting it into multiple frames (fragmentation), in order to conform to the format / limitations of the frame's unique extension area.

[0034] The metadata storage processing unit 11f can arbitrarily set the timing for storing metadata in a frame. For example, the storage timing can be set to each time the metadata is updated, or the metadata can be stored after a certain period of accumulation rather than sequentially. Furthermore, when metadata has been accumulated for a certain period, the metadata storage processing unit 11f may store the record (log) of that data or the results of specific calculations / statistical processing in a frame.

[0035] The type and timing of metadata stored in a frame may be fixed or variable. The type and timing of metadata storage may be dynamically changed based on the terminal 11's own judgment or instructions from the management node 13.

[0036] The communication protocol operation unit 11d1 transmits a frame containing sensing data and device information stored in a predetermined area and metadata stored in an extended area or optional area to the network device 12 using a lightweight, standardized communication protocol such as LLDP or HTIP (Home network Topology Identifying Protocol). The communication protocol of the frame containing the sensing data and the communication protocol of the frame containing the device information may be the same or different. In the latter case, the metadata storage processing unit 11f may store the metadata in a frame of either one of the communication protocols (either the frame containing the sensing data or the frame containing the device information), or it may store it in frames of both communication protocols (the frame containing the sensing data and the frame containing the device information).

[0037] Furthermore, terminal 11 also has the function of operating in accordance with instructions from management node 13, etc. Specifically, terminal 11 has an instruction interpretation unit 11g, and when it changes the BLE beacon signal or metadata information (information to be transmitted, radio wave strength, transmission frequency, etc.) transmitted by the terminal itself in accordance with instructions from management node 13, it transmits that information to the outside. When transmitting information using the same protocol as communication with network device 12, it operates the communication protocol operation unit 11d1. When transmitting information using a different protocol than communication with network device 12, in addition to the communication protocol operation unit 11d1, it is equipped with a communication protocol operation unit 11d2, and operates the communication protocol operation unit 11d2.

[0038] This also includes cases where terminal 11 itself is a beacon signal source for other terminals to grasp metadata. For example, terminal 11 may be a beacon signal source for identifying location metadata, or it may be a beacon terminal carried by a worker to identify nearby people.

[0039] The network device 12 is, for example, a network switch, a wireless access point, or a wireless repeater. The network device 12 sends the frame group uploaded from the lower-level data collection network 15 directly to the management node 13. Here, the network device 12 may have a metadata processing unit (metadata detection unit 11e and metadata storage processing unit 11f) for metadata possessed by the terminal 11. Even if the network device 12 does not have a sensor device 11a, it can add unique information such as its own MAC address and metadata such as connection port to the frame sent from the terminal 11 and forward it to the management node 13. If the network device 12 has a metadata processing unit, it becomes possible to understand the logical connections from the management node 13 to the terminal 11, and to create a more accurate logical / physical network management map. In other words, even if the network device 12 is a network switch (switching hub) or wireless repeater that does not have Layer 3 or higher functionality, this technology operates at Layer 2, making it possible to manage and understand the connections of network devices, including the network device 12.

[0040] Figure 3 illustrates the management node 13. The management node 13 comprises a communication protocol operation unit 13a, an information processing unit 13b, and an information storage unit 13c. The management node 13 extracts and stores information from frames received from the network device 12 and uses it for analysis. In particular, the management node 13 is characterized by its function of storing combinations of two or more collected pieces of information in the information storage unit 13c.

[0041] The communication protocol operation unit 13a receives frames containing sensing data and metadata from terminals 11 and network devices 12. The information processing unit 12b extracts sensing data, device information, and metadata from the received frames and organizes them in the information storage unit 13c based on information that identifies the individual terminal 11 (e.g., MAC address). (1) Physical information of the terminal (information such as the characteristics of the casing, image information, information on attached labels, the object the worker is pointing to, the object the worker is looking at, etc.) (2) Identifier of a terminal on the logical network (MAC address, UUID, etc.) (3) Main data (sensing data such as temperature, image, acceleration, sound, light, CO2, etc.) (4) Various metadata (data such as location, time, people, things, events, etc.) For example, the management node 13 refers to metadata about a location and stores primary data obtained from the same location or within a certain area in the format [location metadata, primary data]. [supplement] Let me add some information about location metadata. In some cases, like GPS information, the location metadata is directly determined at the time it is sensed by terminal 11. On the other hand, in some cases, such as signals from BLE beacons, visible light, or sound information, it is not determined whether the information is location information at the time it is sensed by terminal 11 and sent as metadata, and the management node 13 recognizes / understands the metadata as location metadata. [End of supplementary information]

[0042] Figure 4 illustrates a frame 41 transmitted from terminal 11 to management node 13. The network device 12 is omitted from Figure 4. Frame 41 is a Layer 2 communication frame, such as an Ethernet® frame or a Wi-Fi communication frame. Frame 41 consists of a logical identifier 41a of the communication device, such as a MAC address; a source and destination identifier 41b, such as an IP address; an area 41c where sensing data such as temperature and images are stored; and an extended area 41d where metadata is stored. Of these, the identifier 41b and area 41c constitute a Layer 3 communication packet.

[0043] The management node 13, for example, combines the MAC address of the logical identifier 41a with the location metadata of the extended area 41d to create [MAC address, location metadata], and combines the MAC address of the logical identifier 41a with the installer metadata of the extended area 41d to create [MAC address, installer metadata], and organizes this information in the information storage unit 13c.

[0044] Thus, the data collection system 301 can acquire network configuration information, device information, sensing data, and metadata of terminals and devices using a communication protocol that does not require high performance.

[0045] (Embodiment 2) Figure 5 shows an example of how the sensor terminals are used. Sensor terminal 11-1 is a sensor terminal in which the sensor device 11a is a camera and transmits image data as sensing data. Sensor terminal 11-2 is a sensor terminal in which the sensor device 11a is a weight sensor and transmits weight data as sensing data. Sensor terminal 11-2 is installed on a product shelf and detects the load applied to the shelf by the products placed on it.

[0046] In an unmanned store / unmanned checkout system, image data I1, I2, and I3 captured by sensor terminal 11-1 are used to detect the event "a customer picked up a product from the shelf and placed it in a basket." When this event is detected, the changes in weight data W1, W2, and W3 detected by sensor terminal 11-2 are used to identify the "product on the shelf." This matching of sensing data at the time of the event is performed based on the timestamp attached to the sensing data.

[0047] However, this timestamp is added by a network device 12, such as an IoT-GW, located in the data collection network 15. Therefore, if delays, data loss, or reordering occur in the local network from the sensor terminal 11 to the network device 12, errors will occur in determining the order of the sensing data. Consequently, if delays or data loss occur in the sensing data along the network, it becomes difficult to match sensing data between different sensor terminals.

[0048] Figure 6 shows an example of a data acquisition system equipped with multiple sensor terminals. The data acquisition system of this embodiment includes sensor terminals 11-1 and 11-2, and a time information source 16 that functions as a time-series information source. The time information source 16 broadcasts a time signal containing time information. The time signal is a signal that sensor terminals 11-1 and 11-2 can receive as metadata, and for example, a beacon signal that can reach only sensor terminals 11-1 and 11-2 located in close proximity can be used.

[0049] The time signal can be any signal broadcast within the same space, including beacon signals containing time information, as well as GPS signals and other signals containing time information.

[0050] Sensor terminals 11-1 and 11-2 are equipped with a time receiving unit 11e4 that receives a time signal from a time information source 16. The time receiving unit 11e4 can be the metadata detection unit described in Embodiment 1. Sensor terminals 11-1 and 11-2 store the time information received by the time receiving unit 11e4 in the extended area 41d of the Layer 2 communication protocol and transmit it. This transmission can be performed, for example, using the metadata storage processing unit 11f and the communication protocol operation unit 11d1 described in Figure 2.

[0051] In the frame 41 transmitted from sensor terminal 11-1, image data, which is sensing data, is stored in region 41c, and time information is stored in the extended region 41d. In the frame 41 transmitted from sensor terminal 11-2, weight data, which is sensing data, is stored in region 41c, and time information is stored in the extended region 41d.

[0052] The management node 13 receives frame 41 and uses the time information stored in extended area 41d to determine the order of the sensing data stored in area 41c. The management node 13 compares the sensing data transmitted from sensor terminals 11-1 and 11-2 and controls devices such as register devices that are different from sensor terminals 11-1 and 11-2 based on the results of the comparison.

[0053] The time information stored in the extended area 41d is time information from a common time information source 16 stored in sensor terminals 11-1 and 11-2. Therefore, this embodiment is not affected by packet delays, loss, or reordering within the local network, thus improving the accuracy of matching events as the same occurrence.

[0054] Furthermore, the metadata stored in the extended area 41d of frame 41 may be moved from the extended area 41d to the same area 41c as the main data by a protocol processing unit such as the IoT-GW12G. In this case, the management node 13 uses the time information stored in area 41c to determine the order of the sensing data.

[0055] This embodiment allows for the assignment of highly accurate timestamps to time-series weight data W1-W3 from the weight sensor terminal 11-2 and image data I1-I3 from the camera, making it easy to identify them as representing the same event (e.g., "Customer A picked up an item from shelf B"). Therefore, in unmanned stores and the like, it is possible to accurately extract which user picked up which item from the weight changes of the shelf weight sensor and the image data from the surveillance camera.

[0056] Although an example has been shown in which multiple sensor terminals 11-1 and 11-2 are arranged in the same space and acquire time information from a common time information source 16, this disclosure is not limited to this. For example, multiple sensor terminals 11-1 and 11-2 may acquire time information from different time information sources 16 that are time-synchronized. Furthermore, the time information may be any time-series information that can identify the time series, such as an event identification code or serial number.

[0057] Furthermore, although this embodiment shows examples where different sensor terminals 11 are sensor terminals 11-1 and 11-2 of different sensor types, this disclosure is not limited thereto. For example, different sensor terminals 11 may be sensor terminals 11 of the same sensor type.

[0058] This disclosure enables the accurate acquisition of the time series and timing of sensing data detected by multiple sensor terminals 11-1 and 11-2. Therefore, metadata regarding the location and state of workers in a workspace (office, warehouse, etc.) (moving, stopped, strong impact, sound, etc.) can be collected in real time. By using these metadata as event occurrence flags (identifiers for identifying the event) in multiple nearby sensors, the searchability of large amounts of accumulated data is improved.

[0059] (Other embodiments) The sensor terminal 11 and management node 13 described above can also be implemented using a computer and a program, and the program can be recorded on a recording medium or provided via a network. [Explanation of Symbols]

[0060] 11: Terminal 11-1, 11-2: Sensor terminals 11a: Sensor device 11b: Sensing data storage processing unit 11c: Device information storage processing unit 11d1, 11d2: Communication protocol operation unit 11e, 11e1, 11e2, 11e3, ...: Metadata detection unit 11f: Metadata storage processing unit 11g: Instruction Interpretation Section 12: Network devices 12G: IoT-GW 13: Management Node 13a: Communication protocol operation unit 13b: Information Processing Section 13c: Information storage unit 15, 15-1, 15-2: Data Collection Network 41: Frame 41a: Logical identifier 41b: Source / Destination Identifier 41c: Main data area 41d: Extended area 301-302: Data collection system

Claims

1. A data collection system that collects sensing data acquired by multiple sensor terminals into a management node, The data acquisition system further comprises a network device that transfers the sensing data transmitted from the sensor terminal to the management node. The sensor terminal transmits the time-series information from a time-series information source that distributes time-series information capable of identifying time series using the extended domain of the Layer 2 communication protocol, and transmits the sensing data using the domain of the Layer 3 communication protocol. The network device transfers the time-series information to the same area as the sensing data and transfers it to the management node. A data collection system characterized by the following features.

2. The data collection system according to claim 1, characterized in that the time-series information source is a beacon that transmits a time signal containing time information.

3. The management node receives the time-series information from the sensor terminal using an extended area of ​​the Layer 3 communication protocol, The order of the sensing data is determined using the received time-series information. The data acquisition system according to claim 1.

4. The management node is characterized in that it matches the sensing data and controls a device different from the plurality of sensor terminals based on the results of the matching. The data acquisition system according to claim 3.

5. A method for a data acquisition system that collects sensing data acquired by multiple sensor terminals into a management node, The data acquisition system further comprises a network device that transfers the sensing data transmitted from the sensor terminal to the management node. The sensor terminal transmits the time-series information from a time information source that distributes time-series information capable of identifying time series using an extended area of ​​the Layer 2 communication protocol, and the sensing data using the area of ​​the Layer 3 communication protocol. The network device transfers the time-series information to the same area as the sensing data and transfers it to the management node. method.

6. The management node receives the time-series information from the sensor terminal using an extended area of ​​the Layer 3 communication protocol, The order of the sensing data is determined using the received time-series information. The method according to claim 5.

Images