System, method and computer program for edge management of geographically dispersed sensors

Abstract

Systems, methods, and computer programs for edge management of geographically dispersed sensors are provided as described herein. An edge device in a network accesses observations of a plurality of geographically dispersed sensors. Further, the edge device processes the observations to determine overlapping portions of the observations. Further, the edge device optimizes the observations to form optimized observations for transmission to a cloud processing system, where the optimization is based on the determined overlapping portions of the observations.

Description

Technical Field

[0001] This invention relates to a technique for collecting data from geographically dispersed sensors. Background Technology

[0002] A sensor is a hardware device that senses properties of the real world and reports these sensed properties to a processing system that can use the sensed information for various purposes. Sensors can be cameras, radar, lidar, motion detectors, temperature or other environmental instruments, etc. These sensors can be used for safety purposes, for vehicle operation purposes (e.g., for autonomous driving), for augmented reality applications, and so on.

[0003] As mentioned above, sensors typically report the information they sense to a separate processing system. These processing systems are usually located far from the sensors (e.g., and have better processing capabilities than the sensors), therefore, some network connection is usually required between the sensors and the processing systems. As sensors are increasingly used in the real world, not only is the network bandwidth consumed by these sensors increasing, but the storage capacity required to store all the sensed information at the receiving end is also increasing.

[0004] Therefore, it is necessary to address these and / or other issues related to existing technologies. Summary of the Invention

[0005] As described herein, systems, methods, and computer programs are provided for edge management of geographically dispersed sensors. Edge devices within a network access observations from multiple geographically dispersed sensors. Furthermore, the edge devices process the observations to determine overlapping portions of the observations. Additionally, the edge devices optimize the observations to form optimized observations for transmission to a cloud processing system, wherein the optimization is based on the determined overlapping portions of the observations. Attached Figure Description

[0006] Figure 1 A method for edge management of geographically dispersed sensors, according to one embodiment, is illustrated.

[0007] Figure 2 The diagram shows the overlap of observations from two geographically dispersed sensors according to one embodiment.

[0008] Figure 3 A system with distributed edge devices for managing geographically dispersed sensors is shown according to one embodiment.

[0009] Figure 4 A network architecture according to one possible embodiment is shown.

[0010] Figure 5 An exemplary system according to one embodiment is shown. Detailed Implementation

[0011] Figure 1 A method 100 for edge management of geographically dispersed sensors according to one embodiment is illustrated. The method 100 can be performed by a static or fixed edge device within a network. The edge device can communicate with the geographically dispersed sensors via the network. The edge device and / or the geographically dispersed sensors can also communicate with a cloud processing system via the network or another network. Although edge devices are mentioned below, it should be noted that any other computer system with a processor can be used to perform method 100.

[0012] In one embodiment, the edge device can be independent of multiple geographically dispersed sensors and within a defined proximity range of the sensors. The edge device uses local transmission near the sensors, which is faster and cheaper (e.g., low-power, such as WiFi or Bluetooth) and saves costs compared to transmission to the cloud (e.g., via cellular). In another embodiment, the edge device can be one of multiple geographically dispersed sensors. In this case, the sensor's resource availability (e.g., availability of computing, storage, power, bandwidth, etc.) can be selected as the edge device based on the resource availability of the sensor compared to the resource availability of the other sensors.

[0013] In yet another embodiment, the edge device can be located within a defined geographic area that includes the sensor, or vice versa. If the edge device and / or sensor are mobile, it can be determined which edge device serves which sensor within defined time intervals. This ensures that the edge device is within the defined geographic area of ​​the sensor it serves.

[0014] In operation 102, observations from geographically dispersed sensors are accessed. These sensors can be cameras, video cameras, motion detectors, or other types of sensors capable of observing the surrounding environment (e.g., measuring, capturing, etc.). For this purpose, the observations can be measurements, images, videos, or any other data observed by the sensors. For example, a sensor can be a camera, and the observations can be images of the environment in which the sensors are placed.

[0015] Each sensor can be mobile (e.g., located in a drone, vehicle, a person in an augmented reality camera, mobile phone, etc.) or stationary (e.g., attached to a wall, lamppost, etc., in a city, a home, etc.). As mentioned earlier, these sensors are geographically dispersed, or in other words, located in different geographical locations. Therefore, each sensor can observe its assigned area.

[0016] In one embodiment, observations may be performed by multiple geographically dispersed sensors that transmit the observations to an edge device. Alternatively, the resolution of the observations may be reduced by the sensors before transmission to the edge device. This, in turn, can reduce the bandwidth used for transmitting the observations. Of course, the observations can be accessed in any desired manner (e.g., retrieved, received, etc.). However, in one embodiment, the observations can be accessed at defined time intervals.

[0017] Alternatively, observations can be accessed only within the region of interest (i.e., the local area) of interest for the edge device. For example, the edge device can be assigned a region of interest. Alternatively, this assignment can be a one-time, static configuration. The edge device can then notify the sensors within the region of interest so that multiple geographically dispersed sensors only provide the edge device with observations within its region of interest. This can be useful when sensors are observing locations both inside and outside the edge device's region of interest.

[0018] As an alternative, each sensor can apply machine learning algorithms to dynamically determine whether observations need to be provided for every area captured by the sensor. For example, if there is no movement in a dead-end street captured by the sensor, then it may not be necessary to upload the observations of that dead-end street to the edge device (and a cloud processing system similar to that described below). However, once such movement occurs, the machine learning algorithm can determine the nature of the movement and whether observations for that area need to be uploaded to the edge device and cloud processing system.

[0019] As mentioned above, edge devices and / or any sensors can be mobile, and it may be desirable for the edge device to serve only sensors within a defined geographic area, or in other words, within a defined vicinity of the edge device. In this case, the edge device may only access the observations of sensors within a defined vicinity of the edge device.

[0020] Additionally, in operation 104, the edge device processes the observation results to determine overlapping portions of the observation results. In the context of this embodiment, overlapping portions can be parts of different observation results that have a threshold similarity. Therefore, overlapping portions can be repetitive of each other.

[0021] In one embodiment, the process may include stitching the observations together to determine overlapping portions of the observations. Any desired stitching algorithm may be used. In another embodiment, the process may include accessing which portions of the previously determined observations overlap, particularly where the sensors are fixed (and therefore, the areas captured by the observations of these fixed sensors are invariant). In any case, for each observation from each sensor, the process determines whether any portion of the observation overlaps with an observation from another sensor.

[0022] Furthermore, in operation 106, the edge device optimizes the observations to form optimized observations for transmission to the cloud processing system, wherein the optimization is based on a determined overlap of the observations. The optimization ensures that duplicate observations are prevented from being transmitted to the cloud processing system. For example, the optimization may include, for each overlap, selecting a portion of the overlap to transmit to the cloud processing system. This optimization can reduce the bandwidth required to transmit the observations to the cloud processing system and can reduce the storage resources required for the cloud processing system to store the received observations.

[0023] In one embodiment, the edge device can transmit optimized observations to a cloud processing system. In another embodiment, the edge device can enable sensors to transmit optimized observations to a cloud processing system. For example, in this embodiment, the edge device can instruct each sensor to transmit which portion(s) of its observations to the cloud processing system.

[0024] Once the optimized observations are received, the cloud processing system can apply them to any application. For example, the cloud processing system can use the optimized observations for security applications, tracking applications, augmented reality applications, and so on.

[0025] Based on user expectations, further illustrative information will now be provided regarding various alternative architectures and uses that may or may not implement the aforementioned methods. It should be particularly noted that the following information is presented for illustrative purposes and should not be construed as limiting in any way. Any of the following features may be optionally combined, excluding or not excluding other features described.

[0026] Figure 2 The diagram illustrates an overlap of observations from two geographically dispersed sensors according to one embodiment. In one embodiment, the overlap may be determined by an edge device independent of the two geographically dispersed sensors. In another embodiment, the overlap may be determined by an edge device that is one of the two geographically dispersed sensors. In either case, the overlap may be as described above regarding... Figure 1 As described in method 100.

[0027] As shown in the figure, the first observation 202 is captured by the first sensor, while the second observation 204 is captured by the second sensor. The first observation 202 and the second observation 204 capture the same object 206 (e.g., a car) from different angles. However, the first observation 202 and the second observation 204 also capture an overlapping area, which is shown by the overlapping portion 208A-B of the observations 202 and 204.

[0028] Figure 3 A system 300 with distributed edge devices for managing geographically dispersed sensors is shown according to one embodiment. Alternatively, system 300 can be configured to... Figure 1 The method 100 is implemented in the context of its details. Of course, however, system 300 can be implemented in any desired environment. Furthermore, the above definitions also apply to the description below.

[0029] As shown in the figure, observations 306A-C captured by geographically dispersed sensors are provided to distributed edge devices 302 and 304. Edge device 302 is a fixed camera, and therefore is itself a sensor (e.g., selected due to its better computing, storage, power, bandwidth, etc. capabilities). Edge device 304 is a fixed edge node independent of the sensor.

[0030] As shown in the figure, one or more geographically dispersed sensors can report to multiple edge devices because different edge devices can cover (e.g., be assigned) different shared areas under observation. For example, observation 306B captured by a sensor is provided to two edge devices 302 and 304, while observation 306A captured by a sensor is provided to edge device 302, and observation 306C captured by a sensor is provided to edge device 304.

[0031] When a stationary sensor reports to a stationary edge device, the edge device may only be "interested" in a portion of the sensor's image (i.e., the portion that the edge device is "responsible" for optimizing). The intersection of the stationary sensors (i.e., their overlapping observations) can be determined once and reused for subsequent observations. This is because the area captured by each stationary sensor is invariant.

[0032] In another embodiment, the edge device is fixed (and may have high bandwidth placed there along with, for example, a fiber optic network), while some sensors may be mobile (e.g., cameras mounted on a vehicle). In this case, the fixed edge device has an area it is "responsible for." Due to the mobile nature of the sensors, the intersection cannot be calculated in advance. Mobile sensors near the fixed edge device are informed of its presence and are instructed to transmit their observations to that edge device. The edge device calculates the overlap and optimizes the observations for transmission to the cloud processing system.

[0033] In yet another embodiment, the sensors and edge devices can be mobile. This may occur, for example, when there are many connected cars with computing capabilities. In this case, cameras mounted on the cars observe the world and upload their images / videos. The mobile sensors can decide, for example, using some form of distributed leader selection algorithm, which sensors are the edge leaders among the devices willing to perform observation optimization. This decision can be made every few seconds, with each location (defined by boundaries) selecting its own leader. After a leader is selected, the leader operates as an edge device for a specified period of time, deciding how to optimize the observations of its nearby sensors.

[0034] Figure 4 A network architecture 400 according to one possible embodiment is illustrated. As shown, at least one network 402 is provided. In the context of this network architecture 400, the network 402 can take any form, including but not limited to telecommunications networks, local area networks (LANs), wireless networks, wide area networks (WANs) such as the Internet, peer-to-peer networks, wired networks, etc. Although only one network is shown, it should be understood that two or more similar or different networks 402 can be provided.

[0035] Multiple devices are coupled to network 402. For example, server computer 404 and end-user computer 406 may be coupled to network 402 for communication purposes. Such end-user computer 406 may include desktop computer, laptop computer, and / or any other type of logic. Furthermore, various other devices may be coupled to network 402, including personal digital assistant (PDA) device 408, mobile phone device 410, television 412, and so on.

[0036] Figure 5 An exemplary system 500 according to one embodiment is shown. Alternatively, system 500 can be configured to... Figure 4 The network architecture 400 can be implemented in the context of any device. Of course, the system 500 can be implemented in any desired environment.

[0037] As shown in the figure, a system 500 is provided, including at least one central processing unit 501 connected to a communication bus 502. The system 500 also includes a main memory 504 [e.g., random access memory (RAM)]. The system 500 also includes a graphics processor 506 and a display 508.

[0038] System 500 may also include secondary storage 510. Secondary storage 510 includes, for example, solid-state drives (SSDs), flash memory, removable storage drives, etc. Removable storage drives read from and / or write to removable storage units in a well-known manner.

[0039] In this regard, computer programs or computer control logic algorithms can be stored in main memory 504, secondary storage 510, and / or any other memory. When such a computer program is executed, it enables system 500 to perform various functions (e.g., as described above). Memory 504, storage 510, and / or any other storage are possible examples of non-transitory computer-readable media.

[0040] System 500 may also include one or more communication modules 512. Communication modules 512 are operable to facilitate communication between system 500 and one or more networks, and / or communication with one or more devices via various possible standard or proprietary communication protocols (e.g., via Bluetooth, Near Field Communication (NFC), Cellular Communication, etc.).

[0041] As used herein, "computer-readable medium" includes one or more suitable media for storing executable instructions of a computer program, such that an instruction executor, system, apparatus, or device can read (or retrieve) the instructions from the computer-readable medium and execute the instructions to perform the methods. Suitable storage formats include one or more electronic, magnetic, optical, and electromagnetic formats. A non-exhaustive list of conventional exemplary computer-readable media includes: portable computer disks; RAM; ROM; erasable programmable read-only memory (EPROM or flash memory); optical storage devices, including portable optical discs (CDs), portable digital video discs (DVDs), and high-definition DVDs (HD-DVDs). TM Blu-ray discs, etc.

[0042] It should be understood that the arrangement of components shown in the figures is exemplary, and other arrangements are possible. It should also be understood that the various system components (and devices) defined by the claims, described below, and shown in the various block diagrams represent logical components in some systems configured according to the subject matter disclosed herein.

[0043] For example, one or more of these system components (and devices) may be implemented wholly or partially by at least some of the components shown in the arrangement illustrated in the figures. Furthermore, while at least one of these components is implemented at least partially as an electronic hardware component and thus constitutes a machine, other components may be implemented in software and, when included in an execution environment, constitute a machine, hardware, or a combination of software and hardware.

[0044] More specifically, at least one component as defined in the claims is at least partially implemented as an electronic hardware component, such as an instruction execution machine (e.g., a processor-based or processor-containing machine), and / or implemented as a special-purpose circuit or loop (e.g., interconnected discrete logic gates to perform a specific function). Other components may be implemented in software, hardware, or a combination of software and hardware. Furthermore, some or all of these other components may be combined, some may be omitted entirely, and additional components may be added while still achieving the functionality described herein. Therefore, the subject matter described herein can be embodied in many different variations, and all such variations are considered to be within the scope of the claims.

[0045] In the foregoing description, unless otherwise stated, the subject matter is described with reference to the actions and symbolic representations of operations performed by one or more devices. Therefore, it is understood that such actions and operations, sometimes referred to as those performed by a computer, include the processor's manipulation of data in a structured form. Such operations transform data or store it in a location within the computer's memory system, which reconfigures or otherwise alters the operation of the device in a manner well known to those skilled in the art. Data is stored as data structures in a physical location in memory, these data structures having specific properties defined by the data format. However, while the subject matter is described in the foregoing context, it is not intended to be limiting, as those skilled in the art will understand that several of the actions and operations described below can also be implemented in hardware.

[0046] To facilitate understanding of the subject matter described herein, many aspects are described as sequences of actions. At least one of these aspects as defined in the claims is performed by an electronic hardware component. For example, it will be appreciated that various actions can be performed by specialized circuits or loops, program instructions executed by one or more processors, or a combination of both. The description of any sequence of actions herein does not imply that a specific order in which that sequence is performed must be followed. All methods described herein can be performed in any suitable order unless otherwise stated herein or clearly contrary to the context.

[0047] In the context of describing the subject matter (particularly in the context of the following claims), the terms "a," "an," and "the," and similar designations, should be interpreted to cover both singular and plural forms, unless otherwise stated herein or clearly contradicted by the context. Unless otherwise stated herein, the enumeration of numerical ranges herein is merely intended as a shorthand method for individually referring to each individual value falling within the range, and each individual value is incorporated into the specification as if it were separately stated herein. Furthermore, the foregoing description is merely illustrative and not restrictive, as the scope of protection sought is defined by the claims described below and any equivalents thereof. The use of any and all examples or exemplary language (such as "such as") provided herein is merely for the purpose of better illustrating the subject matter and does not constitute a limitation on the scope of that subject matter, unless otherwise stated. The use of the term "based on" and other similar phrases indicating conditions that produce the result in the claims and the written description is not intended to exclude any other conditions that produce that result. No language in the specification should be construed as indicating that any unclaimed element is essential for the practice of the claimed invention.

[0048] The embodiments described herein include one or more modes known to the inventors for implementing the claimed subject matter. Variations of those embodiments will, of course, become apparent to those skilled in the art upon reading the foregoing description. The inventors intend that those skilled in the art will appropriately employ these variations, and the inventors intend to implement the claimed subject matter in ways other than those specifically described herein. Therefore, the claimed subject matter includes all modifications and equivalents of the subject matter listed in the appended claims, where permitted by applicable law. Furthermore, unless otherwise stated herein or clearly contradicted by the context, any combination of all possible variations of the foregoing elements is included.

[0049] While various embodiments have been described above, it should be understood that they are presented by way of example only and not as limitations. Therefore, the breadth and scope of preferred embodiments should not be limited by any of the exemplary embodiments described above, but should be defined solely by the following claims and their equivalents.

Claims

1. A non-transitory computer-readable medium storing processor-executable computer code to perform a method, the method comprising: Access to observations from multiple geographically dispersed sensors by edge devices within the network; The edge device processes the observation results to determine the overlapping portion of the observation results; The edge device determines the optimization of the observation results to form optimized observation results for transmission to the cloud processing system, which includes: For the overlapping portions of the observation results, only one of the overlapping portions of the observation results is selected to be transmitted to the cloud processing system; The edge device enables the multiple geographically dispersed sensors to transmit the optimized observations to the cloud processing system, thereby preventing duplicate observations from being transmitted to the cloud processing system.

2. The non-transitory computer-readable medium of claim 1, wherein the edge device is independent of the plurality of geographically dispersed sensors and is within a defined proximity range of the plurality of geographically dispersed sensors.

3. The non-transitory computer-readable medium of claim 2, wherein the observation results are accessed by the plurality of geographically dispersed sensors that transmit the observation results to the edge device.

4. The non-transitory computer-readable medium of claim 3, wherein the resolution of the observation is reduced for transmission to the edge device.

5. The non-transitory computer-readable medium of claim 1, wherein the edge device is one of the plurality of geographically dispersed sensors.

6. The non-transitory computer-readable medium of claim 5, wherein the one sensor is selected as the edge device based on a comparison of the resource availability of the one sensor among the plurality of geographically dispersed sensors with the resource availability of the remaining sensors among the plurality of geographically dispersed sensors.

7. The non-transitory computer-readable medium of claim 6, wherein the resource availability includes the availability of one or more computing resources.

8. The non-transitory computer-readable medium of claim 6, wherein the plurality of geographically dispersed sensors are mobile, and wherein the selection is performed at defined time intervals.

9. The non-transitory computer-readable medium of claim 1, wherein the edge device is assigned a region of interest.

10. The non-transitory computer-readable medium of claim 9, further comprising: The edge device notifies the plurality of geographically dispersed sensors of the region of interest, so that the plurality of geographically dispersed sensors provide the edge device with the observation results of the edge device within the region of interest.

11. The non-transitory computer-readable medium of claim 1, wherein one or more of the plurality of geographically dispersed sensors are mobile.

12. The non-transitory computer-readable medium of claim 11, wherein the edge device accesses the observations of one or more of the plurality of geographically dispersed sensors when one or more of the sensors are within a defined vicinity of the edge device.

13. The non-transitory computer-readable medium of claim 1, wherein the plurality of geographically dispersed sensors are cameras, and wherein the observations of the plurality of geographically dispersed sensors are images of the environment in which the geographically dispersed sensors are placed.

14. The non-transitory computer-readable medium of claim 1, wherein the overlapping portion comprises portions of different observations having a threshold similarity.

15. The non-transitory computer-readable medium according to claim 1, wherein, The multiple geographically dispersed sensors are stationary, allowing the overlapping portions identified in the observations to be reused to generate additional optimized observations for subsequent observations from the multiple geographically dispersed sensors.

16. A method for edge management of geographically dispersed sensors, comprising: Access observations from multiple geographically dispersed sensors via edge devices within the network; The edge device processes the observation results to determine the overlapping portion of the observation results; The edge device determines the optimization of the observation results to form optimized observation results for transmission to the cloud processing system, which includes: For the overlapping portions of the observation results, only one of the overlapping portions of the observation results is selected to be transmitted to the cloud processing system; The edge device enables the multiple geographically dispersed sensors to transmit the optimized observations to the cloud processing system, thereby preventing duplicate observations from being transmitted to the cloud processing system.

17. A system for edge management of geographically dispersed sensors, comprising: An edge device within the network, the edge device having a processor, the processor being used for: Access observations from multiple geographically dispersed sensors; The observation results are processed to determine the overlapping portion of the observation results; as well as Determining the optimization of the observation results to form optimized observation results for transmission to a cloud processing system includes: For the overlapping portions of the observation results, only one of the overlapping portions of the observation results is selected to be transmitted to the cloud processing system; The edge device enables the multiple geographically dispersed sensors to transmit the optimized observations to the cloud processing system, thereby preventing duplicate observations from being transmitted to the cloud processing system.

18. The system of claim 17, wherein the edge device communicates with the plurality of geographically dispersed sensors via the network.

Images