Real-time search and retrieval of streaming sensor data

Abstract

Systems and technologies for processing streaming data are provided. For example, the process may involve an edge device acquiring multiple data streams corresponding to objects, activities, or events registered in the edge environment associated with the edge device. One or more machine learning networks implemented on the edge device can generate multiple features and / or embeddings corresponding to each of the multiple data streams. Based on analyzing the multiple features and / or embeddings generated for each of the multiple data streams, the edge device can determine a subset of prominent content from the multiple data streams. The edge device can generate index information corresponding to the determined subset of prominent content. The subset of prominent content and the generated index information can be stored locally on the edge device for retrieval and searching based on local queries received on the edge device.

Description

Technical Field

[0001] The present disclosure relates to edge computing, and more specifically, to systems and techniques for performing real-time search and retrieval of streaming data or content at the edge without requiring a cloud connection. Background Art

[0002] Edge computing is a distributed computing paradigm that can be used to distribute data processing and other computational operations by bringing computational power and data storage closer to the edge (e.g., locations where computation and / or data storage are required, often the "edge" of a network such as the Internet). Edge computing systems are often deployed at the same location where input data is generated and / or the same location where the output results of computational operations are required. Because data is captured and processed locally at the edge rather than being transmitted to a more centralized location for processing, the use of edge computing systems can reduce latency and bandwidth usage. Edge computing systems can provide a secure and private environment that can store and process sensitive data locally.

[0003] In many existing cloud computing architectures, data generated at endpoints (e.g., mobile devices, Internet of Things (IoT) sensors, robots, industrial automation systems, security cameras, etc., among various other edge devices and sensors) is sent to a centralized data center for processing. The processed results are then sent from the centralized data center to endpoints requesting those results. The centralized processing approach can present challenges for growing use cases such as real-time applications and / or artificial intelligence (AI) and machine learning (ML) workloads. For example, centralized processing models and traditional cloud computing architectures can face constraints in the areas of latency, availability, bandwidth usage, data privacy, network security, and the ability to process large amounts of data in a timely manner.

[0004] In the context of edge computing, "edge" refers to the network edge closest to endpoint devices and data sources. In edge computing architectures, computation and data storage are distributed across a network of edge nodes located near endpoint devices and data sources. Edge nodes can be configured to perform a variety of tasks related to data processing, storage, and analysis. Based on processing data locally using edge nodes, the amount of data transferred from the edge to the cloud (or other centralized data centers) can be significantly reduced. Therefore, the use of edge computing is becoming increasingly popular to implement a diverse range of AI and ML applications, as well as to provide other use cases that require real-time processing, minimal latency, and high reliability. Generally, such applications and use cases may rely on high-bandwidth sensors capable of generating data at massive rates (e.g., on the order of 50 Gbit / s or 22 TB / hour). [Overview of the project]

[0005] The following provides a simplified overview relating to one or more embodiments disclosed herein. Therefore, this overview should not be considered a broad overview of all intended embodiments, nor should it be considered to identify any major or significant elements relating to all intended embodiments, or to define the scope relating to any particular embodiment. Accordingly, the sole purpose of this overview is to provide, in a simplified form, certain concepts relating to one or more embodiments of the mechanisms disclosed herein, prior to the detailed descriptions presented below.

[0006] Systems, methods, apparatus, and computer-readable media for processing streaming data at the edge are disclosed. For example, the systems and technologies may be used to perform real-time retrieval of streaming sensor data and / or streaming content at the edge, based on processing, storing, and / or indexing sensor data or other content in-situ (e.g., after it is generated at the edge) using edge computing devices. In some embodiments, the systems and technologies may be used to perform real-time retrieval at the edge without relying on cloud connectivity, storage, and / or computing resources. In some examples, a hybrid approach can be implemented to retrieve and rank content based on a combination of indexes and content stored at the edge and indexes and content residing in the cloud.

[0007] A method for processing streaming data is provided by at least one exemplary example. The method includes: an edge device acquiring multiple data streams corresponding to objects, activities, or events registered in an edge environment associated with the edge device; using one or more machine learning networks implemented on the edge device to generate multiple features corresponding to each of the multiple data streams; the edge device determining a subset of prominent content from the multiple data streams, the subset of prominent content being determined based on an analysis of multiple features generated for each of the respective data streams; the edge device generating index information corresponding to the determined subset of prominent content; and locally storing the subset of prominent content and the generated index information on the edge device for retrieval and search based on local queries received on the edge device.

[0008] In another exemplary example, a device for processing streaming data is provided. The device includes at least one memory and at least one processor coupled to at least one memory, the at least one processor being configured such that, by an edge device, multiple data streams corresponding to objects, activities, or events registered in the edge environment associated with the edge device, one or more machine learning networks implemented on the edge device to generate multiple features corresponding to each of the multiple data streams, the edge device to determine a subset of prominent content from the multiple data streams, the subset of prominent content being determined based on an analysis of multiple features generated for each of the respective data streams, the edge device to generate index information corresponding to the determined subset of prominent content, and the subset of prominent content and the generated index information being stored locally on the edge device for retrieval and search based on local queries received on the edge device.

[0009] In another exemplary example, a non-temporary computer-readable storage medium is provided, which stores instructions, and when the instructions are executed by at least one processor, it causes at least one processor to retrieve multiple data streams corresponding to objects, activities, or events registered in the edge environment associated with the edge device, to use one or more machine learning networks implemented on the edge device to generate multiple features corresponding to each of the multiple data streams, to cause the edge device to determine a subset of prominent content from the multiple data streams, the subset of prominent content being determined based on an analysis of the multiple features generated for each of the respective data streams, the edge device generating index information corresponding to the determined subset of prominent content, and storing the subset of prominent content and the generated index information locally at the edge.

[0010] Another exemplary example provides a device for processing streaming data. The device includes means for an edge device to acquire multiple data streams corresponding to objects, activities, or events registered in an edge environment associated with the edge device; means for generating multiple features corresponding to each of the multiple data streams using one or more machine learning networks implemented on the edge device; means for the edge device to determine a subset of prominent content from the multiple data streams, wherein the subset of prominent content is determined based on analyzing multiple features generated for each of the respective data streams; means for the edge device to generate index information corresponding to the determined subset of prominent content; and means for locally storing the subset of prominent content and the generated index information on the edge device for retrieval and search based on local queries received on the edge device.

[0011] As used herein, the terms “User Equipment” (UE) and “Network Entity” are not intended to be specific to, or otherwise limited to, any particular radio access technology (RAT) unless otherwise stated. Generally, a UE may be any wireless communication device (e.g., mobile phones, routers, tablet computers, laptop computers, and / or tracking devices) used by a user to communicate over a wireless communication network, wearables (e.g., smartwatches, smart glasses, wearable rings, and / or extended reality (XR) devices such as virtual reality (VR) headsets, augmented reality (AR) headsets or glasses, or mixed reality (MR) headsets), vehicles (e.g., automobiles, motorcycles, bicycles, etc.), robotic units (e.g., unmanned ground vehicles, unmanned aerial vehicles, articulated arms, visual inspection systems, cobots, etc.), and / or Internet of Things (IoT) devices. A UE may be mobile or stationary (for example, at a particular time) and may communicate with a radio access network (RAN). The term “UE” as used herein may be interchangeably referred to as “access terminal” or “AT,” “client device,” “wireless device,” “subscriber device,” “subscriber terminal,” “subscriber station,” “user terminal” or “UT,” “mobile device,” “mobile terminal,” “mobile station,” or variations thereof. A UE can typically communicate with the core network via the RAN, through which the UE may connect to external networks such as the Internet and other UEs. Of course, other mechanisms for connecting to the core network and / or the Internet are also possible for a UE, such as via a wired access network or a wireless local area network (WLAN) network (for example, based on the IEEE 802.11 communication standard).

[0012] The terms “network entity” or “base station” can refer to a single physical transmit / receive point (TRP) or to multiple physical transmit / receive points (TRPs) that may or may not be colocated. For example, when the terms “network entity” or “base station” refer to a single physical TRP, the physical TRP may be the antennas of a base station (e.g., a satellite constellation ground station / Internet gateway) corresponding to a cell (or several cell sectors) of the base station. When the terms “network entity” or “base station” refer to multiple colocated physical TRPs, the physical TRPs may be an array of antennas of the base station (e.g., as in a multiple-input multiple-output (MIMO) system, or if the base station employs beamforming). When the terms “base station” refer to multiple uncolocated physical TRPs, the physical TRPs may be a distributed antenna system (DAS) (a network of spatially separated antennas connected to a common source via a transport medium) or a remote radio head (RRH) (a remote base station connected to a serving base station). Since a TRP is the point from which a base station transmits and receives wireless signals, as used herein, references to transmission from or reception at a base station should be understood to refer to a specific TRP of the base station. An RF signal includes electromagnetic waves of a given frequency that transport information through the space between a transmitter and a receiver. As used herein, a transmitter may transmit a single “RF signal” or multiple “RF signals” to a receiver. However, a receiver may receive multiple “RF signals” corresponding to each transmitted RF signal due to the propagation characteristics of RF signals through multipath channels. The same transmitted RF signal on different paths between a transmitter and a receiver may be called a “multipath” RF signal. As used herein, an RF signal may also be called a “wireless signal” or simply a “signal” where it is clear from the context that the term “signal” refers to a wireless signal or an RF signal.

[0013] This summary is not intended to identify any significant or essential features of the claimed subject matter, nor is it intended to be used alone to determine the scope of the claimed subject matter. The subject matter should be understood by referring to the appropriate parts of the entire specification of this patent, any or all of the drawings, and each claim. The foregoing will become clearer by referring to the following specification, claims, and accompanying drawings, along with other features and embodiments. [Brief explanation of the drawing]

[0014] To illustrate how the above and other advantages and features of this disclosure can be obtained, a more specific description of the principles briefly described above is made by reference to the specific embodiments shown in the accompanying drawings. The use of the same reference numeral in different drawings indicates similar or identical items or features. With understanding that these drawings only illustrate exemplary embodiments of this disclosure and should therefore not be considered to limit its scope, the principles of this specification are described and explained with additional specificities and details through the use of the accompanying drawings. [Figure 1] This figure shows exemplary implementations of a system-on-a-chip (SoC) using several examples. [Figure 2A] This section presents an example of a fully connected neural network using several examples. [Figure 2B] This section presents an example of a locally connected neural network using several examples. [Figure 3] This diagram illustrates an example of an edge computing system for statistical processing, machine learning (ML), and / or artificial intelligence (AI) workloads, with several examples illustrating that the edge computing system includes one or more local sites, each having one or more edge computing units. [Figure 4]This figure shows some examples of content storage and indexing that can be performed at the edge (for example, by an edge computing system or device and / or at an edge location), using several examples. [Figure 5] This figure shows some examples of content search and ranking analysis that can be performed at the edge (for example, by an edge computing system or device and / or at an edge location), using several examples. [Figure 6] This figure shows examples of content search and ranking analysis that can be performed in the cloud based on local query information acquired at the edge (e.g., by an edge computing system or device and / or at an edge location), with several examples. [Figure 7] This block diagram shows an example of a computing system architecture that may be used to implement one or more embodiments described herein, with some examples. [Modes for carrying out the invention]

[0015] Specific embodiments of this disclosure are provided below for illustrative purposes. Alternative embodiments can be devised without departing from the scope of this disclosure. In addition, well-known elements of this disclosure are not described in detail or are omitted so as not to obscure the relevant details of this disclosure. As will be apparent to those skilled in the art, some of the embodiments described herein can be applied independently, and some of them can be applied in combination. In the following description, certain details are given for illustrative purposes to provide a complete understanding of the embodiments of this application. However, it will be apparent that various embodiments can be carried out without these specific details. The drawings and description are not intended to limit.

[0016] The following description provides only exemplary embodiments and does not limit the scope, applicability, or configuration of the disclosure. Rather, the following description of exemplary embodiments provides a possible description for carrying out the exemplary embodiments for those skilled in the art. It should be understood that various modifications can be made to the function and configuration of the elements without departing from the scope of this application as set forth in the appended claims.

[0017] overview This specification describes systems, apparatus, methods (also called processes), and computer-readable media (collectively referred to herein as “Systems and Technologies”) for real-time retrieval of streaming sensor data and / or streaming content at the edge. In one exemplary example, real-time retrieval of streaming sensor data and / or streaming content can be performed using one or more edge computing devices (e.g., also referred to herein as “Edge Devices”) deployed at an edge site or edge location. In some examples, the edge computing devices may be provided as High Performance Computing and Storage (HPCS) edge devices, including elastic HPCS (E-HPCS) edge devices.

[0018] According to aspects of this disclosure, real-time retrieval and access to streaming sensor data and / or streaming content can be performed at the edge without requiring cloud connectivity, cloud storage, and / or cloud computing. For example, one or more edge devices (e.g., HPCS, E-HPCS, etc.) can be used to process, store, and index each stream of sensor data or other content in situ at the edge. Each stream of sensor data or other content at the edge can be processed, stored, and indexed substantially in real time after or as soon as each stream of sensor data or content is generated.

[0019] Performing real-time search and retrieval at the edge can otherwise reduce or eliminate the latency associated with conventional implementations in which data generated at the edge must be transmitted to a centralized cloud or data center for processing, and then processing results are sent back from the centralized cloud or data center to the edge. Further, the systems and techniques described in this specification for real-time search and retrieval can reduce search and retrieval times associated with querying data indexed and / or stored at the edge. For example, a query may be received locally (e.g., at the edge) and analyzed against index information corresponding to data stored and / or previously ingested at the edge. In some aspects, the edge device may generate the index information in the form of an inverted index (also referred to as, for example, a "reverse index").

[0020] In some aspects, the systems and techniques may be configured to perform real-time index generation in-situ at an edge location or edge device that receives streaming data and / or content. For example, as data (e.g., video, images, text and / or numerical data, audio files or transcripts, sensor readings, etc.) is streamed from multiple sensors to the edge device, the edge device can generate corresponding index information for the streamed data substantially in real time. Streaming data ingested into the edge device and processed by the edge device can be immediately indexed and stored locally at the edge. For example, streaming data may be stored onboard the edge device, may be stored using one or more local edge databases associated with the edge device, or may otherwise be deployed at the same edge location, and the like. The streaming data may be stored in combination with corresponding index information generated by the edge device as each respective portion of the streaming data is ingested.

[0021] In one illustrative example, an edge device may be implemented as a high-performance computing (HPC) device. The HPC edge device may perform indexing (e.g., real-time or near real-time index generation) based on using one or more machine learning (ML) and / or artificial intelligence (AI) models, networks, algorithms, etc., to perform tasks that may include, but are not limited to, feature extraction, embedding generation, tokenization, and the like for data streaming from connected sensors or other edge assets associated with the HPC edge device.

[0022] These aspects and further aspects are described in more detail below with reference to the drawings.

Examples

[0023] Figure 1 shows an exemplary implementation of a system-on-chip (SOC) 100 that may include a central processing unit (CPU) 102 or a multi-core CPU, configured to perform one or more of the functions described herein. Parameters or variables (e.g., neural signals and synaptic weights), system parameters associated with a computing device (e.g., neural networks with weights), delays, frequency bin information, and task information, among other information, may be stored in a memory block associated with a neural processing unit (NPU) 108, a memory block associated with the CPU 102, a memory block associated with a graphics processing unit (GPU) 104, a memory block associated with a digital signal processor (DSP) 106, a memory block 118, and / or may be distributed across a plurality of blocks. Instructions executed on the CPU 102 may be loaded from a program memory associated with the CPU 102, or may be loaded from the memory block 118.

[0024] The SOC 100 may also include a GPU 104, a DSP 106, a connectivity block 110 which may include fifth-generation (5G) connectivity, fourth-generation long-term evolution (4G LTE) connectivity, WiFi connectivity, USB connectivity, Bluetooth connectivity, etc., and additional processing blocks tailored to specific functions, such as a multimedia processor 112 which can detect and recognize gestures. In one implementation, the NPU is implemented within the CPU 102, DSP 106, and / or GPU 104. The SOC 100 may also include a navigation module 120 which may include a sensor processor 114, an image signal processor (ISP) 116, and / or a global positioning system. In some examples, the sensor processor 114 may be associated with or connected to one or more sensors to provide sensor inputs (one or more) to the sensor processor 114. For example, one or more sensors and the sensor processor 114 may be located in the same computing device, coupled, or otherwise associated.

[0025] The SOC 100 may be based on the ARM instruction set. In one aspect of this disclosure, instructions loaded into the CPU 102 may include code for retrieving a stored multiplication result in a lookup table (LUT) corresponding to the product of the multiplication of the input value and the filter weights. Instructions loaded into the CPU 102 may also include code for disabling the multiplier during the multiplication operation of the product of the multiplications when a lookup table hit is detected for the product of the multiplications. Furthermore, instructions loaded into the CPU 102 may include code for storing the calculated product of the multiplications of the input value and the filter weights when a lookup table miss is detected for the product of the multiplications. The SOC 100 and / or its components may be configured to perform image processing using machine learning techniques according to aspects of this disclosure described herein. For example, the SOC 100 and / or its components may be configured to perform semantic image segmentation and / or object detection according to aspects of this disclosure.

[0026] Machine learning (ML) can be considered a subset of artificial intelligence (AI). ML systems can include algorithms and statistical models that computer systems can use to perform various tasks by relying on patterns and reasoning without using explicit instructions. An example of an ML system is a neural network (also called an artificial neural network), which may include an interconnected group of artificial neurons (e.g., neuron models). Neural networks can be used in a variety of applications and / or devices, including, among others, image and / or video coding, image analysis and / or computer vision applications, Internet Protocol (IP) cameras, Internet of Things (IoT) devices, autonomous vehicles, and service robots.

[0027] Individual nodes within a neural network can emulate biological neurons by acquiring input data and performing simple operations on it. The results of these operations are selectively passed to other neurons. Weight values ​​are associated with each vector and node in the network, and these values ​​constrain how input data is associated with output data. For example, the input data of each node can be multiplied by the corresponding weight value, and the products can be summed. The sum of the products may be adjusted by an arbitrary bias, and an activation function may be applied to the result to produce an output signal or "output activation" (sometimes called a feature map or activation map) for the node. The weight values ​​may initially be determined by an iterative flow of training data through the network (for example, the weight values ​​are established during the training phase, when the network learns how to identify a particular class by its typical input data characteristics).

[0028] In particular, there are different types of neural networks, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), generative adversarial networks (GANs), multilayer perceptron (MLP) neural networks, and transformer neural networks. For example, a convolutional neural network (CNN) is a type of feedforward artificial neural network. A convolutional neural network can contain a set of artificial neurons that collectively tile the input space, each having a receptive field (e.g., a spatially localized region of the input space). RNNs work on the principle of storing the output of a layer and feeding this output back into the input to help predict the outcome of the layer. GANs are a form of generative neural network that can learn patterns in input data so that the neural network model can produce a new synthetic output that could reasonably have been obtained from the original dataset. A GAN can include two neural networks working together, one that includes a generative neural network that produces the synthetic output and another that includes a discriminative neural network that evaluates the output for authenticity. In an MLP neural network, data may be fed into the input layer, and one or more hidden layers provide a level of abstraction to the data. Predictions can then be made on the output layer based on the abstracted data.

[0029] Deep learning (DL) is an example of machine learning techniques and can be considered a subset of ML. Many DL approaches are based on neural networks such as RNNs or CNNs and utilize multiple layers. The use of multiple layers in a deep neural network allows progressively higher levels of features to be extracted from a given input of raw data. For example, the output of the first layer of an artificial neuron becomes the input to the second layer of the artificial neuron, the output of the second layer of the artificial neuron becomes the input to the third layer of the artificial neuron, and so on. Layers located between the inputs and outputs of the entire deep neural network are often called hidden layers. Hidden layers learn (e.g., are trained) to transform intermediate inputs from preceding layers into slightly more abstract and complex representations that can be provided to subsequent layers until the final representation or desired representation is obtained as the final output of the deep neural network.

[0030] As described above, a neural network is an example of a machine learning system that can include an input layer, one or more hidden layers, and an output layer. Data is provided from the input nodes of the input layer, processing is performed by the hidden nodes of one or more hidden layers, and the output is produced via the output nodes of the output layer. Deep learning networks typically include multiple hidden layers. Each layer of a neural network can include a feature map or activation map that can include artificial neurons (or nodes). A feature map can be an output that applies one or more filters, kernels, weights, etc., to the input. A node can include one or more weights used to indicate the importance of one or more nodes in the layer. In some cases, a deep learning network can have a series of many hidden layers, with the initial layers used to determine simple and low-level characteristics of the input, and later layers building a hierarchy of more complex and abstract characteristics.

[0031] Deep learning architectures can learn feature hierarchies. For example, when visual data is presented, the first layer may learn to recognize relatively simple features such as edges in the input stream. In another example, when auditory data is presented, the first layer may learn to recognize spectral power at a particular frequency. A second layer may take the output of the first layer as input and learn to recognize combinations of features, such as simple shapes for visual data or combinations of sounds for auditory data. For example, higher layers may learn to represent complex shapes in visual data or words in auditory data. Even higher layers may learn to recognize common visual objects or spoken phrases.

[0032] Deep learning architectures can perform particularly well when applied to problems with natural hierarchical structures. For example, classifying powered vehicles can benefit from initial learning to recognize wheels, windshields, and other features. These features can then be combined in different ways in higher layers to recognize cars, trucks, and airplanes.

[0033] Neural networks can be designed with various connection patterns. In a feedforward network, information is passed from lower layers to higher layers, and each neuron in a given layer communicates with neurons in higher layers. Hierarchical representations can be constructed from consecutive layers of feedforward networks, as described above. Neural networks can also have recurrent or feedback (also called top-down) connections. In recurrent connections, the output from a neuron in a given layer can communicate with another neuron in the same layer. Recurrent architectures can be useful for recognizing patterns that span two or more chunks of input data delivered sequentially to the neural network. Connections from neurons in a given layer to neurons in lower layers are called feedback (or top-down) connections. Networks with many feedback connections can be useful when the recognition of a higher-level concept can help distinguish specific low-level features of the input.

[0034] The connections between layers of a neural network may be fully connected or locally connected. Figure 2A shows an example of a fully connected neural network 202. In the fully connected neural network 202, neurons in the first layer may communicate their outputs to all neurons in the second layer, so that each neuron in the second layer receives input from all neurons in the first layer. Figure 2B shows an example of a locally connected neural network 204. In the locally connected neural network 204, neurons in the first layer may be connected to a limited number of neurons in the second layer. More generally, the locally connected layer of the locally connected neural network 204 may be configured such that each neuron in the layer has the same or similar connection pattern but has a connection strength that can have different values ​​(e.g., 210, 212, 214, and 216). Since higher-layer neurons in a given region can receive inputs that are trained to adapt to the characteristics of a limited portion of the total input to the network, local connectivity patterns can create spatially distinct receptive fields in the higher layers.

[0035] Streaming sensor data and / or streaming content at the edge Various AI and ML applications (also known as workloads, workflows, tasks, etc.) can benefit from edge computing or from being implemented at the edge in other ways. Edge computing can play a crucial role in delivering a wide range of AI and ML applications, including (but not limited to) use cases that leverage real-time processing, high reliability, and low latency, all of which are characteristics of edge computing and edge computing units as described herein.

[0036] For example, edge-deployed AI and ML applications may heavily utilize one or more high-bandwidth sensors (e.g., high-speed and / or HD cameras, stereo cameras, 3D scanners, LiDAR cameras and / or sensor systems, RADAR, ultrasonic sensors, vibrometers, accelerometers, gyroscopes, and other inertial sensor packages, fiber optic sensors, radar, ultrasonic sensors, etc.). Furthermore, multimodal sensor packages may include multiple sensors operating across multiple different modalities and / or sensor domains. These multimodal sensor packages can generate and / or stream data at rates exceeding 50 Gbit / s (e.g., 22 TB / hr).

[0037] High-bandwidth or other sensor data streams may be supplied to one or more (e.g., trained) AI or ML models configured to process such sensor data for purposes such as real-time decision-making and analysis (among a variety of other purposes). For example, ML and AI models that may be associated with ingesting and / or processing large or high-bandwidth data streams may include, but are not limited to, deep neural networks (DNNs), convolutional neural networks (CNNs), region-based CNNs (R-CNNs), recurrent neural networks (RNNs), long short-term memory (LSTM) networks, vision transformers (ViTs), variational autoencoders (VAEs), generative adversarial networks (GANs), autoencoders, transformers, bidirectional encoder representations from transformers (BERT), stable diffusion, attention mechanisms, and / or large-scale language models (LLMs).

[0038] In one exemplary example, a typical offshore drilling platform generates a considerable amount of data, ranging from 2 to 4 TB per day. Of this 2 to 4 TB of raw data generated daily, approximately 80% may remain unused (e.g., 1.6 to 3.2 TB / day). Even a single oil rig operating remotely in a northern environment can generate over 1 TB of data per day, and in this example, less than 1% of that data is used for analysis and / or decision-making purposes. The challenge of the volume of data generated can become exponentially more difficult with increasing remoteness of the operating environment. In particular, the difficulty of managing and utilizing large amounts of generated sensor data can increase non-linearly with the distance from existing data networks and other communication infrastructure for uploading / transmitting the generated sensor data.

[0039] For example, sensor data generated in a remote operating environment often cannot be transmitted via conventional fiber optics or other physical / wired internet communication links, largely due to the lack of such infrastructure within or near the remote operating environment. Therefore, sensor data generated in a remote operating environment often has to be transmitted via much slower (and often more expensive) wireless communication links, such as cellular and / or satellite communication links.

[0040] A satellite communication link with an upload speed of 25 Mbps takes approximately 90 hours (about 4 days) to transmit 1 TB of data, meaning that an exemplary oil rig generating 1 TB / day would rapidly hinder any data upload from the oil rig to a data center. As the amount of data generated increases, this challenge becomes more severe and increasingly unacceptable. For example, a large refinery can easily generate more than 10 TB of raw data per day.

[0041] One common type of unstructured data that can be generated daily (or otherwise regularly) is video data captured by cameras local to the operating site. Video data captured by cameras often falls into the category of unstructured data because such video data consists of raw visual information without a predefined structure or format. The increasing use and availability of high-resolution cameras for tasks such as video-based monitoring, scene understanding, and / or navigation applications has led to a surge in unstructured video data generation. For example, a 4K camera capturing 30 frames per second (fps) will generate 5.4 TB of uncompressed video data in less than an hour.

[0042] A similar increase in raw data generation can be seen in the context of autonomous vehicles (AVs) equipped with multiple camera and sensor packages (e.g., lidar, radar, ultrasonic sensors, IMU / INS, etc.). The data generation rate of a single AV can reach or exceed 50 Gbit / s. In AV use cases, a significant portion of this 50 Gbit / s of raw data generation may require local and real-time processing to enable low-latency decisions for the AV's navigation and control as it moves through its environment. In particular, even a single IP camera can contribute considerably to the overall sensor data stream described above, streaming at rates ranging from 0.01 to 1.20 Mbit / s, and a single IP camera can generate somewhere between 5 and 500 MB of data per hour.

[0043] Sensors such as IP cameras are often deployed in large numbers, and as a result, these deployments, which at first glance seem to be low-bandwidth contributors, can have a significant impact when considered as a whole. Consider the example of a stadium equipped with IP cameras as part of a large-scale security or monitoring system with a total deployment of 900 IP cameras. In just one hour, the 900 IP camera security system alone can generate 1 / 2 terabyte (0.5 TB) of video data. In security and monitoring scenarios, IP camera video data needs to be processed in near real-time for purposes such as event logistics, threat detection, and safety monitoring. While various forms of unstructured data (e.g., IP camera video data) can be indexed and stored for later retrieval, the common use cases of such unstructured data themselves are often the primary driving force behind the need for near real-time processing and analysis capabilities.

[0044] Table 1 below summarizes various exemplary scenarios / use cases for AI and ML applications in the context of edge computing. Table 1 also presents exemplary sensing, bandwidth, computation, and storage requirements corresponding to each exemplary use case (however, note that the information in Table 1 is provided for illustrative and illustrative purposes only and is not intended to be construed as limiting).

[0045] [Table 1]

[0046] Figure 3 shows an example of an edge computing system 300 that may be used to implement or perform one or more aspects of the present disclosure. For example, a system and technology for real-time retrieval of streaming sensor data may be implemented at a local edge site (e.g., an edge environment) 302 using one or more edge computing units 330. In some embodiments, the edge computing units 330 may also be referred to as “edge devices”. In some embodiments, the edge computing units 330 may be provided as high-performance computing and storage (HPCS) and / or elastic-HPCS (E-HPCS) edge devices.

[0047] For example, local site 302 may be one of several edge environments / edge deployments associated with edge computing system 300. Multiple local sites may include local site 302 and an additional number N of local sites 302-N, each of which may be the same as or similar to local site 302. Local site 302 may be a geographical location associated with an enterprise user or other users of edge computing. Local site 302 may be an edge location in terms of data network connectivity (i.e., edge environment 302 is the local geographical location of an enterprise user and an edge location in the corresponding data network topography).

[0048] In the example in Figure 3, the edge environment 302 includes one or more edge computing units 330. Each edge computing unit 330 may be configured as a containerized edge computing unit or data center for performing sensor data generation or acquisition and inference for one or more trained ML / AI models provided on the edge computing unit 330. For example, the edge computing unit 330 may include computing hardware components configured to perform inference for one or more trained AI / ML models. As illustrated, a first portion of the hardware resources of the edge computing unit 330 may be associated with or used to perform inference for a first AI / ML model 335-1, ..., and N AI / ML models 335-N. In other words, the edge computing unit 330 may be configured with computing hardware and computing power for performing inference using multiple different AI / ML models. Inference for multiple AI / ML models may be performed simultaneously or in parallel for multiple of the N AI / ML models 335-1, ..., 335-N. In some embodiments, inference may be performed on a first subset of N AI / ML models during a first time segment, and on a second subset of N AI / ML models during a second time segment, and so on. The first and second subsets of AI / ML models may be disjoint or overlap.

[0049] In some embodiments, the edge computing unit 330 may be associated with performing one or more (or all) on-premises training (or retraining) of one or more AI / ML models from a group of AI / ML models, fine-tuning one or more AI / ML models from a group of AI / ML models, and / or performing instruction tuning of one or more AI / ML models from a group of AI / ML models. For example, a subset of AI / ML models deployed (or otherwise deployable) to the edge computing unit 330 may be trained or fine-tuned on-premises at the local edge site 302 without relying on the cloud (e.g., without relying on a cloud-based AI / ML training cluster implemented within a cloud user environment 370). In some embodiments, the edge computing unit 330 can perform on-premises training or retraining, fine-tuning, and / or instruction tuning of one or more AI / ML models from a group of AI / ML models to account for model degradation or drift over time. In some examples, the edge computing unit 330 can perform on-premise training or retraining, fine-tuning, and / or instruction adjustments on one or more AI / ML models from a group of AI / ML models to adapt each AI / ML model from the task to which each model was initially trained (e.g., pre-trained) to a new or differentiated task.

[0050] In some cases, fine-tuning of an AI / ML model can be performed in the cloud (e.g., using a cloud-based AI / ML training cluster implemented within a cloud user environment 370), at the edge (e.g., in a local edge environment 302, using edge computing units 330 and AI / ML model fine-tuning 334-1, ..., 334-M), and / or using a distributed combination across the cloud and one or more edge computing units 330. In some cases, fine-tuning of an AI / ML model can be performed in either the cloud or the edge environment 302 (or both), based on the significantly less computing power and data required to perform fine-tuning and / or instruction tuning of an AI / ML model trained for a particular task compared to the computing power and data required to initially train the AI / ML model for either the specific task or a broader class of tasks including the specific task.

[0051] In some embodiments, the edge computing unit 330 may include computing hardware components that can be configured to perform training, retraining, fine-tuning, etc., on one or more trained AI / ML models. In some embodiments, at least some of the computing hardware components of the edge computing unit 330 used to implement AI / ML model inference 335-1, ..., 335-N may also be used to perform AI / ML model retraining 333-1, ..., 333-K and / or AI / ML model fine-tuning 334-1, ..., 334-M. For example, computing hardware components included in the edge computing unit 330 (e.g., CPU, GPU, NPU, hardware accelerator, etc.) may be configured to perform various combinations of model inference, model retraining, and / or model fine-tuning at the edge (e.g., at the local edge site 302). At least some of the K AI / ML models 333-1, ..., 333-K associated with model retraining at the edge may be included in the N AI / ML models associated with model inference at the edge. Similarly, at least some of the M AI / ML models 334-1, ..., 334-M related to model fine-tuning at the edge may be included in the N AI / ML models related to model inference at the edge.

[0052] In some embodiments, for a given pre-trained AI / ML model received by the edge computing unit 330 (for example, received from an AI / ML training cluster in a cloud user environment 370), the edge computing unit 330 may be configured to perform one or more (or all) of the following at the edge: model inference 335, model retraining 333, and / or model fine-tuning 334.

[0053] As shown in Figure 3, retraining of multiple AI / ML models can be performed simultaneously or in parallel on multiple of the K AI / ML models 333-1, ..., 335-K (which may be the same as, similar to, or different from, N AI / ML models 335-1, ..., 335-N, and / or the same as, similar to, or different from, M AI / ML models 334-1, ..., 334-M, as described above). In some embodiments, retraining may be performed on a first subset of K AI / ML models during a first time portion, on a second subset of K AI / ML models during a second time portion, and so on. The first and second subsets of K AI / ML models may be disprime or overlap. Additionally or alternatively, fine-tuning of multiple AI / ML models can be performed simultaneously or in parallel on multiple of the M AI / ML models 334-1, ..., 334-M (which may be the same as, similar to, or dissimilar to, N AI / ML models 335 and / or K AI / ML models 333). In some embodiments, fine-tuning may be performed on a first subset of M AI / ML models during a first time portion, on a second subset of M AI / ML models during a second time portion, and so on. The first and second subsets of M AI / ML models may be dissimilar or overlap.

[0054] Each edge computing unit 330, which comprises one or more edge computing units provided in each of the multiple edge environments 302-N, may further include a cloud service 332, a high-performance computing (HPC) engine 334, and a local database 336. In some embodiments, the HPC engine 334 may be used to perform and / or manage inferences associated with each of the trained AI / ML models 335-1, ..., 335-N provided on the edge computing unit 330.

[0055] In one exemplary example, the edge computing unit 330 can receive trained AI / ML models 335-1, ..., 335-N from a centralized AI / ML training cluster or engine provided by one or more cloud user environments 370. The AI / ML training cluster in the cloud user environment 370 may be used to perform training (e.g., pre-training) of AI / ML models that may later be deployed to the edge computing unit 330 for inference and / or other implementations in the edge environment 302. Data network connectivity between the edge computing unit 330 and the cloud user environment 370 may be provided using one or more internet backhaul communication links 340. For example, the internet backhaul 340 may be implemented as fiber communication links (e.g., wired fiber optic connections to an internet infrastructure that can connect from the edge environment 302 / edge computing unit 330 to the cloud user environment 370, direct or point-to-point wired fiber optic connections from the edge environment 302 / edge computing unit 330 to the cloud user environment 370, etc.).

[0056] The internet backhaul 340 may be implemented using one or more satellite communication links, either additionally or alternatively. For example, the internet backhaul 340 may be a wireless communication link between the edge computing unit 330 / edge environment 302 and the satellites of a satellite internet constellation. In some embodiments, the edge computing unit 330 is intended to include (or otherwise associated with) one or more satellite transceivers for implementing satellite connectivity to and / or from the edge computing unit 330. In some embodiments, one or more satellite transceivers may be integrated or coupled to the housing of the edge computing unit 330 (e.g., a container in an example where the edge computing unit 330 is a containerized data center) and used to provide satellite connectivity that can implement the internet backhaul link 340. In another embodiment, one or more satellite transceivers may be provided in the edge environment 302 where the edge computing unit 330 is deployed, either additionally or alternatively.

[0057] In some embodiments, the internet backhaul link 340 between the edge computing unit 330 and the cloud user environment 370 may be used to provide an uplink (e.g., from the edge computing unit 330 to the cloud user environment 370) for scheduled batch uploads of information corresponding to one or more AI / ML models 335-1, ..., 335-N implemented by the edge computing unit 330, one or more features (intermediate or output) generated by the AI / ML models implemented by the edge computing unit 330, and / or one or more sensor data streams generated by edge assets 310 associated with the edge computing unit 330 and provided in the edge environment 302. The internet backhaul link 340 may further be used to provide a downlink (e.g., from the cloud user environment 370 to the edge computing unit 330) of the AI / ML models, such as updated, retrained, and fine-tuned models. For example, an updated, retrained, or fine-tuned AI / ML model transmitted from the cloud user environment 370 to the edge computing unit 330 via the internet backhaul link 340 can be updated, retrained, or fine-tuned based on scheduled batch upload data transmitted uplink from the edge computing unit 330 to the cloud user environment 370. In some embodiments, the updated AI / ML model transmitted from the cloud user environment 370 to the edge computing unit 330 may be an updated version of the same AI / ML model 335-1, ..., 335-N already implemented on the edge computing unit 330 (for example, already stored in the local database 336 for implementation on the edge computing unit 330). In another example, an updated AI / ML model sent from the cloud user environment 370 to the edge computing unit 330 may include one or more new AI / ML models that are not currently included (and / or were not previously included) in the set of AI / ML models 335-1, ..., 335-N, which are either implemented on the edge computing unit 330 or stored in the local database 336 for potential implementation on the edge computing unit 330.

[0058] In some cases, the AI / ML distributed computing platform 300 can perform local data capture and transmission using one or more edge computing units 330 provided in each edge environment 302. In particular, locally captured data can be obtained from one or more local sensors and / or other edge assets 310 provided in the edge environment 302. For example, in the example of Figure 3, the local edge assets / sensors 302 may include, but are not limited to, one or more autonomous robots 316, one or more local site cameras 314, one or more environmental sensors 312, etc. The local sensors and edge assets 310 can communicate with the edge computing units 330 in or via a local network 320 implemented for the edge environment 302.

[0059] For example, the local network 320 may be used to provide one or more communication links between the edge computing unit 330 and each of the edge assets 310. In one exemplary example, the local network 320 may be implemented as a private 5G, 4G, LTE, or other private cellular network; as a public 5G or other public cellular network; as a WiFi, Bluetooth, Zigbee, Z-wave, Long Range (LoRa), Sigfox, Narrowband-IoT (NB-IoT), LTE for Machines (LTE-M), IPv6 thread, or other short-range wireless network; as a local wired or fiber optic network, etc. As shown in the example in Figure 3, the edge computing unit 330 can receive different types of data from different edge assets / sensors 310 and can transmit different types of configurations / controls to different edge assets / sensors 310. For example, the edge computing unit 330 can receive onboard camera feeds and other sensor information (including 3D range scanner, LiDAR, or odometry information for simultaneous localization and mapping) from the autonomous robot 316, and in response can send routing commands to the autonomous robot 316. The routing commands may be generated or otherwise determined based on processing the onboard camera feed data from the autonomous robot 316 using one (or more) suitable trained AI / ML models 335-1, ..., 335-N implemented on the edge computing unit 330 and / or using the HPC engine 334 of the edge computing unit 330.

[0060] In another example, the edge computing unit 330 may receive local camera feed information from the local site camera 314 and, in response, transmit camera configuration and / or control information to the local site camera 314. In some cases, the edge computing unit 330 may receive local camera feed information from the local site camera 314 and transmit nothing in response. For example, camera configuration and / or control information may be used to rearrange or reconfigure one or more image capture parameters of the local site camera 314. If rearrangement or image capture parameter reconfiguration is not required, the edge computing unit 330 may not transmit camera configuration / control information in response. In some embodiments, camera configuration and / or control information may be generated or otherwise determined based on processing local camera feed data from the local site camera 314 using one or more suitable trained AI / ML models 335-1, ..., 335-N implemented on the edge computing unit 330 and / or using the HPC engine 334 of the edge computing unit 330.

[0061] In another example, the edge computing unit 330 may receive one or more environmental sensor data streams from the environmental sensor 312 and, in response, transmit sensor configuration / control information to the environmental sensor 312. In some cases, the edge computing unit 330 may receive sensor data stream information from the environmental sensor 312 and transmit nothing in response. For example, sensor configuration and / or control information may be used to adjust or reconfigure one or more sensor data acquisition parameters of the environmental sensor 312. If adjustment or reconfiguration of the environmental sensor 312 is not required, the edge computing unit 330 may not transmit sensor configuration / control information in response. In some embodiments, sensor configuration and / or control information may be generated based on or otherwise determined by processing the local environmental sensor data stream from the environmental sensor 312 using one or more suitable trained AI / ML models 335-1, ..., 335-N implemented on the edge computing unit 330 and / or using the HPC engine 334 of the edge computing unit 330.

[0062] In some examples, the systems and technologies described herein can be used to drive local storage, inference, prediction, and / or response performed by an edge computing unit (e.g., edge computing unit 330) with minimal or no reliance on cloud communication or cloud offloading of computing workloads (e.g., to a cloud user environment 370). The edge computing unit 330 may be further used to locally perform tasks such as background / batch data cleaning, ETL, and feature extraction. The local edge computing unit 330 can locally perform inference and generate predictions or inference results using one or more trained (e.g., pre-trained) AI / ML models 335-1, ..., 335-N received by the edge computing unit 330 from the cloud user environment 370. The local edge computing unit 330 may perform further fine-tuning or instruction tuning of the pre-trained model for a specified task (e.g., one or more corresponding AI / ML model fine-tuning instances 333-1, ..., 4333-M as described above).

[0063] The prediction or inference results (and / or intermediate features, related data, etc.) may be compressed by the edge computing unit 330 and periodically uploaded to the cloud or another centralized location (e.g., a cloud user environment 370). In one exemplary example, the compressed prediction or inference results may be uploaded to the cloud via a satellite communication link, such as a communication link to a satellite internet constellation configured to provide wireless satellite connectivity between the edge computing unit and existing terrestrial internet infrastructure. For example, the compressed prediction or inference results may be included in a scheduled batch upload transmitted from the edge computing unit 330 to the cloud user environment 370 via the internet backhaul link 340. In some cases, the prediction or inference results may be immediately available in the edge computing unit 330 and later transmitted (in a compressed form) to the cloud or a centralized location (e.g., a cloud user environment 370). In some embodiments, satellite connectivity may be used to periodically transmit or upload the compressed prediction or inference results, such as periodic transmissions during high-bandwidth or low-cost availability times of the satellite internet constellation. In some cases, some (or all) of the compressed prediction or inference results may be transmitted and / or retransmitted using wired or wireless backhaul means, if available, including fiber optic connections for internet backhaul, etc.

[0064] In particular, the system and technology can perform the tasks and operations described above locally onboard one or more edge computing units 330, while offloading more computationally intensive and / or less time-dependent tasks from the edge computing units to an AI / ML training cluster in the cloud user environment 370. For example, the AI / ML training cluster may be used to provide on-demand AI / ML model training and fine-tuning corresponding to the updated AI / ML model shown in Figure 3, which is transmitted from the cloud user environment 370 to the edge computing units 330 via the internet backhaul 340. In some embodiments, the AI / ML training cluster may implement thousands of GPUs or other high-performance computing hardware capable of training or fine-tuning an AI / ML model using thousands of GPUs over a long period of time (e.g., days, weeks, or more). In some embodiments, the AI / ML training cluster may be used to perform on-cloud model compression and optimization before sending data representing trained AI / ML models 335-1, ..., 335-N to the edge computing unit 330 for local implementation using sensor data generated by the associated edge asset 310, either additionally or alternatively. In some embodiments, the edge computing unit 330 may be configured to perform scheduled or periodic downloads of fresh (e.g., updated or new) AI / ML models from the AI / ML training cluster 370 via an internet backhaul link 340 (e.g., updated or new AI / ML models can be distributed in a pull manner from the AI / ML training cluster in the cloud user environment 370 to the edge computing unit 330). In another example, updated or new AI / ML models may be distributed via a push mechanism from the AI / ML training cluster in the cloud user environment 370 to the edge computing unit 330, and the AI / ML training cluster 370 sends the updated or new AI / ML model to the edge computing unit 330 via the internet backhaul link 340 as soon as the updated or new AI / ML model becomes available in the AI / ML training cluster.

[0065] Training AI / ML models 335-1, ..., 335-N may require massive amounts of data and processing power, which can be more efficiently performed in a cloud user environment 370 (and shared across edge computing units 330 in multiple edge environments 302-N) rather than individually in each of the corresponding edge computing units 330. In some aspects, the quality of an AI / ML model can be directly correlated to the size of the training and test (e.g., validation) data used to perform training and subsequent fine-tuning. Furthermore, training large-scale AI / ML models often requires running thousands of GPUs, ingesting hundreds of terabytes of data, and running these processes over several weeks. Therefore, training large-scale ML / AI models is often best suited to cloud or on-premises infrastructure and advanced MLOp. For example, the training datasets associated with training large-scale AI / ML models can be on the order of hundreds of TB to tens of petabytes (PB), or even more. This may require thousands of GPUs and several hours to several weeks of training time, and the resulting uncompressed trained model could exceed hundreds or thousands of gigabytes in size.

[0066] On the other hand, ML or AI inference (e.g., inference using a trained ML or AI model) can be implemented using far fewer resources than training and can be efficiently performed at the edge (e.g., by edge computing units 330 associated with local sites 302 or 302-N). In fact, edge inference often offers better latency than cloud inference because input sensor data generated at the edge (e.g., using edge asset 310) does not need to be migrated to the cloud domain (e.g., cloud user environment 370 associated with the AI / ML training cluster) via the internet backhaul link 340 before inference can begin. Accordingly, in this specification, trained AI / ML models 335-1, ..., 335-N may be created and trained in the cloud (e.g., in an AI / ML training cluster implemented within a cloud user environment 370), and may be further significantly optimized and compressed. The systems and techniques described herein enable the distribution of the optimized, compressed, and trained AI / ML models 335-1, ..., 335-N to edge locations associated with local site 302 and corresponding edge computing units (one or more) 330, and the optimized, compressed, and trained AI / ML models are intended to be implemented for inference at the edge using local sensor data from edge assets 310. As already stated above, in some embodiments, one or more of the trained models (e.g., one or more of the trained AI / ML models 335-1, ..., 335-N deployed to edge computing units 330 for local edge inference) may be fine-tuned or instructively tailored to a specific task, which is a technique that requires significantly less data and computation than their training. For example, a trained (e.g., pre-trained) AI / ML model can be fine-tuned or instructed to perform specific tasks that include new and / or differentiated tasks compared to the tasks (one or more) that the trained model originally or previously addressed.In some examples, a trained (e.g., pre-trained) AI / ML model can be fine-tuned or instructively adjusted for a specific task using one or more model retraining instances 333-1, ..., 333-K, and / or one or more model fine-tuning instances 334-1, ..., 334-M, which are also implemented locally by the edge computing unit 330, as described above.

[0067] For example, the edge computing unit 330 can perform edge inference based on input data including locally / edge-generated sensor data streams acquired from edge assets 310 provided in the same edge environment 302 as the edge computing unit 330, using one or more of the trained AI / ML models 335-1, ..., 335-N. In some embodiments, the input dataset for edge inference performed by the edge computing unit 330 can include real-time data feeds from edge assets / sensors 310, which can be between tens of Mbps and tens of Gbps (or more). In at least some embodiments, the edge computing unit 330 can include tens of GPUs for performing local inference using the trained AI / ML models 335-1, ..., 335-N. By performing local inference on the edge computing unit 330, inference response times or latency on the order of milliseconds (ms) can be achieved, significantly exceeding the inference response times or latency achievable using cloud-based or on-premises remote inference solutions.

[0068] In some embodiments, the system and technology may be configured to implement a continuous feedback loop between an edge computing unit(s) 330 in a cloud user environment 370 and an AI / ML training cluster. For example, the continuous feedback loop may be implemented using the edge computing unit(s) and associated edge assets / sensors 310 to locally capture data, perform inferences locally, and respond locally (e.g., based on the inferences). The edge computing unit(s) 330 may be additionally used to compress and transmit features generated during inference from source data, and / or to efficiently compress and transmit inference results to the AI / ML training cluster in the cloud user environment 370 (among other cloud or on-premises locations). In the continuous feedback loop, training and fine-tuning may then be performed in the cloud, for example, by the AI / ML training cluster and using batch-uploaded sensor data and / or features uploaded to the AI / ML training cluster by the edge computing unit(s) 330. Based on training and fine-tuning performed in the cloud by the AI / ML training cluster, the new or updated AI / ML model is distributed from the AI / ML training cluster to the edge (e.g., to edge compute units 330 and local sites 302). This continuous feedback loop for training and fine-tuning the AI / ML model is found to optimize the use of cloud, edge, and bandwidth resources. The same AI / ML model can be fine-tuned across multiple edge nodes to optimize the use of compute resources available on the nodes and in the cloud. For example, the AI / ML model may be fine-tuned across a set of edge nodes including at least edge compute unit 330 and one or more edge compute units located in additional local edge sites 302-N. In some cases, the distributed fine-tuning of the AI / ML model across multiple edge nodes may be mediated, supervised, and / or controlled, etc., by the AI / ML training cluster implemented within the cloud user environment 370 (e.g., or various other cloud entities).In some examples, distributed fine-tuning of an AI / ML model across multiple edge nodes may be overseen and / or controlled by one or more selected edge nodes from a set of edge nodes associated with the distributed fine-tuning of the model. In one exemplary example, distributed fine-tuning or retraining of an AI / ML model across multiple edge nodes may be organized by each of the multiple edge nodes or by each of the multiple edge nodes' respective fleet management clients.

[0069] Figure 4 is a flowchart illustrating an example of a content storage and indexing process 400 that may be executed at the edge (for example, by an edge computing system or device and / or at an edge location), using several examples. For example, the content storage and indexing process 400 may be executed by an edge computing unit or HPCS edge device, including the edge computing unit 330 in Figure 3.

[0070] The content storage and indexing process 400 may be performed for streaming sensor data and / or streaming content that is performed at the edge, generated, acquired, or otherwise ingested or otherwise made available at the edge. For example, the content storage and indexing process 400 may be performed with streaming data 402 that is generated by and / or acquired from edge assets associated with one or more connected sensors and / or edge devices. For example, the streaming data 402 may be sensor data acquired (e.g., streamed) from one or more of the edge assets 310 in Figure 3 to the edge computing unit 330 in Figure 3. In some embodiments, the context storage and indexing process 400 may be performed for various different types of streaming data 402.

[0071] For example, the streaming data 402 may include corresponding data generated by various sensors, which may include (but are not limited to) one or more of the following: cameras and video cameras, multispectral and hyperspectral cameras, infrared or thermal cameras, stereo cameras, document scanners, 3D scanners, 3D range scanners, LiDAR, RADAR, accelerometers, gyroscopes, vibrometers, inertial measurement units, inertial navigation systems, GPS, microphones, and other acoustic devices, ultrasonic devices, barcode scanners, IoT sensors and devices. Data generated by different sensors may be represented as different data or content streams within the streaming data 402. Sensor data / content of different types or modalities may be represented as discrete data or content streams within the streaming data 402. In some embodiments, a single sensor may generate one or more data streams contained within the streaming data 402. For example, a video camera may be configured to generate a first data stream containing a series of video frames and a second data stream containing a series of audio data. The various outputs of a given sensor or edge asset can also be combined into a single data / content stream within the streaming data 402. The given data / content stream within the streaming data 402 may also contain data corresponding to multiple different sensors.

[0072] Connected edge sensors and / or other assets can register or otherwise represent content (e.g., streaming data) of objects, activities, events, etc., in the surrounding environment. The generated content may additionally or alternatively represent the surrounding environment and various environmental parameters detected within it. As used herein, the terms “data” and “content” may be used interchangeably (e.g., streaming data may be used interchangeably with streaming content).

[0073] Much of the content generated by connected edge sensors and other edge assets takes the form of raw data such as text data or reports, images, 3D point clouds or meshes, video, audio, geolocation, or location information. In many existing approaches to edge computing, data or content generated at the edge must be sent to the cloud or other centralized location (e.g., data center, on-premises location) for processing. In particular, the storage of edge data content for search and retrieval purposes is indexed and performed in the cloud, not on the edge device. For example, in many cases, the edge device can act as a relay between the edge sensor / asset and the cloud computing resources that are actually used to process, index, store, etc., the collected edge data content. In such examples, sensor data or other content generated by the edge asset can be ingested by the edge device. However, when sending edge data content to the cloud, the edge device is typically configured to transfer the data without performing or providing local (e.g., edge) long-term storage of the transferred data.

[0074] Existing approaches that send edge data to the cloud for processing can be associated with various limitations and challenges. For example, sending edge data to the cloud for processing can result in increased latency between the time the edge device first acquires the raw sensor data / content and the time the final processed results from that raw sensor data / content become available at some point later (corresponding to a round-trip time of approximately [edge-cloud uplink time / latency] + [cloud processing time] + [cloud-edge downlink time / latency]). For example, the increased latency associated with hybrid edge-cloud approaches where edge data processing is offloaded to the cloud can make it difficult or impossible to achieve virtually real-time indexing, retrieval, and / or retrieval of streaming sensor data from specific edge locations as the number of edge locations served by the cloud increases.

[0075] Existing approaches that offload edge data processing to the cloud may be further associated with high bandwidth requirements because raw sensor data must be sent entirely from the edge to the cloud. High bandwidth requirements can exist at multiple stages of traditional indexing, retrieval, and retrieval processes performed in the cloud for edge data from one or more edge locations / environments.

[0076] For example, the first bandwidth is required for sending raw sensor data to the cloud, which is indexed and / or otherwise processed. This first bandwidth can be reduced by sending only a portion of the raw sensor data captured or generated at the edge device, but such an approach typically reduces the quality of the resulting searchable index for the simple reason that the portion of sensor data not sent from the edge to the cloud is not indexed and therefore not searchable or retrieval based on queries.

[0077] Therefore, there is a need for systems and technologies that can be used to implement indexing of streaming edge data or content with reduced bandwidth between edge devices and the cloud or other centralized computing locations. There is a further need to reduce bandwidth from the edge to the cloud without reducing the scope or comprehensiveness of the resulting index (e.g., the need to reduce bandwidth from the edge to the cloud while still indexing the desired edge data streams in their entirety).

[0078] Existing approaches that offload edge data processing to the cloud may require a second bandwidth to perform local searches or queries on cloud-hosted indexes from the edge. For example, if an index can be generated in the cloud from sensor data / content transferred from an edge device, and the index is stored in the cloud without being sent back to the edge, queries received at the edge must be forwarded to the cloud for processing against the constructed index. Sending queries from the edge to the cloud (and receiving query results sent from the cloud to the edge) consumes a second bandwidth. Furthermore, if the data network connection between the edge and the cloud is interrupted or degraded, the edge user's ability to query the index will be similarly interrupted or degraded. Even if the index is generated in the cloud and sent back to the edge for local storage, a second bandwidth may still be consumed when the edge user wants to view the relevant portion of the data / content identified based on the index query, because the relevant data / content requested by the edge user is sent from the cloud data store to the edge device.

[0079] Therefore, there is a need for systems and technologies that can be used to perform retrieval of streaming sensor data while leveraging reduced bandwidth between edge devices and the cloud or other centralized computing locations. In at least some cases, there is a need for systems and technologies that can be used to perform retrieval of indexed streaming sensor data without relying on cloud connectivity, cloud computing, or cloud storage. As mentioned above, limitations associated with offloading processing from edge devices to the cloud or other centralized computing entities can negatively impact the indexing, retrieval, and retrieval of streaming sensor data / content captured at the edge, because each of the three is only available after the content has been stored in the cloud. For example, there is a further need for systems and technologies that can be used to perform substantially real-time retrieval of streaming sensor data captured at the edge, based on reducing or eliminating latency associated with sending data from the edge to the cloud, and vice versa.

[0080] As described above, in one exemplary example, process 400 in Figure 4 can be performed by a high-performance computing and storage (HPCS) edge device, which may be the same as or similar to the edge computing unit 330 in Figure 3. As disclosed herein, the use of HPCS edge devices to perform indexing, searching, and / or retrieval of streaming sensor data and / or streaming content at the edge may be seen as addressing the challenges associated with conventional cloud-based approaches.

[0081] In particular, HPCS edge devices can be configured to process edge data / content in-situ with minimal latency or delay between the time edge data is generated or captured by the edge device and the time corresponding index information is made available for retrieval and / or retrieval of the captured edge data. For example, streaming data 402 may be captured at the edge device or otherwise received and stored on the local edge in block 412. In some embodiments, storing streaming data 402 on the local edge may include storing streaming sensor data 402 in one or more local edge databases 450. In some embodiments, streaming data 402 may be stored in a content portion 452 of the local edge database 450, which may be separate from or combined with an index portion 455 used to store corresponding index information for the content 452.

[0082] In block 414, the edge device can analyze streaming data 402 (e.g., stored content 452 from content 452 of the local edge database 450) and extract or otherwise generate one or more features, embeddings, and / or metadata information for each portion or stream contained in the streaming data 402. For example, the streaming data 402 / stored content 452 can be analyzed, processed, and indexed locally at the edge in block 414, and the indexing is performed based on one or more features / embeddings generated using the streaming data 402 / stored content 452 and / or based on metadata information associated with or extracted from the streaming data 402 / stored content 452.

[0083] Local (e.g., edge) search and retrieve applications may require repeated access to the same content (e.g., content that is repeatedly searched or retrieved over time). In some embodiments, since extracted features, embeddings, and metadata (e.g., also called “index information” or “index data”) are stored locally and available without network latency, local search and retrieve applications that require repeated access to the same content can be faster based on performing local indexing at the edge in block 414 of the real-time search and retrieve process 400.

[0084] In some embodiments, in block 414, the edge device may be configured to analyze the streaming data 402 / content 452 in-situ and substantially in real time (or near real time). Based on the analysis, the edge device may generate one or more tags for each portion of the data / content, or otherwise determine, as will be described in more detail below. The one or more corresponding tags generated for each portion of the data / content may be used to generate a reverse index corresponding to some or all of the respective portions of the data / content stored in the local edge content database 452.

[0085] A reverse index, also known as an inverted index, is a data structure that can be used to make the search for complete documents, data, or files within a database more efficient. Reverse indexes can be used to achieve faster full-file searches by performing increased preprocessing when indexing each file added to a searchable database. For example, consider a scenario where a reverse index is built for text documents (though it should be noted that reverse indexes can be built for various other files, file types, documents, data or content streams, data structures, etc.). An inverted index for a database of text documents may include a corpus list of all unique words that appear in any given document in the database. For each word in the corpus of the inverted index, the inverted index further includes a corresponding list of all documents in which that word appears.

[0086] In some examples, a reverse index may be generated based on performing tokenization on each input file or document to break down the input into subparts or constituent blocks. In an example of a reverse index spanning multiple text documents, each text document can be parsed or tokenized word by word. After tokenization, each unique token extracted from the document is used as a key in the index. The corresponding key value stored in the index is a list of each document (e.g., each document ID) in which a particular token (e.g., a unique word) appears. In some examples, the reverse index may further include frequency information, so that the corresponding key value in the index indicates the document ID in which the unique token (e.g., a word) appears, and further indicates how many times the unique token appears in each of the listed documents.

[0087] Searching and retrieving based on a reverse index can be performed quickly and efficiently. For example, when a search or query is received, the query data can be parsed in the same or similar way as that used to parse or tokenize the documents represented in the reverse index. More generally, the query may be parsed, processed, analyzed, etc., to extract or generate one or more tokens from the query that can be checked against the unique tokens that are keys in the reverse index. The tokenized query expression can be compared to the unique tokens (e.g., keys) in the reverse index, and the query result can be determined based on specified matching parameters (e.g., exact match, similar or most similar match, relevance, splendor, minimizing the difference or distance between token expressions, etc.). Each unique token / key in the reverse index identified as potentially matching or potentially related to the received query can be associated with one or more document IDs in which the unique token appears. Thus, content retrieval can be performed by retrieving some (or all) of the document IDs listed in the reverse index for the unique tokens / keys in the reverse index that are identified as relevant during the search / query process.

[0088] Returning to the explanation of Figure 4, in block 414, the process may further include “tagging” each portion of the received streaming data / content using the extracted features and metadata. Tagging portions of streaming data / content may include associating the corresponding document ID of the streaming data / content portion with one or more of the extracted features or metadata.

[0089] The information used to tag each portion of the streaming data / content can be used to serve as the key for the inverse index of the local edge content. Streaming data / content may have one tag and / or multiple different tags. In one exemplary example, streaming data / content may be tagged (and indexed in the inverse index) based on corresponding features generated based on providing the streaming data / content as input to machine learning (ML) or artificial intelligence (AI) models, networks, algorithms, etc.

[0090] For example, the corresponding features used to tag sensor data or content may include one or more feature vectors generated by a machine learning classifier, encoder, autoencoder, etc. Feature vectors may also be called “extracted features.” In some embodiments, extracted features may be output features or output embeddings generated by an ML / AI classification model. In some examples, extracted features may additionally or alternatively be intermediate (e.g., non-output) features or intermediate (non-output) embeddings generated by an intermediate layer of the ML / AI classification model.

[0091] The extracted features (e.g., feature vectors) generated in block 414 may be generated as relatively high-dimensional data (e.g., n-dimensional feature vectors) representing a given portion of the sensor data / content. The structure and / or dimensions of the generated features or embedding vectors may vary based on factors such as the type of ML model used for feature extraction, the type of sensor, or the type / format of input data provided to the ML model. Generally, various combinations of these factors may be referred to as different “modes” or “modalities.”

[0092] In one exemplary example, performing feature extraction in block 414 may include performing multimodal feature extraction. For example, based on the modality of the given sensor data, features can be generated from given sensor data using an appropriate or corresponding ML model. Different ML models can generate different types of feature vectors (or other feature representations) for different types of data and modalities. For example, an autoencoder-based neural network may include a hidden layer corresponding to a compressed representation of the input data provided to the network. In some cases, the compressed hidden layer representation can be used as extracted features for indexing the sensor data / content.

[0093] In another example, in the context of image processing, a CNN can be used to generate feature maps of an input image (e.g., a still image, a frame of video data, etc.). Feature maps can be generated in each convolutional layer of the CNN. In some embodiments, the convolutional layer feature maps can be used as extracted features for indexing sensor data / content. The fully connected layers in an image processing CNN can generate a feature vector output or feature map that represents the original input image as a whole, which can also be used as extracted features for indexing image-based sensor data / content.

[0094] In yet another example, an RNN or transformer can be used for natural language processing to generate feature vectors of words or tokens in a text-based input sequence. In such an example, each feature vector corresponding to a word or token may be used for indexing in block 414 of process 400. In some examples, it is possible, either additionally or alternatively, to generate extracted features corresponding to a larger sequence of text-based inputs, up to including each text-based input as a whole.

[0095] In some cases, feature extraction in block 414 may be additionally used to generate one or more embeddings corresponding to features of the input sensor data / content. Embeddings are representations of features of a portion of the content (e.g., in images, videos, text documents, audio clips, etc.). In particular, embeddings may be generated to represent relatively high-dimensional features in a lower-dimensional vector space. Embeddings may be used to capture the underlying relationships and characteristics of content in a way that facilitates effective searching, retrieval, or matching of two (or more) portions of content. Embeddings can be generated based on mapping high-dimensional data to a lower-dimensional space. For example, one or more neural networks and / or dimensionality reduction techniques can be used to generate one or more embeddings corresponding to indexed data (e.g., streaming data 402 at the edge and / or stored content 452). The resulting embedding vectors are dense and continuous and represent the essential characteristics of the original portion of the content. Thus, the extracted features used for indexing (e.g., inverse indexing) of streaming sensor data and streaming content at the edge may, in at least some embodiments, include the embeddings generated for the input streaming data 402.

[0096] In one exemplary example, extracted features generated in block 414 (and used in block 416 to determine significant content, and / or used in block 418 to generate an index of significant content) may be generated using a comprehensive and consistent embedding space shared across various ML models / networks and / or different data modalities. For example, by generating extracted features using a shared embedding space, systems and technologies can support integrated indexing, retrieval, and retrieval in multimodal implementations spanning different types of sensor data and content 402 captured at the edge. In some embodiments, multimodal implementations may be provided on the basis of using one or more transformation layers between modalities (e.g., to transform features / embedding from a first modality to a second modality, or to transform features / embedding from a first modality to a shared embedding space). For example, in the context of a transformer-based architecture or implementation, each modality-specific transformation layer may be implemented using corresponding attention heads within the transformer layer.

[0097] In block 416, process 400 may include determining significant content represented within the streaming data and content 402 / stored local edge content 452. For example, significant content may be determined at least in part on using features and metadata extracted from block 414, as described above.

[0098] In some embodiments, determining significant content may involve identifying related patterns or anomalies across a network of multiple sensors (either multimodal or homogeneous) registering the same object or event. Such approaches may be used to achieve more reliable and robust information retrieval when used downstream in the real-time search and retrieve pipeline of this disclosure. For example, portions of retrieved information returned in response to a search query may be more reliable based on using multiple sensor data streams (and their extracted features) to confirm the objects or events represented in the retrieved information.

[0099] In other words, if an object or event is represented by multiple sensor data streams, confirmation or verification of the object or event as a distinguishing feature can be performed during the index building process itself, rather than waiting to perform confirmation or verification as a ranking metric at a later point in time after the search and retrieval steps. In this approach, distinguishing data identified in block 416 includes data / content that indicates an object or event successfully confirmed by one or more other sensor data streams. In particular, the distinguishing information identified in block 416 proceeds to storage at the edges of index generation 418 and block 420.

[0100] On the other hand, some of the streaming data / content 402 that is not identified as significant information in block 416 may be removed or refrained from proceeding to index generation 418 and from being stored on the edge in block 420. In some examples, content identified as not containing significant information in block 416 may be streaming sensor data or streaming content that has been determined to be inconsistent and / or anomalous. For example, streaming sensor data may be acquired from multiple cameras, each providing a different perspective view of an oil well cover. Features and / or embeddings may be generated for video frames acquired from each camera (in block 414) and analyzed for significant content (in block 416).

[0101] In the context of this example, notable content or information may include the presence of unknown or unexpected objects, entities, etc., and / or the occurrence of various events. For example, the presence of a crack in the oil well cover can be identified as a notable feature in the streaming camera data, and can be identified based on analyzing the features corresponding to each streaming camera data feed using one or more ML models. Other notable information in the context of this example may include foreign objects on or near the oil well cover, unknown or unauthorized individuals appearing in the camera frame, etc. If a first camera feed (corresponding to a first viewpoint of the oil well cover) is analyzed and the extracted features indicate that the video frame can be classified as containing a crack, then a comparison or validation can be performed to validate the seemingly notable information from the first camera feed with some (or all) of the remaining camera feeds of the same oil well cover (in addition to any other sensor or edge data available and corresponding to the same oil well cover, e.g., either in the same video-based modality as the camera feed or in a completely different modality).

[0102] If patterns are identified across multiple camera feeds corresponding to the same well cover, such as patterns in corresponding camera feeds from different cameras, each indicating some classification probability of a crack present in the well cover, then potential cracks may be identified as prominent features in block 416 and can proceed to block 418 to index them in that manner within the inverse index information constructed at the edge.

[0103] If no pattern is identified across multiple camera feeds corresponding to the same oil well cover (for example, the first camera feed shows a potential crack, but the second, third, and fourth cameras show no potential cracks), then in block 416, the potential crack from the first camera feed stream can be identified as non-significant information, and it can be refrained from indexing it in block 418 and storing it locally at the edge in block 420.

[0104] In another exemplary example of the splendor analysis in block 416, consider a scenario in which image and video data are streamed from respective RGB and thermal cameras mounted on one or more drones inspecting a bridge. Both streams of data (e.g., RGB camera data and thermal camera data) can be processed on-premises at the edge using the edge computing unit 330, for example by streaming the camera data to the edge computing unit 330 in Figure 3 to perform process 400 in Figure 4. In particular, the RGB and thermal camera data streams can be processed in-situ at the edge using process 400, and a series of region-based convolutional neural networks (CNNs) can be used to perform one or more of the feature extraction 414 and / or splendor content determination 416. For example, the CNNs can be used to detect or classify defects such as cracks and fissures in both the RGB data stream and the thermal camera data stream. In some embodiments, only defects that are consistent across both modalities of data are identified as splendor information in block 416, then indexed in block 418, and stored locally at the edge in block 420. For example, defects consistent across two modalities of data may be indexed and stored for on-site (e.g., edge) retrieval. In one exemplary example, a civil engineer, inspector, or other user may query locally stored images or video frames using queries to the local edge database 450 and / or to prominent content and index 420 stored on the edge (for example, prominent content and index 420 may be the same as, or different from, the respective content 452 and index 455 shown as stored in the local edge database 450). For example, a user may query locally stored images or video frames for potential problems such as "corrosion," "penetration," "crack," "creep," "carbonation," "shrinkage," "water seepage," or various other structural anomalies.

[0105] In some embodiments, the queries enumerated in the above examples may be examples of different tags applied to the data indexed in block 418 when an index of outstanding content is generated. For example, in some embodiments, the index of outstanding content may be based on one or more tags applied to the streaming data, and the one or more tags are intended to be selected from a classification space such as one or more ML or AI classifiers or classification models.

[0106] For example, process 400 can be implemented based on aggregating and chunking streaming content 402 within a predetermined time window (e.g., 1 minute, 5 minutes, 10 minutes, etc.). In some cases, streaming content 402 can be paginated for temporary storage on the edge. Using local computing at the edge (e.g., the edge computing unit 330 in Figure 3 and / or other edge devices related to the implementation of process 400), features and metadata related to user queries on the edge can be extracted in block 414. The extracted features or embeddings may not be human-interpretable, but metadata tags can be used directly to generate an inverted index. For example, streaming video may be chunked into segments with a duration of 1 minute, and given segments may be tagged with metadata tags such as "person, walking, existing path", "vehicle, speeding, 80 mph", "railroad crossing, person, tractor trailer, stalled". In some embodiments, metadata tags may be single tags (e.g., "person" or "vehicle") and / or composite metadata tags, such as the three examples above.

[0107] In block 416, prominent segments are identified and may be identified as metadata-tagged segments with associated tags. In blocks 418 and 420, prominent segments are indexed and stored on the edge, respectively. In one exemplary example, a subset of identified prominent content from block 416 may be selected based on corresponding metadata, features, and / or embeddings and sent in block 434 for persistent storage in the cloud using a backhaul connection from the edge (e.g., the internet backhaul 340 in Figure 3).

[0108] In some embodiments, the distinctive content and corresponding indexes identified and generated in block 418 may be stored locally at the edge (e.g., in a local edge database 450) without being transmitted to the cloud or other centralized location. Storing sensitive data and their corresponding indexes locally at the edge may be used to address privacy and security concerns. For example, consider another exemplary example where the input data / content 402 includes health or medical data such as MRI, CT scans, X-rays, ultrasounds, PET scans, EEGs, ECGs, blood panels, psychiatric assessments, EHRs, and clinician notes. In such an example, the systems and technologies described herein may be used to store the health data locally (e.g., at the edge) at the healthcare provider's facility. The index information generated in block 418 for the identified distinctive content can be made available for retrieval and / or to anyone with access to the local network. For example, the local network may be the same or similar local edge network as the local network 320 in Figure 3, corresponding to the local site / edge environment 302 (in the above example, the healthcare provider's office).

[0109] By minimizing data transfer over the Internet or other open / public networks while making reverse indexes of significant information locally available, systems and technologies can minimize the risk of data breaches and unauthorized access. In some embodiments, search requests for reverse indexes of significant information may be restricted to local edge networks or identified external sites.

[0110] In some examples, a search request for a reverse index of significant information may be provided to one or more users or entities located remotely from the edge. For example, the system and technology may, in some embodiments, provide the ability to enable searching of indexed edge content in the cloud (for example, a cloud user environment 370 may, with limited permissions or applicability, search for indexed edge content generated and stored in the local site / edge environment 302 of an edge compute unit 330 implementing process 400 in Figure 4).

[0111] In this case, process 400 may include analyzing backhaul bandwidth availability and persistent storage requirements in block 432 to determine which data and indexes (if any) should be sent to the cloud. In some cases, block 432 may be used to identify edge data and indexes that should be sent to the cloud for cloud storage as persistent data and indexes. In some cases, the identification of persistent data and indexes to be sent to the cloud may exceed backhaul bandwidth availability, in which case a smaller amount (e.g., a subset) of the identified data and indexes for cloud persistence may be sent to the cloud.

[0112] In block 434, identified persistent data and indexes (or a subset thereof) can be sent from the edge (e.g., from the local edge database 450) to the cloud. In block 440, identified persistent data and indexes are stored in the cloud for cloud-based searching and retrieval of queried information.

[0113] In one exemplary example, persistent content stored in the cloud in block 440 may include only extracted features and / or generated embeddings corresponding to the underlying raw sensor data stream 402. For example, extracted features / embeddings generated in block 414 can be sent to the cloud as a representation of the raw sensor data stream, while the raw sensor data itself does not leave the edge and is not transmitted over the internet or other public data networks. In some embodiments, the sensor data embeddings can be sent to the cloud along with corresponding inverse index information generated at the edge. Queries received in the cloud can be processed against the inverse index, and the relevant content can be requested / pulled from the edge to be presented in response to the cloud query. In some embodiments, the cloud query results may indicate matches found in the inverse index, but the raw content is not transmitted from the edge. In some embodiments, in block 434, the edge device can send extracted features / embeddings generated against significant data without sending the inverse index generated at the edge. Using the extracted features / embeddings, the cloud can generate a cloud-based inverse index in the same or similar process as described above for the edge.

[0114] In some embodiments, queries can be processed on-site where the content is generated and indexed. For example, queries can be processed at the edge location where the streaming sensor data (or streaming content) is generated and indexed, as described above with respect to process 400 in Figure 4. Such approaches and implementations allow for the retrieval of content that responds to the query or is associated with the query by processing the query on-site where the content is generated, while ensuring that the retrieved content does not leave the local network. In one exemplary example, relevance and ranking associated with a query can be performed as close as possible to the sensor that generated the data, i.e., the sensor where the corresponding index was created and stored. In another exemplary example, personalization of retrieved content can be performed locally based on factors such as the query issuer's preferences, intent information associated with the query and / or the query issuer, and contextual information associated with the query and / or the query issuer.

[0115] Figure 5 shows an example of content search and ranking 500 that may be performed at the edge (for example, by an edge computing system or device and / or at an edge location), using several examples. The local edge database 550 in Figure 5 may be the same as or similar to the local edge database 450 in Figure 4. The content 552 and index 555 in Figure 5 may be the same as or similar to the content 452 and index 455 in Figure 4, respectively.

[0116] In block 510, one or more local user queries are received at the edge and may be processed in block 512 based on the query modality. For example, the query modality may indicate a specific type or modality of sensor data being retrieved. In some cases, the query modality may be included in, indicated, or specified in the local user query 510. In other cases, the query modality may be determined or inferred based on an analysis of the local user query 510.

[0117] In block 514, the query may be matched against content index information in a reverse index generated using process 400 in Figure 4. For example, the query may be processed at the edge by comparing the query information with a reverse index 555 stored in the local edge database 550. In some embodiments, the query is compared with a prominent reverse index identified at the edge during process 400 for generating the index information.

[0118] In block 516, relevant content for a query (for example, in response to a query) can be retrieved based on a corresponding relevance score calculated between the relevant content and the query information. For example, the relevance score may be predetermined in block 514 or determined in block 516 as part of the content retrieval operation. The relevant content identified based on the relevance score can be retrieved from the local edge database 550 and included in the stored content 552 within the local edge database 550. In some cases, the relevant content retrieved in block 516 may be content with a corresponding calculated relevance score, such as being greater than one or more thresholds and / or satisfying one or more configured rules. In some cases, relevance may be based on an evaluation of the most likely or high-priority content that could potentially respond to the query. In some cases, relevance may be based on the strength of the correspondence between the indexed content portion and the query 510.

[0119] In block 518, process 500 may include ranking the content retrieved (from block 516) with respect to the user query and intent. In some cases, the user query intent information may be determined in block 512 or in block 518. In some cases, the relevance score used for content retrieval in block 516 may include a first path evaluation of the content for the query, and the ranking of the retrieved content in block 518 may include a second path or improvement of the first path for a more detailed aspect of the query information of the local user query 510.

[0120] In block 520, ranked relevant content can be presented on the edge in response to a local user query 510.

[0121] In one exemplary example, when a query is received, the retrieved content can be personalized. For example, the personalization of the retrieved content can be done locally based on the query issuer's preferences, the intent of the query or query user, and the context of the query or query user. In some cases, the personalization of the retrieved content may be performed in blocks 516 and / or 518 of process 500.

[0122] Relevance can be user-specific, edge environment-specific, or domain-specific. For example, a user in an edge environment monitoring oilfield equipment for signs of failure may have a higher relevance to queries about potential cracks in edge sensor data than a user in an edge environment corresponding to a healthcare provider. In other words, relevance and ranking decisions in the search and retrieval process corresponding to local edge queries 510 can be personalized based on contextual information at the edge or user contextual information. In the example above, queries about metadata tags "crack" or "crack occurrence" may have a higher relevance in the oilfield edge environment than in a healthcare provider edge environment, for example.

[0123] The content ranking in block 518 can be further adjusted for relevance or otherwise adjusted based on personalized / customized information corresponding to the query issuer, and / or information specific to the edge context, edge environment / location, region, technology or enterprise sector, etc. For example, ranking and sorting in block 518 may be performed based on personalized (customized) adjusted filtering for the query issuer or edge location, such that types of content that are more commonly queried or retrieved at a particular edge location (or a larger region, etc.) are ranked higher in the results returned for local edge queries 510. In some embodiments, personalization can be performed based on a learning process over time, and the system described herein can evaluate the query process performed by the user to determine which retrieved content (or which part of the reverse index) was most useful for the actual queries being processed by the system. Based on historical information of the correspondence between input queries and the retrieved content / index that was ultimately most useful (or most accurate) for those queries, a personalization process can be implemented to provide continuous learning or improvement of the ranking behavior in 518 for presenting retrieved content for a given local edge query 510.

[0124] In some cases, personalization or customization (e.g., of extracted content, ranking, relevance scoring) can be implemented based on modifications to the reverse index itself. For example, personalization can be provided by controlling the composition of the content on which the index is built. Personalization can also be implemented based on adjustments to a classification scheme or schema implemented by one or more ML models used to extract features and generate the embeddings used to generate the index. In some cases, as mentioned above, separate indexes can be built in the edge location and in the cloud. For example, a cloud index can be generated for a selection of prominent content that is sent to the cloud to be stored as persistent cloud content. In some cases, the persistent cloud content may be identical to the content stored at the edge, but the edge and the cloud may generate different indexes for the same information. For example, the edge index and the cloud index can be personalized (e.g., change) based on a subset(s) of the complete set of content actually indexed, based on the tagging or key scheme used to generate the index, etc.

[0125] For example, if queries received in the cloud differ in type, scope, nature, context, intent, etc., from queries received locally at the edge, it may be desirable to implement separate indexes for the edge (used to process queries at the edge) and the cloud (used to process queries in the cloud). Cloud indexes can be generated based on a compact representation of significant content sent from the edge to the cloud. This compact representation may include document IDs for each part of the significant content and one or more corresponding embeddings generated (at the edge) for the content of each document ID.

[0126] In one exemplary example, a hybrid-based implementation for retrieving and ranking content may be performed across one or more edge locations and cloud environments associated with one or more edge locations. For example, Figure 6 shows a hybrid content search and ranking analysis 600 that may be performed in the cloud based on local query information obtained at the edge (e.g., by an edge computing system or device and / or at the edge location), in several examples. In the example in Figure 6, content can be retrieved from local edge storage, persistent cloud storage, or both, based on a relevance score determined based on local user queries 610.

[0127] In some cases, the local user query 610 in Figure 6 may be the same as or similar to the local user query 510 in Figure 5. The query processing block 612 in Figure 6 may be the same as or similar to block 512 in Figure 5. The index stored at edge 655 and the content stored at edge 652 in Figure 6 may be the same as or similar to the edge content 552 and edge index 555 in Figure 5, and / or the prominent content and index stored at edge 420 in Figure 4, the content 452 and index 455 of the local edge database 450 in Figure 4, etc.

[0128] As described above, in some embodiments, indexed content may be stored across both the edge and the cloud, with the edge intended to store local data and the cloud intended to store persistent data. Persistent cloud data may be a subset of local edge data, but may differ from it. For example, persistent cloud data may have a longer lookback time window compared to local edge data, which may be deleted or archived at shorter intervals. Persistent cloud data may also contain data received from multiple different edge locations. For example, persistent cloud data may correspond to significant data and indices from 10 edge locations, with each piece of local edge data at each edge location not containing significant information identified at the other 9 edge locations. In this example, persistent cloud data may be a larger or more comprehensive dataset than the local edge dataset, and vice versa.

[0129] In some embodiments, process 600 in Figure 6 may be used to perform content retrieval and ranking on cloud or local queries, taking into account hybrid storage spanning persistent cloud data and local edge data. As described below, content from both edge and cloud storage can be ranked together. In particular, persistent cloud content does not need to be retrieved immediately, thereby saving bandwidth on the backhaul. Instead, only content that is ranked sufficiently high and selected by the user (query issuer) is retrieved from the cloud and sent to the edge in response to the local edge query 610 to be presented to the user (query issuer).

[0130] In block 614, process 600 includes determining a relevance score which may be used to perform content retrieval from edge storage, persistent cloud storage, or both. The relevance score is compared to a threshold. In one exemplary example, the relevance score for edge index information 655 is compared to a threshold. For example, the relevance score may be based on evaluating a local user query 610 against the edge index information 655 and indicate whether the indexed content available at the edge (such as that represented by the edge index information 655) is sufficiently relevant to the local edge user query 610.

[0131] If block 614 determines that the edge index-local query relevance score is greater than a threshold, the process can proceed to block 616 and retrieve the relevant content from the edge content store 652. Block 616 may be the same as or similar to block 516 in Figure 5. In block 618, process 600 includes ranking the retrieved relevant edge content with respect to the user query and intent. Block 618 may be the same as or similar to block 518 in Figure 5. Subsequently, in block 660, process 600 includes presenting the ranked content on the edge (for example, the same as or similar to the presentation of ranked content on the edge in block 520 in Figure 5). In other words, if block 614 determines that the edge relevance score for query 610 is greater than a threshold, process 600 may proceed in the same or similar manner as described above with respect to process 500 in Figure 5.

[0132] However, if block 614 determines that the edge relevance score of query 610 is below a threshold, the indexed content stored at the edge is not sufficiently relevant to resolve the local edge query 610. Therefore, process 600 can proceed to block 622, where query 610 and / or one or more (or both) of the generated or extracted features of query 610 are sent to the cloud. As shown in Figure 6, the edge environment 602 is separate from the cloud environment 605. For example, edge environment 602 may be the same as or similar to local site / edge environment 302 in Figure 3, and user cloud environment 605 may be the same as or similar to user cloud environment 370 in Figure 3. In some cases, edge environment 602 may be associated with various other edge devices, including the edge compute unit 330 in Figure 3 and / or HPCS edge devices.

[0133] In block 632, which is performed in the cloud, the query / feature 630 received from the edge (in block 622) can be processed and analyzed to match the edge query / feature 630 with the persistent index 675 stored in the cloud. As described above, the cloud persistent index 675 can be different from the edge stored index 655 (and accordingly, the edge stored content 652 can be different from the persistent cloud storage content 672).

[0134] The process that matches edge queries / features 630 with persistent cloud index 675 (for example, the process in block 632) may be the same as or similar to block 514 of process 500 where local edge queries are matched with local edge indexes.

[0135] In block 634, persistent content can be retrieved from the persistent cloud content store 672 in the same or similar manner as the retrieval of related edge content in block 616 of Figure 6 and / or the retrieval of related content in block 516 of Figure 5.

[0136] In block 636, the relevant and / or ranked persistent cloud content may be determined based on the persistent cloud content previously retrieved in block 634. The relevance and ranking decisions in the cloud may be the same as, or different from, those performed at the edge. For example, in the context of the hybrid search and retrieve process 600, the edge and the cloud may perform the same relevance and ranking decisions (but applied to the retrieved edge content and retrieved cloud content, respectively), and the relevance and ranking decisions may be based on the local edge query 610 and / or the relevant parameters of the query 610 (such query parameters and related query information are determined in block 612 and included in the transmission of the edge query / feature from the edge to the cloud in block 622).

[0137] In some embodiments, the cloud-based ranking step 636 performed in the cloud for persistent cloud content may be the same as or similar to the edge-based ranking step 618 performed at the edge for local edge content. In some examples, the cloud-based ranking step 636 performed in the cloud for persistent cloud content may be the same as or similar to the edge-based ranking step 518 performed for local edge content in Figure 5.

[0138] In block 638, the cloud can compress the ranked persistent content determined in block 636. In block 642, the cloud environment 605 may be configured to send the compressed content (generated in block 638) and the corresponding relevance score (determined, e.g., in block 634 and / or block 636) to the edge environment 602. In particular, the cloud content and relevance score 650 can be determined in the cloud based on analyzing edge queries / features 630 against the persistent cloud data store of the cloud index 675 and cloud content 672. Cloud execution searches and retrievals may be performed using cloud-specific personalization, which may differ from edge-specific personalization used in the edge environment 602 and blocks 612-618.

[0139] Returning to the description of the implementation forms of process 600 at the edge (in the edge environment 602), after sending the edge query / feature 6430 to the cloud 605 in step 622, process 600 can continue at the edge in block 624, and the edge content relevance score 625 may be cached or temporarily stored at the edge. In some embodiments, the edge content relevance score 625 may be cached or temporarily stored while the edge device 602 waits to receive the relevant cloud content and relevance score 650 identified in the cloud 605 (identified in the cloud based on the cloud performing cloud-based lookup and retrieval of the local edge query 610 against persistent cloud data 672, 675 as described above).

[0140] As mentioned above, in the hybrid edge cloud search and retrieval performed in process 600, content from both edge and cloud storage can be ranked together for the local edge query 610. Also, as mentioned above, content from the cloud (e.g., cloud content and relevance score 650) does not need to be retrieved immediately, thereby saving bandwidth on the backhaul between edge 602 and cloud 605. Only persistent cloud content that is ranked sufficiently high for the local edge query 610 (e.g., determined by the cloud in block 634 and / or block 636) is retrieved from the cloud.

[0141] In some embodiments, the cloud data transmission 650 may include only the cloud content relevance score and other identifiers or index information that enable the retrieval of the complete persistent cloud content from the cloud 605 if necessary or requested. For example, the cloud data transmission 650 may indicate the relevant persistent cloud content identified for the edge query / feature 630, and as a result, in block 652, the edge device can determine a combined or hybrid ranking of the cloud content and edge content. The combined ranking may mix the cloud content relevance score with the edge content relevance score. That is, the output of block 652 may be a ranking that mixes the portion of local edge content identified and indexed as relevant at edge 602 with the portion of persistent cloud content identified and indexed as relevant at cloud 605. In some embodiments, the ranking in block 652 may be a sort based on the edge content score 625 and the cloud content score 650, and no further ranking is utilized (for example, in cases where the edge content score 625 and the cloud content score 650 use the same ranking system or metric). In other examples, ranking in block 652 may include additional analysis, processing, and / or ranking of cloud and edge data based on metadata or additional information associated with identified relevant persistent cloud content, identified relevant local edge content, or some (or all) of a combination thereof. For example, persistent cloud content may include additional metadata and / or tag information indicating that a particular portion of the persistent cloud content has been identified as prominent in location / region / edge environment / use case x or time y, i.e., metadata information relating to the scope of the persistent cloud data store across multiple edge environments but not included in any single given scope of any one of the multiple edge environments.Therefore, in such cases, hybrid ranking performed at the edge in block 652 may take into account additional metadata information specific to the context of the cloud environment 605.

[0142] In block 660, the ranked content (in this example, the ranked hybrid cloud and edge content from block 652) may be presented on the edge in the same or similar manner as the presentation of edge-only content described above with respect to the output of block 618.

[0143] In some embodiments, persistent cloud content index information is displayed on the edge in the presentation of ranked content 660. For example, user selection or input may be required before the system retrieves the underlying persistent cloud content from the cloud 605 and sends it to edge 602 for presentation to the user. In some embodiments, the presentation of ranked content 660 may show document IDs, ranking or relevance information, index information, etc., associated with each part of the persistent cloud content, allowing the user (query issuer) to view an overview of the persistent cloud content and decide whether to request that a particular part of the persistent cloud content be pulled down from the persistent cloud content store 672 for further viewing at edge 602.

[0144] In some cases, the systems and technologies described herein may be implemented or otherwise performed by a computing device, apparatus, or system. In one example, the systems and technologies described herein may be implemented or performed by a computing device or system having the computing device architecture 700 of Figure 7. The computing device, apparatus, or system may include any suitable device such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, AR glasses, a network-connected watch or smartwatch, or other wearable device), a server computer, an autonomous vehicle or computing device for an autonomous vehicle, a robotic device, a laptop computer, a smart television, a camera, and / or any other computing device having the resource capacity to perform the processes described herein. In some cases, the computing device or apparatus may include various components such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components (one or more) configured to perform the steps of the processes described herein. In some examples, a computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components (one or more). The network interface may be configured to communicate and / or receive Internet Protocol (IP) based data or other types of data.

[0145] Components of a computing device may be implemented in a circuit. For example, a component may include an electronic circuit or other electronic hardware that includes one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuits), and / or may be implemented using them, and / or may include computer software, firmware, or any combination thereof to perform the various operations described herein, and / or may be implemented using them.

[0146] The processes described herein may include sequences of actions that can be performed by hardware, computer instructions, or a combination thereof. In the context of computer instructions, an action represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the enumerated action. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a particular function or implement a particular data type. The order in which actions are described is not intended to be interpreted as limiting, and any number of described actions can be combined in any order and / or in parallel to perform a process.

[0147] Furthermore, the processes described herein may be executed under the control of one or more computer systems consisting of executable instructions, or as code (e.g., executable instructions, one or more computer programs, or one or more applications) executed collectively by hardware or in combination thereof on one or more processors. As described above, the code may be stored in a computer-readable or machine-readable storage medium in the form of a computer program containing multiple instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-temporary.

[0148] Figure 7 shows an exemplary computing device architecture 700 of an exemplary computing device capable of implementing various technologies described herein. In some examples, the computing device may include a mobile device, a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a video server, a vehicle (or a computing device in a vehicle), or other devices. The components of the computing device architecture 700 are shown to communicate with each other using connections 705, such as a bus. The exemplary computing device architecture 700 includes a processing unit (CPU or processor) 710 and a computing device connection 705 that connects various computing device components to the processor 710, including computing device memory 715 such as read-only memory (ROM) 720 and random access memory (RAM) 725.

[0149] The computing device architecture 700 may include a cache of high-speed memory directly connected to, adjacent to, or integrated as part of the processor 710. The computing device architecture 700 may copy data from memory 715 and / or storage device 730 to cache 712 for rapid access by the processor 710. In this way, the cache can provide a performance improvement that avoids delays in the processor 710 while waiting for data. These and other engines may control or be configured to control the processor 710 to perform various actions. Other computing device memories 715 may also be available. Memory 715 may include multiple different types of memory with different performance characteristics. The processor 710 may include any general-purpose processor, hardware or software services such as service 1 732, service 2 734, and service 3 736 stored in storage device 730, and a dedicated processor with software instructions incorporated into the processor design, configured to control the processor 710. The processor 710 may be a self-contained system including multiple cores or processors, buses, memory controllers, caches, etc. A multicore processor may be symmetrical or asymmetrical.

[0150] To enable user interaction with the computing device architecture 700, the input device 745 can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, or voice. The output device 735 can also be one or more of several output mechanisms known to those skilled in the art, such as a display, projector, television, or speaker device. In some examples, a multimodal computing device can allow the user to provide multiple types of inputs for communication with the computing device architecture 700. The communication interface 740 can generally control and manage user inputs and computing device outputs. There are no restrictions on operating on any particular hardware configuration, and therefore the basic features here can be easily replaced with improved hardware or firmware configurations as they are developed.

[0151] The storage device 730 is a non-volatile memory and may be a hard disk or other type of computer-readable medium capable of storing computer-accessible data, such as a magnetic cassette, flash memory card, solid-state memory device, digital versatile disk, cartridge, random access memory (RAM) 725, read-only memory (ROM) 720, and hybrids thereof. The storage device 730 may include services 732, 734, and 736 for controlling the processor 710. Other hardware or software modules or engines are intended. The storage device 730 may be connected to the computing device connection 705. In one embodiment, a hardware module performing a particular function may include software components stored on a computer-readable medium in relation to the necessary hardware components such as the processor 710, the connection 705, and the output device 735 in order to perform the function.

[0152] Aspects of this disclosure are applicable to any suitable electronic device (such as a security system, smartphone, tablet, laptop computer, vehicle, drone, or other device) that includes or is coupled to one or more active depth sensing systems. While a device having or being coupled to one optical projector is described below, aspects of this disclosure are applicable to devices having any number of optical projectors and are therefore not limited to any particular device.

[0153] The term “device” is not limited to one or a specific number of physical objects (such as one smartphone, one controller, or one processing system). As used herein, a device can be any electronic device having one or more parts that can implement at least some parts of the Disclosure. The following descriptions and examples use the term “device” to illustrate various aspects of the Disclosure, but the term “device” is not limited to a specific configuration, type, or number of objects. Furthermore, the term “system” is not limited to multiple components or a specific configuration. For example, a system may be mounted on one or more printed circuit boards or other boards and may have movable or static components. The following descriptions and examples use the term “system” to illustrate various aspects of the Disclosure, but the term “system” is not limited to a specific configuration, type, or number of objects.

[0154] Specific details are provided in the above description in order to provide a complete understanding of the embodiments and examples provided herein. However, it will be understood by those skilled in the art that these embodiments may be carried out without these specific details. For clarity in the description, in some examples the technology may be presented as including individual functional blocks, which include devices, device components, steps or routines in a method embodied in software, or combinations of hardware and software. Additional components other than those shown in the figures and / or described herein may be used. For example, circuits, systems, networks, processes, and other components may be shown as components in the form of block diagrams in order to avoid obscuring the embodiments with unnecessary details. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary details in order to avoid obscuring the embodiments.

[0155] Individual embodiments may be described above as processes or methods represented as flowcharts, flow diagrams, data flow diagrams, structural diagrams, or block diagrams. While flowcharts may describe operations as sequential processes, many operations may be performed in parallel or simultaneously. Furthermore, the order of operations may be rearranged. A process terminates when its operations are complete, but it may have additional steps not shown in the diagram. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. If a process corresponds to a function, its termination may correspond to the function's return to the calling function or main function.

[0156] The processes and methods described above can be implemented using computer-executable instructions stored on or otherwise available from computer-readable media. Such instructions may include instructions and data that cause a general-purpose computer, a dedicated computer, or a processing device to perform a particular function or group of functions, or otherwise configure them to perform them. Some of the computer resources used may be accessible via a network. Computer-executable instructions may include, for example, binaries, intermediate format instructions such as assembly language, firmware, source code, and the like.

[0157] The term “computer-readable media” includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or transporting instructions (one or more) and / or data. Computer-readable media may also include non-temporary media on which data can be stored and which do not contain carrier waves and / or temporary electronic signals that propagate wirelessly or via wired connections. Examples of non-temporary media include, but are not limited to, magnetic disks or tapes, optical storage media such as flash memory, memory or memory devices, magnetic or optical disks, flash memory, USB devices with non-volatile memory, network storage devices, compact discs (CDs) or digital multipurpose discs (DVDs), and any appropriate combination thereof. Computer-readable media may store code and / or machine-executable instructions that may represent procedures, functions, subprograms, programs, routines, subroutines, modules, engines, software packages, classes, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc., may be passed, transferred, or transmitted via any suitable means, including memory sharing, message passing, token passing, network transmission, etc.

[0158] In some embodiments, computer-readable storage devices, media, and memory may include cables or wireless signals, such as bitstreams. However, non-transient computer-readable storage media, as referred to, explicitly exclude media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0159] Devices implementing the processes and methods described herein may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing the required tasks may be stored in computer-readable or machine-readable media. A processor(s) may perform the required tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet devices, or other smaller form factor personal computers, personal digital assistants, rack-mount devices, and standalone devices. The functions described herein may also be embodied in peripheral devices or add-in cards. Such functions may also, as a further example, be implemented on a circuit board between different chips or different processes running in a single device.

[0160] Instructions, a medium for transmitting such instructions, computing resources for executing them, and other structures for supporting such computing resources are exemplary means for providing the functionality described herein.

[0161] In the foregoing description, embodiments of this application are described with reference to specific embodiments, but those skilled in the art will recognize that this application is not limited thereto. Therefore, while exemplary embodiments of this application are described in detail herein, it should be understood that the concepts of the present invention may be embodied and adopted in various other ways, and the appended claims should be construed to include such variations, unless limited by the prior art. The various features and embodiments of the applications described above may be used individually or in combination. Furthermore, embodiments may be used in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of this specification. Therefore, the specification and drawings should be considered illustrative, not restrictive. For illustrative purposes, the methods are described in a particular order. In alternative embodiments, it should be understood that the methods may be performed in a different order than described.

[0162] Those skilled in the art will understand that the symbols or terms “less than” ("<") and “greater than” (">") used herein may be replaced with the symbols “less than or equal to” ("≦") and “greater than or equal to” ("≧"), respectively, without departing from the scope of this description.

[0163] When a component is described as being "configured" to perform a certain operation, such configuration can be achieved, for example, by designing an electronic circuit or other hardware to perform the operation, by programming a programmable electronic circuit (e.g., a microprocessor or other suitable electronic circuit) to perform the operation, or by any combination thereof.

[0164] The term "connected" refers to any component that is directly or indirectly physically connected to another component, and / or any component that communicates directly or indirectly with another component (for example, connected to another component via a wired or wireless connection, and / or other suitable communication interface).

[0165] The wording of a claim that includes "at least one of" and / or "one or more" of a set, or any other wording, indicates that one member or multiple members of a set (in any combination) satisfy the claim. For example, the wording of a claim that includes "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, the wording of a claim that includes "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The wording "at least one of" and / or "one or more" of a set does not limit the set to items listed in the set. For example, the wording of a claim that includes "at least one of A and B" or "at least one of A or B" can mean A, B, or A and B, and may further include items not listed in the set A and B.

[0166] Various exemplary logic blocks, modules, engines, circuits, and algorithmic steps described in relation to the embodiments disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly demonstrate this hardware- and software compatibility, various exemplary components, blocks, modules, engines, circuits, and steps have been schematically described above in relation to their functions. Whether such functions are implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. A person skilled in the art may implement the described functions in various ways for each specific application, but such implementation decisions should not be construed as causing a departure from the scope of this application.

[0167] The technologies described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such technologies may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication device handsets, or integrated circuit devices having multiple applications, including applications in wireless communication device handsets and other devices. Any feature described as a module or component may be implemented together within an integrated logic device, or separately as discrete but interoperable logic devices. When implemented in software, the technology may be at least partially implemented by a computer-readable data storage medium having program code that, when executed, performs one or more of the methods described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include memory or data storage media such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electro-erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. The technology may be implemented, at least in part, by a computer-readable communication medium, such as a propagated signal or wave, that carries or communicates program code in the form of instructions or data structures, and which can be accessed, read, and / or executed by a computer.

[0168] The program code may be executed by a processor which may include one or more processors such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated circuits or discrete logic circuits. Such a processor may be configured to perform any of the techniques described herein. The general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Accordingly, the term “processor” as used herein may refer to any of the aforementioned structures, any combination thereof, or any other structure or device suitable for implementing the techniques described herein.

[0169] Exemplary embodiments of this disclosure include:

[0170] Embodiment 1. A method comprising: an edge device acquiring multiple data streams corresponding to objects, activities, or events registered in an edge environment associated with the edge device; generating multiple features corresponding to each of the multiple data streams using one or more machine learning networks implemented on the edge device; the edge device determining a subset of prominent content from the multiple data streams, wherein the subset of prominent content is determined based on analyzing multiple features generated for each of the respective data streams; the edge device generating index information corresponding to the determined subset of prominent content; and locally storing the subset of prominent content and the generated index information on the edge device for retrieval and search based on local queries received on the edge device.

[0171] Embodiment 2. The method according to Embodiment 1, wherein generating multiple features includes determining learned embeddings of one or more machine learning networks for multiple data streams.

[0172] Embodiment 3. The method according to Embodiment 1 or 2, wherein the index information is generated and stored locally on the edge device without using a cloud connection and without the edge device communicating with a cloud computing entity.

[0173] Embodiment 4. The method according to any one of Embodiments 1 to 3, wherein the index information is generated using one or more machine learning networks implemented on an edge device, based on one or more of the following performed on multiple data streams: feature extraction, embedding generation, or tokenization.

[0174] Embodiment 5. The method of Embodiment 4, wherein determining a subset of prominent content comprises analyzing a plurality of features to identify duplicate data streams, wherein the duplicate data streams have the same or similar features as the respective features of previously indexed and stored data streams, and generating a subset of prominent content that includes a portion of a plurality of data streams that are not identified as duplicate data streams.

[0175] Embodiment 6. The method according to Embodiment 4 or 5, wherein determining a subset of outstanding content includes analyzing a plurality of features to classify objects or events present in a first data stream of a plurality of data streams; analyzing a plurality of features to classify objects or events present in a second data stream of a plurality of data streams, wherein the second data stream is different from the first data stream; and including the first and second data streams in a subset of outstanding content based on classifying the same objects or events present in the first and second data streams.

[0176] Embodiment 7. The method according to Embodiment 6, wherein the first data stream and the second data stream are associated with the same modality data type or sensor type.

[0177] Embodiment 8. The method according to Embodiment 6 or 7, wherein the first data stream is associated with a first modality of data type or sensor type, and the second data stream is associated with a second modality of data type or sensor type, the second modality being different from the first modality.

[0178] Embodiment 9. The method according to any one of Embodiments 1 to 8, wherein the index information includes a reverse index of a subset of prominent content.

[0179] Embodiment 10. The method according to Embodiment 9, wherein the keys of the inverse index include unique embeddings for multiple features, and one or more values ​​mapped to each of the keys of the inverse index include corresponding identifiers for each portion of content in a subset of prominent content associated with the unique embedding containing each key.

[0180] Embodiment 11. The method of Embodiment 10, further comprising: an edge device receiving a local query for stored content; an edge device processing the local query against generated index information to obtain corresponding identifiers for one or more portions of the stored content; generating a ranked order for one or more portions of the stored content, the ranked order being based on one or more of the local query or intent information determined for the local query; and the edge device outputting a user interface display showing one or more portions of the stored content in the ranked order.

[0181] Embodiment 12. The method according to Embodiment 11, wherein processing a local query on the generated index information includes determining a reverse index key corresponding to one or more search terms expressed in the local query, determining the relevance score of each piece of content mapped to each reverse index key corresponding to one or more search terms expressed in the local query, and outputting the corresponding identifier for each piece of content having a relevance score greater than a configured threshold.

[0182] Embodiment 13. The method according to any one of embodiments 1 to 12, further comprising transmitting identified content and corresponding index information of the identified content to a cloud entity, wherein the cloud entity is configured to store the identified content and corresponding index information in persistent cloud storage.

[0183] Embodiment 14. The method according to Embodiment 13, wherein the identified content comprises at least a portion of a subset of prominent content and is identified based on one or more of the following: backhaul bandwidth availability between an edge device and a cloud entity, or relevance information determined about the identified content.

[0184] Embodiment 15. The method according to any one of embodiments 1 to 14, further comprising: an edge device transmitting a plurality of embeddings corresponding to a subset of prominent content, the plurality of embeddings being transmitted to a cloud entity configured to generate a cloud-based reverse index using the plurality of embeddings, the cloud entity being configured to store the cloud-based reverse index as persistent cloud content.

[0185] Embodiment 16. The method of Embodiment 15, further comprising: receiving a local query for stored content by an edge device; determining that the relevance score of the local query for a subset of significant content stored on the edge device is below a configured threshold; and in response, sending one or more of the local query or generated features corresponding to the local query to a cloud entity associated with the edge device.

[0186] Embodiment 17. The method according to Embodiment 16, further comprising an edge device receiving a set of persistent cloud content identified by a cloud entity based on a local query or generated features, wherein the set of persistent cloud content is not included in a subset of prominent content stored on the edge device.

[0187] Embodiment 18. The method according to Embodiment 17, further comprising: analyzing a subset of prominent content and a set of persistent cloud content stored on an edge device to generate a composite hybrid ranking of edge content and persistent cloud content identified as relevant to a local query; and having the edge device output a user interface display showing the composite hybrid ranking of edge content and persistent cloud content.

[0188] Embodiment 19. The method of Embodiment 18, further comprising: receiving user input indicating the selection of specific cloud content to an edge device via a user interface display; the edge device sending a request to the cloud for underlying content corresponding to the selection stored in persistent cloud storage; the edge device receiving the underlying content corresponding to the selection; and the edge device outputting the underlying content within a user interface display.

[0189] Embodiment 20. The method according to any one of Embodiments 1 to 19, wherein the features are generated in-situ on an edge device, and the features are generated substantially in real time.

[0190] Embodiment 21. The method according to any one of Embodiments 1 to 20, wherein multiple features are generated based on analyzing each respective data stream using a corresponding machine learning classifier, encoder, representation learner, or feature generator.

[0191] Embodiment 22. The method according to Embodiment 21, wherein the corresponding machine learning network used to generate multiple features of each data stream is selected from multiple trained machine learning networks based on the identified data type modality of each data stream.

[0192] Embodiment 23. The method according to Embodiment 22, wherein a first machine learning network is used to perform one or more feature extraction or embedding extraction on each data stream in a first subset of multiple data streams, including image data or video frames; a second machine learning network is used to perform one or more feature extraction or embedding extraction on each data stream in a second subset of multiple data streams, including text data; and a third machine learning network is used to perform one or more feature extraction or embedding extraction on each data stream in a third subset of multiple data streams, including non-image or non-text data.

[0193] Embodiment 24. The method according to any one of Embodiments 1 to 23, wherein each data stream of the plurality of data streams includes a set of data content representing objects, activities, or events registered in the surrounding environment of the edge device or the plurality of edge sensors associated with the generation of the data content.

[0194] Embodiment 25. The method according to any one of Embodiments 1 to 24, wherein the edge device is included in a local edge environment, and each of the data streams of the multiple data streams is obtained from a sensor or asset included in the same local edge environment as the edge device.

[0195] Embodiment 26. The method according to any one of Embodiments 1 to 25, wherein the multiple data streams include multiple different data types and are acquired from multiple different edge sensors, each of which is different.

[0196] Embodiment 27. The method according to Embodiment 26, wherein the multiple different edge sensors include one or more of the following: a camera, a document scanner, a range or distance scanner, an accelerometer, a gyroscope, a vibrometer, a GPS, an inertial measurement unit, a LiDAR, a RADAR, an ultrasonic sensor, or a barcode scanner.

[0197] Embodiment 28. An edge device for processing streaming data, comprising at least one memory and at least one processor coupled to the at least one memory, wherein the at least one processor is configured to: acquire multiple data streams corresponding to objects, activities, or events registered in an edge environment associated with the edge device by the edge device; generate multiple features corresponding to each of the multiple data streams using one or more machine learning networks implemented on the edge device; determine a subset of prominent content from the multiple data streams by the edge device, the subset of prominent content being determined based on analysis of multiple features generated for each of the respective data streams; generate index information corresponding to the determined subset of prominent content by the edge device; and locally store the subset of prominent content and the generated index information on the edge device for retrieval and search based on local queries received on the edge device.

[0198] Embodiment 29. The apparatus according to Embodiment 28, wherein at least one processor is configured to determine learned embeddings of one or more machine learning networks for multiple data streams in order to generate multiple features.

[0199] Embodiment 30. The apparatus according to Embodiment 28 or 29, wherein the index information is generated and stored locally on the edge device without using a cloud connection and without the edge device communicating with a cloud computing entity.

[0200] Embodiment 31. The apparatus according to any one of Embodiments 28 to 30, wherein the index information is generated using one or more machine learning networks implemented on an edge device, based on one or more of the following performed on multiple data streams: feature extraction, embedding generation, or tokenization.

[0201] Embodiment 32. A non-temporary computer-readable medium storing instructions, wherein, when executed by one or more processors, the instructions cause one or more processors to perform actions including: obtaining a plurality of data streams by an edge device corresponding to objects, activities, or events registered in an edge environment associated with the edge device; generating a plurality of embeddings corresponding to one or more of the plurality of data streams using one or more machine learning networks implemented on the edge device; determining a subset of prominent content from the plurality of data streams, wherein the subset of prominent content is determined based on analyzing a plurality of features generated for each respective data stream; generating index information corresponding to the determined subset of prominent content by the edge device; and locally storing the subset of prominent content and the generated index information on the edge device for retrieval and search based on local queries received on the edge device.

[0202] Embodiment 33. An apparatus comprising means for performing any of the operations described in Embodiments 1 to 27.

[0203] Embodiment 34. A non-temporary computer-readable storage medium that, when executed by one or more processors, stores instructions causing one or more processors to perform any of the operations described in Embodiments 1 to 27.

Claims

1. It is a method, The edge device retrieves multiple data streams corresponding to objects, activities, or events registered in the edge environment associated with the edge device, Using one or more machine learning networks implemented on the edge device, generate multiple features corresponding to each of the multiple data streams, The edge device determines a subset of prominent content from the plurality of data streams, the subset of prominent content being determined based on an analysis of the plurality of features generated for each of the respective data streams. The edge device generates index information corresponding to the determined subset of prominent content, For searching and retrieving based on local queries received at the edge device, the subset of prominent content and the generated index information are stored locally at the edge device. Methods that include...

2. Generating the aforementioned multiple features means The method according to claim 1, comprising determining the learned embeddings of one or more machine learning networks for the plurality of data streams.

3. The method according to claim 1, wherein the index information is generated and stored locally on the edge device without using a cloud connection and without the edge device communicating with a cloud computing entity.

4. The method according to claim 1, wherein the index information is generated using one or more machine learning networks implemented on the edge device, based on one or more of the feature extraction, embedding generation, or tokenization performed on the plurality of data streams.

5. Determining the subset of the aforementioned prominent content is Identifying duplicate data streams by analyzing the aforementioned multiple features, wherein the duplicate data streams have the same or similar features as the respective features of the previously indexed and stored data streams. The method of claim 4, comprising generating a subset of the prominent content to include the portion of the plurality of data streams that are not identified as duplicate data streams.

6. Determining the subset of the aforementioned prominent content is By analyzing the aforementioned multiple characteristics, the objects or events present in the first data stream among the multiple data streams are classified. The process involves analyzing the aforementioned multiple characteristics to classify objects or events present in the second data stream among the multiple data streams, wherein the second data stream is different from the first data stream. The method according to claim 4, further comprising including the first data stream and the second data stream in a subset of the prominent content based on classifying the same objects or events as existing in the first data stream and the second data stream.

7. The method according to claim 6, wherein the first data stream and the second data stream are associated with the same modality data type or sensor type.

8. The method according to claim 6, wherein the first data stream is associated with a first modality of data type or sensor type, and the second data stream is associated with a second modality of data type or sensor type, the second modality being different from the first modality.

9. The method according to claim 1, wherein the index information includes an inverse index of a subset of the prominent content.

10. The key of the inverse index includes a unique embedding for the plurality of features, One or more values ​​mapped to each of the keys of the inverse index include the corresponding identifier of each portion of the content of the prominent content subset associated with the unique embedding containing each of the keys. The method according to claim 9.

11. The edge device receives local queries for stored content, The edge device processes the local query against the generated index information to obtain the corresponding identifiers of one or more portions of the stored content. The method involves generating a ranked order of one or more portions of the stored content, wherein the ranked order is based on one or more of the local query or intent information determined for the local query, and the edge device outputs a user interface display showing one or more portions of the stored content in the ranked order. The method according to claim 10, further comprising:

12. Processing the local query against the generated index information is: Determining a reverse index key corresponding to one or more search terms expressed in the local query, Determining the relevance score of each of the contents mapped to each reverse index key corresponding to the one or more search terms expressed in the local query, The method according to claim 11, comprising outputting the corresponding identifier for each portion of content having a relevance score greater than a configured threshold.

13. The edge device transmits identified content and corresponding index information of the identified content to a cloud entity, wherein the cloud entity is configured to store the identified content and corresponding index information in persistent cloud storage. The method according to claim 1, further comprising:

14. The method according to claim 13, wherein the identified content comprises at least a portion of the prominent content and is identified based on one or more of the following: backhaul bandwidth availability between the edge device and the cloud entity, or relevance information determined for the identified content.

15. The edge device transmits a plurality of embeddings corresponding to a subset of the prominent content, the plurality of embeddings are transmitted to a cloud entity configured to generate a cloud-based reverse index using the plurality of embeddings, and the cloud entity is configured to store the cloud-based reverse index as persistent cloud content. The method according to claim 1, further comprising:

16. The edge device receives local queries for stored content, Determining that the relevance score of the local query for the subset of prominent content stored on the edge device is below a configured threshold, In response, one or more of the local query or the generated features corresponding to the local query are sent to the cloud entity associated with the edge device. The method according to claim 15, further comprising:

17. The edge device receives a set of persistent cloud content identified by the cloud entity based on the local query or generated features, wherein the set of persistent cloud content is not included in the subset of prominent content stored on the edge device. The method according to claim 16, further comprising:

18. Analyzing the subset of prominent content and the set of persistent cloud content stored on the edge device to generate a composite hybrid ranking of edge content and persistent cloud content identified as relevant to the local query, The edge device outputs a user interface display showing the composite hybrid ranking of edge content and persistent cloud content, The method according to claim 17, further comprising:

19. The user interface display allows the edge device to receive user input indicating the selection of specific cloud content, The edge device sends a request to the cloud for the underlying content corresponding to the selection stored in persistent cloud storage. The edge device receives the underlying content corresponding to the selection, The edge device outputs the underlying content within the user interface display, The method according to claim 18, further comprising:

20. The method according to claim 1, wherein the plurality of features are generated in-situ on the edge device, and the plurality of features are generated substantially in real time.

21. The method according to claim 1, wherein the plurality of features are generated based on analyzing each respective data stream using a corresponding machine learning classifier, encoder, representation learner, or feature generator.

22. The method according to claim 21, wherein the corresponding machine learning network used to generate the plurality of features of each data stream is selected from a plurality of trained machine learning networks based on the identified data type modality of each data stream.

23. The first machine learning network is used to perform one or more feature extraction or embedding extraction on each data stream of the first subset of the plurality of data streams, which include image data or video frames. The second machine learning network is used to perform one or more feature extraction or embedding extraction on each data stream in the second subset of the plurality of data streams, which include text data. The third machine learning network is used to perform one or more feature extraction or embedding extraction on each data stream of the third subset of the plurality of data streams, which include non-image or non-text data. The method according to claim 22.

24. The method according to claim 1, wherein each of the plurality of data streams includes a series of data contents indicating objects, activities, or events registered in the surrounding environment of the edge device or of a plurality of edge sensors associated with the generation of data content.

25. The edge device is included in the local edge environment. Each of the aforementioned data streams is acquired from a sensor or asset included in the same local edge environment as the edge device. The method according to claim 1.

26. The aforementioned multiple data streams include multiple different data types and are acquired from multiple different edge sensors, The aforementioned multiple different edge sensors include one or more of the following: a camera, a document scanner, a range or distance scanner, an accelerometer, a gyroscope, a vibration meter, a GPS, an inertial measurement unit, a LiDAR, a radar detector, an ultrasonic sensor, or a barcode scanner. The method according to claim 1.

27. An edge device for processing streaming data, At least one memory, The system comprises at least one processor coupled to the at least one memory, and the at least one processor is The edge device retrieves multiple data streams corresponding to objects, activities, or events registered in the edge environment associated with the edge device. Using one or more machine learning networks implemented on the edge device, multiple features corresponding to each of the multiple data streams are generated. The edge device determines a subset of prominent content from the multiple data streams, and the subset of prominent content is determined based on analyzing the multiple features generated for each of the data streams. The edge device generates index information corresponding to the determined subset of prominent content, A device configured to locally store a subset of prominent content and the generated index information on the edge device for searching and retrieving based on local queries received on the edge device.

28. In order to generate the aforementioned multiple features, the at least one processor, The apparatus according to claim 27, configured to determine the learned embeddings of one or more machine learning networks for the plurality of data streams.

29. The aforementioned index information is generated and stored locally on the edge device without using a cloud connection and without the edge device communicating with a cloud computing entity. The index information is generated using one or more machine learning networks implemented on the edge device, based on one or more of the feature extraction, embedding generation, or tokenization performed on the plurality of data streams. The apparatus according to claim 27.

30. A non-temporary computer-readable medium storing instructions, wherein when an instruction is executed by one or more processors, the instructions are sent to the one or more processors. The edge device retrieves multiple data streams corresponding to objects, activities, or events registered in the edge environment associated with the edge device, Using one or more machine learning networks implemented on the edge device, generate multiple embeddings corresponding to one or more of the multiple data streams, The edge device determines a subset of prominent content from the plurality of data streams, the subset of prominent content being determined based on an analysis of the plurality of features generated for each of the respective data streams. The edge device generates index information corresponding to the determined subset of prominent content, A non-temporary computer-readable medium that causes the edge device to perform actions including storing locally on the edge device a subset of the prominent content and the generated index information for searching and retrieving based on local queries received on the edge device.

Images