Overlay metadata onto video streams on demand for intelligent video analytics

Abstract

Disclosed is on-demand overlaying of metadata onto video streams for intelligent video analysis. In various examples, a cloud computing system may store frames of a video stream and metadata generated from the frames in separate data stores, where each type of data is indexed using a shared timestamp. Thus, frames of a video stream may be stored and / or processed across any number of devices (e.g., edge devices and / or core devices) of a cloud computing system and corresponding metadata for these frames may be stored and / or generated, while being linked by timestamps. A client device may provide a request or query to dynamically annotate a video stream with a specific subset of metadata. When processing the request or query, the timestamp may be used to retrieve video data representing frames of the video stream and metadata extracted from those frames across the data store. The retrieved metadata and video data may be used to annotate these frames for display on a client device.

Description

Background Art

[0001] Intelligent video analysis (IVA) systems can be used to analyze video streams and provide analysis results in real time. For example, an IVA system can help a user identify and track specific objects, such as people or vehicles, that appear in a video stream. To this end, the IVA system can extract metadata about the object from the video stream and annotate the video stream with corresponding visual indicators (such as labels, bounding boxes, alarms, and / or colors). When an IVA system is implemented using a distributed computing architecture with edge and cloud, a module that processes the video stream and generates metadata may be followed by other modules that also generate additional metadata. When annotating frames of a video stream using this metadata, the annotations should match the frames that were analyzed to generate the metadata. A challenge arises when annotating a video stream because the analysis may be distributed across many devices that may not be in the same location, and the user may not be interested in viewing annotations for all metadata at once.

[0002] To address these challenges, conventional IVA systems generate multiple copies of a video stream with different annotations baked into the frames for each type of analysis engine used to process the source video data. When a user wishes to view a specific type of annotation, the user may be provided with an appropriate copy of the video stream. These copies are stored pre-processed, pre-annotated, and pre-labeled, which can create significant storage requirements considering each variant of the video stream where different annotations may be required. By including more annotations in the same video, fewer annotated videos can be created, but this can result in visually cluttered images due to the number of visual indicators that can be included within the video frames. This can reduce the usefulness of the annotated videos. Summary of the invention

[0003] Embodiments of the present disclosure relate to overlaying metadata on video streams on demand for intelligent video analysis. Systems and methods are disclosed in which video data of a video stream and metadata of the video stream can be stored in separate data stores (e.g., on different cloud devices), wherein timestamps associate frames of the video stream with corresponding portions of the metadata. When processing a request (e.g., from a client) to view a video stream annotated with metadata, the timestamps can be used to retrieve relevant frames and metadata across the data stores, and the metadata can be overlaid on the frames for viewing.

[0004] Compared to conventional systems, the present disclosure provides a cloud computing system that can store frames of a video stream and metadata generated from these frames in separate data stores, indexing each type of data using a shared timestamp. Therefore, frames of a video stream can be stored and / or processed across any number of devices (e.g., edge devices and / or core devices) of the cloud computing system and corresponding metadata for these frames can be stored and / or generated, while being linked by timestamps. A client device can provide a request or query to dynamically annotate a video stream (e.g., live video or historical video) using a specific subset of metadata (e.g., indicated by one or more metadata attributes). When processing a request or query, a timestamp can be used to retrieve video data representing frames of a video stream and metadata extracted from those frames across a data store. The retrieved metadata and video data can be used to annotate frames for display on a client device.

[0005] The disclosed method provides flexibility about how and where an intelligent video analysis (IVA) system implemented using a distributed computing architecture with edge and / or cloud components can analyze a video stream to generate metadata and store metadata and video data of the video stream. For example, any number of edge devices and / or core devices of a cloud computing system can be used to analyze the video stream and to generate and / or store metadata with timestamps. Any number of edge devices and / or core devices of a cloud computing system can be used to process and store corresponding video data with timestamps, and one or more of the edge devices and / or core devices can be different from those that process metadata. In processing a request or query, the corresponding metadata and video data can be retrieved from a data store (e.g., by a core device and / or an edge device) using a timestamp, and annotations can be performed by an edge device, a core device, and / or a client device. Wherein the video data is annotated on a cloud device, and the annotated video can be provided to a client device for display. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The present system and method for on-demand annotation of metadata on a video stream of the present invention are described in detail below with reference to the accompanying drawings, wherein:

[0007] Figure 1 is a diagram of an example of a video annotation system according to some embodiments of the present disclosure;

[0008] Figure 2A is an illustration of an example video stream according to some embodiments of the present disclosure;

[0009] Figure 2B is presented with an overlay according to some embodiments of the present disclosure and may correspond to Figure 2A An example of an annotated video stream of a video stream;

[0010] Figure 3is a diagram illustrating an example of an interface that may be used to request video annotations using a query according to some embodiments of the present disclosure;

[0011] Figure 4 is a diagram illustrating an example of an interface that may be used to request and view video annotations according to some embodiments of the present disclosure;

[0012] Figure 5 is a diagram illustrating an example of an interface that may be used to request and view video annotations according to some embodiments of the present disclosure;

[0013] Figure 6 is a flow chart illustrating an example of communications involving a client device and a cloud computing system, wherein an edge device retrieves metadata and video data to generate an overlay of a video stream, according to some embodiments of the present disclosure;

[0014] Figure 7 is a flow chart illustrating an example of communications involving a client device and a cloud computing system, wherein the cloud computing system retrieves and sends metadata for generating an overlay on a video stream, according to some embodiments of the present disclosure;

[0015] Figure 8 is a flow chart illustrating an example of communications involving a client device and a cloud computing system, wherein the client device retrieves metadata for overlaying on a video stream, according to some embodiments of the present disclosure;

[0016] Fig. 9 is a flow chart illustrating a method of generating a video overlay according to some embodiments of the present disclosure, which an edge device may use to retrieve metadata and video data for generating an overlay of a video stream;

[0017] Fig.10 is a flow chart illustrating a method of generating a video overlay according to some embodiments of the present disclosure, which a cloud computing system may use to retrieve and send metadata for generating an overlay for a video stream;

[0018] Fig.11 is a flow chart illustrating a method of generating a video overlay according to some embodiments of the present disclosure, which a client device may use to retrieve metadata for generating an overlay for a video stream;

[0019] Fig.12 is a block diagram of an example computing device suitable for implementing some embodiments of the present disclosure;

[0020] Fig.13 is a block diagram of an example data center suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0021] Systems and methods are disclosed that relate to overlaying metadata on video streams on demand for intelligent video analytics. Systems and methods are disclosed that can store video data of a video stream and metadata of the video stream in separate data stores (e.g., on different cloud devices), wherein timestamps associate frames of the video stream with corresponding portions of the metadata. When processing a request to view a video stream annotated with metadata, the timestamps can be used to retrieve relevant frames and metadata across the data stores, and the metadata can be used to generate an overlay of the frames for viewing.

[0022] The disclosed embodiments may be implemented using a variety of different systems, such as automotive systems, robotics, aviation systems, ship systems, smart area monitoring, simulations, and / or other technical fields. The disclosed methods may be used for any perception-based or more generally image-based analysis, monitoring, and / or tracking of objects and / or environments.

[0023] For smart area monitoring, various disclosed embodiments may be incorporated into the systems and / or methods described in U.S. Non-Provisional Application No. 16 / 365,581, filed on March 26, 2019, and entitled “Smart Area Monitoring with Artificial Intelligence,” which is incorporated herein by reference in its entirety.

[0024] In contrast to conventional systems, an annotated version of a video can be generated in response to a request, where the request provides (e.g., implicitly, explicitly, or inherently) one or more timestamps that are used to retrieve relevant video data and corresponding metadata generated during the video analysis process. As a result, video data retrieved using timestamps (e.g., video from 1 p.m. to 4 p.m. only) can be annotated with metadata that is relevant to the request (e.g., annotate only people and not other objects, color code school bags, mark vehicle license plates, etc.) and that corresponds to content depicted in frames represented by the retrieved video data.

[0025] In some examples, the request can be provided by a client device to a cloud computing system. For example, a core device of the cloud computing system can receive the request. When one or more portions of video data and / or metadata to be used for annotation are located in a core network, the core device can retrieve the portions and provide the data (directly or indirectly) to a video annotator (e.g., an edge device and / or a client device) for generating one or more overlays of the video data.

[0026] One or more portions of the video data and / or metadata to be used for annotation may additionally or alternatively be located outside the core network, such as at one or more edge devices of a cloud computing system and / or an external storage system (e.g., on a remote network or video management system that may be implemented on an external cloud storage system). In these cases, the core device may provide sufficient information for other devices to retrieve the requested data.

[0027] In some non-limiting embodiments, the data used for annotation can be distributed across any number of devices, systems, and / or locations, while annotation can occur at a single device, system, and / or location. In some examples, the device, system, and / or location that annotates the video using the retrieved information sends (e.g., streams) the annotated data to the client device for display thereon.

[0028] For example, if a portion of the video data is located at and / or managed by an edge device, the core device can provide one or more timestamps that the edge device can use to retrieve the corresponding video data. In some examples, the core device can use an index and / or search system to locate the appropriate edge device. The edge device can then provide the retrieved portion of the video data to a client device, a core device, and / or another edge device that performs the annotation. In other examples, the edge device that retrieves the video data can perform the annotation.

[0029] Similarly, if a portion of the metadata is located at and / or managed by the edge device, the core device can provide one or more timestamps that the edge device can use to retrieve the corresponding metadata. In addition, one or more metadata criteria (e.g., identification of attributes and / or other indicators of metadata) can be provided (e.g., indicators of metadata describing a person object in a frame, metadata describing a blue pickup truck, etc.). In some examples, the core device can use an index and / or search system to locate the appropriate edge device based on the timestamp and / or metadata criteria or indicators. For example, a request or query to find "blue pickup trucks on garage cameras between 2pm and 2:30pm" may result in retrieval of metadata corresponding to a time range between 2pm and 3:30pm with a timestamp, but only the metadata portion of the object classified as a blue pickup truck is included within the timestamp range (other portions of the metadata may be stored in the same or different metadata storage). The edge device can then provide the retrieved portion of the metadata to a client device, a core device, and / or another edge device that performs annotations. In other examples, the edge device that retrieves the metadata can perform the annotations.

[0030] Thus, a request may be received that specifies or otherwise indicates a timestamp for identifying one of a plurality of frames of a video stream. Each timestamp may correspond to a frame of the video stream (e.g., a video stream having a frame rate of 30fps may have a first frame with a timestamp and subsequent frames may have timestamps 33ms later). A device and / or system (e.g., an edge device) storing video data of a video stream may store the video data indexed or otherwise associated with the timestamps (e.g., locally, such as on-site or remote to an external location), such as in real time as the video data is generated. In some examples, the device (e.g., an edge device) may be a camera that captures a video stream, and the camera in a shared geographic location may store the corresponding video stream to a common storage system.

[0031] Time synchronization may be implemented across devices of a cloud computing system so that timestamps and video data are synchronized across the cloud computing system. For example, a device (e.g., a camera device) that stores and / or processes video data for storage may be connected to an Internet Protocol (IP) network and run a Network Time Protocol (NTP) daemon to synchronize time with other connected devices and determine synchronized timestamps for incoming video data. When video data is processed and / or generated, timestamps may be associated with corresponding frames for storage.

[0032] Similarly, a device and / or system (e.g., an edge device) that analyzes, stores, and / or generates metadata for annotation of a video stream can associate the metadata with a timestamp so that the metadata is indexed or otherwise associated with the timestamp in storage (e.g., local storage, such as on-site or remotely at an external location). A time synchronization system for video data can similarly be implemented on a device for metadata. This can also occur in real time as the video data is generated, received, and / or analyzed. In some examples, the device (e.g., an edge device) that analyzes and generates metadata from video data can be the same device (or a device in a shared geographic location) that generates or captures the video data, such as a camera that captures the video stream. The camera in the shared geographic location can store the corresponding metadata for the video stream into a common storage system.

[0033] When video stream data is analyzed (e.g., an analysis engine performs object tracking, facial recognition, object classification, etc.), metadata can be created that indicates the results of the analysis process performed. Metadata can be generated with respect to objects or aspects identified in one or more frames of a video. For example, metadata can represent the position and / or size of a bounding box of an object (e.g., a geometric shape whose position and size can change from frame to frame, indicating temporal tracking of an object) and / or can represent other properties of an object that has been identified in a video. Metadata can represent any detectable aspect of a video frame, such as a visual feature of a video frame. Each portion of metadata can be associated with a frame of video data, such as by using a timestamp that is synchronized with the video frame described by that portion of metadata.

[0034] Therefore, compared to conventional systems, matching the timestamps and metadata of video data can achieve the ability to store each across multiple devices and / or locations. The video data of a video stream can reside in one or more video data stores, which can be located in a location (e.g., geographic location) or device different from the metadata residing in one or more metadata stores. This can decouple the video analysis and metadata generation of the video stream from the video storage, thereby providing significant flexibility in how to implement a distributed computing architecture. For example, video data storage and / or video generation can be located on an edge device, while metadata storage and / or metadata generation and video analysis can be located on a core device or other edge device of a cloud computing system. The metadata storage can be co-located with the video data storage, or it can reside elsewhere because the timestamps of the stored video data are synchronized with the timestamps of the corresponding stored metadata.

[0035] One or more overlays of the video may be constructed using the retrieved video data and the retrieved metadata to display a version of the video frame with a visual indication of the metadata. Since the retrieved metadata may be limited to metadata reflecting the request, only metadata indicated or otherwise related to the request may be reflected in the generated overlay. The generated overlay may be baked into the video frame or sent with and / or in association with the corresponding video data to be displayed with the video data. Therefore, compared to conventional systems, a copy of the video stream does not need to be stored pre-processed, pre-annotated, and pre-marked for each type of annotation or each set of annotations. As a result of some embodiments of the systems and methods described herein, the storage requirements placed on the IVA system may be reduced while supporting multiple overlays to be generated for the video stream.

[0036] refer to Figure 1 , Figure 1is an example system diagram of a video annotation system 100 according to some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are set forth by way of example only. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to or in place of those shown, and some elements may be omitted entirely. In addition, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in combination with other components, and in any suitable combination and location. The various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, the various functions may be performed by a processor executing instructions stored in a memory.

[0037] The video annotation system 100 may include a client device 102, a cloud computing system 108, an edge device 104, and / or a core device 106, etc. In any example, there may be any number of client devices 102, edge devices 104, and / or core devices 106. The video annotation system 100 (and its components and / or features) may be implemented using one or more computing devices, such as those described in more detail below. Fig.12 A computing device 1200 is provided.

[0038] The client device 102 may include a smartphone, a laptop, a tablet computer, a desktop computer, and / or another type of device capable of requesting and / or presenting annotated video streams, such as the annotated video stream 140. By way of example and not limitation, the client device 102 may be embodied as a personal computer (PC), a laptop computer, a mobile device, a smartphone, a tablet computer, a smart watch, a wearable computer, a personal digital assistant (PDA), an MP3 player, a virtual reality head mounted display device, a global positioning system (GPS) or device, a video player, a virtual machine, a camera, a surveillance device or system, an entertainment system, a vehicle computer system, an embedded system controller, a remote control, an appliance, a consumer electronic device, a workstation, an edge device, any combination of these described devices, or any other suitable device.

[0039] The client device 102 may include an annotation requester 122B, a metadata store 118C, a video annotator 120C, a communication manager 136C, and a presentation manager 138. Although only a few components and / or features of the client device 102 are described in detail, Figure 1 For example, the client device 102 may include additional or alternative components, such as Fig.12 Those described by the computing device 1200.

[0040] The cloud computing system 108 may include one or more edge devices 104, one or more core devices 106, and / or one or more networks 110B. Figure 1 Only a few components and / or features of the cloud computing system 108 are shown, but this is not intended to be limiting. For example, the cloud computing system 108 may include additional or alternative components, such as the following Fig.12 In at least one embodiment, the cloud computing system 108 may be at least partially Fig.13 is implemented in data center 1300.

[0041] The edge device 104 may include a video management system 150 having one or more video managers 112 and one or more video data stores 116, one or more stream analyzers 114, one or more metadata stores 118A, and / or one or more communication managers 136A. Figure 1 Only a few components and / or features of the edge device 104 are shown, but this is not intended to be limiting. For example, the edge device 104 may include additional or alternative components, such as Fig.12 Those described by the computing device 1200.

[0042] The components of the video annotation system 100 may communicate via networks 110A and 110B (collectively or individually referred to herein as "networks 110"). The networks 110 may include wide area networks (WANs) (e.g., the Internet, public switched telephone networks (PSTN), etc.), local area networks (LANs) (e.g., Wi-Fi, ZigBee, Z-Wave, Bluetooth, Bluetooth Low Energy (BLE), Ethernet, etc.), low power wide area networks (LPWANs) (e.g., LoRaWAN, Sigfox, etc.), global navigation satellite system (GNSS) networks (e.g., global positioning systems (GPS)), and / or other network types. In any example, each component of the video annotation system 100 may communicate with one or more of the other components via one or more of the networks 110. In some examples, the network 110B includes one or more core and / or edge networks of the cloud computing system 108, and the network 110A includes the Internet.

[0043] The video manager 112 may include one or more components and features for managing video stream data. The video manager 112 may be implemented or instantiated using one or more edge devices 104. In at least some embodiments, the video manager 112 may be instantiated at least in part using a video or imaging device (e.g., a camera device, a sensor device, etc.). The video manager 112 may store the video stream data in one or more video data stores 116 so that the video stream data is synchronized across any number of devices using a timestamp associated with a frame of the video stream data. For example, the video manager 112 may be connected to an Internet Protocol (IP) network and run a network time protocol (NTP) daemon to synchronize time with other connected devices (e.g., other video managers 112 and / or one or more stream analyzers 114 for storing metadata in one or more metadata stores 118) across a cloud computing system. As an example, the video manager may process and / or generate a video with a timestamp associated with a corresponding frame of the video. Video data with associated timestamps may be stored in one or more video data stores 116A on the edge device 104 of the video manager 112 or in one or more video data stores 116 that may be located on any number of connected edge devices 104 or client devices 102 (or on a cloud computing system external to the cloud computing system 108).

[0044] In at least some embodiments, the video manager 112 may include a video streaming server, such as a real-time streaming protocol (RTSP) server, which may be used to establish and control a media session between any of the various endpoints of the video annotation system 100. For example, the video annotator 120 may communicate with the streaming server to receive image data and / or video data for annotation (e.g., using a representational state transfer application program interface). In various examples, communication between devices may be provided through an Open Network Video Interface Forum (ONVIF) bridge, which may define rules for how software should query a device for its name, settings, streams, etc. An ONVIF "call" or message may be sent to an ONVIF-compliant device, and the device may return an RTSP address to retrieve the corresponding video and / or image data via the RTSP server. In some embodiments, the video manager 112 may also provide video data to the stream analyzer 114 using RTSP communications.

[0045] Stream analyzer 114 may include one or more components and features for analyzing video stream data (eg, an analysis engine that performs object tracking, facial recognition, object classification, etc.).

[0046] The stream analyzer 114 may be configured to receive and process video data. For example, the stream analyzer 114 may receive video data from the video manager 112 and / or from the video data storage 116. The stream analyzer 114 may be configured to process the video data to detect objects from the video data, such as objects depicted in one or more frames of the video stream. The stream analyzer may be configured to determine object properties, such as the properties of the objects detected by the stream analyzer 114. The stream analyzer 114 may be configured to track objects represented in the video data. In an embodiment where there are multiple stream analyzers 114, different stream analyzers 114 may perform different analysis processes on the video data and may generate different metadata using different analysis processes. For example, one stream analyzer 114 may use one or more machine learning models (MLMs) to detect vehicles and / or vehicle positions in video frames and store the corresponding metadata in the metadata storage 118. Another stream analyzer 114 may use one or more MLMs to extract features of the detected vehicles and store the corresponding metadata in the metadata storage 118. Yet another stream analyzer 114 may use one or more MLMs to detect pedestrians and / or pedestrian locations in video frames and store corresponding metadata in metadata storage 118. These stream analyzers 114 may operate serially and / or in parallel on the same or different video streams. As described herein, MLMs may be used (e.g. Fig.13 This is accomplished by one or more GPUs and / or vGPUs of the data center 1300).

[0047] The stream analyzer 114 may include one or more components and features for generating metadata. Metadata may be generated with respect to an object or aspect identified in one or more frames of video stream data. For example, metadata may represent the position and / or size of a bounding box of an object (e.g., a geometric shape whose position and size may change frame by frame, indicating the temporal tracking of an object) and / or may represent other properties of an object identified in a video. The metadata generated by the stream analyzer 114 may represent any detectable aspect of a video frame, such as a visual feature of a video frame. The stream analyzer 114 may use a timestamp to generate metadata that may be associated with a frame of video data. For example, the stream analyzer 114 may generate metadata for a frame of video data having a specific timestamp, and may synchronize frames and metadata by associating the metadata with the timestamp. As an example, the timestamp associated with the metadata may originate from the video stream data and / or may originate from the stream analyzer 114 (e.g., via an IP network, an NTP daemon, etc.). In various examples, the timestamps of the video data and metadata may be synchronized across each of the video manager 112 and the stream analyzer 114 (e.g., using the same time synchronization system or method). The stream analyzer 114 may send the metadata generated for each frame (eg, with a timestamp) that has been associated with the timestamp to one or more metadata stores 118 for storage therein.

[0048] The metadata store 118A, metadata store 118B, and metadata store 118C (collectively or individually referred to herein as "metadata stores 118") may be located on edge devices 104, core devices 106, and / or client devices 102. In some non-limiting embodiments, metadata for annotation may be distributed across any number of metadata stores 118 across any number of devices, systems, and / or locations. For example, metadata corresponding to a video stream may be located on a combination of metadata store 118A on edge device 104 and metadata store 118B on one or more core devices 106. Metadata may be indexed on one or more metadata stores 118 based on any number of methods, which may include, but are not limited to, timestamps of the metadata, classifications of objects described by the metadata, locations where the metadata was generated, attributes of the associated video data, or any combination of methods. As an example, metadata describing a vehicle object depicted in the corresponding video data may be indexed and located in a first metadata store, while metadata describing a person object may be indexed and located in a second metadata store.

[0049] Annotation requesters 122A and 122B (collectively or individually referred to herein as "annotation requesters 122") may generate a request that provides one or more timestamps for retrieving corresponding video data (e.g., only video from 1 to 4 p.m.) and corresponding metadata generated during the video analysis process. For example, annotation requesters 122 may generate a request that is sent to one or more video annotators 120, which may use the provided timestamps to retrieve video data from one or more video data stores 116 and metadata from one or more metadata stores 118, as described herein. For example, the request may indicate an implicit or explicit timestamp corresponding to video data in one of the plurality of video data stores 116. Annotation requesters 122 may also generate a request that provides one or more criteria or indicators of metadata corresponding to metadata in one or more metadata stores 118 (e.g., metadata indicating only people and not other objects, metadata describing vehicles, etc.). In some examples, the request may be generated in response to a query from which one or more timestamps may be determined (e.g., searching for objects within a time window to find frames depicting the objects). In some examples, the request may be predetermined or otherwise derived without requiring a user to provide a query. For example, one or more devices of the cloud computing system 108 may include a media server for providing annotated video data using WebRTC. One or more devices of the cloud computing system 108 may also provide a video management system API that can be used to request annotated video data.

[0050] In an embodiment, the annotation requester 122 and / or the presentation manager 138 may communicate with one or more devices of the cloud computing system 108 to request and / or present annotated video data using a REST API and / or a Web Real-Time Communications (WebRTC) API.

[0051] The video data store 116 may be located on one or more edge devices 104. In some non-limiting embodiments, the video data may be distributed across any number of video data stores 116 across any number of devices, systems, and / or locations. For example, video data corresponding to one or more video streams may be located on a combination of video data stores 116 on multiple edge devices 104. Video from a video stream may be indexed in one or more video data stores 116 based on any number of factors, which may include, but are not limited to, a timestamp of the video data, a source of the video stream, an attribute of the video data, or any combination thereof. As an example, video data generated from a first camera or imaging device may be indexed and located in a first video data store 116, while video data generated from a second camera or imaging device may be indexed and located in a second video data store 116.

[0052] The video annotator 120A and the video annotator 120C (collectively or individually referred to herein as "video annotator 120") may be configured to generate overlay data associated with frames of video data (e.g., using one or more GPUs and / or vGPUs). The video annotator 120 may use metadata from one or more metadata stores 118 and video data from one or more video data stores 116 to construct one or more overlays to the video frames. In some examples, the overlay data may be incorporated into the video frames associated with the video data and may take the form of a visual indication of the metadata (e.g., a bounding box, a bounding shape, a color highlight, a change in pixels of the video frame, etc.). In other examples, the overlay data may be sent to the presentation manager 138 on one or more client devices 102 along with and / or otherwise associated with the corresponding video data for display with the video data. As shown in the example, the overlay data may represent a bounding box 144 or other visual indicator for annotating the corresponding video data. In various examples, the overlay data may include image data representing at least one or more visual indicators and may or may not be stored with the video data being annotated.

[0053] In various embodiments, the video annotator 120 may receive one or more requests from the annotation requester 122 indicating one or more timestamps and indications of metadata. The timestamps and indications of the metadata may be used to retrieve relevant video data and metadata from the metadata store 118 and the video data store 116. In some embodiments, only metadata reflecting the request may be retrieved from the metadata store 118, and only metadata indicated or otherwise relevant to the request may be reflected in an overlay generated by the video annotator 120. In some examples, the metadata reflected in the generated overlay may be a subset of the metadata retrieved from the metadata store 118.

[0054] Communications manager 136A, communications manager 136B, and communications manager 136C (collectively or individually referred to herein as "communication managers 136") may include one or more components and features for communicating across one or more networks (e.g., network 110). Communications manager 136 may be configured to communicate via any number of networks 110 described herein. For example, to communicate between Figure 1In order to communicate in the video annotation system 100 of the embodiment of the present invention, the client device 102 can access the Internet through a router using an Ethernet or Wi-Fi connection to communicate with the edge device 104, the core device 106, other client devices 102, and / or any other device within the cloud computing system 108. In some examples, the core device 106 and the edge device 104 of the cloud computing system 108 can communicate with other core devices 106 and / or edge devices 104 located within the cloud computing system 108.

[0055] The communication manager 136 may be configured to manage communications received by and / or provided by its respective system (e.g., requests, queries, video data and / or metadata, overlay data, etc.), such as in any of the client devices 102, edge devices 104, core devices 106, and / or other components that may be included in the video annotation system 100. Additionally or alternatively, the communication manager 136 may manage communications within the respective system.

[0056] The presentation manager 138 of the client device 102 may include one or more components and features for managing the presentation of frames of a video stream, such as frame 140A of annotated video stream 140. The frames of annotated video stream 140 may be represented in video data and may be associated with overlay data. For example, the presentation manager 138 may manage the presentation of overlay data sent from the video annotator 120, where the overlay data may be incorporated into the video frame associated with the visual indication of the metadata (e.g., in the video data). In some examples, the presentation manager 138 may use a combination of the video data and the overlay data to present and generate the frames of the annotated video stream 140.

[0057] Reference now Figure 2A , Figure 2A 2 is a diagram of an example video stream 200 according to some embodiments of the present disclosure. A frame 202 of the video stream 200 may contain any number of objects, such as Figure 1 The stream analyzer 114 tracks, classifies, detects, and / or identifies the objects 210 , 220 , and 230 (eg, tracking moving objects, color detection, facial recognition, text from vehicle license plates, etc.) to extract metadata therefrom.

[0058] Reference now Figure 2B , Figure 2B According to some embodiments of the present disclosure, Figure 2A1 is a diagram of an example of an annotated video stream 140 with an overlay of the video stream 200. A frame of the annotated video stream 140, such as frame 140A, may be visualized using overlay data that is incorporated into the video data and / or provided as metadata. Figure 1 The overlay data provided by the video annotator 120 may be provided by Figure 1 The rendering manager 138 of the video stream 140 is used to render one or more frames of the annotated video stream 140 with visualizations corresponding to the metadata, such as the bounding box 144 and the text information 232. Although the bounding box 144 and the text information 232 are shown, the visualization may be rendered as a geometric shape, a boundary, text data, a color, and / or any other suitable visual indication of the metadata.

[0059] Reference now Figure 3 , Figure 3 is a diagram showing an example of an interface 300 that can be used to request and visualize video annotations using queries according to some embodiments of the present disclosure. A user can use the interface 300 to cause a client device 102 to request annotated videos, such as by using Figure 1 The annotation requester 122 of the embodiment of the present invention provides a query to generate an annotated video. As a specific and non-limiting example, the query may require finding all blue vehicles identified in one or more video streams in the past 7 days. The user can provide the query to the interface 300 using any number of methods. For example, the user can provide the query and / or one or more parts of the information used to generate the query to the search bar 320 and / or the time interval selector 310. In response to the query, the interface 300 can provide a display 330 indicating video data and associated metadata that meet the query (e.g., as query results). For example, the display 330 can display which days (or other time intervals) and at which times (e.g., corresponding to the start time, end time, etc.) have been found in the video stream data. The query results may include video clips and / or time indicators that enable playback of at least a portion of the video stream corresponding to the query results. In some examples, the interface 300 can display the generated annotated video in the video display area 340 (e.g., based on the selection of the query result indicator 332). Interface 300 may also include video controls 350 that allow a user to control playback of video display area 340 .

[0060] Reference now Figure 4 , Figure 44 is a diagram illustrating an example of an interface 400 that can be used to request and view video annotations according to some embodiments of the present disclosure. A user can use the interface 400 to view annotated videos and related information. For example, a user can use the interface 400 to view a generated annotated video in a video display area 410. The interface 400 may include video controls 420A, 420B, 420C, 420D, and / or 420E, which allow a user to control playback of the video display area 410. In various examples, a user can use one or more of the video controls 420A, 420B, 420C, 420D, and / or 420E to provide a request or query for annotated video data.

[0061] In some examples, video display area 410 may present a video of a detected object (e.g., object 210). In some examples, video display area 410 may present a video of a detected object (e.g., object 210). Figure 1 The overlay data may be generated in response to a request and / or query sent by the annotation requester 122 of the video annotator 120. As described herein, the overlay data may be visualized in a variety of ways, such as an enclosing shape 440 or other indicator that may visually emphasize an object identified in the video stream and captured by the associated metadata. In some examples, information corresponding to an object in the video stream may be retrieved from one or more of the metadata stores 118 and may be provided to the video annotator 120 for use in generating annotated video data. The annotated video data may be provided by the video annotator 120 to the presentation manager 138 for displaying the annotated video data on the interface 400. As described herein, the presentation manager 138 may receive one or more portions of the overlay data baked into a frame of the video data, and / or one or more portions of the overlay data may be in the form of metadata for generating a visualization. As an example, the metadata may be used to generate text information 450 that may be separate from or included in the video display area 410.

[0062] Reference now Figure 5 , Figure 5 is a diagram showing an example of an interface 400 that may be used to request and view video annotations, according to some embodiments of the present disclosure. Figure 5 The interface 400 may be similar to Figure 4 interface 400, but from a vehicle view rather than a hallway view. For example, the user may have used video control 420E to select a vehicle view of object 210, rather than a hallway view. Figure 4 . By way of example and not limitation, bounding box 144 is displayed around object 210. Bounding boxes may be displayed for other requested objects, such as vehicle 510 having bounding box 544. Also by way of example and not limitation, at least a portion of text information 450 is shown as being displayed at a location corresponding to the location of bounding box 144.

[0063] Reference now Figure 6 , Figure 6 6 is a flow chart showing an example of communication involving a client device and a cloud computing system according to some embodiments of the present disclosure, in which an edge device retrieves metadata and video data for generating an overlay of a video stream. It should be understood that this and other arrangements described herein are presented only as examples. At a high level, communication 600 may involve one or more of a client device 102, an annotation requester 122, a metadata store 118, a video management system 150, a video manager 112, a video data store 116, and / or a video annotator 120A executed on one or more edge devices 104 and / or instantiated using one or more edge devices 104. Although certain data is shown as being sent or received by a client device 102 and an edge device 104, this does not mean that one or more other components shown cannot be included in the client device 102 and / or edge device 104. In the example, any number of edge devices 104 and / or core devices 106 may be used to implement communication 600 and may involve additional communications between devices, which are not shown.

[0064] In some examples, the communication 600 may include sending timestamp data 610 between one or more devices. For example, the annotation requester 122 may send the timestamp data 610 to the video annotator 120A located on the edge device 104. The timestamp data 610 may represent any indication of one or more timestamps that may be used to identify one or more frames of a video stream. For example, the timestamp data 610 may represent at least one timestamp corresponding to a frame of a video stream. As another example, the timestamp data 610 may represent information for deriving or determining at least one timestamp. For example, the information may include a request type and / or query indicating at least one timestamp, or may include a start timestamp of a video stream requested for annotation. In some examples, the timestamp data 610 may be part of a request for an annotated video from the annotation requester 122A of the client device 102, as described herein. For example, the request may indicate metadata for annotation, one or more timestamps, one or more video streams, etc.

[0065] Upon receiving the timestamp data 610 sent from the annotation requester 122, the video annotator 120A may use the timestamp data 610 to generate request data 612 representing one or more requests that may be sent to the metadata store 118 so that metadata 616 corresponding to at least one timestamp indicated by the timestamp data 610 may be retrieved. In some examples, the request may also be based on an indication of metadata included in the request from the annotation requester 122 (e.g., one or more attributes included in the request). The video annotator 120A may also use at least the timestamp data 610 to generate request data 614 representing one or more requests that may be sent to the video data store 116 so that image data 618 (e.g., video data) corresponding to the at least one timestamp may be retrieved. The order in which the request data 612 and 614 are shown as being transmitted is not intended to be limiting.

[0066] Upon receiving the request data 612 sent from the video annotator 120A, the metadata store 118 may retrieve metadata 616 corresponding to at least the timestamp data 610. The metadata store 118 may store metadata indexed or otherwise associated with at least one timestamp indicated by the timestamp data 610. The metadata store 118 may further store metadata indexed based on the indication or other attributes of the metadata, which may be included in the request data 612. The retrieved metadata 616 may be sent by the metadata store 118 to the video annotator 120A.

[0067] Upon receiving the request data 614 sent from the video annotator 120A, the video management system 150 may access the video data store 116 to retrieve image data 618 corresponding to at least one timestamp indicated by the timestamp data 610. The video data store 116 may store the image data 618 (or video data) indexed or otherwise associated with the at least one timestamp. The retrieved image data 618 may be sent by the video management system 150 to the video annotator 120A. In various examples, the request data 614 may be received by the video manager 112 and the image data 618 may be retrieved and provided by the video manager 112 from the video data store 116.

[0068] Upon receiving metadata 616 sent from metadata store 118 and image data 618 sent from video management system 150, video annotator 120A may generate overlay data 622 using metadata 616 and image data 618. Overlay data 622 may be sent by video annotator 120A to one or more client devices 102 for display by presentation manager 138.

[0069] Reference now Figure 7 , Figure 7700 is a flow chart showing an example of communication involving a client device and a cloud computing system according to some embodiments of the present disclosure, wherein the cloud computing system retrieves and sends metadata to generate an overlay on a video stream. It should be understood that this and other arrangements described herein are presented only as examples. At a high level, communication 700 may involve an annotation requester 122B and a presentation manager 138 located on a client device 102. Communication 700 may also involve metadata storage 118, a video management system 150, a video manager 112, a video data storage 116, a video annotator 120, and a cloud computing system 108. Although certain data is shown as being sent or received by a client device 102 and a cloud computing system 108, this does not mean that one or more of the other components shown cannot be included in the client device 102 and / or the cloud computing system 108. In the example, any number of edge devices 104 and / or core devices 106 may be used to implement communication 700 and may involve additional communications between devices, which are not shown.

[0070] In some examples, communication 700 may include sending timestamp data 710 between one or more devices. For example, annotation requester 122B may send timestamp data 710 representing one or more timestamps to cloud computing system 108. Timestamp data 710 may be similar to Figure 6 Describes the timestamp data 610 .

[0071] After receiving the timestamp data 710 sent from the annotation requester 122B, the cloud computing system 108 can use the timestamp data 710 to generate request data 712, which can be sent to the metadata storage 118 so that metadata 714 corresponding to the at least one timestamp can be retrieved. The request data 712 can be similar to Figure 6 Request data 612.

[0072] Upon receiving the request data 712 sent from the cloud computing system 108, the metadata storage 118 can retrieve metadata 714 corresponding to the timestamp data 710. The metadata storage 118 can store metadata indexed or otherwise associated with at least one timestamp. The metadata storage 118 can also store metadata indexed based on the indication and / or other attributes of the metadata, which can be indicated by the request for annotation from the client device 102. The retrieved metadata 714 can be sent by the metadata storage 118 to the cloud computing system 108.

[0073] After receiving the metadata 714 sent from the metadata storage 118, the cloud computing system 108 may send the metadata 716 and the timestamp data 718 to the video annotator 120. The video annotator 120 may be located, for example, on the edge device 104 or the core device 106. The timestamp data 718 may correspond to the timestamp data 710. For example, the timestamp data 718 may indicate at least one timestamp indicated by the timestamp data 710.

[0074] Upon receiving the timestamp data 718 from the cloud computing system 108 , the video annotator 120 may generate request data 720 representing one or more requests that may be sent to the video data store 116 and / or the video manager 112 so that image data 722 corresponding to the timestamp data 718 may be retrieved.

[0075] Upon receiving the request data 720 from the video annotator 120, the video management system 150 may access the video data store 116 to retrieve image data 722 corresponding to at least one timestamp indicated by the timestamp data 718. The video data store 116 may store the image data 722 indexed or otherwise associated with the at least one timestamp. The retrieved image data 722 may be sent by the video management system 150 to the video annotator 120.

[0076] After receiving the image data 722 and the metadata 716, the video annotator 120 may generate overlay data 726, which may be similar to Figure 6 The overlay data 622. The overlay data 726 may be sent by the video annotator 120 to the client device 102 for presentation by the presentation manager 138.

[0077] Reference now Figure 8 , Figure 88 is a flow chart showing an example of communication involving a client device and a cloud computing system according to some embodiments of the present disclosure, in which the client device retrieves metadata for superimposition on a video stream. It should be understood that this and other arrangements described herein are presented only as examples. At a high level, communication 800 may involve an annotation requester 122B and a presentation manager 138 located on a client device 102. Communication 800 may further involve metadata storage 118, a video annotator 120, and a cloud computing system 108, which has a video management system 150, a video manager 112, and a video data storage 116A. Although certain data is shown as being sent or received by the client device 102 and the cloud computing system 108, this does not mean that one or more of the other components shown cannot be included in the client device 102 and / or the cloud computing system 108. In the example, any number of edge devices 104 and / or core devices 106 may be used to implement communication 800 and may involve additional communications between devices, which are not shown.

[0078] In some examples, communication 800 may include sending timestamp data 810 between one or more devices. For example, annotation requester 122B may send timestamp data 810 to cloud computing system 108. Timestamp data 810 may represent at least one timestamp and may be used by video management system 150 of cloud computing system 108 to retrieve corresponding image data 812 from video data storage 116A of cloud computing system 108. In various examples, timestamp data 810 may be similar to Figure 7 Timestamp data 710 (eg, received in a request for annotation).

[0079] Upon receiving the timestamp data 810 sent from the annotation requester 122B, the video management system 150 of the cloud computing system 108 may use the timestamp data 810 to retrieve the image data 812 from the video data store 116 corresponding to the timestamp data 810. The video data store 116A may be located on one or more of the edge device 104 and / or the core device 106. For example, the timestamp data 810 may be received by the core device 106, and the image data 812 may be retrieved by the same or different core device 106 and / or edge device 104. The video data store 116 may store the image data 812 indexed or otherwise associated with at least one timestamp indicated by the timestamp data 810. The retrieved image data 812 may be sent by the video management system 150 to the annotation requester 122B.

[0080] In some examples, the annotation requester 122B may send timestamp data 814 representing at least one timestamp to the metadata store 118 so that metadata 816 corresponding to the at least one timestamp may be retrieved. The timestamp data 814 may also have one or more criteria or attribute indicators for identifying or selecting metadata 816 to be retrieved from the metadata store 118. In embodiments where the metadata store 118 is located on the cloud computing system 108, the timestamp data 814 may not be sent to the metadata store 118, as the timestamp data 810 may be sufficient. In examples where the metadata store 118 includes a metadata store 118C located on the client device 102, the sending of the timestamp data 814 may be at least partially within the client device 102.

[0081] In a further example, the annotation requester 122B may send the image data 818 to the video annotator 120. The annotation requester 122B may additionally send metadata 820 to the video annotator 120. The image data 818 may correspond to (e.g., include) the image data 812 and the metadata 820 may correspond to (e.g., include) the metadata 816. In embodiments where the video annotator 120 corresponds to the video annotator 120C of the client device 102, the metadata 820 and the image data 818 may be sent at least partially within the client device 102. In embodiments where the video annotator 120 is located on the cloud computing system 108, the timestamp data 820 and / or the image data 818 may not be sent to the video annotator 120, as the metadata 816 and the image data 812 may be sufficient.

[0082] Using the image data 818 and the metadata 820, the video annotator 120 may generate overlay data 822, which may be similar to Figure 7 The overlay data 726. The overlay data 822 may be sent by the video annotator 120 to the presentation manager 138 located on the client device 102.

[0083] Reference now Fig. 9 , each block of method 900 and other methods described herein includes a computing process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in a memory. The method can also be embodied as computer usable instructions stored on a computer storage medium. The method can be provided by a stand-alone application, a service or a hosted service (standalone or in combination with another hosted service), or a plug-in for another product, to name a few. In addition, by way of example, with respect to Figure 1 and Figure 6The method 900 is described with reference to a system of FIG. However, the method may additionally or alternatively be performed by any one system or any combination of systems, including but not limited to those described herein.

[0084] Fig. 9 900 is a flow chart illustrating a method 900 for generating a video overlay, which an edge device may use to retrieve metadata and video data to generate an overlay of a video stream, according to some embodiments of the present disclosure. At block B 902, the method 900 includes receiving one or more timestamps and at least one indicator of metadata. For example, the video annotator 120A may receive timestamp data 610 indicating one or more timestamps, and may also receive an indicator of metadata 616 from the annotation requester 122.

[0085] At block B904, the method 900 includes retrieving image data representing one or more frames using the one or more timestamps. For example, the video annotator 120A may use the timestamps to request image data 618 representing one or more frames from one or more video data stores 116. The video data stores 116 may be executed and / or instantiated using one or more devices. For example, the video data stores 116 may be executed and / or instantiated using one or more edge devices 104, core devices 106, and / or client devices 102.

[0086] At block B906, the method 900 includes retrieving metadata using the one or more timestamps based on at least one indicator of the metadata. For example, the video annotator 120A requests metadata 616 from one or more metadata stores 118 based on the one or more timestamps and at least one indicator of the metadata 616. The metadata stores 118 may be located on one or more further devices. For example, the metadata stores 118 may be located on one or more edge devices 104, core devices 106, and / or client devices 102.

[0087] At block B 908 , the method 900 includes generating image data representing one or more overlays. For example, the video annotator 120A may use metadata 616 retrieved from one or more metadata stores 118 and image data 618 from one or more video data stores 116 to generate overlay data 622 , which may include image data representing an overlay.

[0088] At block B 910 , the method 900 includes sending image data representing one or more overlays. For example, the video annotator 120A may send overlay data 622 including image data representing one or more overlays to the client device 102 .

[0089] Fig.10is a flow chart showing a method 1000 for generating a video overlay according to some embodiments of the present disclosure, which a cloud computing system may use to retrieve and send metadata for generating an overlay of a video stream. The method 1000 is illustrated by way of example with reference to Figure 1 and Figure 7 However, the method may additionally or alternatively be performed by any one system or any combination of systems, including but not limited to those described herein.

[0090] At block B1002, method 1000 includes receiving a request indicating one or more timestamps and metadata. For example, cloud computing system 108 may receive timestamp data 810, which may be included in a request from annotation requester 122B. The request may include data representing an indicator of one or more timestamps and metadata 816. In some examples, annotation requester 122B may be located on client device 102 or may correspond to annotation requester 122 located on one or more core devices 106 or edge devices 104 of cloud computing system 108.

[0091] At block B1004, the method 1000 includes retrieving metadata using the one or more timestamps. For example, the video annotator 120 may use the timestamp data 710 to request metadata 714 from one or more metadata stores 118 using request data 712. In some examples, the metadata stores 118 may be located on one or more devices. For example, the metadata stores 118 may be located on one or more edge devices 104, core devices 106, and / or client devices 102.

[0092] At block B1006, the method 1000 sends data representing metadata and one or more timestamps to cause at least one remote device to retrieve image data representing one or more frames and generate image data representing one or more overlays. For example, a video annotator 120 that can be executed or instantiated using components of the cloud computing system 108 can send timestamp data 718 and metadata 716 to the video annotator 120, so that the video annotator 120 retrieves image data 722 from one or more video data stores 116 and generates overlay data 726, which can include image data representing one or more overlays. In some examples, the metadata store 118 can be executed or instantiated using one or more devices. For example, the metadata store 118 can be executed or instantiated using one or more edge devices 104, core devices 106, and / or client devices 102. In addition, the video data store 116 can be executed or instantiated using one or more devices. For example, the video data store 116 can be executed or instantiated using one or more edge devices 104, core devices 106, and / or client devices 102.

[0093] Fig.11 is a flow chart illustrating a method 1100 for generating a video overlay according to some embodiments of the present disclosure, which a client device may use to retrieve metadata for generating an overlay of a video stream. Figure 1 and Figure 8 The method 1100 is described in detail with reference to a system of the present invention. However, the method may additionally or alternatively be performed by any system or any combination of systems, including but not limited to those described herein. At block B 1102, the method 1100 includes sending one or more timestamps and at least one indicator of metadata. For example, the annotation requester 122B executed or instantiated using the client device 102 may send a request including the timestamp data 810 and at least one indicator of the metadata 816 to the cloud computing system 108.

[0094] At block B1104 , the method 1100 includes receiving image data representing one or more frames using the one or more timestamps. For example, the annotation requester 122B executed or instantiated using the client device 102 may receive image data 812 representing one or more frames of a video stream from one or more video data stores 116A of the cloud computing system 108 .

[0095] At block B 1106, the method 1100 includes retrieving metadata using the one or more timestamps based on the at least one indicator. For example, the annotation requester 122B executed or instantiated using the client device 102 may transmit a request including the timestamp data 814 to one or more metadata stores 118 to retrieve data representing metadata 816 of the one or more frames using the one or more timestamps.

[0096] At block B1108, the method 1100 includes generating image data representing one or more overlays. For example, the video annotator 120 may use the image data 818 and the metadata 820 to generate overlay data 822, which may include image data representing one or more overlays for one or more frames of the video stream.

[0097] At block B1110 , method 1100 includes displaying image data representing one or more overlays. For example, presentation manager 138 executed or instantiated using client device 102 may cause display of one or more frames having one or more overlays using overlay data 822 .

[0098] Example computing device

[0099] Fig.12 1 is a block diagram of an example computing device 1200 suitable for implementing some embodiments of the present disclosure. The computing device 1200 may include an interconnect system 1202 that directly or indirectly couples the following devices: a memory 1204, one or more central processing units (CPUs) 1206, one or more graphics processing units (GPUs) 1208, a communication interface 1210, an input / output (I / O) port 1212, an input / output component 1214, a power supply 1216, one or more presentation components 1218 (e.g., a display), and one or more logic units 1220.

[0100] In at least one embodiment, computing device 1200 may include one or more virtual machines, and / or any of its components may include virtual components (e.g., virtual hardware components). For example, one or more GPUs 1208 may include one or more vGPUs, one or more CPUs 1206 may include one or more vCPUs, and / or one or more logic units 1220 may include one or more virtual logic units.

[0101] although Fig.12The various blocks of are shown as being connected via an interconnect system 1202 having wires, but this is not intended to be limiting and is merely for clarity. For example, in some embodiments, a presentation component 1218 such as a display device may be considered an I / O component 1214 (e.g., if the display is a touch screen). As another example, CPU 1206 and / or GPU 1208 may include memory (e.g., memory 1204 may represent a storage device in addition to the memory of GPU 1208, CPU 1206, and / or other components). In other words, Fig.12 The computing devices described herein are illustrative only. No distinction is made between categories such as "workstations," "servers," "laptops," "desktops," "tablets," "client devices," "mobile devices," "handheld devices," "game consoles," "electronic control units (ECUs)," "virtual reality systems," and / or other device or system types, as all are considered Fig.12 within the range of computing devices.

[0102] The interconnection system 1202 may represent one or more lines or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnection system 1202 may include one or more bus or link types, such as an industry standard architecture (ISA) bus, an extended industry standard architecture (EISA) bus, a video electronics standard association (VESA) bus, a peripheral component interconnect (PCI) bus, a peripheral component interconnect express (PCIe) bus, and / or another type of bus or line. In some embodiments, there is a direct connection between components. As an example, CPU 1206 may be directly connected to memory 1204. In addition, CPU 1206 may be directly connected to GPU 1208. In the case where there is a direct or point-to-point connection between components, the interconnection system 1202 may include a PCIe link to perform the connection. In these examples, the PCI bus does not need to be included in the computing device 1200.

[0103] Memory 1204 may include any of a variety of computer-readable media. Computer-readable media may be any available media that can be accessed by computing device 1200. Computer-readable media may include volatile and nonvolatile media and removable and non-removable media. By way of example and not limitation, computer-readable media may include computer storage media and communication media.

[0104] Computer storage media may include volatile and nonvolatile media and / or removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 1204 may store computer-readable instructions (e.g., representing programs and / or program elements, such as an operating system). Computer storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage devices, cassettes, magnetic tape, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by computing device 1200. As used herein, computer storage media does not include the signals themselves.

[0105] Computer storage media may contain computer readable instructions, data structures, program modules, and / or other data types in a modulated data signal such as a carrier wave or other transmission mechanism, and include any information delivery media. The term "modulated data signal" may refer to a signal that has one or more of its characteristics set or changed in such a way that information is encoded into the signal. By way of example and not limitation, computer storage media may include wired media such as a wired network or a direct wired connection, and wireless media such as sound, RF, infrared, and other wireless media. Any combination of the above should also be included within the scope of computer readable media.

[0106] The CPU 1206 may be configured to execute at least some computer-readable instructions to control one or more components of the computing device 1200 to perform one or more of the methods and / or processes described herein. Each of the CPUs 1206 may include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of processing a large number of software threads simultaneously. The CPU 1206 may include any type of processor, and may include different types of processors, depending on the type of computing device 1200 implemented (e.g., a processor with fewer cores for mobile devices and a processor with more cores for servers). For example, depending on the type of computing device 1200, the processor may be an Advanced RISC Machine (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). The computing device 1200 may also include one or more CPUs 1206 in addition to one or more microprocessors or supplementary coprocessors such as math coprocessors.

[0107] In addition to or in place of CPU 1206, GPU 1208 may be configured to execute at least some computer-readable instructions to control one or more components of computing device 1200 to perform one or more methods and / or processes described herein. One or more GPUs 1208 may be integrated GPUs (e.g., with one or more CPUs 1206) and / or one or more GPUs 1208 may be discrete GPUs. In an embodiment, one or more GPUs 1208 may be coprocessors of one or more CPUs 1206. Computing device 1200 may use GPU 1208 to render graphics (e.g., 3D graphics) or perform general purpose computing. For example, GPU 1208 may be used for general purpose computing (GPGPU) on GPU. GPU 1208 may include hundreds of thousands of cores capable of processing hundreds of thousands of software threads simultaneously. GPU 1208 may generate pixel data for outputting an image in response to a rendering command (e.g., a rendering command received from CPU 1206 via a host interface). GPU 1208 may include graphics memory, such as display memory, for storing pixel data or any other suitable data, such as GPGPU data. Display memory may be included as part of memory 1204. GPU 1208 may include two or more GPUs operating in parallel (e.g., via a link). The link may connect the GPUs directly (e.g., using NVLINK) or through a switch (e.g., using NVSwitch). When combined together, each GPU 1208 may generate pixel data or GPGPU data for different parts of the output or different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU may include its own memory or may share memory with other GPUs.

[0108] In addition to or in lieu of the CPU 1206 and / or the GPU 1208, the logic unit 1220 may be configured to execute at least some computer-readable instructions to control one or more components of the computing device 1200 to perform one or more methods and / or processes described herein. In embodiments, the CPU 1206, the GPU 1208, and / or the logic unit 1220 may perform any combination of methods, processes, and / or portions thereof, separately or jointly. The one or more logic units 1220 may be a part of and / or integrated with one or more of the CPU 1206 and / or the GPU 1208, and / or the one or more logic units 1220 may be a discrete component or otherwise external to the CPU 1206 and / or the GPU 1208. In embodiments, the one or more logic units 1220 may be a coprocessor of the one or more CPUs 1206 and / or the one or more GPUs 1208.

[0109] Examples of logic unit 1220 include one or more processing cores and / or components thereof, such as a tensor core (TC), a tensor processing unit (TPU), a pixel vision core (PVC), a vision processing unit (VPU), a graphics processing cluster (GPC), a texture processing cluster (TPC), a streaming multiprocessor (SM), a tree traversal unit (TTU), an artificial intelligence accelerator (AIA), a deep learning accelerator (DLA), an arithmetic logic unit (ALU), an application specific integrated circuit (ASIC), a floating point unit (FPU), an input / output (I / O) element, a peripheral component interconnect (PCI) or a peripheral component interconnect express (PCIe) element, etc.

[0110] The communication interface 1210 may include one or more receivers, transmitters, and / or transceivers that enable the computing device 1200 to communicate with other computing devices via an electronic communication network, including wired and / or wireless communications. The communication interface 1210 may include components and functionality that enable communication over any of a number of different networks, such as a wireless network (e.g., Wi-Fi, Z-wave, Bluetooth, Bluetooth LE, ZigBee, etc.), a wired network (e.g., via Ethernet or Infiniband communications), a low power wide area network (e.g., LoRaWAN, SigFox, etc.), and / or the Internet.

[0111] I / O ports 1212 can enable computing device 1200 to be logically coupled to other devices including I / O components 1214, presentation components 1218, and / or other components, some of which can be built into (e.g., integrated into) computing device 1200. Illustrative I / O components 1214 include microphones, mice, keyboards, joysticks, game pads, game controllers, satellite dishes, scanners, printers, wireless devices, and the like. I / O components 1214 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by a user. In some instances, the input can be sent to an appropriate network element for further processing. The NUI can implement any combination of voice recognition, stylus recognition, facial recognition, biometric recognition, gesture recognition on and adjacent to the screen, air gestures, head and eye tracking, and touch recognition associated with the display of computing device 1200 (as described in more detail below). Computing device 1200 may include depth cameras such as stereo camera systems, infrared camera systems, RGB camera systems, touch screen technology, and combinations of these for gesture detection and recognition. Additionally, computing device 1200 may include an accelerometer or gyroscope to enable motion detection (e.g., as part of an inertial measurement unit (IMU)). In some examples, the output of the accelerometer or gyroscope may be used by computing device 1200 to render immersive augmented reality or virtual reality.

[0112] The power supply 1216 may include a hardwired power supply, a battery power supply, or a combination thereof. The power supply 1216 may provide power to the computing device 1200 to enable the components of the computing device 1200 to operate.

[0113] The presentation component 1218 may include a display (e.g., a monitor, a touch screen, a television screen, a head-up display (HUD), other display types, or a combination thereof), a speaker, and / or other presentation components. The presentation component 1218 may receive data from other components (e.g., GPU 1208, CPU 1206, etc.) and output the data (e.g., as an image, video, sound, etc.).

[0114] Sample Data Center

[0115] Fig.13 An example data center 1300 is shown in which at least one embodiment may be used. In at least one embodiment, the data center 1300 includes a data center infrastructure layer 1310, a framework layer 1320, a software layer 1330, and an application layer 1340.

[0116] In at least one embodiment, Fig.13As shown, the data center infrastructure layer 1310 may include a resource coordinator 1312, grouped computing resources 1314, and node computing resources ("node CRs") 1316(1)-1316(N), where "N" represents any whole positive integer. In at least one embodiment, the node CRs 1316(1)-1316(N) may include, but are not limited to, any number of central processing units ("CPUs") or other processors (including accelerators, field programmable gate arrays (FPGAs)), graphics processors, etc.), memory devices (e.g., dynamic read-only memories), storage devices (e.g., solid-state or disk drives), network input / output ("NW I / O") devices, network switches, virtual machines ("VMs"), power modules, and cooling modules, etc. In at least one embodiment, one or more of the node CRs 1316(1)-1316(N) may be a server having one or more of the computing resources mentioned above.

[0117] In at least one embodiment, the grouped computing resources 1314 may include separate groups of node CRs housed in one or more racks (not shown) or many racks (also not shown) housed in data centers located in different geographic locations. The separate groups of node CRs within the grouped computing resources 1314 may include computing, network, memory or storage resources that can be configured or allocated to support groupings of one or more workloads. In at least one embodiment, several node CRs including CPUs or processors may be grouped in one or more racks to provide computing resources to support one or more workloads. In at least one embodiment, one or more racks may also include any number of power modules, cooling modules and network switches in any combination.

[0118] In at least one embodiment, resource coordinator 1322 may configure or otherwise control one or more nodes CR 1316(1)-1316(N) and / or grouped computing resources 1314. In at least one embodiment, resource coordinator 1322 may include a software design infrastructure ("SDI") management entity for data center 1300. In at least one embodiment, resource coordinator 1322 may include hardware, software, or some combination thereof.

[0119] In at least one embodiment, Fig.13As shown, the framework layer 1320 includes a job scheduler 1332, a configuration manager 1334, a resource manager 1336, and a distributed file system 1338. In at least one embodiment, the framework layer 1320 may include a framework for supporting software 1331 of the software layer 1330 and / or one or more applications 1342 of the application layer 1340. In at least one embodiment, the software 1331 or the application 1342 may include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. In at least one embodiment, the framework layer 1320 may be, but is not limited to, a type of free and open source software web application framework, such as Apache Spark. TM (hereinafter referred to as "Spark"), which can use the distributed file system 1338 for large-scale data processing (e.g., "big data"). In at least one embodiment, the job scheduler 1332 can include a Spark driver to facilitate the scheduling of workloads supported by various layers of the data center 1300. In at least one embodiment, the configuration manager 1334 can be capable of configuring different layers, such as the software layer 1330 and the framework layer 1320, including Spark and the distributed file system 1338 for supporting large-scale data processing. In at least one embodiment, the resource manager 1336 can be capable of managing clustered or grouped computing resources mapped to or allocated to support the distributed file system 1338 and the job scheduler 1332. In at least one embodiment, the clustered or grouped computing resources can include grouped computing resources 1314 at the data center infrastructure layer 1310. In at least one embodiment, the resource manager 1336 can coordinate with the resource coordinator 1312 to manage these mapped or allocated computing resources.

[0120] In at least one embodiment, software 1331 included in software layer 1330 may include software used by node CRs 1316(1)-1316(N), grouped computing resources 1314, and / or at least a portion of distributed file system 1338 of framework layer 1320. One or more types of software may include, but are not limited to, Internet web page search software, email virus scanning software, database software, and streaming video content software.

[0121] In at least one embodiment, the applications 1342 included in the application layer 1340 may include one or more types of applications used by the node CRs 1316(1)-1316(N), the grouped computing resources 1314, and / or at least a portion of the distributed file system 1338 of the framework layer 1320. The one or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), or other machine learning applications used in conjunction with one or more embodiments.

[0122] In at least one embodiment, any of configuration manager 1334, resource manager 1336, and resource coordinator 1312 can implement any number and type of self-modification actions based on any number and type of data acquired in any technically feasible manner. In at least one embodiment, the self-modification actions can save a data center operator of data center 1300 from making potentially poor configuration decisions and potentially avoiding underutilized and / or underperforming portions of the data center.

[0123] In at least one embodiment, data center 1300 may include tools, services, software, or other resources for training one or more machine learning models or predicting or inferring information using one or more machine learning models according to one or more embodiments described herein. For example, in at least one embodiment, a machine learning model may be trained by calculating weight parameters according to a neural network architecture using the software and computing resources described above with respect to data center 1300. In at least one embodiment, a trained machine learning model corresponding to one or more neural networks may be used to infer or predict information using the resources described above with respect to data center 1300 using weight parameters calculated by one or more training techniques described herein.

[0124] The present disclosure may be described in the general context of machine-usable instructions or computer codes executed by a computer or other machine such as a personal digital assistant or other handheld device, including computer executable instructions such as program modules. Typically, program modules including routines, programs, objects, components, data structures, etc. refer to codes that perform specific tasks or implement specific abstract data types. The present disclosure may be practiced in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, and the like. The present disclosure may also be practiced in a distributed computing environment, where tasks are performed by remote processing devices linked through a communication network.

[0125] As used herein, the description of "and / or" about two or more elements should be interpreted as referring to only one element or a combination of elements. For example, "element A, element B and / or element C" can include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or elements A, B and C. In addition, "at least one of element A or element B" can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Further, "at least one of element A and element B" can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0126] The subject matter of the present disclosure is described in detail herein to meet statutory requirements. However, the description itself is not intended to limit the scope of the present disclosure. On the contrary, the inventors have contemplated that the claimed subject matter may also be embodied in other ways to include steps different from the steps described herein in conjunction with other current or future technologies or combinations of similar steps. Moreover, although the terms "step" and / or "frame" may be used herein to imply different elements of the method employed, these terms should not be interpreted as implying any particular order among or between the various steps disclosed herein, unless the order of the steps is explicitly described.

Claims

1. A method for intelligent video analysis, comprising: In response to receiving one or more timestamps corresponding to one or more frames of a video stream and metadata, and at least one criterion of the metadata corresponding to a location of one or more identified objects depicted in the one or more frames: retrieving first image data representing the one or more frames from a video data store on a first device based at least on sending the one or more timestamps to the video data store, the first device identifying the first image data in the video data store using the one or more timestamps and a first index to the first image data generated at least in part by the first device and indexed by the one or more timestamps; retrieving metadata corresponding to the at least one criterion of the metadata for the one or more frames from the metadata storage based at least on sending the one or more timestamps to a metadata storage on a second device, the second device identifying the metadata in the metadata storage using the one or more timestamps and a second index to the metadata and the one or more timestamps generated at least in part by the second device; as well as Second image data is generated based on the metadata retrieved from the metadata store and the first image data retrieved from the video data store, the second image data representing one or more overlays corresponding to the locations of the one or more identified objects.

2. The method of claim 1 , wherein at least one edge device sends the one or more timestamps to the video data store and the metadata store to retrieve the first image data and the metadata, and wherein the at least one edge device generates the second image data representing the one or more overlays using the first image data and the metadata.

3. The method of claim 1 , wherein receiving the one or more timestamps comprises: The one or more timestamps are received in a request sent from a client device, and the retrieval of the metadata and the retrieval of the first image data are performed in response to the request. 4 . The method of claim 3 , wherein generating the second image data is performed using a third device different from the first device, the second device, and the client device.

5. The method of claim 1, wherein retrieving the metadata is based at least on selecting the second device from a plurality of devices storing different portions of the plurality of metadata for the one or more frames using the at least one criterion.

6. A method as claimed in claim 1, wherein the first device uses a time synchronization system to determine the one or more timestamps associated with the first image data to generate the first index, and the second device uses the time synchronization system to determine the one or more timestamps associated with the metadata to generate the second index.

7. The method of claim 1, wherein when the first device receives the one or more frames, the first device generates a first index, and when the second device receives the one or more frames and analyzes the one or more frames, the second device generates the second index.

8. The method of claim 1, wherein the first device generates first data representing the one or more timestamps and generates the first index using the first data, and the second device generates second data representing the one or more timestamps and generates the second index using the second data.

9. The method of claim 1, wherein the first device is a first edge device of a cloud computing system, and the second device is a second edge device of the cloud computing system, and the retrieval of the one or more timestamps is implemented through a core device of the cloud computing system.

10. A method for intelligent video analysis, comprising: receiving, using a cloud computing system, a request from a client device indicating one or more timestamps of one or more frames of a video stream and metadata corresponding to locations of one or more identified objects depicted in the one or more frames; in response to the request, retrieving, using the cloud computing system, the metadata for the one or more frames from a metadata store on a first device based at least on sending the one or more timestamps to the first device, the first device identifying the metadata in the metadata store using the one or more timestamps and a first index to the metadata generated at least in part by the first device and indexed by the one or more timestamps; as well as Using the cloud computing system, the metadata retrieved in response to the request and the one or more timestamps is transmitted to at least one remote device, the transmitting causing the at least one remote device to perform operations comprising: retrieving the first image data from a video data store on a second device using the one or more timestamps, the second device identifying the first image data in the video data store using the one or more timestamps and a second index to the first image data generated at least in part by the second device and indexed by the one or more timestamps, and Second image data is generated using the metadata retrieved from the metadata store and the first image data retrieved from the video data store, the second image data representing one or more overlays corresponding to the locations of the one or more identified objects.

11. The method of claim 10, wherein the receiving of the request is performed using at least one core device of the cloud computing system, wherein the at least one remote device comprises at least one edge device of the cloud computing system.

12. The method of claim 10, wherein sending the metadata retrieved in response to the request and the one or more timestamps to the at least one remote device is performed using at least one edge device of the cloud computing system, wherein the at least one remote device includes the client device.

13. The method of claim 10, wherein the first device is a core device of the cloud computing system, and the second device is an edge device of the cloud computing system.

14. The method of claim 10, wherein the at least one remote device is distinct from the client device and comprises a component of the cloud computing system, and the method further comprises: The second image data is transmitted, the transmitting causing the one or more frames with the one or more overlays to be displayed using the client device.

15. A system for intelligent video analysis, comprising: One or more processing devices configured to perform a method comprising: sending a request to a cloud computing system, the request indicating one or more timestamps of one or more frames of a video stream; receiving, in response to the request, first image data from a video data store on a first device of the cloud computing system, the first device identifying the first image data in the video data store using the one or more timestamps and a first index to the first image data generated at least in part by the first device and indexed by the one or more timestamps; retrieving metadata from a metadata store on a second device based at least on the one or more timestamps sent to the metadata store, the metadata corresponding to locations of one or more identified objects depicted in the one or more frames, the second device identifying the metadata in the metadata store using the one or more timestamps and a second index to the metadata and the one or more timestamps generated at least in part by the second device; and Second image data is generated using the first image data received from the video data store and the metadata retrieved from the metadata store, the second image data representing one or more overlays corresponding to the locations of the one or more identified objects.

16. The method of claim 15, wherein the method is performed at least in part using one or more virtual machines.

17. The method of claim 15, wherein the method is performed by a client device.

18. The method of claim 15, wherein the request includes at least one indicator of the metadata of the one or more frames.

19. The method of claim 15, wherein the request comprises a query received from a user interface of a client device, and the query comprises identifying one or more of the one or more timestamps.

20. The method of claim 15, wherein the metadata represents one or more of object location data, object classification data, or object segmentation data.

Images