Video content analysis

Abstract

Video content analysis is disclosed. Techniques for analyzing stored video based on requests are described. For example, a method of receiving a first application programming interface (API) request to analyze a stored video, the API request including a location of the stored video and at least one analysis action to perform on the stored video, accessing the location of the stored video to retrieve the stored video, segmenting the accessed video into chunks, processing each chunk with a chunk processor to perform the at least one analysis action, each chunk processor utilizing at least one machine learning model to perform the at least one analysis action, combining results of the processing of each chunk to generate a final result, storing the final result, and providing the final result to a requestor in response to a second API request are described.

Description

[0001] This application is a divisional application of the invention patent application entitled "Video Content Analysis", which has PCT international application number PCT / US2018 / 061496, international application date of November 16, 2018, and Chinese national phase application number 201880075463.0.

[0002] Cross-reference to related applications

[0003] This application claims the benefit of U.S. Nonprovisional Application No. 15 / 926,745, filed March 20, 2018, and U.S. Provisional Patent Application No. 62 / 590,208, filed November 22, 2017, both of which are incorporated herein by reference. Background Technology

[0004] Video is becoming increasingly prevalent in everyday use. For example, stores that are open use video to track shoppers, and camera feeds are used to detect license plates, etc. The amount of video is enormous and constantly changing. Attached Figure Description

[0005] Various embodiments according to this disclosure will be described with reference to the accompanying drawings, in which: Figure 1 An implementation scheme for a web service provider capable of performing video analytics is shown.

[0006] Figure 2 An implementation scheme of the method is shown.

[0007] Figure 3 An implementation scheme for a video analytics service used for archived videos is shown.

[0008] Figure 4 An implementation scheme for analyzing stored video is shown. In some implementations, at step 401, the request is placed into a job queue.

[0009] Figure 5 This illustrates various aspects of an implementation scheme for a video processing (analysis) architecture for video.

[0010] Figure 6 An implementation scheme of data flow between components according to some implementation schemes is shown.

[0011] Figure 7 An implementation scheme of data flow between components according to some implementation schemes is shown.

[0012] Figure 8 An implementation scheme for a video analytics service for streaming video processing is shown.

[0013] Figure 9An implementation scheme for the frame processor is shown.

[0014] Figure 10 A sequence diagram (method flow) for initiating face recognition for a video stream is shown according to some implementation schemes.

[0015] Figure 11 A sequence diagram (method flow) for stopping face recognition in a video stream is shown according to some implementation schemes.

[0016] Figure 12 A sequence diagram (method flow) of the streaming job status for obtaining a video stream is shown according to some implementation schemes.

[0017] Figure 13 An exemplary data center is shown according to some implementation schemes, which uses IP tunneling technology to implement an overlay network on the network substrate.

[0018] Figure 14 This is a block diagram of an exemplary provider network that provides storage virtualization services and hardware virtualization services to customers according to some implementation schemes.

[0019] Figure 15 This is a block diagram illustrating an exemplary computer system that can be used in some implementations.

[0020] Figure 16 A logical arrangement of a set of common components of an exemplary computing device that can be utilized according to various implementation schemes is shown.

[0021] Figure 17 Examples of environments for implementing various aspects are shown, according to various implementation schemes. Detailed Implementation

[0022] Various implementations of methods, apparatus, systems, and non-transitory computer-readable storage media for video analytics (archived and streaming) as a web service are described. In some implementations, application programming interfaces (APIs) for object, face, and / or activity detection in video are described in detail. Video analytics as a web service allows analysis to be performed on video captured from end-user / edge devices. For example, webcams typically do not have hardware for analyzing the video they capture. Video analysis can be performed by offloading the analysis processing to a web service provider. In some implementations, the analysis is performed in real-time or near real-time. Additionally, web service providers allow for the automatic configuration and flexible scaling of the infrastructure required to ingest and process streaming video data. Furthermore, video streams from the same entity can be analyzed together (e.g., tracking a person one camera at a time in a store).

[0023] Implementations of video analytics services capable of operating on large-scale, resilient, distributed, and highly available platforms are described, enabling developers to apply pre-trained ML models for purposes including, but not limited to, content moderation, object and activity classification, face detection and recognition, people tracking, and celebrity identification. In some implementations, web service providers extend to accommodate additional processing elements such as optical character recognition (OCR), automatic speech recognition (ASR), and other capabilities.

[0024] In some implementations, the video analytics service uses a microservices-style approach to market video recognition capabilities isolated within the video segmenter, and enables aggregator workers to coarsely mimic a distributed aggregation approach to establish effective service level agreements (SLAs) around latency and throughput. In some implementations, the video analytics service supports both batch and streaming media inputs and reuses aspects related to keyframe selection, containerization of video processing elements, inference service pipelines, and supplementary aspects such as authentication and authorization, billing, metrics, and logging. For batch-oriented jobs, in some implementations, the video is distributed into configurable-sized chunks that can be processed independently. This enables parallel processing within the video, thereby facilitating better end-to-end latency. In subsequent steps, feature-specific aggregation of the individual chunks is achieved through parametric smoothing in both space and time. The video analytics service can tolerate failures (e.g., underlying hardware failures or service outages) and, in some implementations, can restart batch or streaming job processing from a recently known good checkpoint state. In some implementations, the video analytics service has an integrated notification component to send messages about how the tracking job is being processed, including scheduling events, any partial failures, lag conditions, etc.

[0025] Figure 1 An implementation scheme of a web service provider capable of performing video analytics is illustrated. An end-user / edge device 111 communicates with the web service provider 101 to initiate video analytics and / or retrieve the results of the video analytics. In some implementations, the communication takes the form of application programming interface (API) calls.

[0026] Web service provider 101 includes several different components. In some embodiments, each of these components is software stored in memory and executed by a processor. Users can select specific hardware for most components. For example, video analytics service 105 can be performed better on dedicated hardware or using an accelerator instead of a general-purpose central processing unit. In other embodiments, one or more of the components are determined by the web service provider based on end-user requirements such as latency, cost, etc.

[0027] Front-end 103 receives communications from end-user / edge device 111. In some embodiments, front-end 103 forwards the communications to video analytics service 105 to initiate video analytics and / or obtain the results of the video analytics. In other embodiments, front-end 103 converts the communications into a format understood by video analytics service 105 so that video analytics service 105 can initiate video analytics and / or obtain the results of the video analytics.

[0028] Video analytics service 105 analyzes videos. In some implementations, video analytics service 105 performs at least one or more of the following functions as instructed by end-user / edge device 111: personnel tracking, face detection, tag detection, celebrity recognition, content moderation, and / or search.

[0029] The video can be streaming (dynamic) video from streaming video source 109, or stored (static) video from streaming video source 109 or non-streaming video source 113 in video storage device 107. In some embodiments, the video stored in video storage device 107 is encrypted and protected against unauthorized access. Additionally, in some embodiments, the video is externally stored in video storage device 119.

[0030] In some implementations, streaming video source 109 provides streaming video directly to video analytics service 105. In other implementations, streaming video source 109 provides streaming video to video capture service 115, which makes the streaming video accessible to video analytics service 105. In some cases, video capture service 115 stores streaming video in video storage device 107 (making the stream static). In some implementations, video capture service 115 securely connects to streaming video source 109 and encrypts and indexes the video it receives. As described above, video storage device 107 can also receive video from non-streaming video sources 113. For example, uploading H.264 files to video storage device.

[0031] Any analysis results performed by the video analytics service 105 are stored as analysis result 121. In some implementations, analysis result 121 can be accessed via front-end 103. In some implementations, analysis result 121 can be accessed more directly.

[0032] In some implementations, the video analytics service 105 provides a completion notification to the notification service 117, which then notifies the requesting user / device 111 that the result is available.

[0033] Figure 2An implementation of the method is illustrated. In some implementations, the method is performed by a video analytics service. In some implementations, the requests discussed below are received at the front end of a web service provider. At 201, one or more API requests to begin analyzing video content are received. Examples of API requests include, but are not limited to: begin content moderation, begin face detection, begin tag detection, begin people tracking, begin celebrity identification, create a stream processor, and begin a stream processor. It should be noted that in some implementations, multiple API requests can be submitted at once.

[0034] The "Initiate Content Moderation Request" initiates the detection of explicit or implicit adult content in a video. The request provides the video's location (such as storage location or streaming). In some implementations, the request includes a location or manner for reporting notification using a job tag from the request upon completion of the detection. A job tag is provided to identify the job. The video analytics service provides a job identifier used to locate the results of the content detection.

[0035] A "Start Face Detection" request initiates the detection of faces in a video. The request provides the video's location (such as storage location or streaming). In some implementations, the request includes a job tag for reporting notifications when detection is complete. A job identifier, provided by a video analytics service, is used to locate the face detection results. If faces have been previously labeled, the results of people tracking may include labels for both the face and name. Furthermore, in some implementations, the request includes face attributes to be returned (such as bounding boxes, confidence scores, poses, quality, and landmarks).

[0036] A people tracking request initiates the detection of people in a video. The request provides the location of the video (such as a storage location or stream). In some implementations, the request includes a location or manner for reporting notifications using the requested job tags when tag detection is complete. If people have been previously tagged, the results of people tracking may include tags for the people and their names. The video analytics service provides a job identifier, which is used to locate the results of people tracking.

[0037] The "Start Tag Tracking" request initiates the detection of tags in a video. Tags are instances of real-world entities and include objects such as flowers, trees, and tables; events such as weddings, graduations, and birthday parties; concepts such as landscapes, evenings, and nature; and activities such as people getting out of a car or skiing. The request provides the location of the video (such as a storage location or stream). In some implementations, the request includes the location or manner in which a notification is reported using the requested job tag upon completion of tag detection. The video analytics service provides a job identifier, which is used to locate the results of tag tracking.

[0038] A request to initiate celebrity identification begins the detection of celebrities in a video. The request provides the location of the video (such as a storage location or stream). In some implementations, the request includes a location or manner for reporting notifications using the requested job tags when tag detection is complete. The results of celebrity tracking may include tags of people in the video. The video analytics service can use the tags to determine who the celebrity is (e.g., based on the request). The video analytics service provides a job identifier, and this job identifier is used to look up the results of celebrity identification.

[0039] In some implementations, a stream processor is created and then started before the above requests can be executed.

[0040] The video content is processed based on the API request and the results stored at 203. Exemplary details of how this processing is performed will be explained later.

[0041] At point 205, the requester is notified once processing is complete. For example, the notification service alerts the requester that a specific task has been completed.

[0042] At a later point in time, at point 207, at least one API request is received to retrieve the results of the video content analysis. Examples of API requests include, but are not limited to: obtaining content moderation, obtaining face detection, obtaining tag detection, obtaining people tracking, obtaining celebrities, etc. It should be noted that in some implementations, multiple API requests can be submitted at once. Each "obtain" request includes a job identifier from the corresponding "start" request.

[0043] At position 209, retrieve the results based on the job identifier and provide the results to the requester.

[0044] Figure 3 An implementation scheme for a video analytics service used for archived videos is shown. The video analytics service front-end 301 receives a request to analyze the video stream. As mentioned above, this request is typically received via an API call.

[0045] The video analytics service frontend 301 places the request as a job into the job queue 303 for later processing. In some implementations, the video analytics service frontend 301 returns the job status to the API consumer. In some implementations, the frontend 301 verifies credentials, obtains the object's metadata to verify its size, and generates a temporary link for inputting content into the video segmenter 307.

[0046] The job manager / poller 305 polls the job queue 303 to obtain jobs and provides them to the video segmenter 307 to begin analysis. In some implementations, the job poller 305 is a library that uses an underlying data repository to keep track of user-submitted jobs and ensure fair sharing of service processing bandwidth. In some implementations, the number of video minutes to be analyzed per second is a measure of user bandwidth. In some implementations, the job manager 305 is responsible for one or more of the following: tracking the status of individual jobs; the boundary number of user-submitted jobs in a pending state; job prioritization; aging; and supporting the use of face-to-face visibility APIs, such as obtaining job status and obtaining all user-submitted jobs within a time frame.

[0047] Video segmenter 307 accesses data storage devices 107 / 119 (e.g., a database) that store the video. In some embodiments, video segmenter 307 segments the video into smaller chunks that can be processed independently to optimize for throughput and latency. In some embodiments, a secondary role is to extract metadata about the video (e.g., duration, bitrate, frames per second, etc.). For overlapping video segments, certain processing elements (such as activity detection) may require overlapping video segments. Creating a video for each segment and uploading it instead of storing start and end byte references of the segments in the video, and using byte-range-based access, can optimize the upload. Video segmenter 307 also determines the size of each segment.

[0048] Video segmenter 307 feeds one or more blocks to block processor 309. In some embodiments, block processor 309 performs all aspects of video content analysis, including object and face detection on frames extracted from the blocks. In some embodiments, block processor 309 is containerized and invokes processing services to obtain tag detection, face detection, face feature extraction, and activity classification. In some embodiments, intermediate results are retained along with frame information from the reduce phase.

[0049] In some implementations, each processing unit in block processor 309 defines contracts for the following: inputs, outputs, errors and exceptions, runtime footprint, and required resources. Block processor 309 can also perform orchestration of unit workers (e.g., frame-based analysis, activity detection, tracking, etc.) to pass the derivation of one step to another unit.

[0050] In some implementations, the reducer / aggregator 311 collects the results from the block processor 309 (e.g., from frame-based detection), applies the combined information, and stores the results in the analysis results 121.

[0051] In some implementations, the video segmenter 307, each block processor 309, and reducer 311 are containers and include models for label detection, face detection, face feature extraction, and activity classification as needed. In other implementations, the video segmenter 307, each block processor 309, and reducer 311 are virtual machines and invoke label detection, face detection, face feature extraction, and activity classification models as needed.

[0052] Figure 4 An implementation scheme for analyzing stored video is illustrated. In some implementations, at step 401, a request is placed in a job queue. For example, a request to start face recognition is placed in the job queue. In some implementations, a job ID is provided to the requester for retrieving the analysis results of the stored video.

[0053] At error 403, due to resource inactivity, the job queue is polled to obtain a job. Using the "Start Face Recognition Request" job as a subsequent example, this job is retrieved.

[0054] Access video content associated with a retrieved job at checkpoint 405. For example, request video files from a database that run face recognition.

[0055] At point 407, the accessed video is segmented into blocks. In some implementations, the block size is determined by the front end of the video analytics service. A block is one or more frames.

[0056] At position 409, each block is processed by a block processor. The results of each block processing are combined at position 411 to form the final result, which is then stored for retrieval. In some implementations, a notification is also sent to the requester.

[0057] At 413, in response to the "Get" API request, the stored result is provided to the requester.

[0058] Figure 5 Aspects of an implementation of a video processing (analysis) architecture for video are illustrated. In this example, the video analytics service front end is not shown. In some implementations, as described above, job queues and pollers are also utilized. Video segmenter 501 accesses a data storage device 515 (e.g., a database) that has video (such as storage device 107) and / or a block table. Segmenter 501 updates the block table using the number of blocks used for a job. In some implementations, the block table has a job ID as a hash key and uses a bitmap to track processed blocks.

[0059] In some implementations, segmenter 501 and / or work queue 503 are responsible for dispatching segments to block processors, keeping track of processed blocks, and notifying reducer 509 once all segments for the job are completed.

[0060] Block processor 505 operates as described above. In some implementations, after processing a block, block processor 505 publishes it to stream processor 507 using the job ID as the partition key. The partition key is used to group data by shards within the stream. Each consumer reads from a specific shard, meaning that the same shard processor will process all block completion events for a given job. This helps ensure consistency when updating job status. Furthermore, in some implementations, reads are batched.

[0061] In some implementations, reducer / aggregator 509 collects the results from block processor 505 and applies the combined information. Reducer / aggregator 509 marks the blocks that have been processed in database 515. After the job is completed, a job completion event is sent to reducer work queue 511.

[0062] Figure 6 An embodiment of data flow between components according to some implementation schemes is shown. Storage device 601 stores video content. Segmenter 603 accesses the video in storage device 601 and provides blocks to block processors (such as those described above).

[0063] The block processor's decoder 605 receives each block and decodes each block into a block frame. In some embodiments, the block frames are provided to the people detector and tracker 607. For example, when people detection and tracking are requested, the people detector and tracker 607 analyzes the blocks to find and track people across frames. In some embodiments, the people detector and tracker 607 analyzes the blocks at a per-frame resolution and outputs tracking segments. In some embodiments, this data, along with sampled frames (such as 10% of the original frame), is fed to a face detector and extractor 609, which proposes face matching for people (including celebrities).

[0064] In some implementations, aggregator 611 processes each block output to produce results over the entire video upon request, the results including person bounding boxes, face bounding boxes (in sampled frames), and face matching for each frame. In some implementations, aggregator 611 performs matching between faces and people (including celebrities).

[0065] In some implementations, the block processor's people detector and tracker 607, face detector and extractor 609, and aggregator 611 are machine learning models executed in a virtual machine. In some implementations, the block processor's people detector and tracker 607, face detector and extractor 609, and aggregator 611 are machine learning models as part of a container.

[0066] Figure 7An embodiment of data flow between components according to some implementation schemes is shown. Storage device 701 stores video content. Segmenter 703 accesses the video in storage device 701 and provides blocks to block processor.

[0067] The block processor's decoder 705 receives each block and decodes each block into a block frame. In some embodiments, the block frames are provided to the tag detector 707. In some embodiments, the blocks are processed and tags are output at per-frame resolution.

[0068] In some implementations, aggregator 709 processes each block output to produce results over the entire video, including the tag matching for each sampled frame.

[0069] In some implementations, the block processor's label detector 707 is a machine learning model executed in a virtual machine. In some implementations, the block processor's label detector 707 is a machine learning model that is part of a container.

[0070] In some implementations, workflow orchestration is decoupled from activity logic. In these implementations, the video segmenter, block processor, and reducer detailed above are activities. Each activity implements a defined interface and is independent of on-chain / off-chain activities. Furthermore, these activities are idempotent and do not contain knowledge of workflow state. Activities are concerned with how their contracts are implemented and are independent of workflow orchestration.

[0071] In some implementations, each activity is associated with a work queue. The activity is wrapped around a work queue client, which polls the work queue and invokes the activity when it receives a task.

[0072] In some implementations, job orchestration is distributed, and each activity worker depends on an orchestrator package. The orchestrator picks up task events, verifies state, performs transformations, and deserializes before invoking a task. Each activity exposes an invocation contract that specifies the input work queue, output work queue, timeout, and retry policy. After an activity is invoked, the orchestrator looks up the job definition to find the next task to execute and updates the workflow state.

[0073] Figure 8 An implementation scheme for a video analytics service used for streaming video processing is shown. Front-end service 801 receives API requests (such as those described above).

[0074] Frontend service 801 records entries in database 803. Database 803 is used to track the stream processing status, such as which worker has acquired the lock to process the stream, and what the last successfully processed segment number was. In some implementations, workers will asynchronously attempt to acquire the lock to pick up and process the stream. In some implementations, workers will send periodic heartbeats to indicate that they are actively processing a specific stream. This will help us manage the process in a fault-tolerant manner. If one worker fails, another worker will acquire the lock and begin processing the stream.

[0075] In some implementations, the front-end service 801 places streaming requests into a queue for streaming processing 805. The queue 805 helps workers obtain a task rather than scanning the entire database to pick one.

[0076] Stream processor worker 811 accesses database 803 and acquires a lock for the stream request, picks up jobs from queue 805, consumes video stream 807 (in some embodiments, this is delivered via video capture service 115), and processes it by invoking services 813 and 815, and in some embodiments, checks the results as checkpoints. Services 813 and 815 are face detection and collection services, respectively. In some embodiments, worker 811 and services 813 and 815 are virtual machines executing machine learning models. In other embodiments, worker 811 and services 813 and 815 are containers executing machine learning models.

[0077] Output stream 809 is sent to the endpoint that provides the results of video processing (analysis result 121).

[0078] In most implementations, this architecture is based on an asynchronous worker system, where workers process real-time streams, apply image processing capabilities to them, and return stream results. In some implementations, the stream processor periodically checks the processing results on the database 803 as checkpoints. When a worker heartbeat fails (deployment / failure, etc.), another worker picks up a request from the queue, acquires the database lock, and begins processing the stream from the previous checkpoint.

[0079] An exemplary schema for database 803 is:

[0080] The stream processor worker 811 is responsible for processing the input video stream and generating output metadata. In some implementations, the stream processor has four components: 1) a stream consumer; 2) a frame extractor; 3) a frame processor; and 4) a metadata output producer. In some implementations, each of these components consists of one or more machine learning models. In some implementations, each of these components is a container. In some implementations, each of these components is a virtual machine.

[0081] In some implementations, once a stream processor worker 811 is assigned to an input stream, the worker initializes a stream consumer to read from the stream. In some implementations, the stream consumer may or may not have any checkpoint data for the stream. The stream processor consumer invokes a frame extraction server to stream the video data. In some implementations, the stream consumer is responsible for handling throttling to keep up with frame extraction / processing and output generation.

[0082] Frame extraction components can be constructed in several ways. In some implementations, a library is built for extracting frames and making calls to start / stop extraction. In some implementations, a service is used for extracting frames and making service calls to start / stop extraction. In some implementations, an executable is used for extracting frames.

[0083] In some implementations, the frame extraction unit operates as a separate server running on the same host as each worker. The frame extraction unit is responsible for decoding the video input and extracting frames at a specific number of frames per second (FPS). In some implementations, during a call to the stream, it returns two ports (p1, p2) as acknowledgments, where the first port is used for streaming within the video, and the second port is used for streaming the frames outside the video. Subsequently, the consumer opens the output socket to p1, and the frame extractor opens the input socket to p1.

[0084] In some implementations, the frame processor is responsible for processing image frames and assigning person identities at 1 FPS, grouped by fragment number. Frame extraction is performed at a frame rate higher than 1 FPS (e.g., @10 FPS). In some implementations, these frames are used for face tracking to ensure quality and optimize the number of search face invocations collected from the frame processor module to maintain a list of current faces in the scene and to invoke the search face service when a new face enters the field of view.

[0085] Figure 9 An implementation scheme of the frame processor is shown. Input from the frame extractor is received at 901. For example, the image frame rate is 10 FPS. The video sampler 903 selects the best frame from the input and outputs it to the face detection module 905. Typically, the frame rate input to the face detection module 905 is less than the input rate.

[0086] The face detection module 905 sends the list of detected faces to the face verification module 907. In some implementations, the face verification module 907 calls the face search module 909 to search for any unknown faces, and those results are returned to the face verification module 907.

[0087] In some implementations, face reviewer 907 sends face indications in frames at a low FPS to output aggregator 911, which generates and outputs faces in frames grouped by segment number 913.

[0088] In some implementations, image frames are sent to a face tracker module 915, which returns a list of tracked faces to a face reviewer 907. In some implementations, the face reviewer 907 outputs this list 917.

[0089] In some implementations, the metadata output component is responsible for writing the output of each fragment to the output stream and checking the fragment number as a checkpoint. In some implementations, this component iterates through each item in the queue sent from the frame processor, obtains the results (grouped by fragment number), and writes them out. In some implementations, if a request times out, the failure is abstracted into a meaningful message and produced as output. When a generator call fails or times out (even after a retry policy), the output stream is considered an error, and stream processing terminates, and the status is updated accordingly in the database 803. If output is successfully produced for a fragment, the fragment number and any necessary data are checked as a checkpoint.

[0090] Figure 10 The diagram illustrates a sequence (method flow) for initiating face recognition for a video stream according to some implementation schemes. Frontend service 801 receives an API request to begin face recognition.

[0091] The front-end service adds the job to the Stream Processing Queue (SPQ) 805, creates a record entry in the database 803, and returns the job ID to the requester.

[0092] At some point, stream processor worker 811 polls SPQ 805 to obtain a job, and once a job is obtained, it retrieves a lease for the stream from database 803.

[0093] Then, the stream processor worker 811 accesses the input stream 807 and begins searching for faces by invoking the face detection service 813. During processing, in some implementations, a heartbeat check is performed on the database 803 and the SPQ 805.

[0094] For each “face” detected by the stream processor worker 811, the face detection service 813 determines whether a face exists and provides the recognition response back to the stream processor worker 811.

[0095] The “face” is then matched against a set of faces by a collection service 815, which provides the face with “who this is”. When identifying each person, the detection against the database 803 is checked by a stream processor worker 811 as a checkpoint.

[0096] Finally, once the video has been fully analyzed, the stream processor worker 811 will publish a response to the output stream 809.

[0097] Figure 11 The diagram illustrates a sequence (method flow) for stopping face recognition in a video stream according to some implementation schemes. Frontend service 801 receives an API request to stop face recognition.

[0098] The front-end service informs the database that face recognition is stopping (803) and returns a response to the requester that the service has stopped.

[0099] Stream worker 811 terminates its lease with database 803 and publishes any running results to output stream 809. Then, stream worker 811 removes the request from SPQ.

[0100] Figure 12 The diagram illustrates a sequence diagram (method flow) for obtaining the streaming job status of a video stream according to some implementation schemes. Frontend service 801 receives an API request for the streaming job status.

[0101] Frontend service 801 accesses database 803 using the job ID provided by the API request and returns the status to the requester. Similarly, a list of streaming jobs can be provided to the user.

[0102] Figure 13 An exemplary data center according to some embodiments is illustrated, which implements an overlay network on a network substrate using IP tunneling technology. Provider data center 1300 may include a network substrate including networking nodes 1312 such as routers, switches, and Network Address Translation (NAT), which may be implemented as software, hardware, or a combination thereof. Some embodiments may employ Internet Protocol (IP) tunneling technology to provide the overlay network, through which encapsulated packets can tunnel through the network substrate 1310. IP tunneling technology can provide overlay network functionality for (e.g., ...) Figure 13A mapping and encapsulation system for overlay networks is created on the local network in data center 1300, and separate namespaces can be provided for the overlay layer (public IP address) and the network base layer 1310 (local IP address). Packets in the overlay layer can be examined against a mapping catalog (e.g., provided by mapping service 1330) to determine what their tunnel base target (local IP address) should be. IP tunneling technology provides a virtual network topology (overlay network); the interface presented to the client (e.g., service API) is attached to the overlay network, so that when the client provides the IP address to which it wants to send packets, the IP address operates in the virtual space by communicating with the mapping service (e.g., mapping service 1330) that knows where the IP overlay address is.

[0103] In some implementations, IP tunneling can map an IP overlay address (public IP address) to a base IP address (local IP address), encapsulate packets within a tunnel between two namespaces, and deliver the packets via the tunnel to the correct endpoint, where the encapsulation is stripped from the packets. Figure 13 The diagram illustrates an exemplary overlay network tunnel 1334A from virtual machines (VMs) 1324A (VMs 1324A1-1324A4, via VMM 1322A) on host 1320A to a device on an intermediate network 1340, and an exemplary overlay network tunnel 1334B between VMs 1324A (VMs 1324A1-1324A4, via VMM 1322A) on host 1320A and VMs 1324B (VMs 1324B1-1324B4, via VMM 1322B) on host 1320B. In some embodiments, packets may be encapsulated in an overlay network packet format before transmission, and the overlay network packet may be stripped after reception. In other embodiments, instead of encapsulating packets in an overlay network packet, the overlay network address (public IP address) may be embedded in the base address (local IP address) of the packet before transmission and stripped from the packet address upon reception. As an example, a 32-bit IPv4 (Internet Protocol version 4) address can be used as a public IP address to implement an overlay network, and the IPv4 address can be embedded as part of a 128-bit IPv6 (Internet Protocol version 6) address used as a local IP address on the base network.

[0104] refer to Figure 13 At least some of the networks in which the implementation scheme can be carried out may include hardware virtualization technology, which enables multiple operating systems to operate on a host computer (e.g., Figure 13The virtual machine (VM) 1324 runs simultaneously on hosts 1320A and 1320B, i.e., on host 1320. VM 1324 may execute, for example, in a slot on host 1320 that is leased or rented to a network provider. A virtual machine manager or virtual machine monitor (VMM) 1322 on host 1320 provides a virtual platform to VM 1324 on the host and monitors the execution of VM 1324. One or more local IP addresses may be provided for each VM 1324; VMM 1322 on host 1320 may know the local IP addresses of VM 1324 on host. Mapping service 1330 may know (e.g., via stored mapping information 1332) the network IP prefix and IP address of the router or other device serving the IP address on the local network. This includes the IP address of VMM 1322 serving multiple VMs 1324. Mapping service 1330 may be centralized, for example, on a server system, or alternatively, may be distributed among two or more server systems or other devices on a network. The network can use mapping service technology and IP tunneling technology, for example, to route data packets between VMs 1324 on different hosts 1320 within the data center 1300 network; it should be noted that the Interior Gateway Protocol (IGP) can be used to exchange routing information within such a local network.

[0105] Additionally, networks such as provider data center 1300 networks (sometimes called Autonomous Systems (AS)) can use mapping services, IP tunneling, and routing services to route packets from VM 1324 to Internet destinations, and from Internet sources to VM 1324. It should be noted that External Gateway Protocol (EGP) or Border Gateway Protocol (BGP) is typically used for Internet routing between sources and destinations on the Internet. Figure 13 An exemplary provider data center 1300 according to some embodiments is illustrated, which implements a network that provides resource virtualization technology and provides full Internet access via an edge router 1314 connected to an Internet transit provider. The provider data center 1300 may, for example, provide customers with the ability to implement virtual computing systems (VMs 1324) and virtualized data repositories 1316 on storage resources 1318A-1318N via hardware virtualization services and storage virtualization services.

[0106] The data center 1300 network can implement IP tunneling, mapping service, and routing service technologies to route traffic to and from virtualized resources. For example, it can route packets from VM 1324 on host 1320 in data center 1300 to an Internet destination, and from an Internet source to VM 1324. The Internet source and destination may include, for example, computing system 1370 connected to an intermediate network 1340 and computing system 1352 connected to a local network 1350, which is connected to the intermediate network 1340 (e.g., via an edge router 1314 that connects network 1350 to an Internet transit provider). The provider's data center 1300 network can also route packets between resources within data center 1300, for example, from VM 1324 on host 1320 in data center 1300 to other VMs 1324 on the same host or other host 1320 in data center 1300.

[0107] The service provider offering data center 1300 can also provide an additional data center 1360, which includes hardware virtualization technology similar to data center 1300 and can also be connected to intermediate network 1340. Packets can be forwarded from data center 1300 to other data centers 1360, for example, from VM1324 on host 1320 in data center 1300 to another VM on another host in another similar data center 1360, and vice versa.

[0108] Although hardware virtualization technology has been described above, which enables multiple operating systems to run simultaneously on a host as virtual machines (VMs), where VMs can be instantiated on slots on a host machine leased or rented to a network provider, hardware virtualization technology can also be used to provide other computing resources (e.g., storage resources 1318A-1318N) as virtualization resources to network provider customers in a similar manner.

[0109] Figure 14 This is a block diagram of an exemplary provider network that provides storage virtualization and hardware virtualization services to customers according to some implementation schemes. Hardware virtualization service 1420 provides multiple computing resources 1424 (e.g., VMs) to customers. For example, computing resources 1424 may be leased or rented to customers of provider network 1400 (e.g., customers implementing customer network 1450). One or more local IP addresses may be provided for each computing resource 1424. Provider network 1400 may be configured to route packets from the local IP addresses of computing resources 1424 to public internet destinations, and from public internet sources to the local IP addresses of computing resources 1424.

[0110] Provider network 1400 can provide, for example, the ability of a client network 1450 coupled to an intermediate network 1440 via a local network 1456, and a hardware virtualization service 1420 coupled to the intermediate network 1440 and provider network 1400 to implement a virtual computing system 1492. In some embodiments, hardware virtualization service 1420 may provide one or more APIs 1402, such as network service interfaces, via which client network 1450 can access functionality provided by hardware virtualization service 1420 via, for example, a console 1494 (e.g., a web-based application, a standalone application, a mobile application, etc.). In some embodiments, each virtual computing system 1492 at provider network 1400 and client network 1450 may correspond to computing resources 1424 leased, rented, or otherwise provided to client network 1450.

[0111] Based on an instance of virtual computing system 1492 and / or another client device 1490 (e.g., via console 1494), a client can access the functionality of storage virtualization service 1410, for example, via one or more APIs 1402, to access and store data from storage resources 1418A-1418N of virtual data repository 1416 provided by provider network 1400. In some embodiments, a virtualized data storage gateway (not shown) may be provided at client network 1450, which may locally cache at least some data, such as frequently accessed or critical data, and may communicate with virtualized data storage service 1410 via one or more communication channels to upload new or modified data from the local cache, thereby maintaining the primary data repository (virtual data repository 1416). In some embodiments, a user may install and access the capacity of virtual data repository 1416 via virtual computing system 1492 and / or on another client device 1490, which appears to the user as a local virtualized storage device 1498.

[0112] Although not in Figure 14 As shown, however, virtualization services can also be accessed from resource instances within provider network 1400 via API 1402. For example, a customer, equipment service provider, or other entity can access virtualization services from within a corresponding virtual network on provider network 1400 via API 1402 to request the allocation of one or more resource instances within that virtual network or another virtual network.

[0113] Explanatory System

[0114] In some implementations, a system implementing some or all of the techniques for model adaptation as described herein may include a general-purpose computer system that includes or is configured to access one or more computer-accessible media, such as... Figure 15 The computer system 1500 is shown. In the illustrated embodiment, the computer system 1500 includes one or more processors 1510, which are coupled to system memory 1520 via an input / output (I / O) interface 1530. The computer system 1500 also includes a network interface 1540 coupled to the I / O interface 1530. Although Figure 15 Computer system 1500 is shown as a single computing device, but in various embodiments, computer system 1500 may include a single computing device or any number of computing devices configured to work together as a single computer system 1500.

[0115] In various embodiments, computer system 1500 may be a single-processor system including one processor 1510, or a multiprocessor system including several processors 1510 (e.g., two, four, eight, or another suitable number). Processor 1510 may be any suitable processor capable of executing instructions. For example, in various embodiments, processor 1510 may be a general-purpose or embedded processor implementing any of a variety of instruction set architectures (ISAs), such as x86, ARM, PowerPC, SPARC, or MIPS ISA or any other suitable ISA. In a multiprocessor system, each of processors 1510 may typically, but not necessarily, implement the same ISA.

[0116] System memory 1520 may store instructions and data accessible by processor 1510. In various embodiments, system memory 1520 may be implemented using any suitable memory technology, such as random access memory (RAM), static RAM (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash memory, or any other type of memory. In the illustrated embodiment, program instructions and data implementing one or more desired functions, such as those methods, techniques, and data described above for resizing virtual networks in a provider network environment, are shown stored in system memory 1520 as code 1525 and data 1526.

[0117] In one embodiment, I / O interface 1530 may be configured to coordinate I / O traffic between processor 1510, system memory 1520, and any peripheral devices (including network interface 1540 or other peripheral interfaces) within the device. In some embodiments, I / O interface 1530 may perform any necessary protocols, timing, or other data transformations to convert data signals from one component (e.g., system memory 1520) into a format suitable for use by another component (e.g., processor 1510). In some embodiments, for example, I / O interface 1530 may include support for devices attached via various types of peripheral buses, such as variants of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard. In some embodiments, for example, the functionality of I / O interface 1530 may be divided into two or more separate components, such as a northbridge and a southbridge. Furthermore, in some embodiments, some or all of the functionality of I / O interface 1530 (such as the interface to system memory 1520) may be directly incorporated into processor 1510.

[0118] For example, network interface 1540 can be configured to allow computer system 1500 to connect with other devices 1560 (such as...) attached to one or more networks 1550. Figure 1 The network interface 1540 can exchange data with other computer systems or devices shown. In various embodiments, for example, the network interface 1540 can support communication via any suitable wired or wireless general-purpose data network, such as an Ethernet network type. Furthermore, the network interface 1540 can support communication via telecommunications / telephone networks (such as analog voice networks or digital fiber optic communication networks), via storage area networks (SANs) (such as Fibre Channel SANs), or via I / O of any other suitable type of network and / or protocol.

[0119] In some implementations, computer system 1500 includes one or more offload cards 1570 connected using I / O interface 1530 (e.g., a bus implementing a Peripheral Component Interconnect Version (Express (PCI-E) standard)) or another interconnect (such as a Fast Path Interconnect (QPI) or Hyper Path Interconnect (UPI) bus). These offload cards may include one or more processors 1575 and possibly one or more network interfaces 1540. For example, in some implementations, computer system 1500 may act as a host electronic device hosting computing instances (e.g., operating as part of a hardware virtualization service), and one or more offload cards 1570 may act as a virtualization manager capable of managing the computing instances running on the host electronic device. As an example, in some implementations, offload card 1570 may perform computing instance management operations such as pausing and / or unpausing computing instances, starting and / or terminating computing instances, performing memory transfer / copy operations, etc. In some implementations, these management operations may be performed by the offload card 1570 in collaboration with a manager (e.g., upon request from a manager) executed by other processors 1510A-1510N of the computer system 1500. However, in some implementations, the virtualization manager implemented by the offload card 1570 may accommodate requests from other entities (e.g., from the computing instance itself) and may not coordinate (or service) with any single manager.

[0120] In some embodiments, system memory 1520 may be one embodiment of a computer-accessible medium configured to store program instructions and data as described above. However, in other embodiments, program instructions and / or data may be received, transmitted, or stored on different types of computer-accessible media. Generally, computer-accessible media may include non-transitory storage media or memory media such as magnetic or optical media, for example, a disk or DVD / CD coupled to computer system 1500 via I / O interface 1530. Non-transitory computer-accessible storage media may also include any volatile or non-volatile media such as RAM (e.g., SDRAM, Double Data Rate (DDR) SDRAM, SRAM, etc.), read-only memory (ROM), etc., which may be included as system memory 1520 or another type of memory in some embodiments of computer system 1500. Furthermore, computer-accessible media may include transmission media or signals (such as electrical, electromagnetic, or digital signals) transmitted via communication media such as networks and / or wireless links (such as those implemented via network interface 1540).

[0121] Figure 16A logical arrangement of a set of common components (such as edge devices, web service providers, etc., as detailed above) of an exemplary computing device 1600 is shown. Generally, the computing device 1600 may also be referred to as an electronic device. The techniques shown in the figures and described herein can be implemented using code and data stored and executed on one or more electronic devices (e.g., client stations and / or server stations). Such electronic devices use computer-readable media (such as non-transitory computer-readable storage media (e.g., disks, optical discs, random access memory (RAM), read-only memory (ROM), flash memory devices, phase-change memory)) and transient computer-readable communication media (e.g., electrical, optical, or acoustic or other forms of propagation signals, such as carrier waves, infrared signals, digital signals) to store and transmit code and data (internal and / or via a network with other electronic devices). Additionally, such electronic devices include hardware such as a set of one or more processors 1602 (e.g., where the processors are microprocessors, controllers, microcontrollers, central processing units, digital signal processors, application-specific integrated circuits, field-programmable gate arrays), said processors coupled to one or more other components (e.g., one or more non-transitory machine-readable storage media (e.g., ... The electronic device includes a memory 1604 for storing code (e.g., instructions 1614) and / or data, and a set of one or more wired or wireless network interfaces 1608 that allow the electronic device to transfer and receive data from other computing devices, typically across one or more networks (e.g., a local area network (LAN), the Internet). Coupling between this set of processors and other components is typically via one or more interconnects (e.g., buses and possible bridges) within the electronic device. Therefore, a non-transitory machine-readable storage medium (e.g., memory 1604) of a given electronic device typically stores code (e.g., instructions 1614) for execution on the set of one or more processors 1602 of that electronic device. Various combinations of software, firmware, and / or hardware can be used to implement one or more portions of various implementation schemes.

[0122] Computing device 1600 may include some type of display element 1606, such as a touchscreen or liquid crystal display (LCD), but many devices (such as portable media players) may convey information via other components (such as audio speakers), and other types of devices (such as server stations) may not have a display element 1606 at all. As discussed, some computing devices used in some embodiments include at least one input and / or output component 1612 capable of receiving input from a user. This input component may include, for example, a button, touchpad, touchscreen, scroll wheel, joystick, keyboard, mouse, keypad, or any other such device or element, thereby enabling the user to input commands to the device. However, in some embodiments, such a device may be controlled by a combination of visual and / or audio commands, and utilizes microphones, cameras, sensors, etc., allowing the user to control the device without physical contact with it.

[0123] As discussed, different methods can be implemented in various environments based on the described implementation scheme. For example, Figure 17Examples of environments 1700 for implementing various aspects according to various implementation schemes are shown. For example, in some implementations, API calls are Hypertext Transfer Protocol (HTTP) requests received by a web server (e.g., web server 1706), and via an electronic device, a user can interact with a provider network via a web port provided by web server 1706 and application server 1708. It should be understood that although a web-based environment is used for illustrative purposes, different environments may be used to implement various implementation schemes as appropriate. The system includes an electronic client device 1702, which may also be referred to as a client device, and can be any suitable device operable to send and receive requests, messages, or information over a suitable network 1704 and relay information back to the user of device 1702. Examples of such client devices include personal computers (PCs), cellular phones, handheld messaging devices, laptop computers, set-top boxes, personal data assistants, e-book readers, wearable electronic devices (e.g., glasses, watch straps, monitors), etc. One or more networks 1704 may include any suitable network, including intranets, the Internet, cellular networks, local area networks, or any other such network or combinations thereof. The components used in such a system can depend at least in part on the type of network and / or environment chosen. Protocols and components used for communication via such a network are well known and will not be discussed in detail herein. Communication via a network can be achieved via wired or wireless connections and combinations thereof. In this example, network 1704 includes the Internet because the environment includes a web server 1706 for receiving requests and providing content in response to those requests; however, for other networks, alternative devices for similar purposes can be used, as will be apparent to those skilled in the art.

[0124] The illustrative environment includes at least one application server 1708 and a data repository 1710. It should be understood that several application servers, tiers, or other elements, programs, or components may exist, which may be linked or otherwise configured, and these may interact to perform tasks such as retrieving data from an appropriate data repository. As used herein, the term "data repository" refers to any apparatus or combination of apparatuses capable of storing, accessing, and retrieving data, which may include any combination and number of data servers, databases, data storage devices, and data storage media in any standard, distributed, or clustered environment. The application server 1708 may include any suitable hardware and software for integration with the data repository 1710 as needed to execute aspects of one or more applications for the client device 1702, and to handle most of the application's data access and business logic. Application server 1708 provides access control services in cooperation with data repository 1710 and is capable of generating content such as text, graphics, audio, and video to be transmitted to client device 1702. In this example, the content can be provided to the user via a web server in the form of Hypertext Markup Language (“HTML”), Extensible Markup Language (“XML”), JavaScript Object Notation (JSON), or another suitable unstructured or structured language. All request and response processing, as well as content delivery between client device 1702 and application server 1708, can be handled by web server 1706. It should be understood that web server 1706 and application server 1708 are not necessary and are merely exemplary components, as the structured code discussed herein can be executed on any suitable device or host as discussed elsewhere herein.

[0125] Data repository 1710 may include several separate data tables, databases, or other data storage mechanisms and data for storing data related to specific aspects. For example, the data repository shown includes mechanisms for storing generation data 1712 and user information 1716, which can be used to provide content to the generator. Mechanisms for storing logs or session data 1714 are also shown in data repository 1710. It should be understood that many other aspects may exist that may need to be stored in the data repository, such as page image information and access permission information, which may be stored, as appropriate, in any of the mechanisms listed above or in additional mechanisms within data repository 1710. Data repository 1710 can receive instructions from application server 1708 through associated logical operations and, in response to said receipt, obtain, update, or otherwise process data. In one example, a user can submit a search request for a specific type of item. In this case, data repository 1710 can access user information 1716 to verify the user's identity and can access generation data 1712 to obtain information about that type of item. The information can then be returned to the user, such as in a results list on a web page that the user can view via a browser on user device 1702. Information about specific items of interest can be viewed in a dedicated page or window of the browser.

[0126] Web server 1706, application server 1708, and / or data repository 1710 may be implemented by one or more electronic devices 1720, which may also be referred to as electronic server devices or server stations, and may or may not be located in different geographical locations. Each of the one or more electronic devices 1720 may include an operating system that provides executable program instructions for the general administration and operation of the device, and will generally include a computer-readable medium storing the instructions, which, when executed by the processor of the device, allow the device to perform its intended function. Suitable implementations of the operating system and the general functionality of the device are known or commercially available and readily implemented by those skilled in the art, especially in light of the disclosure herein.

[0127] One implementation scheme utilizes a distributed computing environment employing several computer systems and components interconnected via communication links using one or more computer networks or direct connections. However, those skilled in the art will understand that such a system can be used in environments with... Figure 17 The system shown operates just as well with fewer or more components. Therefore, Figure 17 The description of the environment in 1700 should be considered illustrative in nature rather than limiting the scope of this disclosure.

[0128] The various implementation schemes discussed or suggested herein can be implemented in a wide variety of operating environments. In some cases, these operating environments may include one or more user computers, computing devices, or processing devices that can be used to operate any of a number of applications. User or client devices may include any of a suitable number of general-purpose personal computers, such as desktop or laptop computers running standard operating systems, and cellular, wireless, and handheld devices running mobile software and capable of supporting a variety of network and messaging protocols. Such systems may also include multiple workstations running any of a variety of commercially available operating systems and other known applications for purposes such as development and database management. These devices may also include other electronic devices, such as virtual terminals, thin clients, gaming systems, and / or other devices capable of communicating via a network.

[0129] Most implementations utilize at least one network familiar to those skilled in the art to support communication using any of a variety of commercially available protocols, such as Transmission Control Protocol / Internet Protocol (TCP / IP), File Transfer Protocol (FTP), Universal Plug and Play (UPnP), Network File System (NFS), Universal Internet File System (CIFS), Extensible Messaging and Presentation Protocol (XMPP), AppleTalk, etc. The network may include, for example, a Local Area Network (LAN), a Wide Area Network (WAN), a Virtual Private Network (VPN), the Internet, an intranet, an extranet, a Public Switched Telephone Network (PSTN), an infrared network, a wireless network, and any combination thereof.

[0130] In implementations utilizing a web server, the web server can run any of a variety of server or middleware applications, including HTTP servers, File Transfer Protocol (FTP) servers, Common Gateway Interface (CGI) servers, data servers, Java servers, business application servers, etc. The server may also be able to execute programs or scripts in response to requests from user devices, such as by executing one or more web applications that can be implemented as one or more scripts or programs, written in any programming language (such as Java®, C, C#, or C++) or any scripting language (such as Perl, Python, PHP, or TCL and combinations thereof). The server may also include a database server, including but not limited to those commercially available database servers from Oracle®, Microsoft®, Sybase®, IBM®, etc. The database server can be relational or non-relational (e.g., "NoSQL"), distributed or non-distributed, etc.

[0131] As described above, the environment can include various data repositories and other storage media. These can reside in various locations, such as on storage media of one or more computers (and / or residing therein) or remotely on any or all computers in the network. In a particular set of embodiments, information can reside in a storage area network (SAN) familiar to those skilled in the art. Similarly, any necessary files for performing functions attributed to a computer, server, or other network device can be stored locally and / or remotely, depending on the circumstances. Where the system includes computerized devices, each such device can include hardware elements that can be electrically coupled via a bus, including, for example, at least one central processing unit (CPU), at least one input device (e.g., mouse, keyboard, controller, touchscreen, or keypad), and / or at least one output device (e.g., display device, printer, or speaker). Such a system can also include one or more storage devices, such as disk drives, optical storage devices, and solid-state storage devices such as random access memory (RAM) or read-only memory (ROM), as well as removable media devices, memory cards, flash memory cards, etc.

[0132] Such devices may also include a computer-readable storage medium reader, a communication device (e.g., a modem, a network interface card (NIC) (wireless or wired), an infrared communication device, etc.), and working memory as described above. The computer-readable storage medium reader may be connected to or configured to receive a computer-readable storage medium, which represents a remote, local, fixed, and / or removable storage device, as well as a storage medium for temporarily and / or more permanently containing, storing, transmitting, and retrieving computer-readable information. Systems and various devices will also typically include multiple software applications, modules, services, or other elements residing within at least one working memory device, including operating systems and applications such as client applications or web browsers. It should be understood that alternative embodiments can have a wide range of variations different from those described above. For example, custom hardware and / or specific components may be used, implemented in hardware, software (including portable software such as applets), or both. Furthermore, connectivity with other computing devices (such as network input / output devices) may be employed.

[0133] Storage media and computer-readable media used to contain code or code portions may include any suitable media known or used in the art, including storage media and communication media, such as, but not limited to, volatile and non-volatile, removable and non-removable media, implemented in any way or by any technique for storing and / or transmitting information, such as computer-readable instructions, data structures, program modules or other data, including RAM, ROM, electrically erasable programmable read-only memory (“EEPROM”), flash memory or other memory technologies, optical disc read-only memory (“CD-ROM”), digital versatile optical disc (DVD) or other optical storage devices, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any suitable media that can be used to store desired information and can be accessed by system devices. Based on the disclosure and teachings provided herein, those skilled in the art will understand other ways and / or methods of implementing the various embodiments.

[0134] In the preceding description, various embodiments have been described. For illustrative purposes, specific configurations and details have been set forth to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that the embodiments can be implemented without specific details. Furthermore, well-known features may have been omitted or simplified to avoid obscuring the described embodiments.

[0135] In this document, bracketed text and boxes with dashed borders (e.g., large dashes, small dashes, dotted dashes, and dots) are used to indicate optional operations for adding additional features to some embodiments. However, such notation should not be construed as indicating that these are the only options or optional operations, and / or that boxes with solid borders in some embodiments are not optional.

[0136] In various embodiments, reference numerals with suffix letters can be used to indicate one or more instances of the referenced entity, and when multiple instances exist, each instance need not be identical, but may share some general characteristics or act in a common manner. Furthermore, unless explicitly stated to the contrary, the use of a particular suffix does not imply the existence of a specific number of entities. Therefore, in various embodiments, two entities using the same or different suffix letters may or may not have the same number of instances.

[0137] References to "an implementation," "implementation," "exemplary implementation," etc., indicate that the described implementation may include specific features, structures, or characteristics, but each implementation need not necessarily include specific features, structures, or characteristics. Furthermore, such phrases do not necessarily refer to the same implementation. Additionally, when a specific feature, structure, or characteristic is described in connection with an implementation, whether explicitly stated or not, it is assumed that implementing such a feature, structure, or characteristic in conjunction with other implementations is within the knowledge of those skilled in the art.

[0138] Furthermore, in the various embodiments described above, unless otherwise specifically indicated, disjunctive language such as the phrase "at least one of A, B, or C" should be understood to refer to A, B, or C, or any combination thereof (e.g., A, B, and / or C). Therefore, such disjunctive language generally does not imply and should not be construed as implying that a given embodiment requires the presence of at least one of A, at least one of B, or at least one of C.

[0139] In addition, at least some embodiments of the disclosed technology can be described according to the following terms: 1. A computer-implemented method comprising: Receive a first application programming interface (API) request to analyze stored video, the API request including the location of the stored video and at least one analysis action to be performed on the stored video; Access the location of the stored video to retrieve the stored video; The accessed video is segmented into blocks; Each block is processed by a block processor to perform the at least one analysis action, and each block processor utilizes at least one machine learning model to perform the at least one analysis action; The results of the processing of each block are combined to generate the final result; Store the final result; and In response to the second API request, the final result is provided to the requester.

[0140] 2. The computer-implemented method according to Clause 1, wherein the first API request is one of the following: a start content review request for performing analysis of the content of the stored video, a start face detection request for performing face detection in the stored video, a start tag detection request for performing tag detection in the stored video, a start people tracking request for performing people tracking in the stored video, and a start celebrity recognition request for performing celebrity detection in the stored video.

[0141] 3. A computer-implemented method according to any one of clauses 1 to 3, wherein the block processor includes a block decoder to generate block frames, the block frames being passed to at least one machine learning algorithm to perform the at least one analysis.

[0142] 4. The computer-implemented method according to Clause 3, wherein the at least one machine learning algorithm of the block processor is a face detection algorithm.

[0143] 5. The computer-implemented method according to Clause 3, wherein the at least one machine learning algorithm of the block processor is a label detection algorithm.

[0144] 6. A computer-implemented method according to any one of Clauses 1 to 5, wherein the results of the processing of each block to generate a final result are performed by an aggregator, and the final result includes at least one of person bounding boxes, face bounding boxes, and face matching for each frame.

[0145] 7. A computer-implemented method according to any one of Clauses 1 to 6, wherein the first API request is received by a front end of a video analytics service.

[0146] 8. A computer-implemented method according to any one of Clauses 1 to 7, wherein the stored video is generated by capturing and indexing streaming video.

[0147] 9. A computer-implemented method according to any one of Clauses 1 to 8, wherein the stored video is encrypted and protected against unauthorized access.

[0148] 10. The computer-implemented method according to any one of clauses 1 to 9, further comprising: notifying the requester that the final result is available.

[0149] 11. A system comprising: An end-user device used to send a first request to analyze the stored video; Web service providers, which are used for: Receive a request to analyze stored video, the first request including the location of the stored video and at least one analysis action to be performed on the stored video; Access the location of the stored video to retrieve the stored video; The accessed video is segmented into blocks; Each block is processed by a block processor to perform the at least one analysis action, and each block processor utilizes at least one machine learning model to perform the at least one analysis action; The results of the processing of each block are combined to generate the final result; Store the final result; and In response to the second API request, the final result is provided to the requester.

[0150] 12. The system according to Clause 11, wherein the first request is one of the following: a start content review request to perform analysis of the content of the stored video, a start face detection request to perform face detection in the stored video, a start tag detection request to perform tag detection in the stored video, a start people tracking request to perform people tracking in the stored video, and a start celebrity identification request to perform celebrity detection in the stored video.

[0151] 13. The system according to any one of Clauses 11 to 12, wherein the block processor includes a block decoder to generate block frames, the block frames being passed to at least one machine learning algorithm to perform the at least one analysis.

[0152] 14. The system according to Clause 13, wherein at least one machine learning algorithm of the block processor is one of a face detection algorithm and a label detection algorithm.

[0153] 15. The system according to any one of clauses 11 to 14, wherein the result of the processing of each block to generate a final result is performed by an aggregator, and the final result includes at least one of person bounding boxes, face bounding boxes, and face matching for each frame.

[0154] Therefore, the specification and drawings should be considered illustrative rather than restrictive. However, it will be apparent that various modifications and changes may be made to this disclosure without departing from the broader spirit and scope of this disclosure as set forth in the claims.

Claims

1. A computer-implemented method comprising: Receive a first application programming interface (API) request to analyze stored video, the API request including the location of the stored video and at least one analysis action to be performed on the stored video; Place the request in the queue; The queue is polled by a job poller to retrieve the request; Access the location of the stored video to retrieve the stored video; The accessed video is segmented into blocks by a segmenter operator, and each block comprises more than one frame. The segmenter worker assigns the blocks to the block processor worker; The segmenter worker updates the database, which includes a block table that has the number of blocks for a job and associates a job ID with the job. Each block is received by the block decoder of the block processor worker, and each block is decoded into a block frame; Each block is processed by the block processor worker to perform the at least one analysis action on the block frame of each block, and each block processor worker is used to perform the at least one analysis action using at least one machine learning model; Each block processor worker uses a job ID to publish each processed block to a stream processor worker by using the job ID as a partition key, which can be used to group data by fragments within the stream; The stream processor worker processes all block completion events for the job associated with the job ID; The aggregator worker combines the results of the processing of each block to generate the final result; The aggregator worker marks the blocks that have been processed in the database; Store the final result; and In response to the second API request, the final result is provided to the requester.

2. The computer-implemented method of claim 1, wherein the first API request is one of the following: a start content review request for performing analysis of the content of the stored video, a start face detection request for performing face detection in the stored video, a start tag detection request for performing tag detection in the stored video, a start people tracking request for performing people tracking in the stored video, and a start celebrity recognition request for performing celebrity detection in the stored video.

3. The computer-implemented method of claim 1, wherein the job poller is a library that uses a basic data repository to keep track of user-submitted jobs and ensure fair sharing of service processing bandwidth.

4. A computer-implemented method, comprising: Receive a first application programming interface (API) request to analyze stored video, the API request including the location of the stored video and at least one analysis action to be performed on the stored video; Access the location of the stored video to retrieve the stored video; The accessed video is segmented into blocks by a segmenter operator, and each block comprises more than one frame. The segmenter worker assigns the blocks to the block processor worker; The segmenter worker updates the database, which includes a block table that has the number of blocks for a job and associates a job ID with the job. Each block is received by the block decoder of the block processor worker, and each block is decoded into a block frame; The block processor processes each block to perform the at least one analysis action on the block frame of each block, and each block processor worker is used to perform the at least one analysis action using at least one machine learning model; Each block processor worker uses a job ID to publish each processed block to a stream processor worker by using the job ID as a partition key, which can be used to group data by fragments within the stream; The stream processor worker processes all block completion events for the job associated with the job ID; The aggregator worker combines the results of the processing of each block to generate the final result; The aggregator worker marks the blocks that have been processed in the database; Store the final result; and In response to the second API request, the final result is provided to the requester.

5. The computer-implemented method of claim 4, wherein the first API request is one of the following: a start content review request to perform analysis of the content of the stored video, a start face detection request to perform face detection in the stored video, a start tag detection request to perform tag detection in the stored video, a start people tracking request to perform people tracking in the stored video, and a start celebrity recognition request to perform celebrity detection in the stored video.

6. The computer-implemented method according to claim 4 or 5, wherein the at least one machine learning algorithm of the block processor worker is a face detection algorithm.

7. A system comprising: Web service providers, configured to: Receive a first request to analyze stored video, the first request including the location of the stored video and at least one analysis action to be performed on the stored video; Access the location of the stored video to retrieve the stored video; The accessed video is segmented into blocks by a segmenter operator, and each block comprises more than one frame. The segmenter worker assigns the blocks to the block processor worker; The segmenter worker updates the database, which includes a block table that has the number of blocks for a job and associates a job ID with the job. Each block is received by the block decoder of the block processor worker, and each block is decoded into a block frame; Each block is processed by the block processor worker to perform the at least one analysis action on the block frame of each block, and each block processor worker is used to perform the at least one analysis action using at least one machine learning model; Each block processor worker uses a job ID to publish each processed block to a stream processor worker by using the job ID as a partition key, which can be used to group data by fragments within the stream; The stream processor worker processes all block completion events for the job associated with the job ID; The aggregator worker combines the results of the processing of each block to generate the final result; The aggregator worker marks the blocks that have been processed in the database; Store the final result; as well as In response to the second API request, the final result is provided to the requester.

8. The system of claim 7, wherein the first request is one of the following: a start content review request to perform analysis of the content of the stored video, a start face detection request to perform face detection in the stored video, a start tag detection request to perform tag detection in the stored video, a start personnel tracking request to perform personnel tracking in the stored video, and a start celebrity recognition request to perform celebrity detection in the stored video.

9. The system according to claim 7 or 8, wherein the at least one machine learning algorithm of the block processor is a face detection algorithm.

10. A computer-readable storage medium having a computer program stored thereon, the computer program causing the computer system to perform the computer-implemented method as claimed in any one of claims 1 to 6 when executed by one or more processors of a computer system.

Images