System and method for performing object analysis

Abstract

A system is provided that, in operation, classifies one or more elements within an input signal, wherein the system: receives a compressed version of the input signal, wherein the compressed version comprises at least two sets of compressed data in a hierarchy, wherein each set of compressed data, when decoded, enables reconstruction of the signal to a respective level of quality; decodes the compressed version of the signal to a first level of quality by decoding a first set of compressed data to generate a first reconstructed signal; performs a first classification operation on the first reconstructed signal; decodes the compressed version of the signal to a second level of quality by decoding a second set of compressed data to generate a second reconstructed signal; and performs one or more second classification operations on the second reconstructed signal.

Description

Technical Field

[0001] This disclosure relates to systems for performing object analysis, such as systems for performing object analysis in image sequences. Furthermore, this disclosure relates to methods for operating the aforementioned system to perform object analysis in image sequences (i.e., methods of operating the aforementioned system). Additionally, this disclosure relates to a computer program product comprising a non-transient computer-readable storage medium storing computer-readable instructions thereon, which can be executed by a computerized device including processing hardware to perform the aforementioned methods. The aforementioned systems employ automated deep learning analysis to perform scene / object detection, classification, recognition, and labeling of complex signals, such as image and video signals, by way of non-limiting examples. For simplicity, this disclosure focuses on the case of video signals; however, those skilled in the art can readily deduce how the same concepts can be applied to the automated analysis of other types of signals besides video signals. Background Technology

[0002] Image recognition systems are known, for example, as described in European patent application EP79300903.6; this application is well known in light of EPO Technical Committee decision T208 / 84 (Vicom).

[0003] It should be understood that time-evolving signals, such as video signals, are typically represented by a temporal sequence of samples (e.g., a temporal sequence of samples at a rate of 60 Hz or 60 samples per second), where each sample (e.g., a video frame) comprises multiple color planes (e.g., RGB, YUV, HSV, etc.), where each color plane comprises a large number of pixels (e.g., for an Ultra HD full-resolution plane, 3840 × 2160 pixels or approximately 8 million pixels), and where each pixel value is represented with a given bit depth (e.g., 8 bits, 10 bits, etc.). In summary, it should be understood that uncompressed video signals require extremely high bitrates per second to transmit from one location to another; for example, for a 4:2:2 UHD 10-bit video signal, 12 gigabits (i.e., 1.5 GB) are required for just one second.

[0004] As stated in EP Technical Appeals Board Decision T208 / 84, the automatic analysis and classification of video signals for purposes such as indexing, tagging, object recognition, and OCR is well-known; such automatic analysis and classification typically requires performing multiple classification "services" on a given signal. These services are usually performed by independent nodes in a cluster computing system, each equipped with appropriate computing resources (e.g., typically a graphics processing unit (GPU) or field-programmable gate array (FPGA) optimized for deep learning computation). For the same given signal, different services often require different levels of quality: for example, some services (e.g., face recognition) can be successfully performed by analyzing only one frame per ten frames of a given raw video sequence, while other services (e.g., scene detection for identifying goals in a football match) may require finer temporal granularity; similarly, some services can only be successfully performed by using grayscale, while other services may require all color planes to be available, and so on.

[0005] To enable the delivery of distinct video frames to different services, state-of-the-art classification systems generate common queues where each frame is independently accessible. It should be understood that distributing uncompressed image frames across multiple nodes in a cluster is impractical, as such distribution could create one or more network bandwidth and / or memory bandwidth bottlenecks; most state-of-the-art solutions for distributing frame data (e.g., using...) Figure 1 The transcoder 110 shown compresses each frame independently. Although any compression scheme can theoretically be applied, most implementations choose to use JPEG because more efficient formats such as JPEG2000, AVC-I, WebP, HEVC-I, or AVIF are slow to decode, creating a processing bottleneck. Figure 1 A schematic diagram illustrating a typical state-of-the-art solution is provided.

[0006] refer to Figure 1 The video classification task is performed by, for example, retrieving multiple frames from queue 120 by scheduler 130, decoding the frames, and sending the corresponding decoded frames to the operational memory of a processing chip (e.g., a graphics processing unit (GPU) or a field-programmable gate array (FPGA)). The processing chip optionally preprocesses the decoded frames (e.g., by downsampling them, converting them to grayscale images, etc.) and then uses them as input data for one or more deep neural networks (e.g., implemented in one or more nodes 140).

[0007] A significant limitation of traditional methods for performing object analysis in image frames is that the resolution and color space of the decoded frames must be the least common denominator of all possible classification services of a given system, and any change to the least common denominator is costly in terms of one or more time-consuming preprocessing operations (in addition to the time required to decode multiple frames at a resolution higher than the required resolution). When color is critical for object analysis in an image, state-of-the-art methods use the RGB 4:4:4 format as input to the convolutional neural network detector, consistent with using JPEG as the encoding format; it should be understood that working in the YUV color space is not feasible for JPEG format; this is the objective technical problem that this disclosure attempts to address. Summary of the Invention

[0008] This disclosure attempts to provide an improved system for performing object analysis, such as an improved system for performing object analysis on objects contained in an image sequence. Furthermore, this disclosure attempts to provide an improved method for performing object analysis using the aforementioned improved system (i.e., a method for performing object analysis using the aforementioned improved system). Additionally, this disclosure attempts to provide an improved software product that can be executed on computing hardware to implement the aforementioned improved method.

[0009] According to a first aspect, a method is provided for classifying one or more elements of an input signal using a system (i.e., a method for classifying one or more elements within an input signal using a system), wherein the method includes:

[0010] Receive a compressed version of the input signal, wherein the compressed version includes at least two sets of compressed data in a hierarchical structure, wherein each set of compressed data is capable of reconstructing the signal to a corresponding quality level when decoded;

[0011] The compressed version of the signal is decoded to a first quality level by decoding the first set of compressed data to generate a first reconstructed signal;

[0012] Perform a first classification operation on the first reconstructed signal;

[0013] The compressed version of the signal is decoded to a second quality level by decoding the second set of compressed data to generate a second reconstructed signal; and

[0014] Perform one or more second classification operations on the second reconstructed signal.

[0015] The advantage of this invention is that the compressed version can be decoded very efficiently for classification purposes using the compressed data provided in the hierarchical structure, wherein the compressed version is provided at a first quality in a first stage of classification, and the compressed image is provided at a second quality in a second stage of classification, wherein the selection of the second stage depends on the classification performed in the first stage.

[0016] Optionally, in the method, when the first classification operation is performed, one or more regions of interest in the first reconstructed signal are identified, and the compressed version of the signal is decoded to a second quality level only for those one or more regions of interest.

[0017] Optionally, in the method, the resolution of the first reconstructed signal and the resolution of the second reconstructed signal are the same.

[0018] Optionally, in the method, the resolution of the first reconstructed signal is different from the resolution of the second reconstructed signal.

[0019] Optionally, in the method, the first classification is performed on a first number of frames in the first reconstructed signal, and the one or more second classifications are performed on a second number of frames in the second reconstructed signal.

[0020] Optionally, in the method, the first classification is performed on the entire frame in the first reconstructed signal, and the one or more second classifications are performed on a portion of the frames in the second reconstructed signal.

[0021] Optionally, in the method, the first classification is performed on a first number of color planes in the first reconstructed signal, and the one or more second classifications are performed on at least a second number of color planes in the second reconstructed signal.

[0022] More optionally, in the method, the first number of color planes corresponds to all of the planes in the first reconstructed signal, and wherein the second number of color planes corresponds to a subset of the color planes in the second reconstructed signal.

[0023] More optionally, in the method, one or more second classifications are also performed on at least a third number of color planes in the first reconstructed signal, wherein the color planes included in the second number of color planes are different from the color planes included in the third number of color planes.

[0024] More optionally, in the method, the first number of color planes corresponds to three and includes planes Y, U, and V, the second number of color planes includes plane Y, and the third number of color planes corresponds to color planes U and V.

[0025] More optionally, in the method, a signal includes multiple color planes (YUV), and each color plane is a luminance plane (Y) or a chrominance plane (U, V).

[0026] Optionally, the method further includes:

[0027] The compressed version of the signal is decoded to a third quality level by decoding the third set of compressed data to generate a third reconstructed signal; and

[0028] Perform one or more third classification operations on the third reconstructed signal.

[0029] More optionally, in the method:

[0030] The first classification is performed on a first number of color planes in the first reconstructed signal, and the one or more second classifications are performed on at least a second number of color planes in the second reconstructed signal;

[0031] The first reconstructed signal also performs the one or more second classifications on at least a third number of color planes, wherein the color planes included in the second number of color planes are different from the planes included in the third number of color planes; and

[0032] Perform the one or more third classifications on at least a fourth number of color planes in the third reconstructed signal.

[0033] More optionally, in the method, the third reconstructed signal corresponds to a portion of the input signal. Even more optionally, in the method, the portion corresponds to a region of interest in the input signal.

[0034] Optionally, in the method, the steps of receiving, decoding, and executing are all performed within the same processing unit (e.g., the same graphics processing unit (GPU)).

[0035] Optionally, in the method, the first quality level is lower than the second quality level (e.g., defined in terms of image resolution).

[0036] Optionally, in the method, the second reconstructed signal is also generated in part based on the first reconstructed signal.

[0037] Optionally, in the method, the reconstructed signal corresponds to a portion of the input signal.

[0038] Optionally, in the method, the first category, one or more second categories, and one or more third categories are organized hierarchically, such that the first category is a coarse classification of some elements in the input signal, and the one or more second categories or the one or more third categories further refine the coarse classification of the elements in the input signal.

[0039] More optionally, the method further includes:

[0040] Based on the rough classification, only those parts of the second set of compressed data that are determined to require further classification are decoded, and the second reconstructed signal is generated based on those decoded parts.

[0041] Optionally, in the method, one or more neural network detectors (e.g., multiple neural network detectors) are used to perform the classification. Optionally, the neural network detector includes a data normalization phase and a subsequent data comparison phase. Optionally, the neural network detector is implemented in digital hardware (e.g., in a field-programmable gate array (FPGA), in computing hardware configured to execute software products to implement neural network functions, or in a combination of both.

[0042] According to a second aspect, a method for classifying one or more elements within an input signal (i.e., a method for classifying one or more elements within an input signal) is provided, the method comprising:

[0043] The first classification subtask in the classification task is performed on the first version of the input signal;

[0044] One or more second classification subtasks in the classification task are performed on the second version of the input signal, wherein the second version is generated based on the output of the first classification subtask.

[0045] Optionally, the method further includes:

[0046] One or more third classification subtasks in the classification task are performed on the third version of the input signal, wherein the generated third version is based on the output of the one or more second classification subtasks.

[0047] Optionally, in the method, based on the output of the first classification subtask, the second version includes only a portion of the input signal.

[0048] Optionally, in the method, based on the output of the first classification subtask, the second version includes only a subset of the color planes that form the input signal.

[0049] Optionally, in the method, each version corresponds to a decoded version of the input signal, wherein each decoded version differs from the other versions based on one or more of the following: resolution, the portion of the input signal referenced by the decoded version, and the number of planes used.

[0050] According to a third aspect, a method for reconfiguring a field-programmable gate array (FPGA) that classifies one or more elements within an input signal (i.e., a method for reconfiguring an FPGA) is provided, the method comprising:

[0051] The FPGA is configured using a first decoding process to decode the compressed version of the input signal to a first quality level;

[0052] The first set of compressed data is decoded to generate the first reconstructed signal;

[0053] The FPGA is configured using a first classification process to detect a first element in the first reconstructed signal; and

[0054] The first classification process is performed on the first reconstructed signal.

[0055] Optionally, the method further includes:

[0056] The FPGA is configured using a second decoding process to decode the compressed version of the input signal to a second quality level;

[0057] The second set of compressed data is decoded to generate the second reconstructed signal;

[0058] The FPGA is configured using a second classification process to detect a second element in the second reconstructed signal; and

[0059] The second classification process is performed on the second reconstructed signal.

[0060] Optionally, in the methods of the first aspect and / or the second aspect, decoding is performed on the computer processing unit (CPU) and classification is performed on the graphics processing unit (GPU).

[0061] According to a third aspect, a method for classifying one or more elements within a plurality of input signals (i.e., a method for classifying one or more elements within a plurality of input signals) is provided, the method comprising:

[0062] Receive compressed versions of the plurality of input signals, wherein the compressed versions include at least two sets of compressed data in the hierarchical structure of each of the plurality of input signals, wherein each set of compressed data is capable of reconstructing the corresponding input signal among the plurality of input signals to a corresponding quality level when decoded;

[0063] The compressed version of the first group of multiple input signals is decoded to a first quality level by decoding the first set of compressed data of each of the first group of multiple input signals to generate a first plurality of reconstructed signals;

[0064] Perform a first classification operation on the first plurality of reconstructed signals;

[0065] The compressed versions of the second set of input signals are decoded to a second quality level by decoding the second set of compressed data for each of the second set of input signals, thereby generating a second plurality of reconstructed signals; and

[0066] Perform one or more second classification operations on the second plurality of reconstructed signals.

[0067] Optionally, in the method, the first plurality of reconstructed signals include a first number of color planes, and the second plurality of reconstructed signals include a second number of color planes. More optionally, in the method, the first number of color planes corresponds to three and includes planes Y, U, and V, and the second number of color planes corresponds to one and includes plane Y.

[0068] Optionally, in the method, the compressed version of the signal is decoded to a second quality level only for regions of interest identified in the previously performed classification.

[0069] Optionally, in the method, when the first classification operation is performed, one or more regions of interest in the first reconstructed signal are identified, and the compressed version of the signal is decoded to the second quality level only for those regions of interest.

[0070] Optionally, in the method, the first version and / or the second version of the input signal includes compressed data.

[0071] According to a fourth aspect, a system is provided that is configured to perform the methods of any one of the first, second, and third aspects.

[0072] According to a fifth aspect, a field-programmable gate array (FPGA) is provided, which is configured to perform the methods of any one of the first aspect, the second aspect, and the third aspect.

[0073] According to a sixth aspect, a computer program product is provided, comprising a non-transient computer-readable storage medium having computer-readable instructions stored thereon, the computer-readable instructions being executable by a computerized means including processing hardware to perform the foregoing methods of any one of the first, second, and third aspects.

[0074] The embodiments disclosed herein substantially eliminate or at least partially solve the aforementioned problems in the prior art, and are able to analyze objects present in image sequences more effectively and quickly.

[0075] Additional aspects, advantages, features, and objectives of this disclosure will become apparent from the accompanying drawings and the detailed description of illustrative embodiments as interpreted in conjunction with the appended claims.

[0076] It should be understood that the features of this disclosure are readily combined in various combinations without departing from the scope of this disclosure as defined by the appended claims. Attached Figure Description

[0077] The foregoing summary of the invention and the following detailed description of illustrative embodiments can be better understood when read in conjunction with the accompanying drawings. Exemplary constructions of this disclosure are shown in the drawings for illustrative purposes. However, this disclosure is not limited to the specific methods and tools disclosed herein. Furthermore, those skilled in the art will understand that the drawings are not drawn to scale. Where possible, similar elements are represented by the same numerals.

[0078] Embodiments of this disclosure will now be described by way of example only, with reference to the following figures, wherein:

[0079] Figure 1 This is a schematic diagram of a known system for analyzing and classifying objects contained in an image frame;

[0080] Figures 2 to 4 This is a schematic diagram of a system for analyzing and classifying objects contained in an image frame according to the present disclosure; and

[0081] Figure 5 This is a flowchart of the steps of a method for analyzing and classifying objects contained in an image frame (i.e., a method for analyzing and classifying objects contained in an image frame).

[0082] In the accompanying diagram, underlined numbers indicate the item to which the underlined number belongs or the item adjacent to the underlined number. Ununderlined numbers are associated with the item identified by the line linking the ununderlined number to the item. When a number has no underline and an associated arrow, the ununderlined number is used to identify the general item that the arrow points to. Detailed Implementation

[0083] The following detailed description illustrates embodiments of the present disclosure and ways in which these embodiments may be implemented. Although some modes for implementing the present disclosure have been disclosed, those skilled in the art will recognize that other embodiments for implementing or practicing the present disclosure are also possible.

[0084] According to a first non-limiting embodiment, a classification method is provided that compresses a sequence of frames, such as image frames, using an encoding format suitable for parallel and region of interest (“RoI”) decoding, thereby generating compressed data of the sequence. In a preferred embodiment, this encoding format is a hierarchical encoding format, preferably a hierarchical-based encoding format. The compressed data of the sequence is then advantageously transmitted in compressed form to the operational memory of a processor node responsible for performing the classification task. Depending on the classification task to be performed, the processor node optionally selects to decode only the relevant portions of the compressed data of the sequence. For multiple classification tasks, optionally (by the same or different processing nodes, depending on the embodiment) the same compressed data is used, with different decoded data (e.g., by means of non-limiting examples, different subsets of the frames in the sequence, different image resolutions, etc.) used as input to one or more artificial intelligence (AI) detectors, such as, but not limited to, neural network detectors.

[0085] In a non-limiting example embodiment, the processor node decodes only the frames in the sequence to a first quality level (e.g., at 480×270 resolution), where the first quality level is lower than the highest quality level (e.g., at 1920×1080 resolution). In another non-limiting example embodiment, the processor node decodes only the frames in the sequence in YUV format (i.e., so that no YUV to RGB conversion step is required) and feeds the subsampled chroma plane as input to an artificial intelligence (AI) detector, such as a neural network detector, thereby reducing the number of inputs to the AI ​​detector, such as a neural network detector. In another non-limiting example embodiment, the processor node decodes only the Y (i.e., luminance) color plane and provides only luminance information as input to the AI ​​detector, such as a neural network detector, thereby significantly reducing the number of inputs to the AI ​​detector, such as a neural network detector. In another non-limiting example, the processor node decodes only a specified region of interest (RoI) of a specified frame, which also significantly reduces the number of inputs to the AI ​​detector, such as a neural network detector.

[0086] In embodiments of this disclosure, using YUV information is feasible and therefore allows for direct processing within the YUV, such as for performing classification operations. Using a YUV format utilizing chroma subsampling (e.g., YUV 4:2:2 or YUV 4:2:0) limits the number of pixels analyzed, thus significantly improving the efficiency of the embodiments. It should be understood that video sources are often 4:2:2 (in the case of high-quality mezzanine) or 4:2:0 (more common in the case of compressed H.264 video), therefore the YUV format utilizing chroma subsampling is more closely related to the source video, thus avoiding any loss due to color format conversion in embodiments of this disclosure. Furthermore, the need to decode frames outside the processing chip (e.g., a graphics processing unit (GPU) chip or a field-programmable gate array (FPGA) chip) and then transfer the corresponding series of uncompressed frames to operational memory makes high resolution extremely expensive in terms of memory bandwidth (and therefore expensive in terms of total processing time); embodiments of this disclosure circumvent this objective technical problem, i.e., limitation.

[0087] The unique and innovative method described in this disclosure allows deep learning systems to circumvent the aforementioned limitations by more effectively managing memory bandwidth bottlenecks and enabling efficient automatic classification tasks on high-resolution signals. Specifically, the embodiments described herein significantly reduce total processing time by leveraging hierarchical compression techniques to redefine the entire classification process: by using a massively parallel, hierarchical compression format, the method described herein delivers frames of the highest possible quality to the operating memory of the processing chip in compressed form (instead of decoding these frames and delivering their corresponding uncompressed representations), and then performs decoding only on the regions of interest (RoIs), quality levels, and color planes of frames deemed important to one or more specific classification tasks. Importantly, the decoding of lower quality levels and / or specific RoIs at high resolution is optionally performed entirely on the processing chip (i.e., a GPU or FPGA) and does not require first fully decoding a given frame and then performing a pruning / rescaling operation. The possibility of rapidly decoding much smaller portions of the overall data (i.e., processing time several orders of magnitude less than that of fully decoding and then pruning / rescaling) also allows for the breakdown of a single classification task into multiple deep learning tasks and progressive refinement, enabling the use of simplified neural network topologies while improving overall classification accuracy.

[0088] refer to Figure 2This describes an example of an embodiment according to the present disclosure. In the system, scheduler 200 feeds compressed image frames to GPU node 220, for example, via decoder 210 (but decoder 210 may be a function of GPU node 220 as a whole). The image frames are encoded in a hierarchical structure such that each layer of the structure corresponds to a different quality level, such as a different resolution. GPU node 220 is configured to perform one or more classification tasks on the image frames. Specifically, a first node performs a first classification task #3 on decoded frames of a first quality level, such as a low-resolution version of the image frame (i.e., 480×270). The first level can be processed using all color planes YUV. A second node (which may also be a second instance of the first node after reconfiguration) performs a second classification task #2 on decoded frames of a second quality level, such as a medium-resolution version of the frame (e.g., 960×540). Optionally, only the Y color plane is used to process the second level. In this case, even if the compressed frame contains all YUV planes, only the Y plane is decoded. The third node performs a third classification task #1 on decoded frames of the third quality level, such as the highest resolution version of the frame (e.g., 1920×1080). All color planes (YUV) can be used to process the third level. Optionally, the third classification is performed only on selected regions of interest (RoIs) of the frame, thereby reducing the amount of information to be processed, such as RoIs identified during previous classification tasks.

[0089] In a non-limiting example embodiment, scheduler 200 optionally delivers compressed data to a given processor node, such as a given GPU node 220, and the processor node optionally uses the same compressed data multiple times, each time decoding different portions and / or quality levels of the signal based on the given classification task to be performed. In a non-limiting embodiment, the decoded portions still in the operational memory of the given processing node for previous classification tasks can be reused as a baseline for higher-resolution decoding (or region of interest decoding) in subsequent classification tasks.

[0090] In other non-limiting example embodiments, the scheduler 200 optionally distributes the classification computation across multiple processing nodes in the cluster and only sends a portion of the compressed data required for each processing node to perform its task.

[0091] According to a second non-limiting embodiment, the aforementioned system employs a classification method that effectively utilizes the possibility of fast partial decoding to divide a given overall classification task into a hierarchical structure of subsequent fast classification subtasks comprising at least two subtasks. The first classification task is performed by decoding all image frames contained in an image frame sequence, but at a very low quality level (i.e., very low resolution): this task is performed very quickly due to the small number of pixels associated with the low quality level. The purpose of this task is to detect those portions of the video that do not require further processing for one or more given classification tasks, and to detect and locate regions of interest (ROIs) in which further processing is required for one or more given classification tasks, and to generate metadata to better guide the necessary subsequent classification subtasks. The second classification task is performed on input data obtained by decoding ROIs from a subset of image frames in the image frame sequence, based at least in part on the results of the first classification task (e.g., the identification of one or more designated ROIs to be decoded for the purpose of the second classification task). In this way, the second classification task is also characterized by a limited number of pixels as input, but the selected ROIs are optionally decoded at the maximum available resolution, thereby positively impacting classification accuracy (i.e., providing a benefit).

[0092] Next reference Figure 3 The illustration shows an embodiment according to this disclosure. A compressed series of video frames is fed to multiple GPU nodes via a scheduler. The series of video frames is encoded in a hierarchical structure such that each layer of the structure corresponds to a different quality level, for example, to a different resolution. The GPU nodes are configured to perform one or more classification tasks on the series of video frames. Specifically, a first node performs a first classification task on a series of decoded video frames of a first quality level, such as a low-resolution version of a frame (e.g., 480×270). Optionally, all color planes YUV are used to process the first level. A second node (which may also be a second instance of the first node after reconfiguration) performs a second classification task on a subset of a series of decoded video frames of a second quality level, for example, performing the second classification task on only one of the video frames. The task is optionally performed on the highest resolution version of the frame (e.g., 1920×1080). Optionally, all color planes YUV are used to process the second level. Optionally, the second classification is performed only on selected regions of interest (RoIs) of the frames, thereby reducing the amount of information to be processed, such as RoIs identified during a previous classification task.

[0093] In a non-limiting embodiment, the same first-level classification subtask can prevent multiple second-level classification subtasks from being executed in parallel by multiple processing nodes belonging to the cluster. Only the relevant portion of the compressed data is transmitted to each processing node.

[0094] Next reference Figure 5 In a non-limiting embodiment, the classification task is divided into three sub-tasks. In the first step 500, a first-level "coarse classification" is performed by rapidly detecting portions of the video sequence that require or do not require further processing (e.g., one or more secondary detections) based on the signal presentation at a low quality level. For example, such "coarse classification" can be performed to detect frames in a football match where there is little or no chance of finding a goal, thus excluding these frames from further analysis and reducing the computational effort required for the classification task. In the second step 510, for each detection task, a second-level detection task is triggered for the portions of the video sequence identified by the first-level coarse classification as having a detection probability. The second-level detection task operates on a subset of the frames used for the first-level coarse classification, but at a higher quality level (e.g., higher image resolution). Each second-level detection task, optionally performed at a higher quality level than the first pass, aims to detect and locate regions of interest (RoIs) of the specific frames that should be considered for identification. In the third step 520, when the second-level detection is successful, one or more third-level identification subtasks are triggered. The third-level identification subtasks operate on the specified region of interest (RoI) of the specified frame to decode at a higher quality level.

[0095] Next reference Figure 4 The diagram illustrates an example of an embodiment according to this disclosure. A compressed series of video frames is fed to multiple GPU nodes via a scheduler. The series of video frames is encoded in a hierarchical structure such that each layer of the structure corresponds to a different quality level, such as a different resolution. The GPU nodes are configured to perform one or more classification tasks on the series of video frames. Specifically, a first node performs a first classification task on a series of decoded video frames of a first quality level, such as a low-resolution version of a frame (e.g., 240×135). Optionally, all color planes YUV are used to process the first level. A second node (which may also be a second instance of the first node after reconfiguration) performs a second classification task on a subset of a series of decoded video frames of a second quality level, such as performing a second classification task on only one of the video frames. The task is optionally performed on a medium-resolution version of the frame (e.g., 480×270). A third node (which may also be a second instance of the first node after reconfiguration) performs a third classification task on a subset of a series of decoded video frames of a third quality level, such as performing a third classification task on only one of the video frames. The task is optionally performed on the highest resolution version of the frame (e.g., 1920×1080). The third classification may optionally be performed using all or only some of the YUV color planes. Alternatively, the third classification may be performed only on selected regions of interest (RoIs) of the frame, thereby reducing the amount of information to be processed, such as the RoIs identified during the previous classification task.

[0096] In a non-limiting example embodiment, the classification system is configured to perform face detection and recognition, such as face detection and recognition. The system receives an input video and, for each given time interval, compresses a sequence of frames of the input video at the source resolution of 4:2:0 YUV using a hierarchical, layer-based intra-frame compression format, producing compressed data for the frame sequence. The compressed data, along with a description of the desired classification task, is sent to a processing node; for example, via, for instance... The data communication network transmits compressed data. Processor nodes process the compressed data and produce a presentation of frames in the sequence at a first (low) quality level. The decoded presentations are placed (stored) adjacent to each other in operational memory and provided as input to a first artificial intelligence (AI) detector, such as a neural network detector. The first neural network of the neural network detector performs face detection and identifies multiple regions of interest (RoIs) for each detected face for recognition, each RoI belonging to a different frame. For each face to be recognized, the processing node performs RoI decoding on the identified regions in the indicating frame, places the decoded RoIs adjacent to each other in operational memory, and feeds them as input to a second neural network of the neural network detector, which performs recognition. The process is repeated for each face to be recognized. In other non-limiting embodiments, the face recognition task is performed in parallel (i.e., simultaneously in time) by multiple processing nodes in a cluster, which may or may not include processing nodes performing the detection task. In other non-limiting embodiments, the same methods described for facial recognition can also be used for object / landmark / topic / tag recognition, brand recognition, emotion recognition, and text / OCR recognition, such as vehicle license plate detection, vehicle detection in complex traffic situations, and road environment for autonomous vehicles.

[0097] In a non-limiting example embodiment, instead of using RGB color data as input to an artificial intelligence (AI) detector, such as a neural network detector, that performs classification functions, the processing node stops decoding after obtaining the YUV planes and feeds those planes as input to the AI ​​detector. In other non-limiting example embodiments, the U and V planes are decoded at a lower (including much lower) quality level (e.g., lower resolution) relative to the Y plane, thereby further reducing the amount of input to the AI ​​detector, such as the neural network detector, and thus reducing the overall processing time.

[0098] According to a third non-limiting embodiment, the processing node analysis still involves data in compressed form or only partially decoded. In these embodiments, a hierarchical coding format based on layers allows each signal (i.e., by way of a non-limiting example, an image frame) to be decoded using the following subsequent operations: reconstructing the signal at a first (lower) quality level, then predicting an initial representation of the signal at a second (higher) quality level, then performing parallel decoding of the residual data to modify the initial representation of the signal at the second quality level, then combining the initial representation and the residual data to reconstruct the signal's representation at the second quality level, then predicting the initial representation of the signal at a third (higher) quality level, then performing parallel decoding of the residual data to modify the initial representation of the signal at the third quality level, and so on. It is noteworthy that all unpredictable information resides in the hierarchical layers (also called "echelons") of the residual data, which contain data that the decoder cannot infer on its own based solely on the availability of lower data echelons. Furthermore, according to these embodiments, the residual data is constructed in the form of data blocks (e.g., by means of non-limiting embodiments, using quantized transform coefficients), which can be independently decoded to reconstruct the residual data for each given block: thus, all information about the pixels in a given block is contained in the quantized transform coefficients. According to these embodiments, additional decoding time is saved by feeding data that is still partially compressed (partially or entirely) to an AI detector, such as a neural network detector, instead of the corresponding final decoded pixel plane.

[0099] In a non-limiting example embodiment, the classification task requires processing the signal at a given quality level. Instead of fully decoding the signal to the given quality level, the processing unit decodes the signal to a first (e.g., a lower level) quality level, and then performs entropy decoding on the transform coefficients needed to reconstruct the residual data echelon at the given quality level. The signal and transform coefficient echelon at the first quality level, rather than the complete signal at the given quality level, are provided as input to an AI detector, such as a neural network detector. In such a non-limiting example embodiment, the number and information content of the inputs to the AI ​​detector, such as a neural network detector, are the same as in the case of fully decoding the signal at a given quality level; however, the decoding process is no longer required, thus saving a significant amount of processing time and is technically very advantageous.

[0100] According to a fourth non-limiting embodiment, the processing node uses a field-programmable gate array (FPGA) device that can be reconfigured in real time via dynamic and partial reconfiguration. In these embodiments, subsequent operations such as partial decoding, hierarchical classification, region of interest (RoI) decoding at higher resolution, and classification refinement are performed by loading different subsequent logic onto the FPGA device. For example, the FPGA device includes a data memory in which parameters representing one or more objects are loaded; the FPGA also has a data memory in which an image at a given quality level can be loaded. The FPGA normalizes the image at the given quality level and then partially correlates the parameters representing one or more objects with the normalized image; when this correlation provides a correlation output greater than a detection threshold, the one or more objects are considered to be present in the normalized image. Therefore, an algorithm in the form of a Kalman filter is advantageously employed. The aforementioned normalization can be taught to the FPGA using a learning process.

[0101] In a non-limiting example embodiment, a processing node receives a sequence of classification tasks and signal samples (e.g., video frames, by way of a non-limiting example) in compressed form stored in operational memory. The processing node loads an FPGA device using a first decoder capable of rapidly decoding the compressed data into a sequence presentation at a first quality level. The processing node then performs a decoding task and organizes the decoded data in operational memory to feed it to a first AI detector, for example, using a neural network. The processing node then loads the FPGA device to perform a first neural network classification. The processing node then performs the classification task and writes the output to operational memory. Based on the output of the first classification task, the processing node then loads the FPGA device using a second decoder capable of rapidly decoding the compressed data into a second input to a second AI detector, for example, using a neural network. The processing node then performs a second decoding task and organizes the decoded data in operational memory to feed it to the second AI detector, for example, using a neural network. In a non-limiting embodiment, the second decoding operation includes some of the decoded outputs of the first decoding operation still in operational memory. In other non-limiting embodiments, the second decoding operation includes region of interest (RoI) decoding of one or more frames. In other non-limiting embodiments, chroma planes that are different from each other are optionally decoded at different quality levels. In other non-limiting embodiments, the second decoding operation decodes only a subset of the frames decoded in the first decoding operation at a higher quality level. Based on the output of the first classification task, the processing node then loads the FPGA device to perform a second AI classification, such as a second neural network classification. The processing node then performs the classification task and writes the output to the operating memory.

[0102] In other non-limiting example embodiments, more than two subsequent classification tasks are performed using the same scheme described above. In one such embodiment, the first classification task is performed at a relatively low quality level throughout the sequence and is common to a series of subsequent classification tasks, which are then executed in parallel (i.e., simultaneously in time) on different processing nodes in the cluster, each receiving only a portion of the compressed data required to perform the task.

[0103] In another non-limiting embodiment, the classification task can be performed using a graphics processing unit (GPU), while the decoding can be performed using a computer processing unit (CPU). For example, the GPU can be hardware-based, enabling very high image processing rates, with the CPU executing software, thereby allowing for a high degree of reconfigurability in performing various classification tasks within the GPU.

[0104] The above description and embodiments provide an efficient solution to the problem of classifying elements within an input signal (e.g., a video signal) in an effective manner. Generally, the solution presented herein relies on using hierarchical coding algorithms to break down the broader classification problem into one or more subtasks, each of which can be performed at a different hierarchical level, thereby enabling more accurate and faster detection of objects present in an image.

[0105] Specifically, if decoding cannot be performed without leaving the GPU (where classification algorithms typically run), then it is impossible to decode outside the GPU and then transmit the uncompressed data.

[0106] Specifically, if it's not possible to selectively decode portions of the signal (e.g., a lower resolution portion of the signal, a part of the signal, a region of interest, or a particular color space), then decoding at maximum resolution and / or then rescaling at maximum resolution to any desired size / cropping will consume a significant amount of time each time. For example, the process of decoding, transmitting, and rescaling 20 UHD frames would take 350 to 400 milliseconds; using hierarchical code with the aforementioned characteristics (e.g., handling classification tasks at different resolutions, processing only portions of the signal, etc.) would only require approximately 30 milliseconds.

[0107] Furthermore, the same compressed data can be reused for multiple classification tasks, potentially using the following methods each time:

[0108] (a) Different numbers of frames;

[0109] (b) Different resolutions;

[0110] (c) Full frame and region of interest (RoI);

[0111] (d) Chromaticity plane and luminance plane at different resolutions;

[0112] For example, perform a first pass advantageously with all frames at a low quality level (and / or resolution) using YUV 4:2:0; perform a second pass with only a few frames at a higher quality level (and / or resolution) targeting only the luma and chroma planes still at a lower quality level (and / or resolution); and perform a third pass with three regions of interest at the highest quality level (and / or resolution) using YUV 4:2:0 targeting only the selected frames.

[0113] In embodiments of this disclosure, further efficiency can be achieved by using partially decoded frames (e.g., reaching a certain resolution) for subsequent decoding at a higher quality level; for example, a decoded 270p frame can be reused to decode two additional quality levels (LoQ) to decode the full 1080p or the region of interest (RoI) of a 1080p frame; here, "p" refers to an image pixel.

[0114] Furthermore, because it can flexibly decode any frame / RoI in real time at any resolution, classification operations can be performed in a hierarchy of subsequent classification tasks. The first level of "coarse classification" can be used to very quickly determine which parts of the video are not worth further processing (i.e., it is feasible to very quickly eliminate large portions of the data, where such elimination is particularly useful not only for reducing the time and cost of offline archiving and tagging, but also for reducing the average cost of real-time video processing).

[0115] The above methods, namely the systems, algorithms, and methods of this disclosure, can be executed on a Field Programmable Gate Array (FPGA), thereby enabling advanced object analysis and detection in image data using dynamic and partial reconfiguration. Dynamic and partial FPGA reconfiguration is an existing option, but rarely used. In embodiments of the invention, the various steps are clearly suited for dynamic reconfiguration: first decoding with a hierarchical codec, then execution of a first neural network, then decoding again with the hierarchical codec in a different manner, then execution of a second neural network, and so on. A reconfiguration time of approximately a few milliseconds (e.g., <50 milliseconds) is perfectly acceptable for many practical applications, and the overall method of this disclosure can easily process one-second duration UHD video in less than 100 milliseconds. This improvement has significant technical benefits and major commercial implications for industrial applications such as security systems, robotics, defense equipment, traffic control systems, autonomous vehicles, automated agricultural equipment, missile guidance systems, etc.

[0116] Furthermore, the same first-level classification can prevent multiple second-level classifications on multiple different processing nodes in the cluster. Only the relevant portions of the compressed data are distributed, thereby reducing (e.g., minimizing) memory bandwidth and transmission time.

[0117] Furthermore, the data provided as input to the neural network detector can be partially decoded (e.g., "indexed in compressed space"); reducing the required level of decoding also reduces the computational effort required to implement embodiments of this disclosure. For example:

[0118] (a) Stop at YUV (U and V planes after subsampling) instead of RGB;

[0119] (b) Stop at YUV, where the quality level (LoQ) of U and V is low (typically, the chromaticity information is very coarse, and the LoQ is even lower);

[0120] (c) Y plane only (equivalent to grayscale, but does not require decoding RGB and then converting to grayscale);

[0121] (d) Only the low-resolution decoded image and the high-resolution transform coefficients (thus saving the relatively time-consuming upsampling operation and residual transform and residual addition operations).

[0122] Optionally, one of the latent hierarchical coding algorithms used in the embodiments of this disclosure is the proprietary Perseus from V-Nova International Ltd. TM Pro products (which feature byte stream format elements that allow for partial and parallel decoding, and use static entropy decoding instead of adaptive entropy decoding); proprietary Perseus from V-Nova International Ltd. TM The Pro product is also described in the following U.S. patent applications, which are hereby incorporated by reference:

[0123] 13 / 188,188、13 / 188,201、13 / 188,207、13 / 188,220、13 / 188,226、13 / 352,944、13 / 188,237、13 / 303,554、13 / 744,808、13 / 893,665、13 / 893,669、13 / 894,417、13 / 893,672、13 / 89 3,677, 15 / 783,204, 15 / 779,193, 16 / 077,828, 16 / 103,784, 16 / 078,352, 16 / 126,939, 16 / 252,357, 16 / 252,362, 16 / 324,433, 16 / 324,431, 16 / 295,847, 16 / 295,851, 16 / 295,854,

[0124] And it is described in the following PCT patent application, which is hereby incorporated by reference:

[0125] PCT / GB2017 / 053716, PCT / EP2018 / 075603, PCT / EP2018 / 082350, PCT / GB2018 / 053551, PCT / GB2018 / 053556, PCT / GB2018 / 053553, PCT / GB2019 / 050122, PCT / GB2018 / 053552, PCT / GB2019 / 051104, PCT / GB2018 / 053546, PCT / GB2018 / 053555, PCT / GB2018 / 053547, PCT / GB2018 / 053554, PCT / GB2018 / 053548.

[0126] Embodiments of this disclosure are advantageously implemented using contemporary FPGAs, such as devices from Intel Inc., for example... Agilex TM F-series FPGAs; these FPGAs offer transceiver support up to 58Gbps, enhanced digital signal processing (DSP) capabilities, high system integration, and second-generation support for a wide range of applications. Hyperflex TM Architecture; In implementing embodiments of this disclosure, operation using a CPU such as a quad-core Arm Cortex-A53 processor facilitates high system integration. Other types of FPGAs can be used to implement embodiments of this disclosure. "TM" indicates a trademark, while Indicates a registered trademark.

[0127] The preceding text described the use of an artificial intelligence (AI) detector for performing object classification. Furthermore, as previously mentioned, the AI ​​detector is conveniently implemented using one or more neural networks. Advantageously, the neural network is implemented in layers, where a given layer of a given neural network feeds its output to one or more other layers of the neural network in a hierarchical manner. The lower layers in the hierarchy are advantageously used to perform the aforementioned first classification operation, and are advantageously implemented in an FPGA, where one or more higher layers in the hierarchy are advantageously implemented using computational hardware configured to perform software for second and / or third classification operations. In this way, after most of the processing is performed on the FPGA, the functionality of the AI ​​detector can be reconfigured in software. For many image classification operations, the first few layers of the neural network are often shared by different neural networks to provide a degree of time parallelism. It is the higher layers of the neural network that actually control the specific execution of the AI ​​task, such as face recognition, emotion recognition, smile detection, license plate recognition, vehicle type recognition, and fruit quality recognition in automated fruit sorting equipment in agriculture.

[0128] As previously described, at least one of the first, second, and third classification operations is implemented using a multi-stage process, for example, including a first normalization stage and a second comparison stage. It should be understood that data normalization is a common (optional) preprocessing stage in neural networks, sometimes performed on each layer of the neural network. The data comparison stage typically occurs at the output of neurons, as part of the neuron's nonlinear function (e.g., from the Reduced Linear Unit – ReLU function) or as part of a general AI system (e.g., similar to thresholding soft probability values ​​for object categories, such as "the object is 80% a car and 20% a motorcycle," to make a final / hard decision, i.e., "the object is a car").

[0129] Advantageously, the AI ​​detector is configured such that, when processing data on objects with varying orientations and / or sizes, the neural network detector is built to be rotation / size flexible, so that the neural network detector does not need to perform these operations during operation. Instead, they use rotated / scaled images in earlier training phases.

[0130] While the use of a Kalman filter to perform classification operations has been mentioned for the foregoing as an optional approach to implementing this disclosure, it should be understood that other methods are possible and are part of this disclosure. It should be understood that the combination of hierarchical codecs and AI-based classification systems is a fundamental feature of embodiments of this disclosure.

[0131] It should be understood that any feature described with respect to any embodiment may be used alone or in combination with other described features, and may also be used in combination with at least one feature of any other embodiment or any combination of any other embodiments. Furthermore, equivalents and modifications not described above may be employed without departing from the scope of the invention as defined in the appended statements.

Claims

1. A method for classifying one or more elements within an input image using a system, wherein the method includes: Receive a compressed version of the input image, wherein the compressed version includes at least a first set of compressed data and a second set of compressed data in a hierarchical structure, wherein each set of compressed data is capable of reconstructing the image to a corresponding quality level when decoded; The compressed version of the image is decoded to a first quality level by decoding the first set of compressed data to generate a first reconstructed image; Perform a first object classification operation on the first reconstructed image; The compressed version of the image is decoded to a second quality level by decoding the second set of compressed data to generate a second reconstructed image, the second quality level being higher than the first quality level; as well as Perform one or more second object classification operations on the second reconstructed image; When the first object classification operation is performed, one or more regions of interest in the first reconstructed image are identified, and the compressed version of the image is decoded to a second quality level only for those regions of interest.

2. The method according to claim 1, wherein the resolution of the first reconstructed image and the resolution of the second reconstructed image are the same.

3. The method according to claim 1, wherein the resolution of the first reconstructed image is different from the resolution of the second reconstructed image.

4. The method of claim 1, wherein the first object classification operation is performed on a first number of frames in the first reconstructed image, and the one or more second object classification operations are performed on a second number of frames in the second reconstructed image; or The first object classification operation is performed on the entire frame in the first reconstructed image, and the one or more second object classification operations are performed on a portion of the frame in the second reconstructed image.

5. The method of claim 1, wherein the first object classification operation is performed on a first number of color planes in the first reconstructed image, and the one or more second object classification operations are performed on at least a second number of color planes in the second reconstructed image, and / or Wherein the first number of color planes corresponds to all the planes in the first reconstructed image, and wherein the second number of color planes corresponds to a subset of the color planes in the second reconstructed image, and / or The process further includes performing the one or more second object classification operations on at least a third number of color planes in the first reconstructed image, wherein the color planes included in the second number of color planes are different from the color planes included in the third number of color planes, and / or The first number of color planes corresponds to three and includes planes Y, U, and V; the second number of color planes corresponds to one and includes plane Y; and the third number of color planes corresponds to two and includes color planes U and V.

6. The method according to claim 5, wherein the image comprises a plurality of color planes Y, U, V, and each color plane is a luminance plane Y or a chrominance plane U, V.

7. The method of claim 1, wherein the method further comprises: The compressed version of the image is decoded to a third quality level by decoding the third set of compressed data to generate a third reconstructed image, the third quality level being higher than the second quality level; as well as Perform one or more third object classification operations on the third reconstructed image, and / or wherein: Perform the first object classification operation on a first number of color planes in the first reconstructed image, and perform the one or more second object classification operations on at least a second number of color planes in the second reconstructed image; and / or The one or more second object classification operations are also performed on at least a third number of color planes in the first reconstructed image, wherein the color planes included in the second number of color planes are different from the planes included in the third number of color planes; and Perform the one or more third object classification operations on at least a fourth number of color planes in the third reconstructed image. And / or wherein the third reconstructed image corresponds to a portion of the input image, And / or wherein the portion thereof corresponds to the region of interest of the input image.

8. The method according to claim 1, wherein the steps of receiving, decoding and executing are all performed within the same processing unit.

9. The method according to claim 1, wherein quality is defined as a function of image resolution.

10. The method according to claim 1, wherein the second reconstructed image is further generated in part based on the first reconstructed image.

11. The method according to claim 1, wherein the reconstructed image corresponds to a portion of the input image.

12. The method according to claim 7, wherein the first object classification, one or more second object classifications, and one or more third object classifications are organized hierarchically, such that the first object classification is a coarse classification of some elements in the input image, and the one or more second object classifications or the one or more third object classifications further refine the classification of the elements. The method further includes: Based on the rough classification, only those parts of the second set of compressed data that are determined to require further classification are decoded, and the second reconstructed image is generated based on those decoded parts.

13. The method according to claim 1, wherein an artificial intelligence detector arrangement is used to perform the object classification operation, wherein the artificial intelligence detector arrangement comprises a neural network detector, and The neural network detector is implemented such that the lowest neural network layer of the neural network detector is implemented in at least one FPGA, and one or more higher neural network layers of the neural network detector are implemented using computing hardware configured to execute a software product.

14. A system configured to perform the method according to any one of claims 1 to 13.

15. A computer program product comprising a non-transient computer-readable storage medium having computer-readable instructions stored thereon, the computer-readable instructions being executable by a computerized means including processing hardware to perform the method according to any one of claims 1 to 13.

Images