Feature fusion method for preprocessing input image data of learning model

Abstract

According to the method and the system, preprocessing of input image data of a learning model is realized through image data fuzzy processing based on depth features. Intermediate features are extracted from convolutional layers of the pre-processing model, and each set of intermediate features are fused to produce a fused feature map and magnified to an input image size. Based on the fused feature map, the pre-processing model may configure one or more processors of the input pre-processing computing system to enhance image data with larger corresponding feature values and weaken other image data when performing the blurred pre-processing calculations.

Description

Related applications

[0001] This application claims priority to U.S. patent application No. 63 / 436,817, filed on January 3, 2023, entitled “FEATURE FUSION FOR INPUT PICTURE DATA PREPROCESSING FOR LEARNING MODEL”; U.S. patent application No. 63 / 492,205, filed on March 24, 2023, entitled “FEATURE FUSION FOR INPUT PICTURE DATA PREPROCESSING FOR LEARNING MODEL”; and U.S. patent application No. 18 / 400,954, filed on December 29, 2023, entitled “FEATURE FUSION FOR INPUT PICTUREDATA PREPROCESSING FOR LEARNING MODEL”. The entire contents of these three applications are expressly incorporated into this application by reference. Technical Field

[0002] The present disclosure generally relates to input image data preprocessing, and more particularly, to methods and systems for implementing input image data preprocessing for learning models through deep feature-based image data blurring. Background Art

[0003] Current image coding technologies, such as H.264 / AVC (Advanced Video Coding), H.265 / HEVC (High Efficiency Video Coding), and Versatile Video Coding (VVC), are primarily based on lossy compression, using a framework that includes transform coding, quantization, and entropy coding. For many years, lossy compression has achieved compression ratios suitable for image capture and storage at limited scales. The primary objective of these existing codecs is to achieve improved reconstructed signal quality within the constraints of a limited bit rate.

[0004] Based on machine learning, end-to-end image compression techniques have been developed. Deep neural networks are trained on image and video datasets to learn the parameters of a nonlinear transformation. This nonlinear transformation configures a computing system to map an input image into a latent representation in a latent space. Entropy coding techniques are then applied to the latent representation of the image, improving computational efficiency.

[0005] Furthermore, thanks to the emergence of fields such as computer vision and machine vision, computer systems are increasingly being configured to capture and store images on a larger scale. Machine learning and deep learning are configuring computer systems to perform new tasks driven by large-scale image and video datasets, so machine learning image and video datasets also rely on image compression to improve the efficiency of data storage.

[0006] However, the image and video inputs to machine learning models have different signal quality requirements than those emphasized by existing lossy compression codecs, and different optimization requirements than those emphasized by end-to-end image compression techniques. Techniques beyond conventional lossy compression codecs and end-to-end image compression techniques are needed to achieve compression and computational efficiency while prioritizing the image data used by machine learning models for computation. Summary of the Invention

[0007] In a first aspect, embodiments of the present invention provide a method. The method comprises: performing, by one or more processors of an input preprocessing computing system, multiple convolution operations on input image data; outputting, by the one or more processors, multiple intermediate feature maps from the respective different convolution operations; averaging, by the one or more processors, the absolute values ​​of the multiple intermediate feature maps to generate a fused feature map; and resizing the fused feature map to the size of the input image data.

[0008] In a second aspect, embodiments of the present invention provide a computing system comprising: an execution module configured to perform multiple convolution operations on input image data; an output module configured to output multiple intermediate feature maps from the respective convolution operations; an averaging module configured to average the absolute values ​​of the multiple intermediate feature maps to generate a fused feature map; and a resizing module configured to resize the fused feature map to the size of the input image data.

[0009] In a third aspect, an embodiment of the present disclosure provides a computing system comprising: one or more processors, and a computer-readable storage medium communicatively coupled to the one or more processors, wherein the computer-readable storage medium stores computer-readable instructions that can be executed by the one or more processors, and when the computer-readable instructions are executed by the one or more processors, performs the computing method described in the first aspect.

[0010] In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium having computer-executable instructions stored thereon, wherein when the processor executes the computer-executable instructions, the method described in the first aspect is implemented.

[0011] In a fifth aspect, an embodiment of the present invention provides a computer program product, comprising: computer program instructions, wherein the computer program instructions enable a computer to execute the method described in the first aspect.

[0012] In a sixth aspect, an embodiment of the present invention provides a computer program, which enables a computer to execute the method described in the first aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The detailed description will be set forth with reference to the accompanying drawings. In the drawings, the leftmost digit(s) of a reference number identifies the drawing in which the reference number first appears. The use of the same reference number in different drawings indicates similar or identical items or features.

[0014] Figure 1 A flowchart illustrating an image compression process performed by one or more processors of a computing system configured with a VVC standard encoder and one or more processors of a computing system configured with a VVC standard decoder is shown.

[0015] Figure 2 shows the output to Figure 1 Flowchart of input preprocessing of the image compression process.

[0016] Figure 3 Mask-based image data removal and blur preprocessing techniques are shown.

[0017] Figure 4 A flowchart of intermediate feature extraction and fusion from the convolutional layers of a pre-processing model according to an example embodiment of the present disclosure is shown.

[0018] Figure 5 A feature-based fuzzy pre-processing technique according to an example embodiment of the present disclosure is shown.

[0019] Figure 6 A mask union operation across a sequence of images is shown.

[0020] Figure 7 An exemplary input pre-processing computing system is shown for implementing the processes and methods described herein for implementing intermediate feature fusion extracted from convolutional layers of a pre-processing model.

[0021] Figure 8 A computing system for implementing the exemplary encoding and decoding described herein is shown, which is used to implement the processes and methods described herein for implementing a VVC standard encoder and a VVC standard decoder. DETAILED DESCRIPTION

[0022] The systems and methods discussed in this article are designed to enable data preprocessing for learning models, and more specifically, to perform image preprocessing on image and video datasets used for model training based on the fusion of intermediate features extracted from the convolutional layers of the preprocessed model.

[0023] The learning model according to an exemplary embodiment of the present disclosure includes at least computer-readable instructions that can be executed by one or more processors of a computing system to perform a computing task that includes computing inputs based on the values ​​of various parameters, and outputs a result. The learning model can be, for example, a hierarchical model such as a deep neural network, which can have a fully connected structure, a feedforward structure such as a convolutional neural network (CNN), a backpropagation structure such as a recurrent neural network (RNN), or other architectures suitable for computing specific computing tasks. Computational tasks can include, for example, classification, clustering, matching, regression, and the like.

[0024] One or more processors of the computing system may be further configured to obtain inference results based on the outputs of these computing tasks to solve various problems such as identifying entities in images and / or videos; tracking the movement of entities across multiple video frames; matching identified entities in images and / or videos to other images and / or videos; providing annotations or transcriptions of images, videos and / or audio in real time; and the like.

[0025] A computing system used to train a learning model, or a computing system that performs computational inference based on a trained learning model, can communicate with one or more input devices, wherein the one or more input devices are configured to capture data to be input into the learning model, thereby performing computations associated with various tasks based on the learning model, and analyzing and outputting the desired results of performing these tasks. The input device can store the captured data on a non-transitory computer-readable medium, which can be a component of the input processing computing system. The input device itself can be a component of the input processing computing system, or it can be an external device to the input processing computing system. The input processing computing system can be the same computing system as the computing system used to train the learning model or perform computational inference based on the trained learning model, or it can be a different computing system. In the case of different computing systems, they can communicate with each other via a suitable connection such as a wired or wireless network connection.

[0026] The input device may communicate with one or more processors of the input processing computing system via a data bus connection, a wired network connection, a wireless network connection, etc. The input device may be configured to transmit the captured data to the input processing computing system, and the input processing computing system may be configured to write the captured data to a storage device comprising one or more non-transitory computer-readable media.

[0027] The input device may be a camera that captures still images, video, or other types of image data. By way of example, such a video camera may be a standalone camera; may be a peripheral device connected to the computing system; may be one of any number of cameras integrated into an electronic device, machine, or motor vehicle having one or more integrated processors and computing storage devices; or may be configured in any other suitable manner to communicate with the computing system.

[0028] Regardless of whether the computing system is training a learning model or performing computational inference based on a trained learning model, it can be configured to perform computations on the captured image data based on the learning model. Because the file sizes of image datasets used in deep learning are very large, storing the image datasets may occupy a large amount of storage space, and loading and computing the image datasets may incur a significant amount of computational overhead.

[0029] Furthermore, in computing architectures where training a learning model or performing inference calculations based on a trained learning model is performed by a cloud computing system, unacceptable latency can occur if the input data, collected or stored locally, is delivered to the cloud computing system via a network connection. Furthermore, raw, uncompressed images are highly inefficient for machine learning computations, as they contain many times more data than compressed images and are often redundant for machine learning training and inference purposes. Therefore, it is desirable to compress images captured by input devices before using them in training and inference datasets.

[0030] General-purpose, non-machine learning-oriented image compression techniques are the default solution in video communications, embodied in video coding standards such as High Efficiency Video Coding (HEVC) and Versatile Video Coding (VVC). These standards employ a hybrid coding framework by dividing frames into blocks of variable or fixed sizes. Alternatively, end-to-end image compression has been developed, where images are transformed into their latent representations based on trained deep neural networks that are also designed to output perceptually meaningful representations.

[0031] Ballé et al. proposed an end-to-end image compression framework using generalized division normalization (GDN) and optimized the entire framework using rate-distortion optimization (RDO). Ballé et al. also proposed the concept of super-prior, which is to capture spatial dependencies in the latent representation. Cui et al. proposed a new type of autoencoder called gain variable autoencoder (G-VAE), which is based on continuous rate control, in which a pair of gain units are integrated into the end-to-end image compression framework to obtain continuous variable rate compression without increasing network parameters and computational cost. Choi et al. proposed a new variable rate image compression framework and conditional autoencoder, which adopts the structure of conditional fluctuation and universal quantization. An end-to-end compression framework based on deep learning, namely the deep video compression framework (DVC), and multi-frame prediction learning video compression (M-LVC) have also been developed.

[0032] Known approaches to image coding for machine vision include visual signal compression and compact feature representation. For visual signal compression, Zhang et al. proposed the Learning Image Compression (LIC) framework and the Multi-Scale Progressive (MSP) probabilistic model for lossy image compression based on the spatial and channel correlations of the latent representation of the image.

[0033] Feature compression is hampered by the fact that models are typically tuned for specific tasks, and top-level features are highly correlated with the task and difficult to generalize. Chen et al. proposed mid-level feature compression, where mid-level features are compressed and transmitted instead of the original video or top-level features. End-to-end learning typically enables deep features to have a larger receptive field and be more task-specific. As a result, features from shallow layers often contain more information than deep features.

[0034] Figure 1 A flowchart of an image compression process 100 performed by one or more processors of a computing system configured as an encoder and one or more processors of a computing system configured as a decoder is shown. The image compression process 100 includes at least the general-purpose, non-machine learning-oriented image compression technology and machine vision-oriented image coding described above.

[0035] In accordance with the VVC video coding standard ("VVC standard") and the general image compression technology described therein, a computing system includes at least one or more processors and a computer-readable storage medium communicatively coupled to the one or more processors. The computer-readable storage medium is a non-transitory or non-transitory computer-readable storage medium, as subsequently attached. Figure 8As defined, it stores computer-readable instructions. At least some of the computer-readable instructions stored on the computer-readable storage medium can be executed by one or more processors of the computing system to configure the one or more processors to perform the associated operations of the computer-readable instructions, including at least the operation of the encoder described by the VVC standard and the operation of the decoder described by the VVC standard. These encoder operations and decoder operations according to the VVC standard will be described in more detail later, and these subsequent descriptions should not be understood as an exhaustive list of encoder operations and decoder operations according to the VVC standard. Subsequently, "VVC standard encoder" and "VVC standard decoder" will describe the corresponding computer-readable instructions stored on the computer-readable storage medium, which configure one or more processors to perform these corresponding operations (which may be referred to as the "reference implementation" of the encoder or decoder).

[0036] In addition, according to an exemplary embodiment of the present disclosure, the VVC standard encoder and the VVC standard decoder further include computer-readable instructions stored on a computer-readable storage medium, which can be executed by one or more processors of a computing system to configure the one or more processors to perform operations not specified by the VVC standard. The VVC standard encoder should not be understood as limited to the operation of the reference implementation of the encoder, but rather includes more computer-readable instructions that configure one or more processors of the computing system to perform further operations as described herein. The VVC standard decoder should not be understood as limited to the operation of the reference implementation of the decoder, but rather includes additional computer-readable instructions that configure one or more processors of the computing system to perform further operations as described herein.

[0037] During the encoding process, the VVC standard encoder 104 configures one or more processors of the computing system to receive one or more input images as input from the image source 102. As an example, the VVC standard encoder 104 encodes an image (the image being encoded is referred to as the current image, to be distinguished from any other images received from the image source 102) by configuring one or more processors of the computing system to divide the original image into units and sub-units according to a partitioning structure. In addition, the VVC standard encoder 104 encodes the image by configuring one or more processors of the computing system to perform motion prediction on blocks of the current image.

[0038] After performing various calculations on the blocks of the current image, the VVC standard encoder 104 configures one or more processors of the computing system to output an encoded image consisting of the encoded blocks from the entropy encoder. The encoded image is output to a transmission buffer, where it is ultimately packaged into a bitstream 106 for output from the VVC standard encoder 104. The bitstream 106 is written by one or more processors of the computing system to a non-transitory or non-transitory computer-readable storage medium of the computing system for transmission.

[0039] The VVC standard decoder 108 configures one or more processors of the computing system to receive one or more encoded pictures from the bitstream 106 as input. The VVC standard decoder 108 configures one or more processors of the computing system to perform intra-frame prediction using prediction information specified in the coding parameter set, and configures one or more processors of the computing system to perform motion-compensated prediction using reference pictures from the decoded picture buffer, thereby generating a prediction signal.

[0040] After performing various calculations on the prediction signal, the VVC standard decoder 108 configures one or more processors of the computing system to output the reconstructed image 110 to the input layer of the learning model 112 for training and / or inference.

[0041] Furthermore, both standard image compression codecs and end-to-end, learning-based compression schemes can further benefit from pre-processing techniques, where one or more processors of a computing system are configured to compute on the raw image data before compressing it.

[0042] Figure 2 shows the output to Figure 1 Flowchart of input preprocessing 200 of image compression process 100. Image compression process 100 at least includes the general-purpose, non-machine learning-oriented image compression technology and machine vision-oriented image coding as described above.

[0043] In input preprocessing 200, a preprocessing model 202 configures one or more processors of an input processing computing system (such as described above) to receive input image data from an image source 102. The preprocessing model 202 configures the one or more processors to perform various preprocessing calculations on the input image data and outputs the processed image data, which is always uncompressed, to one or more processors of the same computing system or a different computing system configured by the VVC standard encoder 104.

[0044] The input image data may be captured by an input device in a raw image format. For example, the input device may be a camera such as a standalone camera, a peripheral device, a camera integrated into another electronic system, or similar devices as described above. The input device may capture image data constituting an image dataset in the form of still images or video.

[0045] The input device may be connected to the input processing computing system via a data bus connection, an optical data connection, or other connection suitable for image transmission, and may be connected to the input processing computing system via a wired or wireless network connection. For example, the input processing computing system may be a personal computing system, a cluster of computing systems, a server of a cloud computing system such as an edge server, an embedded computing system of a smart device such as a vehicle or an appliance, etc.

[0046] As an example, the pre-processing module 202 can configure one or more processors of the input processing computing system in the following manner. Wang et al. proposed a pre-processing technique in which a mask of important objects in the image data ("object mask") is derived through segmentation, and then a portion of the background is removed based on the object mask, and the remaining portion of the background is blurred. With this technique, less important image data is weakened by removal or blurring.

[0047] Figure 3 The mask-based image data removal and blur preprocessing technique is shown. In the manner described above, a merge mask M is derived by segmentation. The merge mask M segments the image data into at least one object segment (in Figure 3 ) and background fragments (shown in white in Figure 3 ), matching as much as possible the actual outline of the object depicted in the image data.

[0048] However, if M is directly used to perform image data removal and image data blurring computations, it may result in over-processing of regions immediately adjacent to objects. To improve the image data removal and image data blurring processes to achieve better discrimination for machine vision tasks, this paper further proposes a block-based blurring preprocessing operation that aims to preserve visual information surrounding objects / instances.

[0049] For example, a sliding window s of size n×n pixels n Applied to the input image data I and the merge mask M, where n is determined based on the resolution of the input image data I, when the sliding window s n When there is an intersection with the merge mask M, the merge mask is modified to exclude each pixel of the sliding window from the masked background, retaining the samples of the input image data I at those pixels. The modified block-based mask is denoted as M B , and the block-based masked input image data is represented by M B The point-by-point multiplication between and I is obtained, expressed as I BLOCK .

[0050] In addition, although blur processing can improve the smoothness of input data and thus make the blurred signal easier to compress, excessive blurring of object areas may lead to a decrease in the distinction between different objects. Excessively blurred image data input may also lead to poor performance of inference calculations based on learning models. In order to optimize the trade-off between representation range and machine performance, blur-based preprocessing is further proposed to blur non-object areas. A Gaussian blur transform is performed on the block-based mask input image data. The Gaussian blur preprocessing I blur The output is represented by I blur =(MB -M)(G k,s *I)+MI, Where I is the input image data, G k,s is a Gaussian filter with kernel size k and standard deviation s, and * denotes a convolution operation.

[0051] Despite these approaches, input image data preprocessing based on image data blurring still has limitations. Image data blurring calculations can configure one or more processors of a computing system to perform calculations based on the same parameters for the entire background region of all visual data. This type of blurring calculation cannot adapt to different visual content. Depending on the problem to be solved by the inference result, the inference calculation based on the learned model may rely on part of the background in the image data rather than the entire background, which means that all parts of the background do not need to be equally weakened through preprocessing.

[0052] Therefore, an exemplary embodiment of the present disclosure provides input image data preprocessing for a learning model by blurring image data based on deep features. According to an exemplary embodiment of the present disclosure, intermediate features are extracted from the convolutional layer of the preprocessing model, and each set of intermediate features is fused to obtain a fused feature map, which is then scaled to the size of the input image. Based on the fused feature map, the preprocessing model can configure one or more processors of the input preprocessing computing system to enhance image data with larger feature values ​​and weaken other image data when performing blur preprocessing calculations.

[0053] Figure 4 A flowchart of intermediate feature extraction and fusion from a convolutional layer of a pre-processing model according to an exemplary embodiment of the present disclosure is shown. As shown herein, input image data from an image source 102 is computed by one or more processors of an input pre-processing computing system configured by a pre-processing model 202, and object masks derived from segmentation of the input image data are output by the one or more processors.

[0054] The preprocessing model 202 can be a learning model that is trained to configure one or more processors of the input preprocessing computing system to perform segmentation calculations on the input image data, such as a Mask R-CNN learning model. Segmentation refers to dividing the input image data into a certain number of differently labeled segments with boundaries between the segments. Each segment can convey at least some semantic meaning that is different from other segments. The input image data can be divided based on various aspects of it, such as the similarity between image data samples, the difference between image data samples, the boundaries between image data samples, etc., as well as distinguishing foreground samples from background samples. The segmentation of objects in the image data samples should be understood as strengthening the segmented samples above the background samples and weakening the background samples.

[0055] Preprocessing model 202 includes multiple convolutional layers 206. Each convolutional layer 206 receives image data from the previous layer and performs convolution on the image data to generate a feature map from the image data. The feature map is generated by applying filters to the image data, outputting multiple channels for matrix operations used to perform convolution on the image data. For example, based on the Mask R-CNN learning model, the number of convolutional layers 206 can be arranged according to the Feature Pyramid Network (FPN).

[0056] At each convolutional layer 206 of the preprocessing model 202, strided convolution and / or pooled convolution can be performed on the image data. Such convolution downsamples the image data, reducing the size and resolution of the image data. Each such convolution operation produces intermediate features that can be different from the intermediate features produced by another convolution operation in another convolutional layer 206; thus, the features extracted from the convolution performed on the image data can have a smaller size and a smaller resolution than the input image data; the features extracted from different convolutions can have different sizes and resolutions; and the features extracted from different convolutions can have different numbers of channels.

[0057] The pre-processing model 202 may be branched into a stack of more than one convolutional layer 206, wherein at least one branch of the pre-processing model 202 configures one or more processors of the input pre-processing computing system to output an object mask. The object mask includes one or more segments that distinguish foreground objects of the image data from the background of the image data.

[0058] Furthermore, according to an exemplary embodiment of the present disclosure, the fusion module 402 stored on a non-transitory or non-transitory computer-readable storage medium of the input pre-processing computing system may configure one or more processors of the input pre-processing computing system to receive the intermediate feature maps F1, F2, ..., F output by the N different convolutional layers 206 before the output of one or more branches of the pre-processing model 202. N The fusion module 402 further configures one or more processors to combine the intermediate feature maps F1, F2, ..., F output by the multiple convolutional layers 206. N Fused into fusion feature map F P .

[0059] One or more processors of the input preprocessing computing system are configured to process different feature maps F1, F2, ..., F N The absolute values ​​of the features are averaged to produce a fusion feature map of the single-channel features, and then the single-channel features are normalized, and then the size of the fusion feature map is adjusted to the same size as the input image data, as follows:

[0060] The one or more processors of the input pre-processing computing system are configured to use the resized fused feature map as input to the blur pre-processing operation. The block-based and feature-based blur pre-processing module 404 configures the one or more processors of the input pre-processing computing system to perform mask-based image data removal and blur pre-processing, as subsequently attached. Figure 5 Descriptive.

[0061] In the feature graphs F1, F2, ..., F N (For simplicity, it is expressed as F here) to generate the fusion feature map F P Afterwards, one or more processors of the input preprocessing computing system perform Gaussian blur transformation on each pixel of the input image data I, wherein the Gaussian blur transformation fuses the feature map F P The feature map value corresponding to the pixel in is used as input. For example, the feature map value corresponding to the pixel can be calculated as the standard deviation value in the Gaussian blur transformation: Where (x, y) is the coordinate of the pixel and σ is the standard deviation.

[0062] In addition, the block-based and feature-based fuzzy pre-processing module 404 configures one or more processors of the input pre-processing computing system to perform block-based fuzzy pre-processing as described above, with reference to the accompanying drawings. Figure 3 . Therefore, one or more processors may be configured to perform a Gaussian blur transform on each pixel of the input image data based on the feature map value (i.e., perform feature-based but not block-based blur pre-processing); the one or more processors may be configured to perform a Gaussian blur transform on block-based masked pixels of the input image data (i.e., perform block-based but not feature-based blur processing); and the one or more processors may be configured to perform a Gaussian blur transform on block-based masked pixels of the input feature data based on the feature map value (i.e., perform block-based and feature-based blur pre-processing).

[0063] Furthermore, according to an exemplary embodiment of the present disclosure, the block-based and feature-based blur pre-processing module 404 configures one or more processors of the input pre-processing computing system to perform a union operation on the modified masks of a series of images. For a series of images in a video sequence, the block-based mask may weaken the visual continuity due to the sudden appearance or disappearance of visible blocks for consecutive images. In order to improve the visual continuity of the image sequence, the one or more processors may be configured to perform a union operation on the modified masks of all images in the sequence, and then apply the modified mask to each image, such as Figure 6 shown.

[0064] Image data pre-processing then proceeds as previously described, but using this additional input for subsequent calculations.

[0065] In various aspects, the exemplary embodiments of the present disclosure may be implemented in alternative forms.

[0066] The preprocessing model can replace or jointly use multiple analysis networks according to different task requirements, such as other detection, segmentation methods or tracking methods.

[0067] Features can be derived from the pre-processing model through outputs other than convolutional layers. Features from different layers can be extracted and utilized. At the same time, the gradients of backpropagation can also be applied to feature refinement.

[0068] Feature maps can also be derived from features through computational methods other than fusion, such as hand-crafted methods (e.g., maximum and average) and deep learning-based methods.

[0069] Furthermore, according to an exemplary embodiment, input image data from the image source 102 is adaptively input to the preprocessing model 202 or one of the block-based and feature-based blur preprocessing modules 404, and a segmentation operation is accordingly performed on the input image data to output an object mask, the input image data being configured by the preprocessing model 202 or the block-based preprocessing and feature-based blur preprocessing modules 404. Given the diversity of input video data, the above preprocessing methods may be used alone or in combination to perform block-based preprocessing and feature-based blur preprocessing to improve overall rate-distortion performance.

[0070] According to the preprocessing method described in this article, the block-based preprocessing and feature-based blur preprocessing module 404 stored on a non-transitory or non-temporary computer-readable storage medium of the input preprocessing computing system can configure one or more processors of the input preprocessing computing system to decide on the preprocessing of the video sequence based on the following two indicators: the average object mask ratio of the video sequence and the temporal complexity of the video sequence. The preprocessing decision of one or more processors configured by the block-based and feature-based blur preprocessing module 404 causes the input image data to be subsequently processed, including by the preprocessing model 202 in the input preprocessing 200 as described above, and / or by feature-based and block-based blur preprocessing. The above preprocessing can be used alone or in combination.

[0071] Temporal complexity is an indicator of temporal variation of a video obtained from the mean absolute difference (MAD) of the frames. The block-based and feature-based blur preprocessing modules 404 configure one or more processors of the input preprocessing computing system to calculate the temporal complexity as follows: Where l is the number of frames and k is the interval between frames. Larger temporal motion in a video indicates that the video content of the objects and background is more complex in the temporal domain. To reduce the computational complexity that increases with time, the block-based and feature-based blur pre-processing module 404 configures one or more processors of the input pre-processing computing system to perform a combination of feature-based and block-based blur pre-processing when the temporal complexity exceeds a complexity threshold, thereby removing the background surrounding the one or more objects while preserving the one or more objects.

[0072] The operation of combining feature-based and block-based fuzzy preprocessing includes at least the following steps: Figure 4 The input image shown is segmented to output an object mask, and then performed as described above. Figure 5 The mask-based image data removal and blur preprocessing shown applies an object mask to the input image and blurs the masked image.

[0073] The average object mask ratio of an image quantifies the average proportion of each object in the frame. To calculate the average object mask ratio, the preprocessing model 202 configures one or more processors of the input preprocessing computing system to obtain the number of objects O(I i ) and the corresponding mask ratio m(I i ), and then calculate the average:

[0074] The object count is the number of distinct object fragments in the object mask. The mask ratio is the ratio between the number of pixels of an object fragment and the number of pixels of the entire image and can be derived by dividing the number of pixels of the object fragment by the number of pixels of the image.

[0075] The calculated average object mask ratio is too small, indicating that the object occupies a smaller proportion of the entire frame than the background. In this case, the mask generated according to the preprocessing model 202 may not be completely accurate because the object is too small. Therefore, for average object mask ratios below the ratio threshold, the block-based and feature-based blur preprocessing module 404 configures one or more processors of the input preprocessing computing system to perform block-based blur preprocessing instead of feature-based blur preprocessing in order to preserve the background surrounding the object. In this way, the computational cost of encoding is controlled, and the background is preserved so that critical information is not distorted.

[0076] The complexity threshold and ratio threshold mentioned above can be described as matrices M thres and K thres As an example, M thres Set to 10, and K thresSet to 7%. In addition, the comparison for each threshold can be performed in the following order: First, if the time complexity M≥M thres , then feature-based and block-based blur preprocessing are combined. Otherwise, if the video has small objects K≤K thres , then block-based blur preprocessing is performed without performing feature-based blur preprocessing. For other results, the block-based and feature-based blur preprocessing module 404 configures one or more processors of the input preprocessing computing system to compress the video into a video directly configured by the VVC standard encoder and the VVC standard decoder.

[0077] It should be understood that the block-based and feature-based blur pre-processing module 404 can configure one or more processors of the input pre-processing computing system to decide on the pre-processing of the video sequence based on other traditional or deep learning-based indicator systems, such as object category, video length, and other features.

[0078] It should be understood that, as an alternative, the block-based and feature-based blur pre-processing module 404 can configure one or more processors of the input pre-processing computing system to perform feature-based blur pre-processing instead of block-based blur pre-processing, and to perform Gaussian blur pre-processing and / or other pre-processing techniques, alone or in combination.

[0079] It should be understood that the block-based and feature-based blur pre-processing module 404 can configure one or more processors of the input pre-processing computing system to determine the pre-processing method in different orders and based on different thresholds. For example, pre-processing can be performed in sequence according to the following thresholds: first, for video sequences with a large average object masking rate, feature-based blur pre-processing and block-based blur pre-processing are combined; otherwise, for video sequences with a small average object masking rate and low temporal complexity, block-based blur pre-processing is performed to preserve background information; otherwise, finally, video sequences with small objects and complex temporal domains are not processed.

[0080] Figure 7 An exemplary input pre-processing computing system 700 is shown for implementing the processes and methods described above for implementing the intermediate feature fusion extracted from the convolutional layers 206 of the pre-processing model 202 described above.

[0081] The techniques and mechanisms described herein may be implemented by multiple instances of the input pre-processing computing system 700 , as well as any other computing devices, systems, and / or environments. Figure 7The input pre-processing computing system 700 shown in the figure is only an example of a system and does not limit the scope of use or functionality of any computing device used to perform the above-mentioned processes and / or programs. Other well-known computing devices, systems, environments and / or configurations that may be suitable for the embodiments include, but are not limited to, personal computers, server computers, handheld or laptop devices, multi-processor systems, microprocessor-based systems, set-top boxes, game consoles, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and implementations using field programmable gate arrays (FPGAs) and application-specific integrated circuits (ASICs).

[0082] The input pre-processing computing system 700 may include one or more processors 702 and a system memory 704 communicatively coupled to the processors 702. The processors 702 and the system memory 704 may be physical or virtual and / or distributed. The processor(s) 702 may execute one or more modules and / or processes to enable the processors 702 to perform various functions. In embodiments, the processors 702 may include a CPU processing unit (CPU), a graphics processing unit (GPU), a CPU and a GPU, or other processing units or components known in the art. In addition, each processor 702 may have its own local memory, which may also store program modules, program data, and / or one or more operating systems.

[0083] Depending on the exact configuration and type of input pre-processing computing system 700, system memory 704 may be volatile, such as RAM, non-volatile, such as ROM, flash memory, a micro hard drive, a memory card, etc., or some combination thereof. System memory 704 may include one or more computer-executable modules 706 that can be executed by processor 702.

[0084] Module 706 may include, but is not limited to, a pre-processing module 708 , a fusion module 710 , and a block-based and feature-based blur pre-processing module 712 .

[0085] The pre-processing model 708 may configure one or more processors according to the pre-processing model 202 as described above.

[0086] The fusion module 710 may configure one or more processors according to the fusion module 402 described above.

[0087] The block-based and feature-based fuzzy pre-processing module 712 may configure one or more processors according to the block-based and feature-based fuzzy pre-processing module 404 as described above.

[0088] The input pre-processing computing system 700 may also include an input / output (I / O) interface 740, and a communication module 750 that allows the input pre-processing computing system 700 to communicate with other systems and devices over a network. The network may include the Internet, wired media such as a wired network or a direct wired connection, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0089] Some or all of the operations of the above methods may be performed by executing computer-readable instructions stored on a computer-readable storage medium, as defined below. As used in the specification and claims, the term computer-readable instructions includes routines, applications, application modules, program modules, programs, components, data structures, algorithms, and the like. Computer-readable instructions may be implemented on a variety of system configurations, including single-processor or multi-processor systems, minicomputers, mainframe computers, personal computers, handheld computing devices, microprocessor-based programmable consumer electronic devices, and combinations thereof.

[0090] Computer-readable storage media may include volatile memory (such as random access memory (RAM)) and / or non-volatile memory (such as read-only memory (ROM), flash memory, etc.). Computer-readable storage media may also include additional removable and / or non-removable storage, including but not limited to flash memory, magnetic storage, optical storage, and / or tape storage, which may provide non-volatile storage of computer-readable instructions, data structures, program modules, etc.

[0091] Non-transitory or non-temporary computer-readable storage media are examples of computer-readable media. Computer-readable media include at least two types of computer-readable media: computer-readable storage media and communication media. Computer-readable storage media include volatile and non-volatile, removable and non-removable media implemented by any process or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessed by a computing device. In contrast, communication media can embody computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transmission mechanism. As defined herein, computer-readable storage media does not include communication media.

[0092] When executed by one or more processors, the computer readable instructions stored on one or more non-transitory computer readable storage media may perform the above-mentioned Figures 1 to 6 The operations described. Generally speaking, computer-readable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations can be combined in any order and / or in parallel to implement the process.

[0093] Figure 8 An example encoding and decoding computing system 800 is shown for implementing the above-described process and method for implementing the intra template matching prediction mode.

[0094] The techniques and mechanisms described herein may be implemented by multiple instances of the encoding and decoding computing system 800 , as well as any other computing devices, systems, and / or environments. Figure 8 The system 800 shown is only an example of a system and is not intended to limit the scope of use or functionality of any computing device used to perform the above processes and / or programs. Other well-known computing devices, systems, environments, and / or configurations that may be suitable for the embodiments include, but are not limited to, personal computers, server computers, handheld or laptop devices, multi-processor systems, microprocessor-based systems, set-top boxes, game consoles, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and implementations using field programmable gate arrays (FPGAs) and application-specific integrated circuits (ASICs).

[0095] The encoding and decoding computing system 800 may include one or more processors 802 and a system memory 804 communicatively coupled to the one or more processors 802. The processor 802 may execute one or more modules and / or processes to enable the encoding and decoding computing processor 802 to perform various functions. In some embodiments, the processor 802 may include a CPU processing unit (CPU), a graphics processing unit (GPU), a CPU and a GPU, or other processing units or components known in the art. In addition, each processor 802 may have its own local memory, which may also store program modules, program data, and / or one or more operating systems.

[0096] Depending on the exact configuration and type of the encoding and decoding computing system 800, the system memory 804 can be volatile, such as RAM, non-volatile, such as ROM, flash memory, a micro hard drive, a memory card, etc., or some combination thereof. The system memory 804 can include one or more computer-executable modules 806 that can be executed by the processor 802.

[0097] Module 806 may include, but is not limited to, one or more of an encoder 808 and a decoder 810 .

[0098] The encoder 808 may be a VVC standard encoder implementing exemplary embodiments of any one, some, or all aspects of the present disclosure as described above, and may be executed by the processor(s) 802 to configure the processor 802 to perform the operations described above.

[0099] The decoder 810 may be a VVC standard encoder implemented according to any, some or all aspects of the exemplary embodiments of the present disclosure described above, which may be executed by the processor(s) 802 to configure the processor 802 to perform the operations described above.

[0100] The encoding and decoding computing system 800 may also include an input / output (I / O) interface 840 for receiving image source data and bitstream data, and for outputting reconstructed images to a reference image buffer and / or a display buffer. The encoding and decoding computing system 800 may also include a communication module 850 that allows the encoding and decoding computing system 800 to communicate with other devices (not shown) via a network (not shown). The network may include the Internet, wired media such as a wired network or a direct wired connection, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0101] In some embodiments, a method is provided, comprising: performing, by one or more processors of the input pre-processing computing system, a plurality of convolution operations on the input image data; Outputting, by the one or more processors, a plurality of intermediate feature maps from the respective different convolution operations; averaging, by the one or more processors, absolute values ​​of the plurality of intermediate feature maps to generate a fused feature map; and The fused feature map is resized to the size of the input image data.

[0102] In one implementation, the method further includes performing, by the one or more processors, a Gaussian blur transform on pixels of the input image data, wherein the Gaussian blur transform takes as input a feature map value corresponding to the pixel in the fused feature map.

[0103] In one implementation, the one or more processors calculate a feature map value corresponding to the pixel as a standard deviation value in the Gaussian blur transformation.

[0104] In one implementation, the one or more processors perform a Gaussian blur transform on each pixel of the input image data.

[0105] In one implementation, the plurality of convolution operations are performed by the one or more processors in the process of performing a segmentation calculation on the input image data to output an object mask.

[0106] In one implementation, the method further includes modifying, by the one or more processors, the object mask to exclude pixels of the sliding window.

[0107] In one implementation, the method further includes multiplying, by the one or more processors, the modified object mask and the input image data to output block-based masked input image data.

[0108] In one implementation, the one or more processors perform a Gaussian blur transform on the block-based masked input image data.

[0109] In one implementation, the method further includes: based on an average object mask ratio of a video sequence exceeding a threshold, the one or more processors determining to perform a Gaussian blur transform on the block-based masked input image data.

[0110] In one implementation, the method further includes: based on a video sequence temporal complexity of a video sequence exceeding a threshold, the one or more processors determining to perform a Gaussian blur transform on the block-based masked input image data.

[0111] In some embodiments, a computing system is provided, comprising: an execution module configured to perform a plurality of convolution operations on the input image data; an output module configured to output a plurality of intermediate feature maps from the above respective convolution operations; an averaging module configured to average the absolute values ​​of the plurality of intermediate feature maps to generate a fused feature map; and A resizing module is configured to resize the fused feature map to the size of the input image data.

[0112] In one implementation, the computing system is characterized in that the execution module is further configured to blur the input image data by performing a Gaussian blur transform on the pixels of the input image data, and the Gaussian blur transform takes the feature map value corresponding to the pixel in the fused feature map as input.

[0113] In one implementation, the computing system further includes: A calculation module is configured to calculate the feature map value corresponding to the pixel as the standard deviation value in the Gaussian blur transformation.

[0114] In one implementation, the execution module is further configured to perform a Gaussian blur transformation on each pixel of the input image data.

[0115] In one implementation, the execution module is further configured to perform the multiple convolution operations in the process of performing a segmentation calculation on the input image data to output an object mask.

[0116] In one implementation, the computing system further includes: A modification module is used to modify the object mask to exclude each pixel of the sliding window.

[0117] In one implementation, the computing system further includes: A multiplication module is configured to multiply the modified object mask with the input image data to output block-based masked input image data.

[0118] In one implementation, the execution module is further configured to perform a Gaussian blur transform on the block-based masked input image data.

[0119] In one implementation, the computing system further includes: A decision module is configured to decide to perform a Gaussian blur transform on the block-based masked input image data based on an average object mask ratio of the video sequence exceeding a threshold.

[0120] In one implementation, the computing system further includes: The decision module is configured to decide to perform Gaussian blur transformation on the block-based masked input image data based on a temporal complexity of the video sequence exceeding a threshold.

[0121] In some embodiments, a computing system is provided, comprising: one or more processors, and A computer-readable storage medium is communicatively coupled to the one or more processors, the computer-readable storage medium storing computer-readable instructions executable by the one or more processors, wherein the computer-readable instructions, when executed by the one or more processors, perform the method described above.

[0122] In some embodiments, a non-transitory computer-readable storage medium is provided, on which computer-executable instructions are stored. When the processor executes the computer-executable instructions, the method described above is implemented.

[0123] In some embodiments, a computer program product is provided, comprising computer program instructions, which enable a computer to execute the method described above.

[0124] In some embodiments, a computer program is provided, which causes a computer to execute the method described above.

[0125] Some or all of the operations of the above methods may be performed by executing computer-readable instructions stored on a computer-readable storage medium, as defined below. The term computer-readable instructions, as used in the specification and claims, includes routines, applications, application modules, program modules, programs, components, data structures, algorithms, and the like. Computer-readable instructions may be implemented on a variety of system configurations, including single-processor or multi-processor systems, minicomputers, mainframe computers, personal computers, handheld computing devices, microprocessor-based programmable consumer electronic devices, combinations thereof, and the like.

[0126] Computer-readable storage media may include volatile memory (such as random access memory (RAM)) and / or non-volatile memory (such as read-only memory (ROM), flash memory, etc.). Computer-readable storage media may also include additional removable and / or non-removable storage, including but not limited to flash memory, magnetic storage, optical storage, and / or tape storage, which may provide non-volatile storage of computer-readable instructions, data structures, program modules, etc.

[0127] Non-transitory or non-temporary computer-readable storage media are examples of computer-readable media. Computer-readable media include at least two types of computer-readable media: computer-readable storage media and communication media. Computer-readable storage media include volatile and non-volatile, removable and non-removable media implemented in any process or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc-read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessed by a computing device. In contrast, communication media can embody computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transmission mechanism. The computer-readable storage medium used herein should not be interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (such as light pulses through a fiber optic cable), or an electrical signal propagating through a wire.

[0128] When executed by one or more processors, the computer readable instructions stored on one or more non-transitory or non-transitory computer readable storage media may perform the above-mentioned Figure 1 and Figure 2 The operations described, generally, include computer-readable instructions that include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific abstract data types. The order in which the operations are described should not be construed as a limitation, and any number of the described operations can be combined in any order and / or in parallel to implement the process.

[0129] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts disclosed are example forms of implementing the claims.

Claims

1. A method comprising: performing, by one or more processors of the input pre-processing computing system, a plurality of convolution operations on the input image data; Outputting, by the one or more processors, a plurality of intermediate feature maps from the respective different convolution operations; averaging, by the one or more processors, absolute values ​​of the plurality of intermediate feature maps to generate a fused feature map; as well as The fused feature map is resized to the size of the input image data.

2. The method according to claim 1 further includes performing a Gaussian blur transform on the pixels of the input image data by the one or more processors, wherein the Gaussian blur transform takes the feature map value corresponding to the pixel in the fused feature map as input.

3. The method according to claim 2, characterized in that The one or more processors calculate a feature map value corresponding to the pixel as a standard deviation value in the Gaussian blur transformation.

4. The method according to claim 2, wherein: The one or more processors perform a Gaussian blur transform on each pixel of the input image data.

5. The method according to claim 2, wherein: The plurality of convolution operations are performed by the one or more processors in performing a segmentation calculation on input image data to output an object mask. 6 . The method of claim 5 , further comprising modifying, by the one or more processors, the object mask to exclude individual pixels of a sliding window. 7 . The method of claim 6 , further comprising multiplying, by the one or more processors, the modified object mask and the input image data to output block-based masked input image data.

8. The method according to claim 7, characterized in that The one or more processors perform a Gaussian blur transform on the block-based masked input image data.

9. The method according to claim 8, further comprising: The one or more processors determine to perform a Gaussian blur transform on the block-based masked input image data based on an average object mask ratio of a video sequence exceeding a threshold.

10. The method according to claim 8, further comprising: The one or more processors determine to perform a Gaussian blur transform on the block-based masked input image data based on a video sequence temporal complexity of a video sequence exceeding a threshold.

11. A computing system comprising: an execution module configured to perform a plurality of convolution operations on the input image data; an output module configured to output a plurality of intermediate feature maps from the above respective convolution operations; an averaging module configured to average the absolute values ​​of the plurality of intermediate feature maps to generate a fused feature map; as well as A resizing module is configured to resize the fused feature map to the size of the input image data.

12. The computing system according to claim 11, wherein: The execution module is further configured to perform blur processing on the input image data by performing Gaussian blur transformation on the pixels of the input image data, and the Gaussian blur transformation takes the feature map value corresponding to the pixel in the fused feature map as input.

13. The computing system of claim 12, further comprising: A calculation module is configured to calculate the feature map value corresponding to the pixel as the standard deviation value in the Gaussian blur transformation. 14 . The computing system of claim 12 , wherein the execution module is further configured to perform a Gaussian blur transform on each pixel of the input image data. 15 . The computing system of claim 12 , wherein the execution module is further configured to perform the plurality of convolution operations in a process of performing a segmentation calculation on the input image data to output an object mask.

16. The computing system of claim 15, further comprising: A modification module is used to modify the object mask to exclude each pixel of the sliding window.

17. The computing system of claim 16, further comprising: A multiplication module is configured to multiply the modified object mask with the input image data to output block-based masked input image data.

18. The computing system according to claim 17, wherein: The execution module is further configured to perform a Gaussian blur transform on the block-based masked input image data.

19. The computing system of claim 18, further comprising: A decision module is configured to decide to perform a Gaussian blur transform on the block-based masked input image data based on an average object mask ratio of the video sequence exceeding a threshold.

20. The computing system of claim 18, further comprising: The decision module is configured to decide to perform Gaussian blur transformation on the block-based masked input image data based on a temporal complexity of the video sequence exceeding a threshold.

21. A computing system comprising: one or more processors, and A computer-readable storage medium communicatively coupled to the one or more processors, the computer-readable storage medium storing computer-readable instructions executable by the one or more processors, the computer-readable instructions, when executed by the one or more processors, performing the method of any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium having computer-executable instructions stored thereon, wherein when the processor executes the computer-executable instructions, the method according to any one of claims 1 to 10 is implemented.

23. A computer program product comprising computer program instructions, characterized in that The computer program instructions cause a computer to execute the method according to any one of claims 1 to 10.

24. A computer program, characterized in that The computer program causes a computer to execute the method according to any one of claims 1 to 10.

Images