Feature fusion for preprocessing input picture data for learning models

Abstract

The method and system performs input picture data preprocessing of a learning model by deep feature-based picture data blurring. Intermediate features are extracted from a convolutional layer of the preprocessing model, and each set of intermediate features is fused to generate a fused feature map, which is scaled to the size of the input picture. Based on the fused feature map, the preprocessing model can configure one or more processors of an input preprocessing computation system to emphasize picture data with larger corresponding feature values ​​while weakening other picture data when performing blur preprocessing computation.

Description

[Technical Field]

[0001] [Related Applications] This application claims the benefit of U.S. patent application Ser. No. 63 / 436,817, entitled "Feature Fusion for Input Picture Data Preprocessing of Learning Models," filed on January 3, 2023, and claims the benefit of U.S. patent application Ser. No. 63 / 492,205, entitled "Feature Fusion for Input Picture Data Preprocessing of Learning Models," filed on March 24, 2023, and claims the benefit of U.S. patent application Ser. No. 18 / 400,954, entitled "Feature Fusion for Input Picture Data Preprocessing of Learning Models," filed on December 29, 2023. All three of these applications are expressly incorporated herein by reference in their entireties.

[0002] The present disclosure relates generally to input picture data preprocessing, and more particularly to a method and system for implementing input picture data preprocessing for a learning model with deep feature-based picture data blurring. [Background technology]

[0003] Current image coding technologies, such as H.264 / AVC (Advanced Video Coding), H.265 / HEVC (High Efficiency Video Coding), and Versatile Video Coding (VVC), are primarily based on lossy compression, a framework that includes transform coding, quantization, and entropy coding. For several years, lossy compression has achieved compression ratios suitable for image capture and storage on a limited scale. The main task of these existing codecs is to achieve better reconstructed signal quality under limited bitrate constraints.

[0004] Based on machine learning, end-to-end image compression techniques have been developed in which parameters of a nonlinear transform are learned by training a deep neural network on image and video datasets, and the nonlinear transform configures a computational system to map an input picture to its latent representation in a latent space, and then entropy coding techniques are applied to the latent representation of the image to improve computational efficiency.

[0005] Furthermore, with the emergence of fields such as computer vision and machine vision, computer systems are being configured to capture and store images on a larger scale. Driven by large image and video datasets, machine learning and deep learning configure computer systems to perform new tasks, and machine learning image and video datasets also rely on image compression to improve the efficiency of data storage.

[0006] However, the images and videos input to machine learning models do not have the same signal quality requirements emphasized by existing lossy compression codecs, nor do they have the same optimization requirements emphasized by end-to-end image compression techniques. Achieving compression and computational efficiency while prioritizing the image data used by machine learning models for computation requires techniques that go beyond both traditional lossy compression codecs and end-to-end image compression techniques. Summary of the Invention

[0007] In a first aspect, an embodiment of the present disclosure provides a method, the method including: performing, by one or more processors of an input pre-processing computing system, a plurality of convolutions on input picture data; outputting, by the one or more processors, a plurality of intermediate feature maps from corresponding different convolutions; averaging, by the one or more processors, absolute values ​​of the plurality of intermediate feature maps to generate a fused feature map; and resizing, by the one or more processors, the fused feature map to a size of the input picture data.

[0008] In a second aspect, an embodiment of the present disclosure provides a computing system including an execution module configured to perform a plurality of convolutions on input picture data, an output module configured to output a plurality of intermediate feature maps from corresponding different convolutions, an averaging module configured to average absolute values ​​of the plurality of intermediate feature maps to generate a fused feature map, and a resizing module configured to resize the fused feature map to a size of the input picture data.

[0009] In a third aspect, embodiments of the present disclosure provide a computing system including one or more processors; and a computer-readable storage medium communicatively coupled to the one or more processors, the computer-readable storage medium storing computer-readable instructions executable by the one or more processors that, when executed by the one or more processors, perform a computing method according to the first aspect.

[0010] In a fourth aspect, embodiments of the present disclosure provide a non-transitory computer-readable storage medium having stored thereon computer-executable instructions that, when executed by the processor, perform a method according to the first aspect.

[0011] In a fifth aspect, an embodiment of the present disclosure provides a computer program product comprising computer program instructions that cause a computer to perform a method according to the first aspect.

[0012] In a sixth aspect, an embodiment of the present disclosure provides a computer program causing a computer to carry out the method according to the first aspect. [Brief explanation of the drawings]

[0013] The detailed description will be set forth with reference to the accompanying drawings, in which the left-most digit(s) of a reference number refers to the drawing in which the reference number first appears. The use of the same reference number in different drawings indicates similar or identical items or features.

[0014] [Figure 1] FIG. 1 shows a flow chart of an image compression process performed by one or more processors of a computing system configured with a VVC standard encoder and one or more processors of a computing system configured with a VVC standard decoder.

[0015] [Figure 2] FIG. 2 shows a flowchart of the input pre-processing output to the image compression process of FIG.

[0016] [Figure 3] FIG. 3 illustrates a mask-based picture data removal and blur preprocessing technique.

[0017] [Figure 4] FIG. 4 illustrates a flowchart of intermediate feature extraction and fusion from the convolutional layer of the preprocessing model according to an exemplary embodiment of the present disclosure.

[0018] [Figure 5] FIG. 5 illustrates a feature-based blur pre-processing technique according to an exemplary embodiment of the present disclosure.

[0019] [Figure 6] FIG. 6 illustrates mask association in a picture sequence.

[0020] [Figure 7] FIG. 7 illustrates an exemplary input preprocessing computation system for implementing the processes and methods described herein for performing intermediate feature fusion extracted from the convolutional layers of the preprocessing model.

[0021] [Figure 8]FIG. 8 illustrates an exemplary encoding and decoding computing system for implementing the processes and methods described herein for implementing a VVC standard encoder and decoder. DETAILED DESCRIPTION OF THE INVENTION

[0022] The systems and methods discussed herein relate to performing data preprocessing for learning models, and more particularly, to performing picture preprocessing on image and video datasets for model training based on intermediate feature fusion extracted from convolutional layers of the preprocessing model.

[0023] According to exemplary embodiments of the present disclosure, a learning model includes at least computer-readable instructions executable by one or more processors of a computing system to perform a computational task, the computational task including calculating inputs based on values ​​of various parameters and outputting results. The learning model may be, for example, a multi-layer model such as a deep neural network, and may have a fully connected architecture, a feed-forward architecture such as a convolutional neural network (CNN), a back-propagation architecture such as a recurrent neural network (RNN), or other architecture suitable for computing a particular computational task. For example, the computational task may include classification, clustering, matching, regression, etc.

[0024] One or more processors of the computing system may be further configured to perform inference computations based on the output of these computational tasks to generate inference results related to solving various problems, such as recognizing entities in images and / or videos, tracking the movement of entities across multiple video frames, matching entities recognized in images and / or videos with other images and / or videos, and providing annotations or transcriptions of images, videos and / or audio in real time.

[0025] A computing system that trains a learning model or computes inferences based on the trained learning model can communicate with one or more input devices configured to capture data to be input to the learning model, perform computations based on the learning model in connection with various tasks, and analyze and output results necessary to perform those tasks. The input devices can store the captured data on a non-transitory computer-readable medium, which may be a component of the input processing computing system. The input devices may themselves be components of the input processing computing system or may be devices external to the input processing computing system. The input processing computing system may be the same as or a different computing system from the computing system that trains the learning model or computes inferences based on the trained learning model. If they are different computing systems, they can communicate via an appropriate connection, such as a wired or wireless network connection.

[0026] The input devices may communicate with one or more processors of the input processing computing system via a data bus connection, a wired network connection, a wireless network connection, etc. The input devices may be configured to transmit captured data to the input processing computing system, and the input processing computing system is configured to write the captured data to a storage device including one or more non-transitory computer-readable media.

[0027] The input device may be a video camera that collects still images, video, or other types of picture data. By way of example, such a video camera may be a standalone camera, a peripheral device connected to a computing system, one of any number of cameras integrated into electronic equipment, mechanical equipment, motorized vehicles, and such electronic systems incorporating one or more processors and computational storage, or may be configured to communicate with a computing system in any other suitable manner.

[0028] Whether the computing system is training a learning model or computing inferences based on the trained learning model, it can be configured to perform computations based on the learning model on captured picture data. Because the file sizes of picture datasets used in deep learning are very large, storing the picture datasets can occupy a large amount of storage space, and loading and computing the picture datasets can result in a large amount of computational overhead.

[0029] Furthermore, in a computing architecture in which a cloud computing system is used to train a learning model or compute inferences based on a trained learning model, the large amount of data captured and stored locally at the input side can cause unbearable latency when delivered to the cloud computing system via a network connection. Additionally, images in raw, uncompressed format are highly inefficient for machine learning computations because they typically contain many times more data than compressed images for the purposes of machine learning training and inference. As a result, it is desirable to compress images captured at an input device before using them in training and inference datasets.

[0030] General-purpose, non-machine learning image compression techniques are the default solution in video communications, embodied by video coding standards such as High Efficiency Video Coding (HEVC) and Versatile Video Coding (VVC). These standards employ a hybrid coding framework by dividing frames into variable- or fixed-size blocks. Additionally, end-to-end image compression has been developed as an alternative, in which an image is transformed into its latent representation based on trained deep neural networks, which also aim to output perceptually meaningful representations.

[0031] proposed an end-to-end image compression framework using generalized decomposition normalization (GDN) and optimized the entire framework using rate-distortion optimization (RDO). They also proposed the concept of hyperpriors to represent spatial dependencies in the latent representation. Cui et al. proposed a new autoencoder called gain variational autoencoder (G-VAE) based on continuous rate control, in which a pair of gain units is incorporated into the end-to-end image compression framework to achieve continuous variable-rate compression without increasing network parameters and computational cost. Choi et al. proposed a new variable-rate image compression framework and conditional autoencoder that employs conditional variation and universal quantification structures. Deep learning-based end-to-end compression frameworks, namely the Deep Video Compression Framework (DVC) and Multi-Frame Prediction for Learning Video Compression (M-LVC), have also been developed.

[0032] Known proposals for image coding for machine vision include visual signal compression and compact feature representation. For visual signal compression, Zhang et al. proposed a Learned Image Compression (LIC) framework and proposed a multi-scale progressive (MSP) probabilistic model for lossy image compression based on spatial and channel correlations of the latent representation of a picture.

[0033] Feature compression is hindered by the following factors: Specifically, models are usually tailored for specific tasks, and top-level features are highly task-related, making generalization difficult. proposed mid-layer feature compression, in which mid-layer features are compressed and transmitted instead of the original video or top-layer features. End-to-end learning typically forces deep features to have larger receptive fields and more task-specific features. Therefore, compared to deep features, features from shallow layers generally contain more information.

[0034] 1 shows a flowchart of an image compression process 100 performed by one or more processors of a computing system comprising an encoder and one or more processors of a computing system comprising a decoder. As described above, the image compression process 100 encompasses at least general-purpose, non-machine learning image compression techniques and machine vision image coding.

[0035] In accordance with the VVC video coding standard ("VVC standard") and such general-purpose image compression techniques described therein, a computing system includes at least one or more processors and a computer-readable storage medium communicatively coupled to the one or more processors. As defined below with reference to FIG. 8, the computer-readable storage medium is a non-transitory or non-transitory computer-readable storage medium that stores computer-readable instructions. At least some of the computer-readable instructions stored on the computer-readable storage medium are executable by one or more processors of the computing system to configure the one or more processors to perform associated operations of the computer-readable instructions, including at least the operations of an encoder described in the VVC standard and the operations of a decoder described in the VVC standard. Some of these encoder and decoder operations according to the VVC standard are described in further detail below, although these subsequent descriptions should not be construed as an exhaustive list of encoder and decoder operations according to the VVC standard. Thereafter, "VVC Standard Encoder" and "VVC Standard Decoder" describe corresponding computer-readable instructions stored on a computer-readable storage medium that configure one or more processors to perform these corresponding operations (which may, for example, be referred to as "reference implementations" of the encoder or decoder).

[0036] Additionally, according to exemplary embodiments of the present disclosure, the VVC standard encoder and VVC standard decoder further include computer-readable instructions stored on a computer-readable storage medium and executable by one or more processors of a computing system to configure the one or more processors to perform operations not specified by the VVC standard. The VVC standard encoder should not be understood as limited to the operation of the reference encoder embodiment, but rather includes additional computer-readable instructions to configure one or more processors of a computing system to perform additional operations described herein. The VVC standard decoder should not be understood as limited to the operation of the reference decoder embodiment, but rather includes additional computer-readable instructions to configure one or more processors of a computing system to perform additional operations described herein.

[0037] In the encoding process, the VVC standard encoder 104 configures one or more processors of the computing system to receive as input one or more input pictures from the image source 102. By way of example, the VVC standard encoder 104 encodes a picture (the picture being encoded is referred to as the "current picture," to distinguish it from any other pictures received from the image source 102) by configuring one or more processors of the computing system to divide the original picture into units and sub-units according to a partitioning structure. Furthermore, the VVC standard encoder 104 encodes a picture by configuring one or more processors of the computing system to perform motion estimation on blocks of the current picture.

[0038] After performing various calculations on the blocks of the current picture, the VVC standard encoder 104 configures one or more processors of the computing system to output an encoded picture consisting of the encoded blocks from the entropy coder. The encoded picture is output to a transmission buffer where it is ultimately packed into a bitstream 106 for output from the VVC standard encoder 104. The bitstream 106 is written by one or more processors of the computing system to a non-transient or non-temporary computer-readable storage medium of the computing system for transmission.

[0039] The VVC standard decoder 108 configures one or more processors of the computing system to receive as input one or more coded pictures from the bitstream 106. The VVC standard decoder 108 configures one or more processors of the computing system to perform intra prediction using prediction information specified in a coding parameter set, and configures one or more processors of the computing system to perform motion compensated prediction using reference pictures from a decoded picture buffer, and generates a prediction signal.

[0040] After performing various calculations on the predicted signal, the VVC standard decoder 108 configures one or more processors of the computing system to output a reconstructed picture 110 to the input layer of the learning model 112 for training and / or inference.

[0041] Additionally, both standard image compression codecs and end-to-end learning-based compression proposals can further benefit from preprocessing techniques in which one or more processors of a computing system are configured to compute the raw picture data before the raw picture data is compressed.

[0042] Figure 2 shows a flowchart of input pre-processing 200 output to the image compression process 100 of Figure 1. Similarly, as noted above, the image compression process 100 encompasses at least general-purpose, non-machine learning image compression techniques and machine vision image coding.

[0043] In input pre-processing 200, a pre-processing model 202 configures one or more processors of an input processing computing system (such as those described above) to receive input picture data from an image source 102. The pre-processing model 202 configures one or more processors to perform various pre-processing calculations on the input picture data and outputs the processed picture data, still uncompressed, to one or more processors of the same computing system configured by the VVC standard encoder 104 or to another computing system.

[0044] The input picture data may have been captured in raw image format by an input device, which may be, for example, a video camera, such as a stand-alone camera, a peripheral device, or a camera integrated into another electronic system, as described above. The input device may capture picture data constituting an image dataset in the form of still images or video.

[0045] The input device may be connected to the input processing computing system by a data bus connection, an optical data connection, or other connection suitable for transmitting images, or by a wired or wireless network connection. For example, the input processing computing system may be a personal computing system, a computing system cluster, a server of a cloud computing system such as an edge server, an embedded computing system in a smart device such as a vehicle or a home appliance, etc.

[0046] For example, the preprocessing model 202 may configure one or more processors of the input processing computing system as follows: Wang et al. proposed a preprocessing technique that obtains a mask of an important object ("object mask") in picture data by segmentation, and then removes part of the background according to the object mask and blurs the remaining part of the background. With such a technique, less important picture data is weakened by removal or blurring.

[0047] Figure 3 shows a mask-based picture data removal and blur preprocessing technique. By the above-mentioned division, a merged mask M is obtained. The merged mask M divides the picture data into at least an object segment (shown in white in Figure 3) and a background segment (shown in black in Figure 3), and matches as closely as possible to the actual contours of the objects depicted in the picture data.

[0048] However, if M is directly used to perform picture data removal and picture data blur calculation, it may cause over-processing of the area immediately surrounding the object.To improve picture data removal and picture data blur to achieve better discrimination in machine vision tasks, a block-based blur pre-processing operation is further proposed to preserve the visual information around the object / instance.

[0049] As an example, a sliding window sn of size n×n pixels is applied to the input picture data I and the merged mask M, where n is determined based on the resolution of the input picture data I. If there is an intersection between the sliding window sn and the merged mask M, the merged mask is modified to exclude each pixel of the sliding window from the masked background and preserve samples of the input picture data I at these pixels. The modified block-based mask is denoted as MB, and the block-based masked input picture data is obtained by point-wise multiplication between MB and I, and is denoted as IBlock.

[0050] Furthermore, blurring can improve the smoothness of input data, resulting in blurred signals that are easier to compress, but excessive blurring in object regions can reduce the distinction between different objects. Conversely, inputting excessively blurred picture data can lead to poor performance in inference calculations based on learning models. To improve the tradeoff between representation expansion and machine performance, blur-based preprocessing has been further proposed to blur non-object regions. A Gaussian blur transform is applied to the block-based masked input picture data. The formula for the output Iblur of Gaussian blur preprocessing is:

number

[0051] Despite these proposals, input picture data preprocessing based on picture data blur still suffers from limitations. One or more processors in a computing system may be configured to perform picture data blur calculations based on the same parameters for the entire background region of all visual data. This type of blur calculation cannot adapt to diverse visual content. Depending on the problem that the inference results must solve, inference calculations based on a learning model may depend on some, but not all, of the background of the picture data. This means that preprocessing does not equally weaken all parts of the background.

[0052] As a result, an exemplary embodiment of the present disclosure provides input picture data preprocessing for a learning model performed by picture data blurring based on deep features. According to the exemplary embodiment of the present disclosure, intermediate features are extracted from a convolutional layer of a preprocessing model, and each set of intermediate features is fused to generate a fused feature map, which is expanded to the size of the input picture. Based on the fused feature map, the preprocessing model can configure one or more processors of an input preprocessing calculation system to emphasize picture data with larger corresponding feature values ​​while weakening other picture data when performing blur preprocessing calculation.

[0053] 4 shows a flowchart of intermediate feature extraction and fusion from a convolutional layer of a preprocessing model according to an exemplary embodiment of the present disclosure. As shown herein, input picture data from an image source 102 is computed by one or more processors of an input preprocessing computing system configured by a preprocessing model 202, and object masks obtained by segmenting the input picture data are output by the one or more processors.

[0054] The preprocessing model 202 may be a learning model, such as a Mask R-CNN learning model, trained to configure one or more processors of the input preprocessing computing system to perform segmentation computations on input picture data. Segmentation refers to dividing the input picture data into several differently labeled segments, with boundaries between the segments. Each segment may convey a different semantic meaning from at least some of the other segments. The input picture data may be segmented based on its aspects, such as similarities between picture data samples, differences between picture data samples, boundaries between picture data samples, and distinguishing between foreground and background samples. It should be understood that segmenting objects in picture data samples emphasizes segmented samples over background samples and weakens background samples.

[0055] The preprocessing model 202 includes multiple convolutional layers 206. Each convolutional layer 206 receives picture data from the previous layer, performs convolution on the picture data, and generates a feature map from the picture data. The feature map is generated by applying a filter to the picture data and outputting multiple channels to perform a matrix operation to perform convolution on the picture data. For example, according to the Mask R-CNN learning model, the multiple convolutional layers 206 can be arranged according to a feature pyramid network (FPN).

[0056] Each convolutional layer 206 of the preprocessing model 202 may perform strided convolutions and / or pooling convolutions on the picture data. Such convolutions downsample the picture data, reducing the size and resolution of the picture data. Each such convolution produces intermediate features that may differ from the intermediate features produced by another convolution in another convolutional layer 206. Thus, features extracted from a convolution performed on the picture data may have a smaller size and resolution than the input picture data, features extracted from different convolutions may have different sizes and resolutions, and features extracted from different convolutions may have different numbers of channels.

[0057] The preprocessing model 202 may branch into a stack of more than one convolutional layer 206, and at least one branch of the preprocessing model 202 configures one or more processors of the input preprocessing computing system to output an object mask, the object mask including one or more segments that distinguish a foreground object in the picture data from a background of the picture data.

[0058] Furthermore, according to an exemplary embodiment of the present disclosure, a fusion module 402 stored in a non-transient or non-transient computer-readable storage medium of the input pre-processing computing system may configure one or more processors of the input pre-processing computing system to receive intermediate feature maps F1, F2, ..., FN output by the N different convolutional layers 206 prior to output from one or more branches of the pre-processing model 202. The fusion module 402 further configures the one or more processors to fuse the intermediate feature maps F1, F2, ..., FN output by the multiple convolutional layers 206 into a fused feature map FP.

[0059] The one or more processors of the input pre-processing computing system are configured to generate a fused feature map of single-channel features by averaging absolute values ​​of features of different feature maps F1, F2, ..., FN, then normalize the single-channel features, and resize the fused feature map to the same size as the input picture data, as follows:

number

[0060] One or more processors of the input pre-processing computation system are configured to output the resized fused feature map as input to a blur pre-processing operation. The block-based and feature-based blur pre-processing module 404 configures one or more processors of the input pre-processing computation system to perform mask-based picture data removal and blur pre-processing, as described below with reference to FIG.

[0061] After the feature maps F1, F2, ..., FN (for brevity, denoted as F herein) generate the fused feature map FP, one or more processors of the input pre-processing computing system perform a Gaussian blur transform on each pixel of the input picture data I, where the Gaussian blur transform takes as input the feature map value corresponding to the pixel from the fused feature map FP. As an example, the feature map value corresponding to a pixel can be calculated as the standard deviation value in the Gaussian blur transform.

number

[0062] Additionally, the block-based and feature-based blur preprocessing module 404 configures one or more processors of the input preprocessing computing system to perform block-based blur preprocessing, as described above with reference to Figure 3. Thus, the one or more processors may be configured to perform a Gaussian blur transform on each pixel of the input picture data based on feature map values ​​(i.e., perform feature-based rather than block-based blur preprocessing), the one or more processors may be configured to perform a Gaussian blur transform on block-based masked pixels of the input picture data (i.e., perform block-based rather than feature-based blur preprocessing), and the one or more processors may be configured to perform a Gaussian blur transform on block-based masked pixels of the input feature data based on feature map values ​​(i.e., perform block-based and feature-based blur preprocessing).

[0063] Also according to an exemplary embodiment of the present disclosure, the block-based and feature-based blur preprocessing module 404 configures one or more processors of the input preprocessing computation system to perform joint computation on the modified masks of the picture sequence. In a picture sequence of a video sequence, block-based masking can reduce visual continuity due to the appearance and disappearance of visible blocks in consecutive pictures. To improve the visual continuity of the picture sequence, the one or more processors can be configured to join the modified masks of all pictures in the sequence and apply the modified mask to each picture, as shown in FIG. 6.

[0064] Picture data pre-processing then proceeds as before, but with this additional input used for subsequent calculations.

[0065] In various respects, the exemplary embodiments of the present disclosure can be practiced alternatively.

[0066] The preprocessing model can replace or cooperate with multiple analysis networks, such as other detection and segmentation methods and tracking methods, depending on different task requirements.

[0067] The output of non-convolutional layers can provide features from preprocessing models. Features from different layers can be extracted and utilized. At the same time, backpropagation gradients can be applied to feature refinement.

[0068] Feature maps can also be obtained from features by computational methods different from fusion, such as manually designed methods (maximum and average values) or deep learning-based methods.

[0069] Also, according to an exemplary embodiment, input picture data from the image source 102 is adaptively input to either one of the preprocessing model 202 or the block-based and feature-based blur preprocessing module 404, and the input picture data is accordingly segmented by either one of the preprocessing model 202 or the block-based and feature-based blur preprocessing module 404, and an object mask is output. Inspired by the diversity of input video data, such a preprocessing method selectively or combinedly performs block-based preprocessing and feature-based blur preprocessing to improve the overall rate-distortion performance.

[0070] According to the preprocessing method described herein, a block-based preprocessing and feature-based blur preprocessing module 404 stored in a non-transitory or non-transient computer-readable storage medium of the input preprocessing computing system can configure one or more processors of the input preprocessing computing system to make preprocessing decisions for a video sequence based on two metrics: the average object mask ratio of the video sequence and the temporal complexity of the video sequence. The preprocessing decisions by the one or more processors configured by the block-based and feature-based blur preprocessing module 404 selectively or in combination cause the input picture data to be subsequently processed by the preprocessing model 202 in the input preprocessing 200 as described above, and / or by the feature-based and block-based blur preprocessing.

[0071] Temporal complexity is a metric of video time domain variation and is derived from the frame mean absolute deviation (MAD). The block-based and feature-based blur preprocessing module 404 configures one or more processors of the input preprocessing computation system to calculate temporal complexity as follows:

number

[0072] The combined feature-based and block-based blur pre-processing includes at least performing segmentation on the input picture and outputting an object mask as shown in FIG. 4 as described above, and performing mask-based picture data removal and blur pre-processing as shown in FIG. 5 as described above, applying the object mask to the input picture, and blurring the masked picture.

[0073] The average object mask ratio of an image quantifies the average proportion of each object in the frame. To calculate the average object mask ratio, the preprocessing model 202 configures one or more processors of the input preprocessing computation system to take the object number O(Ii) and the corresponding mask ratio m(Ii), and calculate the average value as follows:

number

[0074] The object number is the number of different object segments in the object mask. The mask ratio is the ratio of the number of pixels in one object segment to the number of pixels in the whole picture, which can be obtained by dividing the number of pixels in the object segment by the number of pixels in the picture.

[0075] A calculation that produces an average object mask ratio that is too small indicates that the object occupies a smaller proportion of the overall frame than the background, in which case the mask produced in response to the preprocessing model 202 may not be completely accurate because the object is too small. Therefore, for average object mask ratios below the proportion threshold, the block-based and feature-based blur preprocessing module 404 configures one or more processors of the input preprocessing calculation system to perform block-based blur preprocessing without performing feature-based blur preprocessing in order to preserve the background around the object. This approach controls the computational cost of encoding and prevents important information from being distorted by background preservation.

[0076] The complexity threshold and percentage threshold as described above can be written as matrices Mthres and Kthres, respectively. For example, Mthres can be set to 10 and Kthres can be set to 7%. Furthermore, the comparison for each threshold can be performed in the following order: First, the time complexity

number

number

[0077] It should be understood that the block-based and feature-based blur preprocessing module 404 can configure one or more processors of the input preprocessing computing system to determine preprocessing of the video sequence based on other traditional or deep learning-based metrics, such as object type, video length, and other features.

[0078] It should be understood that the block-based and feature-based blur preprocessing module 404 can alternatively be configured to perform feature-based blur preprocessing without block-based blur preprocessing, and to perform Gaussian blur preprocessing and / or other preprocessing techniques, selectively or in combination.

[0079] It should be understood that the block-based and feature-based blur preprocessing module 404 can configure one or more processors of the input preprocessing computing system to determine the preprocessing methods in different orders and based on different thresholds. For example, the preprocessing can be determined based on the following thresholds in the following order: First, for video sequences with a large average object mask ratio, a combination of feature-based blur preprocessing and block-based blur preprocessing is performed; otherwise, for video sequences with a smaller average object mask ratio and low temporal complexity, block-based blur preprocessing is performed to preserve background information; otherwise, finally, video sequences with small objects and high temporal complexity are not processed.

[0080] FIG. 7 illustrates an exemplary input preprocessing computation system 700 for implementing the above-described processes and methods for performing intermediate feature fusion extracted from the convolutional layer 206 of the preprocessing model 202.

[0081] The techniques and mechanisms described herein may be implemented by numerous instances of the input preprocessing computing system 700, as well as any other computing device, system, and / or environment. The input preprocessing computing system 700 shown in FIG. 7 is merely one example system and is not intended to impose limitations on the scope of use or capabilities of any computing device utilized to perform the processes and / or procedures described above. Other known computing devices, systems, environments, and / or configurations that may be suitable for use with embodiments include, but are not limited to, personal computers, server computers, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, game consoles, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, implementations using field programmable gate arrays (FPGAs), and application specific integrated circuits (ASICs), etc.

[0082] The input pre-processing computing system 700 may include one or more processors 702 and a system memory 704 communicatively coupled to the processors 702. The processors 702 and the system memory 704 may be physical, virtual, and / or distributed. The processors 702 may execute one or more modules and / or processes to cause the processors 702 to perform various functions. In embodiments, the processors 702 may include a central processing unit (CPU), a graphics processing unit (GPU), both a CPU and a GPU, or other processing units or components known in the art. In addition, each of the processors 702 may have its own local memory, which may also store program modules, program data, and / or one or more operating systems.

[0083] Depending on the exact configuration and type of input pre-processing computing system 700, system memory 704 may be volatile, such as RAM, non-volatile, such as ROM, flash memory, a mini-hard drive, a memory card, or some combination thereof. System memory 704 may include one or more computer-executable modules 706 that are executable by processor 702.

[0084] The modules 706 may include, but are not limited to, a preprocessing model 708 , a fusion module 710 , and a block-based and feature-based blur preprocessing module 712 .

[0085] The preprocessing model 708 may configure one or more processors according to the preprocessing model 202, as described above.

[0086] The fusion module 710 may configure one or more processors in accordance with the fusion module 402, as described above.

[0087] The block-based and feature-based blur preprocessing module 712 may comprise one or more processors in accordance with the block-based and feature-based blur preprocessing module 404, as described above.

[0088] The input-preprocessing computing system 700 may also include an input / output (I / O) interface 740 and a communications module 750 that enables the input-preprocessing computing system 700 to communicate with other systems and devices over a network. Such networks may include wired media, such as the Internet, a wired network, or a direct-wired connection, and wireless media, such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0089] Some or all of the operations of the above-described methods may be performed by the execution of computer-readable instructions stored on a computer-readable storage medium, as defined below. As used in this specification and claims, the term "computer-readable instructions" includes routines, applications, application modules, program modules, programs, components, data structures, algorithms, etc. The computer-readable instructions may be implemented in a variety of system configurations, including single-processor or multi-processor systems, minicomputers, mainframe computers, personal computers, handheld computing devices, microprocessor-based systems, programmable consumer electronic devices, combinations thereof, etc.

[0090] The computer-readable storage medium may include volatile memory (such as random access memory (RAM)) and / or non-volatile memory (such as read-only memory (ROM), flash memory, etc.). The computer-readable storage medium may also include additional removable and / or non-removable storage devices including, but not limited to, flash memory, magnetic storage devices, optical storage devices, and / or tape storage devices that may provide non-volatile storage for computer-readable instructions, data structures, program modules, and the like.

[0091] A non-transitory or non-transient computer-readable storage medium is an example of a computer-readable medium. Computer-readable media includes at least two types of computer-readable media: computer-readable storage media and communication media. Computer-readable storage media includes volatile and non-volatile, removable and non-removable media implemented in any process or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage device, magnetic cassette, magnetic tape, magnetic disk storage device or other magnetic storage device, or any other non-transmission medium that can be used to store information for access by a computing device. Alternatively, communication media may embody computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transmission mechanism. As defined herein, computer-readable storage media does not include communication media.

[0092] Computer-readable instructions stored on one or more non-transitory computer-readable storage media, when executed by one or more processors, may perform the operations described above with reference to Figures 1-6. Generally, computer-readable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as limiting, and a process can be performed in any order and / or in parallel, combining any number of the operations described.

[0093] FIG. 8 illustrates an exemplary encoding and decoding computing system 800 for implementing processes and methods for implementing intra template matching prediction modes.

[0094] The techniques and mechanisms described herein may be implemented by numerous instances of encoding and decoding computing system 800, as well as any other computing device, system, and / or environment. The system 800 shown in FIG. 8 is merely one example system and is not intended to impose limitations on the scope of use or capabilities of any computing device utilized to perform the processes and / or procedures described above. Other known computing devices, systems, environments, and / or configurations that may be suitable for use with embodiments include, but are not limited to, personal computers, server computers, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, game consoles, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, implementations using field programmable gate arrays (FPGAs), and application-specific integrated circuits (ASICs), etc.

[0095] The encoding and decoding computing system 800 may include one or more processors 802 and a system memory 804 communicatively coupled to the processors 802. The processors 802 may execute one or more modules and / or processes to cause the encoding and decoding computing processors 802 to perform various functions. In some embodiments, the processors 802 may include a central processing unit (CPU), a graphics processing unit (GPU), both a CPU and a GPU, or other processing units or components known in the art. In addition, each of the processors 802 may have its own local memory, which may also store program modules, program data, and / or one or more operating systems.

[0096] Depending on the exact configuration and type of encoding and decoding computing system 800, system memory 804 may be volatile, such as RAM, non-volatile, such as ROM, flash memory, a mini-hard drive, a memory card, or some combination thereof. System memory 804 may include one or more computer-executable modules 806 that are executable by processor 802.

[0097] The module 806 may include, but is not limited to, one or more of an encoder 808 and a decoder 810 .

[0098] The encoder 808 may be a VVC standard encoder that implements any one, some, or all aspects of the exemplary embodiments of the present disclosure as described above and is executable by the processor 802 to configure the processor 802 to perform the above operations.

[0099] The decoder 810 may be a VVC standard decoder that implements any one, some, or all aspects of the exemplary embodiments of the present disclosure as described above and is executable by the processor 802 to configure the processor 802 to perform the operations described above.

[0100] The encoding and decoding computing system 800 may also include an input / output (I / O) interface 840 for receiving image source data and bitstream data and outputting reconstructed pictures to a reference picture buffer and / or a display buffer. The encoding and decoding computing system 800 may also include a communications module 850 that enables the encoding and decoding computing system 800 to communicate with other devices (not shown) over a network (not shown). Such networks may include wired media, such as the Internet, a wired network, or a direct-wired connection, and wireless media, such as acoustic, radio frequency (RF), infrared, or other wireless media.

[0101] In some embodiments, performing, by one or more processors of the input pre-processing computing system, a plurality of convolutions on the input picture data; outputting, by the one or more processors, a plurality of intermediate feature maps from corresponding different convolutions; averaging, by the one or more processors, absolute values ​​of the intermediate feature maps to generate a fused feature map; and resizing the fused feature map to the size of the input picture data.

[0102] In one embodiment, the method further includes performing, by the one or more processors, a Gaussian blur transform on pixels of the input picture data using a feature map value corresponding to the pixel from the fused feature map as input.

[0103] In one embodiment, the feature map value corresponding to the pixel is calculated by the one or more processors as a standard deviation value in the Gaussian blur transform.

[0104] In one embodiment, the one or more processors perform a Gaussian blur transform on each pixel of the input picture data.

[0105] In one embodiment, the multiple convolutions are performed by the one or more processors during a segmentation calculation performed on the input picture data to output an object mask.

[0106] In one embodiment, the method further includes modifying, by the one or more processors, the object mask to exclude each pixel of a sliding window.

[0107] In one embodiment, the method further includes multiplying, by the one or more processors, the modified object mask and the input picture data to output block-based masked input picture data.

[0108] In one embodiment, the one or more processors perform a Gaussian blur transform on the block-based masked input picture data.

[0109] In one embodiment, the method further includes determining, by the one or more processors, to perform a Gaussian blur transform on the block-based masked input picture data based on an average object mask ratio of the video sequence exceeding a threshold.

[0110] In one embodiment, the method further includes determining, by the one or more processors, to perform a Gaussian blur transform on the block-based masked input picture data based on the temporal complexity of the video sequence exceeding a threshold.

[0111] In some embodiments, an execution module configured to perform a plurality of convolutions on input picture data; an output module configured to output a plurality of intermediate feature maps from corresponding different convolutions; an averaging module configured to average absolute values ​​of the plurality of intermediate feature maps to generate a fused feature map; a resizing module configured to resize the fused feature map to a size of the input picture data.

[0112] In one embodiment, the execution module is further configured to blur the input picture data by performing a Gaussian blur transform on pixels of the input picture data, the Gaussian blur transform taking as input a feature map value corresponding to the pixel from the fused feature map.

[0113] In one embodiment, the computing system comprises: The method further includes a calculation module configured to calculate the feature map value corresponding to the pixel as a standard deviation value in the Gaussian blur transform.

[0114] In one embodiment, the execution module is further configured to perform a Gaussian blur transformation on each pixel of the input picture data.

[0115] In one embodiment, the execution module is further configured to perform the plurality of convolutions during a segmentation calculation performed on the input picture data to output an object mask.

[0116] In one embodiment, the computing system comprises: The method further includes a modification module configured to modify the object mask to exclude each pixel of a sliding window.

[0117] In one embodiment, the computing system comprises: The apparatus further includes a multiplication module configured to multiply the modified object mask with the input picture data to output block-based masked input picture data.

[0118] In one embodiment, the execution module is further configured to perform a Gaussian blur transform on the block-based masked input picture data.

[0119] In one embodiment, the computing system comprises: The video processing device further includes a decision module configured to decide to perform a Gaussian blur transform on the block-based masked input picture data based on an average object mask ratio of the video sequence exceeding a threshold.

[0120] In one embodiment, the computing system comprises: The video processing device further includes a decision module configured to decide to perform a Gaussian blur transform on the block-based masked input picture data based on a temporal complexity of the video sequence exceeding a threshold.

[0121] In some embodiments, one or more processors; and a computer-readable storage medium communicatively coupled to the one or more processors, the computer-readable storage medium storing computer-readable instructions executable by the one or more processors, that when executed by the one or more processors perform a method in accordance with the method embodiment described above.

[0122] In some embodiments, a non-transitory computer-readable storage medium is provided, the computer-readable storage medium storing computer-executable instructions that, when executed by the processor, perform a method according to the method embodiments described above.

[0123] In some embodiments, a computer program product is provided that includes computer program instructions that cause a computer to perform a method according to the above method embodiments.

[0124] In some embodiments, a computer program is provided that causes a computer to perform the methods according to the above method embodiments.

[0125] Some or all of the operations of the above-described methods may be performed by the execution of computer-readable instructions stored on a computer-readable storage medium, as defined below. As used in this specification and claims, the term "computer-readable instructions" includes routines, applications, application modules, program modules, programs, components, data structures, algorithms, etc. The computer-readable instructions may be implemented in a variety of system configurations, including single-processor or multi-processor systems, minicomputers, mainframe computers, personal computers, handheld computing devices, microprocessor-based systems, programmable consumer electronic devices, combinations thereof, etc.

[0126] The computer-readable storage medium may include volatile memory (such as random access memory (RAM)) and / or non-volatile memory (such as read-only memory (ROM), flash memory, etc.). The computer-readable storage medium may also include additional removable and / or non-removable storage devices including, but not limited to, flash memory, magnetic storage devices, optical storage devices, and / or tape storage devices that may provide non-volatile storage for computer-readable instructions, data structures, program modules, and the like.

[0127] A non-transitory or non-transient computer-readable storage medium is an example of a computer-readable medium. Computer-readable media includes at least two types of computer-readable media: computer-readable storage media and communication media. Computer-readable storage media includes volatile and non-volatile, removable and non-removable media implemented in any process or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage device, magnetic cassette, magnetic tape, magnetic disk storage device or other magnetic storage device, or any other non-transmission medium that can be used to store information for access by a computing device. Communication media, on the other hand, may embody computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transmission mechanism. Computer-readable storage media as employed herein should not be interpreted as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (such as light pulses through fiber optic cable), or electrical signals propagating through wires.

[0128] Computer-readable instructions stored on one or more non-transitory or non-transient computer-readable storage media, when executed by one or more processors, may perform the operations described above with reference to Figures 1 and 2. Generally, computer-readable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and a process can be performed in any order and / or in parallel, combining any number of the operations described.

[0129] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claims.

Claims

1. performing a plurality of convolutions on the input picture data by one or more processors of the input pre-processing computing system; outputting, by the one or more processors, a plurality of intermediate feature maps from corresponding different convolutions; averaging, by the one or more processors, absolute values ​​of the intermediate feature maps to generate a fused feature map; and resizing the fused feature map to a size of the input picture data.

2. The method of claim 1 , further comprising: performing, by the one or more processors, a Gaussian blur transform on pixels of the input picture data using feature map values ​​corresponding to the pixels from the fused feature map as input.

3. The method of claim 2 , wherein the feature map value corresponding to the pixel is calculated by the one or more processors as a standard deviation value in the Gaussian blur transform.

4. The method of claim 2 , wherein the one or more processors perform a Gaussian blur transform on each pixel of the input picture data.

5. The method of claim 2 , wherein the plurality of convolutions are performed by the one or more processors during a segmentation calculation performed on the input picture data to output an object mask.

6. The method of claim 5 , further comprising modifying, by the one or more processors, the object mask to exclude each pixel of a sliding window.

7. The method of claim 6 , further comprising multiplying, by the one or more processors, the modified object mask with the input picture data to output block-based masked input picture data.

8. The method of claim 7 , wherein the one or more processors perform a Gaussian blur transform on the block-based masked input picture data.

9. 9. The method of claim 8, further comprising: determining, by the one or more processors, to perform a Gaussian blur transform on the block-based masked input picture data based on an average object mask ratio of the video sequence exceeding a threshold.

10. 9. The method of claim 8, further comprising: determining, by the one or more processors, to perform a Gaussian blur transform on the block-based masked input picture data based on a temporal complexity of the video sequence exceeding a threshold.

11. an execution module configured to perform a plurality of convolutions on input picture data; an output module configured to output a plurality of intermediate feature maps from corresponding different convolutions; an averaging module configured to average absolute values ​​of the plurality of intermediate feature maps to generate a fused feature map; a resizing module configured to resize the fused feature map to a size of the input picture data.

12. 12. The computing system of claim 11, wherein the execution module is further configured to blur the input picture data by performing a Gaussian blur transform on pixels of the input picture data using feature map values ​​corresponding to the pixels from the fused feature map as input.

13. The computing system of claim 12 , further comprising a computing module configured to compute the feature map value corresponding to the pixel as a standard deviation value in the Gaussian blur transform.

14. The computing system of claim 12 , wherein the execution module is further configured to perform a Gaussian blur transform on each pixel of the input picture data.

15. The computing system of claim 12 , wherein the execution module is further configured to perform the plurality of convolutions during a segmentation calculation performed on the input picture data to output an object mask.

16. The computing system of claim 15 , further comprising a modification module configured to modify the object mask to exclude each pixel of a sliding window.

17. 17. The computing system of claim 16, further comprising a multiplication module configured to multiply the modified object mask with the input picture data to output block-based masked input picture data.

18. The computing system of claim 17 , wherein the execution module is further configured to perform a Gaussian blur transform on the block-based masked input picture data.

19. 20. The computing system of claim 18, further comprising: a decision module configured to decide to perform a Gaussian blur transform on the block-based masked input picture data based on an average object mask ratio of a video sequence exceeding a threshold.

20. 20. The computing system of claim 18, further comprising: a decision module configured to decide to perform a Gaussian blur transform on the block-based masked input picture data based on a temporal complexity of a video sequence exceeding a threshold.

21. one or more processors; a computer-readable storage medium communicatively coupled to the one or more processors, the computer-readable storage medium storing computer-readable instructions executable by the one or more processors and, when executed by the one or more processors, performing the method of any one of claims 1 to 10.

22. The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the method of any one of claims 1 to 10 is performed.

23. A computer program comprising computer program instructions which when executed on a computer cause the computer to carry out the method according to any one of claims 1 to 10.

24. A computer program that causes a computer to execute the method according to any one of claims 1 to 10.

Images