Feature map processing methods and related equipment

Abstract

This application relates to the field of artificial intelligence technology, and provides a method and related device for processing feature maps. The method is implemented by calling a neural network model including multiple input adaptation branches and a post-processing part. The outputs of the multiple input adaptation branches serve as the inputs to the post-processing part. Each input adaptation branch can downsample the feature map at different ratios. Specifically, a first feature map is input into one of the multiple input adaptation branches to obtain a second feature map. One input adaptation branch is determined based on the size of the first feature map and a preset size threshold, where the preset size threshold is the upper limit size of the input feature map in the post-processing part, and the size of the second feature map is not greater than the preset size threshold. The second feature map is then input into the post-processing part for post-processing to obtain the processing result of the first feature map. This application embodiment can reduce computational power.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of artificial intelligence (AI) technology, and in particular to a feature map processing method and related equipment. BACKGROUND

[0002] The existing neural network of a machine task is a visual task neural network (referred to as an image domain task network) with an image as input, that is, the input of the image domain task network is an image, and the image domain task network outputs a processing result after processing the input image. When the image domain task network is combined with video or image coding and decoding applications, the decoding end (for example, an image decoder based on an artificial neural network) needs to first parse the code stream to generate a reconstructed feature map, and then input the reconstructed feature map into the image reconstruction network to generate a reconstructed image, taking the reconstructed image as the input of the image domain task network. Therefore, when the image domain task network is combined with video or image coding and decoding applications, a large amount of computing power is required. SUMMARY

[0003] The present application provides a feature map processing method and related equipment, which can reduce the computing power.

[0004] According to a first aspect, the present application relates to a feature map processing method. The method is executed by an electronic device, and is implemented by calling a neural network model. The neural network model includes a plurality of input adaptation branches and a post-processing part. The outputs of the plurality of input adaptation branches are the inputs of the post-processing part. The plurality of input adaptation branches are used for down-sampling a feature map. Each input adaptation branch in the plurality of input adaptation branches has a different down-sampling rate for the feature map. The method includes: inputting a first feature map into one of the plurality of input adaptation branches to obtain a second feature map. The one of the plurality of input adaptation branches is determined according to a size of the first feature map and a preset size threshold. The preset size threshold is an upper limit size of the input feature map of the post-processing part. The size of the second feature map is not greater than the preset size threshold. The second feature map is input into the post-processing part for post-processing to obtain a processing result of the first feature map.

[0005] In the embodiments of the present application, the visual task neural network for machine task processing is a feature domain task network with a feature map as input. For example, the input of the neural network model designed in the present application can be a feature map reconstructed by a decoder parsing a bitstream, and the output is a task processing result corresponding to the feature map. In this way, compared with an image domain task network, the process of obtaining a reconstructed image from a reconstructed feature map is skipped, thereby reducing the computing power. Moreover, compared with an image, the computing power of the neural network model for processing a feature map is also smaller. Further, the neural network model designed in the present application includes a plurality of input adaptation branches and a post-processing part. The outputs of the plurality of input adaptation branches are the inputs of the post-processing part. Each input adaptation branch in the plurality of input adaptation branches can perform down-sampling on the feature map at different scales. For a first feature map of different sizes (resolutions) input into the neural network model, one of the plurality of input adaptation branches can be selected according to the size of the first feature map and the upper limit size of the input feature map of the post-processing part, and the first feature map is down-sampled by the selected input adaptation branch to generate a second feature map with a size not greater than the upper limit size of the input feature map of the post-processing part. In this way, the second feature map meets the size requirement of the input feature map of the post-processing part, and the processing result of the first feature map can be obtained by inputting the second feature map into the post-processing part for post-processing. Therefore, the present application can maintain high analysis accuracy with similar computing power when the input feature map of the neural network model is of different resolutions. In addition, compared with training an independent neural network model for each resolution of input feature map (each network has similar computing power), and then processing feature maps of different resolutions by the neural network models trained for different resolutions, the neural network model designed in the present application can achieve the same function through the plurality of input adaptation branches. The plurality of input adaptation branches are in front of the backbone network of the neural network model, and there is only one set of post-processing part (core part of the backbone network and the head network) in the neural network model. In this way, the network parameters can be significantly saved, and the model deployment difficulty can be reduced without switching the network model during inference.

[0006] In a possible design, any first input adaptation branch in the plurality of input adaptation branches that down-samples the feature map by an integer multiple includes at least one first convolutional layer configured to down-sample the feature map. The step length of the at least one first convolutional layer is determined according to the down-sampling multiple of the any first input adaptation branch.

[0007] In the design, for any first input adaptation branch with an integer times of down-sampling rate on the feature map, the first input adaptation branch can down-sample the input feature map by at least one first convolution layer, and the down-sampling rate of the feature map by the convolution layer is related to the step length of the convolution layer. After the step length of the at least one first convolution layer is determined according to the down-sampling rate of the feature map by the any first input adaptation branch, the integer times of down-sampling rate on the feature map input into the any first input adaptation branch can be realized. For example, the down-sampling rate of the feature map by the any first input adaptation branch is 2, and the step length of the at least one first convolution layer is 2, so that the at least one first convolution layer can reduce the size of the feature map input into the any first input adaptation branch by 2 times.

[0008] In a possible design, any second input adaptation branch with a non-integer times of down-sampling rate on the feature map in the plurality of input adaptation branches comprises at least one up-sampling module and at least one second convolution layer. The at least one up-sampling module is configured to up-sample the feature map, and the at least one second convolution layer is configured to down-sample the feature map up-sampled by the at least one up-sampling module. The up-sampling rate of the feature map by the at least one up-sampling module and the step length of the at least one second convolution layer are determined according to the down-sampling rate of the feature map by the any second input adaptation branch.

[0009] In the design, for any second input adaptation branch with a non-integer times of down-sampling rate on the feature map, the second input adaptation branch can first up-sample the input feature map by at least one up-sampling module, and then down-sample the feature map up-sampled by the at least one up-sampling module by at least one second convolution layer. The down-sampling rate of the feature map by the convolution layer is related to the step length of the convolution layer. After the up-sampling rate of the feature map by the at least one up-sampling module and the step length of the at least one second convolution layer are determined according to the down-sampling rate of the feature map by the any second input adaptation branch, the non-integer times of down-sampling rate on the feature map input into the any second input adaptation branch can be realized. For example, the down-sampling rate of the feature map by the any second input adaptation branch is 1.5, the up-sampling rate of the feature map by the at least one up-sampling module is 2, and the step length of the at least one second convolution layer is 3. In this way, the any second input adaptation branch can first enlarge the size of the input feature map by 2 times, and then reduce the size by 3 times to output, so as to realize the overall reduction of the size of the feature map by 1.5 times from input to output.

[0010] In a possible design, any third input adaptation branch in the plurality of input adaptation branches, which has a down-sampling ratio greater than 1 for the feature map, includes at least one residual block structure (Resblock) configured to down-sample the feature map, and the at least one residual block structure includes a plurality of third convolution layers, and a step length of any third convolution layer in the plurality of third convolution layers is determined according to the down-sampling ratio of the any third input adaptation branch for the feature map.

[0011] In this design, for any third input adaptation branch having a down-sampling ratio greater than 1 for the feature map, the input feature map can be down-sampled by the at least one residual block structure, and since the at least one residual block structure includes the plurality of third convolution layers, and the down-sampling ratio of the feature map by the convolution layer is related to the step length of the convolution layer, after the step length of any third convolution layer in the plurality of third convolution layers is determined according to the down-sampling ratio of the any third input adaptation branch for the feature map, the down-sampling of the input feature map by the any third input adaptation branch by a ratio greater than 1 can be implemented; in addition, it can be understood that, in addition to the plurality of third convolution layers for down-sampling, the residual block structure also includes other layers, but the other layers do not change the size of the input feature map, and the convolution layers change the size of the input feature map.

[0012] In a possible design, the one of the input adaptation branches is determined according to a maximum value of a first ratio and a second ratio, the first ratio is a ratio between a width of the first feature map and a width of the preset size threshold, and the second ratio is a ratio between a height of the first feature map and a height of the preset size threshold.

[0013] In this design, the ratio between the width of the first feature map and the width of the upper limit size of the input feature map of the post-processing part, and the ratio between the height of the first feature map and the height of the upper limit size of the input feature map of the post-processing part are calculated, and then the one of the input adaptation branches is selected from the plurality of input adaptation branches according to the maximum value of the two ratios, to down-sample the first feature map, so that the width and the height of the feature map obtained by down-sampling the first feature map will not be greater than the width and the height of the upper limit size of the input feature map of the post-processing part, respectively.

[0014] In a possible design, if the size of the second feature map is greater than the preset size threshold, the method further includes: obtaining a third feature map by cropping the second feature map, and the size of the third feature map is not greater than the preset size threshold; and inputting the third feature map into the post-processing part for post-processing, to obtain the processing result of the first feature map.

[0015] In the design, the size of the first feature map inputted can be too large, and after down-sampling by one of the input adaptation branches, for example, after down-sampling by the input adaptation branch with the largest down-sampling rate, the second feature map obtained can still be larger than the upper limit size of the input feature map of the post-processing part. In this case, a third feature map with a size not larger than the upper limit size of the input feature map of the post-processing part is obtained by cutting from the second feature map, and the third feature map is inputted into the post-processing part for post-processing to obtain the processing result of the first feature map. In this way, it can be ensured that after processing of the first feature map inputted into the neural network model, the processing result of the first feature map can be obtained.

[0016] In a possible design, the first feature map is a feature map obtained by decoding a code stream.

[0017] In the design, since the first feature map inputted into the neural network model is a feature map obtained by decoding a code stream, for example, the first feature map is a feature map obtained by decoding a feature map code stream using an entropy decoding module of an end-to-end image decoder; in this way, for video or image coding and decoding, it is not necessary to perform image reconstruction on the feature map obtained by decoding a code stream to obtain a reconstructed image inputted into the neural network model for processing, that is, the process of image reconstruction can be omitted, and the feature map obtained by decoding a code stream can be directly inputted into the neural network model for analysis, thereby reducing the computing power.

[0018] In a possible design, the method further includes: performing N rounds of training on the neural network model according to a training sample set to obtain N first loss values, the training sample set including sample feature maps of at least N sizes, N being an integer and not less than the number of the plurality of input adaptation branches; obtaining a second loss value according to the N first loss values; and updating parameters of the neural network model according to the second loss value.

[0019] In the design, the training sample set used for training the neural network model of the design includes sample feature maps of at least N sizes, and N is not less than the number of the plurality of input adaptation branches. In this way, training of the neural network model using the training sample set can enable each of the plurality of input adaptation branches to be trained, so that the N first loss values obtained by performing N rounds of training on the neural network model include the first loss value corresponding to any of the plurality of input adaptation branches. The second loss value is obtained according to the N first loss values, and the parameters of the neural network model are updated according to the second loss value, so that the parameters in any of the plurality of input adaptation branches are also updated.

[0020] According to a second aspect, the present application relates to a feature map processing apparatus, the advantages of which can be referred to the description of the first aspect, which will not be repeated here. The feature map processing apparatus has functions to implement the behaviors in the method instances of the first aspect. The functions can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. In one possible design, the feature map processing apparatus is implemented by invoking a neural network model, the neural network model includes a plurality of input adaptation branches and a post-processing part, the outputs of the plurality of input adaptation branches are the inputs of the post-processing part, the plurality of input adaptation branches are used for down-sampling the feature map, each input adaptation branch in the plurality of input adaptation branches has a different down-sampling rate for the feature map; the apparatus includes a processing unit, configured to: input a first feature map into one of the plurality of input adaptation branches to obtain a second feature map, the one of the plurality of input adaptation branches is determined according to the size of the first feature map and a preset size threshold, the preset size threshold is an upper limit size of the input feature map of the post-processing part, the size of the second feature map is not greater than the preset size threshold; input the second feature map into the post-processing part for post-processing to obtain a processing result of the first feature map.

[0021] In one possible design, any first input adaptation branch in the plurality of input adaptation branches that has an integer down-sampling rate for the feature map includes at least one first convolutional layer, the at least one first convolutional layer is configured to down-sample the feature map, and a step length of the at least one first convolutional layer is determined according to the down-sampling rate of the any first input adaptation branch for the feature map.

[0022] In one possible design, any second input adaptation branch in the plurality of input adaptation branches that has a non-integer down-sampling rate for the feature map includes at least one up-sampling module and at least one second convolutional layer, the at least one up-sampling module is configured to up-sample the feature map, the at least one second convolutional layer is configured to down-sample the feature map obtained by the up-sampling, and an up-sampling rate of the at least one up-sampling module and a step length of the at least one second convolutional layer are determined according to the down-sampling rate of the any second input adaptation branch for the feature map.

[0023] In a possible design, any third input adaptation branch of the plurality of input adaptation branches, in which a down-sampling rate of the feature map is greater than 1, includes at least one residual block structure (Resblock) configured to down-sample the feature map, and the at least one residual block structure includes a plurality of third convolution layers, a stride of any third convolution layer of the plurality of third convolution layers being determined according to the down-sampling rate of the feature map of the any third input adaptation branch.

[0024] In a possible design, the one of the input adaptation branches is determined according to a maximum value of a first ratio and a second ratio, the first ratio being a ratio between a width of the first feature map and a width of the preset size threshold, and the second ratio being a ratio between a height of the first feature map and a height of the preset size threshold.

[0025] In a possible design, if the size of the second feature map is greater than the preset size threshold, the processing unit is further configured to: obtain a third feature map by cropping the second feature map, the size of the third feature map being not greater than the preset size threshold; and input the third feature map into the post-processing part for post-processing to obtain a processing result of the first feature map.

[0026] In a possible design, the first feature map is a feature map obtained by decoding a bitstream.

[0027] In a possible design, the processing unit is further configured to: perform N rounds of training on the neural network model according to a training sample set to obtain N first loss values, the training sample set including sample feature maps of at least N sizes, the N being not less than a quantity of the plurality of input adaptation branches, and the N being an integer; obtain a second loss value according to the N first loss values; and update parameters of the neural network model according to the second loss value.

[0028] According to a third aspect, the present application relates to an electronic device, including: one or more processors; and a non-transitory computer-readable storage medium coupled to the processors and storing programming for execution by the processors, wherein the programming, when executed by the processors, causes the electronic device to carry out the method in any possible implementation of the first aspect.

[0029] According to a fourth aspect, the present application relates to a non-transitory computer-readable storage medium including program code, when executed by a computer device, for performing the method in any possible implementation of the first aspect.

[0030] According to a fifth aspect, the present application relates to a computer program product including program code, when executed, for performing the method in any possible implementation of the first aspect.

[0031] According to a sixth aspect, the present application relates to a chip comprising a processor configured to call and run a computer program from a memory, so that a device in which the chip is installed performs the method according to any one of the first aspect.

[0032] One or more embodiments are described in detail in the accompanying drawings and the following description. Other features, objects, and advantages are apparent from the description and drawings. BRIEF DESCRIPTION OF DRAWINGS

[0033] The drawings used in the description of the embodiments of the present application are introduced as follows.

[0034] Figure 1 A schematic diagram of an image codec based on artificial neural network;

[0035] Figure 2 An architecture schematic diagram of a neural network model provided by the embodiments of the present application;

[0036] Figure 3 A flow schematic diagram of a feature map processing method provided by the embodiments of the present application;

[0037] Figure 4 A structural schematic diagram of a feature map processing device provided by the embodiments of the present application;

[0038] Figure 5 A structural schematic diagram of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0039] First, some technical knowledge in the embodiments of the present application is introduced to facilitate the understanding of the present application by those skilled in the art.

[0040] (1) Convolutional Neural Network (CNN)

[0041] A convolutional neural network is a typical method in the field of image processing and analysis. The convolutional neural network at least includes a convolutional layer, and can further include an activation layer, a pooling layer, a batch normalization layer (BN layer), a fully connected layer (FC layer), and other functional modules. The activation layer can be a rectified linear unit (ReLU) or a parametric rectified linear unit (PReLU). Typical convolutional neural networks include LeNet, AlexNet, VGGNet, ResNet, Yolo, Faster RCNN, Mask RCNN, and ASLFeat. A basic convolutional neural network can be composed of a backbone network and a head network, such as AlexNet in object recognition. Some complex CNNs, such as the Faster RCNN network with a feature pyramid structure in the target detection field, are composed of a backbone network, a neck network, and a head network.

[0042] The backbone network is the first part of the convolutional neural network, and its function is to extract multiple scale feature maps from the input image. The backbone network is usually composed of convolutional layers, pooling layers, and activation layers, and does not contain fully connected layers. Generally, the layers closer to the input image in the backbone network output feature maps with larger resolution (width and height) and fewer channels. Typical backbone networks include VGG-16, ResNet-50, and ResNeXt-101. The backbone network can be divided into a backbone network front part and a backbone network core part. The backbone network front part, also known as the stem, is a part of the backbone network close to the input. It usually includes a small number of convolutional layers and can also include other forms of layers such as pooling layers. It preliminarily processes the input signal, reduces the spatial resolution, and increases the number of channels. For example, in Resnet-50, the input side is a structure composed of a 7x7 convolutional layer and a maximum pooling layer (Max Pool). The part of the backbone network other than the backbone network front part is the backbone network core part. The backbone network core part usually contains a large number of convolutional layers and some network sub-modules with the same or similar structure connected in series, such as the residual block structure (Resblock) in Resnet.

[0043] Neck network is the middle part of the convolutional neural network, which further integrates and processes the feature map generated by the backbone network to obtain a new feature map. Common neck networks include Feature Pyramid Networks (FPN).

[0044] Head network is the last part of the convolutional neural network, which processes the feature map to obtain the prediction result of the neural network output. Common head networks include fully connected layers, Softmax modules, etc.

[0045] Bottleneck structure is a multi-layer network structure, in which the input data is first processed by one or more neural network layers to obtain intermediate data, and then the intermediate data is processed by one or more neural network layers to obtain output data, wherein the data volume (i.e., the product of width, height, and channel number) of the intermediate data is lower than that of the input data and the output data.

[0046] Feature map is a three-dimensional data output by convolutional neural network layers such as convolutional layer, activation layer, pooling layer, and batch normalization layer, and its three dimensions are called width, height, and channel.

[0047] Neural network needs to determine the parameters of each layer of the neural network through training. During training, forward loss calculation and backpropagation are used to update trainable parameters in the neural network. Through multiple parameter updates, the parameters of each layer of the neural network are finally converged to a relatively optimal analysis precision. After training is completed, the parameters of each layer of the network are usually fixed, and the input signal is passed through the neural network to obtain the result. This process of actually using the neural network is called "inference".

[0048] (2) Artificial neural network-based image encoder-decoder

[0049] Since 2017, artificial neural network-based image encoder-decoders have developed rapidly. From the early autoencoder-based structure and recurrent neural network (RNN)-based structure, they have developed into more advanced variation autoencoder (VAE) structure, whose compression performance can be comparable to that of H.266 / VCC (Versatile Video Coding) encoding. A classic network model is shown in FIG. 1. Figure 1

[0050] At the encoding end, the original image x is input into the feature extraction module, i.e., the input encoding (Enc) module, and the original feature map ​; on the one hand, the original feature map feature extraction through a Hyper-Enc module, and output prior information ; the prior information After quantization, the quantized , the quantized input probability estimation module, output each feature element Probability distribution model parameters are obtained, so as to obtain the probability distribution of each feature element ; the entropy encoding or arithmetic encoding (AE) module encodes each feature element according to the probability distribution to obtain a hyper-prior code stream; the entropy decoding or arithmetic decoding (AD) module entropy decodes the hyper-prior code stream to obtain the value of the feature element , the Hyper-Dec module reconstructs the image to output the reconstructed map of the feature map (that is, the reconstructed feature map y). On the other hand, the original feature map input quantization module, get quantized feature map . The entropy encoding module obtains the probability distribution of each feature element according to the probability distribution model parameters obtained by the Hyper-Dec module, and entropy encodes each feature element in the quantized feature map , to obtain an encoding code stream.

[0051] At the decoding end, the decoding end parses the code stream, and outputs the probability distribution of the to-be-encoded symbol according to the probability distribution model parameters transmitted by the Hyper-Dec module, and the entropy decoding module arithmetic decodes each feature element in the quantized feature map according to the probability distribution of each feature element , to obtain the value of the feature element ; input the feature map into the image reconstruction module to output the reconstructed map.

[0052] In addition, some Variational Auto Encoder (VAE) probability estimation modules also use the decoded feature elements around the current feature element to more accurately estimate the probability distribution of the current feature element.

[0053] (3) Feature domain task network

[0054] The existing neural network for machine tasks is a visual task neural network taking an image as input (referred to as an image domain task network). That is, the input of the image domain task network is an image, and the image domain task network outputs a processing result after processing the input image. The image domain task network is, for example, Resnet, Faster-RCNN, Mask-RCNN, Yolo, etc. When the image domain task network is combined with video or image coding and decoding applications, a decoder (for example, an image decoder based on an artificial neural network) is required to first parse the code stream to generate a reconstructed feature map, and then input the reconstructed feature map into the image reconstruction network to generate a reconstructed image, which is used as the input of the image domain task network.

[0055] The present application proposes a feature map domain visual task neural network (referred to as a feature domain task network) taking a feature map (a feature map obtained based on an artificial neural network encoder and decoder) as input. The advantage is to skip the image decoding network, take the reconstructed feature map generated by parsing the code stream as input, and directly perform visual tasks, so as to significantly reduce the computing power.

[0056] The feature domain task network has a significant disadvantage compared to the image domain task network. When the image domain task network runs on a low-power device, the image to be analyzed (Image) A (for example, a decoded and reconstructed image) can be down-sampled into a smaller resolution image B, and then image B is input into the image domain task network, thereby reducing the computing power required for neural network inference. This is because the computing power required for neural network inference increases with the increase of the input image resolution. Common down-sampling processing includes bilinear interpolation (Bilinear Interpolation), nearest neighbor interpolation (Nearest Interpolation), bicubic interpolation (Bicubic Bnterpolation) down-sampling, such as the torch.nn.functional.interpolate function in Pytorch. The image after down-sampling processing will not significantly reduce the analysis accuracy of the image domain task network, that is, the analysis accuracy obtained by inputting a high-resolution image A into the image domain task network is slightly higher than that obtained by inputting a low-resolution image B into the image domain task network. The difference between the two analysis accuracies is not large; for example, the performance of Resnet-50 network on the Imagenet validation set decreases by about 1%. This feature allows the image domain task network to easily adjust the computing power by down-sampling the input image.

[0057] However, for the feature domain task network, scaling the feature map of the network input by traditional down-sampling methods such as bilinear interpolation can significantly reduce the analysis accuracy, because the feature map is more nonlinear than the image, and using methods such as bilinear, nearest neighbor interpolation for down-sampling of the feature map can significantly destroy the information in the feature map. For example, for the performance of the cResnet-39 network on the Imagenet validation set, using the nearest neighbor interpolation method to down-sample the input feature map by 2 times will cause the analysis accuracy to decrease by more than 10%.

[0058] The technical solutions provided by the present application will be described in detail below in conjunction with the specific embodiments.

[0059] Please refer to Figure 2 , Figure 2 is a schematic diagram of a neural network model architecture provided by an embodiment of the present application. As Figure 2 indicated, the architecture of the neural network model is a feature domain task network structure, for example, it can be a feature domain task network structure for artificial neural network encoder-decoder feature map; in addition, the architecture of the neural network model is also a multi-input adaptive branch network (Multi-stem Networks) for feature map input. The neural network model includes a backbone network and a head network, the backbone network includes a backbone network front part and a backbone network core part (which contains most of the layers of the backbone network), the backbone network front part includes multiple input adaptive branches (Multiple Stems, also called multi-stem branches) and a branch selection module, each input adaptive branch of the multiple input adaptive branches has different feature map size adjustment capabilities; the branch selection module includes a first branch selection module, which can select one of the multiple input adaptive branches according to the size of the input feature map of the neural network model and the size upper limit of the input feature map of the backbone network core part; after selecting one of the input adaptive branches, the input feature map of the neural network model is input into the one of the input adaptive branches for down-sampling processing, and the feature map obtained after the down-sampling processing has a size not greater than the size upper limit of the input feature map of the backbone network core part, and the feature map obtained after the down-sampling processing is the input of the backbone network core part.

[0060] Among them, the backbone network core part is a common technology in the field of machine vision, for example, it can be a VGG network structure composed of multiple convolutional layers, or a Resnet structure composed of multiple Resblocks, or an Inception structure, etc.

[0061] Among them, the head network can be a classification head network containing a fully connected layer, or a detection frame head network for predicting object detection frame disposal, etc.

[0062] The processing result output by the neural network model can be a classification probability vector in an image classification task, which describes the probability value of each classification; the processing result output by the neural network model can also be an image segmentation map, face recognition result, object detection box, etc., and this application does not make specific limitations on this.

[0063] It should be noted that a neck network can also be set between the backbone network and the head network of this neural network model. Figure 2 (Not shown in the image) The neck network fuses the multi-scale feature maps generated by the backbone network, and the processed result is then fed into the head network. Typical neck networks are Faster RCNN with Feature Pyramid Networks (FPN) and YOLOv4.

[0064] As mentioned earlier, this neural network model selects an input adaptation branch based on the size of the input feature map F_A and the upper limit of the size of the input feature map in the core part of the backbone network. It processes F_A, and the size of the output feature map F_B falls within a preset size threshold range; for example, the size of feature map F_B is not greater than the upper limit of the size of the input feature map in the core part of the backbone network. Then, feature map F_B is input into the core part of the backbone network. Afterward, it passes through the core part of the backbone network and the neck network (…). Figure 2 The head network (not shown in the diagram) processes the data and outputs the results. Since the size of the feature map F_B is within a preset size threshold range, the computational power of the backbone network, neck network, and / or head network can be limited to a certain range. Thus, this application can be applied to feature domain task networks that take feature maps of multiple resolutions as input, ensuring that the computational power of the feature domain task network remains at a similar level for input feature maps of multiple resolutions. This allows a single model to support the analysis of input feature maps of multiple resolutions on a device with limited computational power.

[0065] As described above, each input adaptation branch of this neural network model performs feature map downsampling at different ratios, as illustrated below with examples.

[0066] As an example, such as Figure 2 As shown, the neural network model contains four input adaptation branches: input adaptation branch 1 (completing 1x downsampling), input adaptation branch 2 (completing 1.5x downsampling), input adaptation branch 3 (completing 2x downsampling), and input adaptation branch 4 (completing 3x downsampling), as detailed below.

[0067] The input adaptation branch 1: the width and height of the output feature map F_A_1 are the same as the width and height of the input feature map F_A, that is, the input adaptation branch 1 completes 1 times down-sampling. The input adaptation branch 1 comprises at least one convolutional layer, and the stride of the at least one convolutional layer is 1. Optionally, the input adaptation branch 1 can further comprise a Batch Normalization (BN) layer and an activation layer (for example, a ReLU layer), which are all conventional layers used in a convolutional neural network. Generally, the padding parameter of the convolutional layer is set to (K-1) / / 2, where " / / 2" represents integer division by 2, and K is the size of the kernel.

[0068] The input adaptation branch 2: the width and height of the output feature map F_A_2 are 2 / 3 of the width and height of the input feature map F_A, that is, the input adaptation branch 2 completes 1.5 times down-sampling. In the input adaptation branch 2, the feature map F_A is first up-sampled by 2 times (for example, nearest neighbor up-sampling) to obtain a feature map F_A', and then the feature map F_A' is subjected to a convolutional layer with a stride of 3 to complete 3 times down-sampling, thereby achieving the effect of 1.5 times down-sampling as a whole. Optionally, the input adaptation branch 2 can further comprise other convolutional layers with a stride of 1, a BN layer, a ReLU layer, and the like, which are not limited in the present application.

[0069] The input adaptation branch 3: the width and height of the output feature map F_A_3 are 1 / 2 of the width and height of the input feature map F_A, that is, the input adaptation branch 3 completes 2 times down-sampling. In the input adaptation branch 3, the feature map F_A is subjected to a convolutional layer with a stride of 2 to complete 2 times down-sampling. Optionally, the input adaptation branch 3 can further comprise other convolutional layers with a stride of 1, a BN layer, a ReLU layer, and the like, which are not limited in the present application.

[0070] The input adaptation branch 4: the width and height of the output feature map F_A_4 are 1 / 3 of the width and height of the input feature map F_A, that is, the input adaptation branch 4 completes 3 times down-sampling. In the input adaptation branch 4, the feature map F_A is subjected to a convolutional layer with a stride of 3 to complete 3 times down-sampling. Optionally, the input adaptation branch 4 can further comprise other convolutional layers with a stride of 1, a BN layer, a ReLU layer, and the like, which are not limited in the present application.

[0071] As an example, in addition to the above-mentioned down-sampling completed by one or more convolution layers in series, the input adaptation branch with an input down-sampling rate greater than 1 can also use more complex structures such as a Resblock structure. For example, for the input adaptation branch 3 that implements a 2-fold down-sampling, the input feature map F_A is processed by a convolution layer with a 1x1 convolution kernel and a step size of 2 to obtain an intermediate result F_a; in addition, the input feature map F_A is also processed by a convolution layer with a 3x3 convolution kernel, a step size of 2, and a ReLU layer to generate an intermediate result F_b, and F_b is further processed by a convolution layer to generate F_c; F_c is added to F_a to obtain F_d, and the output of the input adaptation branch 3 is F_d or the result of F_d after the ReLU layer.

[0072] As an implementation manner, the first branch selection module selects a specific input adaptation branch according to the upper limit size (Th_W, Th_H) of the input feature map of the core part of the backbone network and the size (In_W, In_H) of the input feature map of the neural network model, so that the size of the feature map output after the input adaptation branch is selected does not exceed (Th_W, Th_H), wherein Th_W is the width of the upper limit size of the input feature map of the core part of the backbone network, Th_H is the height of the upper limit size of the input feature map of the core part of the backbone network, In_W is the width of the input feature map of the neural network model, and In_H is the height of the input feature map of the neural network model. Taking the input adaptation branch 1, the input adaptation branch 2, the input adaptation branch 3, and the input adaptation branch 4 as examples, an example of the input adaptation branch selection manner is as follows:

[0073] Let RW=In_W / Th_W, RH=In_H / Th_H;

[0074] If max(RW, RH)≤1, use the input adaptation branch 1;

[0075] Otherwise, if max(RW, RH)≤1.5, use the input adaptation branch 2;

[0076] Otherwise, if max(RW, RH)≤2, use the input adaptation branch 3;

[0077] Otherwise, use the input adaptation branch 4;

[0078] Wherein, the function max(a, b) represents the maximum value of a and b.

[0079] For example, as shown in FIG. 2, the input adaptation branch selection manner is as follows: Figure 2As shown, when Th_W and Th_H are set as 16, for the input feature map F_A with width and height of 24, the input feature map F_A is input to the input adaptation branch 2 after the judgment of the branch selection module, 1.5 times down-sampling is completed, and the feature map F_B with width and height of 16 is output. Similarly, for the input feature map F_A with width and height of 32, the input adaptation branch 3 is selected, 2 times down-sampling is completed, and the feature map F_B with width and height of 16 is output. Similarly, for the input feature map F_A with width and height of 48, the input adaptation branch 4 is selected, 3 times down-sampling is completed, and the feature map F_B with width and height of 16 is output. For the input feature map F_A with width and height of 42, the input adaptation branch 4 is also selected, 3 times down-sampling is completed, and the feature map F_B with width and height of 14 is output.

[0080] As an example, the front part of the backbone network can also only contain the input adaptation branch 1 and the input adaptation branch 3. At this time, the processing of the branch selection module is, for example: if max(RW, RH)≤1, the input adaptation branch 1 is used; otherwise, the input adaptation branch 1 is used.

[0081] As an example, the front part of the backbone network can also contain the input adaptation branch 1, the input adaptation branch 2 and the input adaptation branch 4. The processing of the branch selection module is similar to the foregoing, which will not be described herein again.

[0082] As an example, for the case that the resolution range of the input feature map F_A is large, the front part of the backbone network can also contain other input adaptation branches with higher down-sampling rates, such as the input adaptation branch with a rate of 4 times down-sampling, which can be realized by inputting the input feature map F_A into a convolutional layer containing 2 convolutional layers with a step of 2 to realize 4 times down-sampling. These methods of realizing down-sampling through multiple convolutional layers with a step greater than 1 can be any one of the ways, which will not be limited herein and will not be described herein again.

[0083] As an implementation manner, the branch selection module can also include a second branch selection module, which can be used to select the output feature map actually executed by the input adaptation branch, and input the output feature map into the core part of the backbone network. In a software-based implementation manner, the second branch selection module can not be needed. In this implementation manner, the program calls the corresponding branch to generate the output feature map according to the decision result of the first branch selection module, stores the output feature map in the public variable P (that is, no matter which branch is called, the output feature map exists in the variable P), and the core part of the backbone network reads the public variable P, so that the second branch selection module does not appear explicitly in the processing process. In a hardware-based implementation manner, the input adaptation branches are usually circuitized; at this time, the first branch selection module (which completes the selection of the input feature map flowing into the input adaptation branch network) and the second branch selection module (which completes the selection of the output feature map flowing out of the input adaptation branch network) are needed.

[0084] As an implementation manner, the front part of the backbone network further comprises a plurality of convolution layers, i.e., the convolution layers between any one of the plurality of input adaptation branches and the second branch selection module, the input of any one of the plurality of convolution layers is the output of any one of the plurality of input adaptation branches, the output of any one of the plurality of convolution layers is the input of the second branch selection module, and any one of the plurality of convolution layers can be used to improve the fitting capability of the input adaptation branch, so that the feature maps output by each input adaptation branch can have similar distributions.

[0085] It should be noted that the present application does not make specific limitations on the specific implementation manners of the number of input adaptation branches in the neural network model and the down-sampling of the feature maps by each input adaptation branch, and the above is only an exemplary description.

[0086] Please refer to Figure 3 , Figure 3 is a flow chart of a process 300 illustrating a method of processing a feature map according to an embodiment of the present application. The process 300 is described as a series of steps or operations, and it should be understood that the process 300 can be executed in various sequences and / or simultaneously, and is not limited to the execution sequence shown in Figure 3 . The process 300 can be executed by an electronic device, specifically, the process 300 can be executed by the electronic device by invoking a neural network model, the neural network model comprising a plurality of input adaptation branches and a post-processing part, the output of the plurality of input adaptation branches being the input of the post-processing part, the plurality of input adaptation branches being used for down-sampling a feature map, and each of the plurality of input adaptation branches having a different down-sampling rate for the feature map. The neural network model can be the neural network model shown in Figure 2 , and the post-processing part is other part of the neural network model except the front part of the backbone network, for example, the post-processing part comprises the backbone network core part and the head network of the neural network model shown in Figure 2 , and optionally further comprises a neck network. The process 300 includes but is not limited to the following steps or operations: Figure 2

[0087] 301、inputting a first feature map into one of the plurality of input adaptation branches to obtain a second feature map, the one of the plurality of input adaptation branches being determined according to the size of the first feature map and a preset size threshold, the preset size threshold being an upper limit size of the input feature map of the post-processing part, and the size of the second feature map being not greater than the preset size threshold.

[0088] In this application, the source of the feature map input into the neural network model is not limited, which can be any source.​

[0089] In a possible design, the first feature map is obtained by decoding a code stream.

[0090] For example, the first feature map of the input neural network model described above can be a feature map obtained by decoding a code stream, i.e., a feature map obtained by decoding a feature map code stream using an entropy decoding module of an end-to-end image decoder (as shown in FIG. 1). The size (resolution) of the feature map obtained by the entropy decoding module is generally large, and the larger the size of the feature map is, the more computing power is required. In the case of a given neural network model, the feature map with too large size needs to be down-sampled before being processed. Figure 1

[0091] For example, the first feature map can be an input feature map F_A in FIG. 2, and the second feature map can be a feature map F_B in FIG. 2, and the preset size threshold can be an upper limit size of an input feature map of a core part of a backbone network in FIG. 2. Figure 2 Figure 2 Figure 2 For example, the first feature map can be an input feature map F_A in FIG. 2, and the second feature map can be a feature map F_B in FIG. 2, and the preset size threshold can be an upper limit size of an input feature map of a core part of a backbone network in FIG. 2.

[0092] In the design, the first feature map input to the neural network model is a feature map obtained by decoding a code stream, for example, a feature map obtained by decoding a feature map code stream using an entropy decoding module of an end-to-end image decoder. In this way, for video or image coding and decoding, the feature map obtained by decoding the code stream does not need to be reconstructed into a reconstructed image to be input to the neural network model for processing, i.e., the process of image reconstruction can be omitted, and the feature map obtained by decoding the code stream can be directly input to the neural network model for analysis, thereby reducing the computing power.

[0093] In a possible design, any first input adaptation branch in the plurality of input adaptation branches that down-samples the feature map by an integer multiple includes at least one first convolutional layer configured to down-sample the feature map, and a step length of the at least one first convolutional layer is determined according to the down-sampling multiple of the any first input adaptation branch.

[0094] For example, the first input adaptation branch in the plurality of input adaptation branches that down-samples the feature map by an integer multiple includes at least one first convolutional layer configured to down-sample the feature map, and a step length of the at least one first convolutional layer is determined according to the down-sampling multiple of the any first input adaptation branch. Figure 2 ​​​The down-sampling ratios of the input adaptation branch 1, the input adaptation branch 3 and the input adaptation branch 4 shown are integer multiples, the first input adaptation branch can be any one of the input adaptation branch 1, the input adaptation branch 3 and the input adaptation branch 4, the first convolutional layer can be a convolutional layer in any one of the input adaptation branch 1, the input adaptation branch 3 and the input adaptation branch 4, and the step of the convolutional layer in any one of the input adaptation branch 1, the input adaptation branch 3 and the input adaptation branch 4 is determined according to the corresponding down-sampling ratio. For example, the down-sampling ratio of the input adaptation branch 1 is 1, and the step of the convolutional layer in the input adaptation branch 1 is 1; the down-sampling ratio of the input adaptation branch 3 is 2, and the step of the convolutional layer in the input adaptation branch 3 is 2; the down-sampling ratio of the input adaptation branch 4 is 3, and the step of the convolutional layer in the input adaptation branch 4 is 3.

[0095] In the design, for any first input adaptation branch with an integer multiple of down-sampling ratio of a feature map, the input feature map can be down-sampled by at least one first convolutional layer, and the down-sampling ratio of the feature map by the convolutional layer is related to the step of the convolutional layer. After the step of the at least one first convolutional layer is determined according to the down-sampling ratio of the feature map by the any first input adaptation branch, the input feature map of the any first input adaptation branch can be down-sampled by an integer multiple. For example, the down-sampling ratio of the feature map by the any first input adaptation branch is 2, and the step of the at least one first convolutional layer is 2, so that the at least one first convolutional layer can reduce the size of the input feature map of the any first input adaptation branch by 2 times.

[0096] In a possible design, any second input adaptation branch with a non-integer multiple of down-sampling ratio of a feature map in the plurality of input adaptation branches includes at least one up-sampling module and at least one second convolutional layer, the at least one up-sampling module is configured to up-sample the feature map, and the at least one second convolutional layer is configured to down-sample the feature map up-sampled by the at least one up-sampling module, and the up-sampling ratio of the feature map by the at least one up-sampling module and the step of the at least one second convolutional layer are determined according to the down-sampling ratio of the feature map by the any second input adaptation branch.

[0097] For example, Figure 2The down-sampling rate of the input adaptation branch 2 is a non-integer, the second input adaptation branch can be the input adaptation branch 2, the at least one up-sampling module can be the up-sampling module in the input adaptation branch 2, the second convolutional layer can be the convolutional layer in the input adaptation branch 2, the up-sampling rate of the up-sampling module in the input adaptation branch 2 is determined according to the down-sampling rate of the input adaptation branch 2, and the step length of the convolutional layer in the input adaptation branch 2 is determined according to the down-sampling rate of the input adaptation branch 2. For example, the down-sampling rate of the input adaptation branch 2 is 1.5, the up-sampling rate of the up-sampling module in the input adaptation branch 2 is 2, and the step length of the convolutional layer in the input adaptation branch 2 is 3. In this way, the input adaptation branch 2 first up-samples the input feature map by 2, and then down-samples the up-sampled feature map by 3 to output, so that the feature map output by the input adaptation branch 2 is reduced by 1.5 times compared with the input feature map.

[0098] It should be noted that the specific implementation of the up-sampling of the up-sampling module is not limited in the present application, and can be any up-sampling method.

[0099] In the present design, for any second input adaptation branch with a non-integer down-sampling rate of the feature map, the feature map can be first up-sampled by the at least one up-sampling module, and then down-sampled by the at least one second convolutional layer, and the down-sampling rate of the feature map by the convolutional layer is related to the step length of the convolutional layer. After the up-sampling rate of the feature map by the at least one up-sampling module and the step length of the at least one second convolutional layer are determined according to the down-sampling rate of the feature map by the any second input adaptation branch, the non-integer down-sampling of the feature map input into the any second input adaptation branch can be realized. For example, the down-sampling rate of the feature map by the any second input adaptation branch is 1.5, the up-sampling rate of the feature map by the at least one up-sampling module is 2, and the step length of the at least one second convolutional layer is 3. In this way, the size of the input feature map can be first enlarged by 2 by the any second input adaptation branch, and then reduced by 3 to output, so that the size of the feature map is reduced by 1.5 from input to output.

[0100] In a possible design, any third input adaptation branch with a down-sampling rate of the feature map greater than 1 in the plurality of input adaptation branches includes at least one residual block structure (Resblock), and the at least one residual block structure is used for down-sampling the feature map. The at least one residual block structure includes a plurality of third convolutional layers, and the step length of any third convolutional layer in the plurality of third convolutional layers is determined according to the down-sampling rate of the feature map by the any third input adaptation branch.

[0101] For example, Figure 2The down-sampling rate of the input adaptation branch 2, the input adaptation branch 3 and the input adaptation branch 4 shown is greater than 1, the third input adaptation branch can be any one of the input adaptation branch 2, the input adaptation branch 3 and the input adaptation branch 4; any one of the input adaptation branch 2, the input adaptation branch 3 and the input adaptation branch 4 can realize down-sampling of the input feature map through at least one residual block structure, the at least one residual block structure realizes down-sampling processing of the input feature map through a plurality of third convolutional layers; the step length of any third convolutional layer in the plurality of third convolutional layers in the at least one residual block structure in the input adaptation branch 2, the input adaptation branch 3 and the input adaptation branch 4 can be determined according to the down-sampling rate of the input adaptation branch 2, the input adaptation branch 3 and the input adaptation branch 4 to the feature map, respectively. For example, taking the input adaptation branch 3 as an example, the down-sampling of the input adaptation branch 3 is 2, and the residual block structure in the input adaptation branch 3 includes a 1x1 convolutional kernel convolutional layer and a 3x3 convolutional kernel convolutional layer, and the step length of the 1x1 convolutional kernel convolutional layer and the 3x3 convolutional kernel convolutional layer is 2.

[0102] In the design, for any third input adaptation branch with a down-sampling rate greater than 1 to the feature map, it can realize down-sampling of the input feature map through at least one residual block structure, since the at least one residual block structure includes a plurality of third convolutional layers, and the down-sampling rate of the feature map through the convolutional layer is related to the step length of the convolutional layer, after determining the step length of any third convolutional layer in the plurality of third convolutional layers according to the down-sampling rate of the feature map through the any third input adaptation branch, it can realize down-sampling of the input feature map through the any third input adaptation branch by a rate greater than 1; in addition, it can be understood that, in addition to the plurality of third convolutional layers for down-sampling, the residual block structure also includes other layers, but the other layers do not change the size of the input feature map, and the convolutional layer changes the size of the input feature map.

[0103] In a possible design, the one of the input adaptation branches is determined according to the maximum of a first ratio and a second ratio, the first ratio is a ratio between the width of the first feature map and the width of the preset size threshold, and the second ratio is a ratio between the height of the first feature map and the height of the preset size threshold.

[0104] For example, Figure 2Taking input adaptation branches 1, 2, 3, and 4 as examples: the first ratio is the aforementioned RW=In_W / Th_W, and the second ratio is the aforementioned RH=In_H / Th_H; where Th_W is the width of the upper limit size of the input feature map of the backbone network core part, that is, the width of the upper limit size of the input feature map of the post-processing part; Th_H is the height of the upper limit size of the input feature map of the backbone network core part, that is, the height of the upper limit size of the input feature map of the post-processing part; In_W is the width of the input feature map of the neural network model, that is, the width of the first feature map; In_H is... The height of the input feature map of the neural network model, that is, the height of the first feature map; first, determine the relationship between max(RW,RH) and 1. If max(RW,RH)≤1, then use input adaptation branch 1; if max(RW,RH)>1, then determine the relationship between max(RW,RH) and 1.5. If max(RW,RH)≤1.5, then use input adaptation branch 2; if max(RW,RH)>1.5, then determine the relationship between max(RW,RH) and 2. If max(RW,RH)≤2, then use input adaptation branch 3; otherwise, use input adaptation branch 4.

[0105] In this design, the ratio between the width of the first feature map and the width of the upper limit dimension of the input feature map of the post-processing part, and the ratio between the height of the first feature map and the height of the upper limit dimension of the input feature map of the post-processing part are calculated. Then, based on the maximum value of the two ratios, one of the multiple input adaptation branches is selected to downsample the first feature map. In this way, it can be ensured that the width and height of the feature map obtained by downsampling the first feature map will not be greater than the width and height of the upper limit dimension of the input feature map of the post-processing part, respectively.

[0106] 302. Input the second feature map into the post-processing part for post-processing to obtain the processing result of the first feature map.

[0107] For example, the post-processing part includes a backbone network core and a head network, and the second feature map is... Figure 2 The feature map F_B is shown; the feature map F_B is input into the core part of the backbone network to obtain the feature map F_C; then the feature map F_C is input into the head network to obtain the processing result.

[0108] Another example is that the post-processing part includes a backbone network core, a neck network, and a head network, and the second feature map is... Figure 2 Feature map F_B is shown; feature map F_B is input into the core part of the backbone network to obtain feature map F_C; then feature map F_C is input into the neck network to obtain feature map F_D; then feature map F_D is input into the head network to obtain the processing result.

[0109] In one possible design, if the size of the second feature map is greater than the preset size threshold, the method further includes: extracting a third feature map from the second feature map, wherein the size of the third feature map is not greater than the preset size threshold; and inputting the third feature map into the post-processing part for post-processing to obtain the processing result of the first feature map.

[0110] It should be noted that when the input feature map resolution is high, the output feature map after the input adaptation branch with the highest downsampling ratio may still have a width greater than Th_W or a height greater than Th_H. In this case, a sub-region of the feature map output by the input adaptation branch can be truncated, such that the area of ​​this sub-region does not exceed Th_W multiplied by Th_H, or that the width and height of this sub-region do not exceed Th_W and Th_H, respectively. Preferably, the sub-region can usually be a sub-region close to the center of the feature map output by the input adaptation branch (i.e., Center Crop).

[0111] For example, the first feature map is feature map F_A, the second feature map is feature map F_B', and the third feature map is feature map F_B; for Figure 2 As shown in the input adaptation branch, when the width and height of the input feature map F_A are 60, the input adaptation branch 4 will be selected to complete the 3x downsampling and output the feature map F_B' with a width and height of 20. However, this still exceeds Th_W and Th_H. At this time, the feature map region with the center of feature map F_B' as the center, the width of Th_W and the height of Th_H can be extracted to obtain feature map F_B. Feature map F_B is used as the input of the core part of the backbone network.

[0112] In this design, the size of the first input feature map may be too large. After downsampling through one of the input adaptation branches, such as the input adaptation branch with the largest downsampling ratio, the resulting second feature map may still be larger than the upper limit size of the input feature map in the post-processing part. In this case, a third feature map with a size no larger than the upper limit size of the input feature map in the post-processing part is extracted from the second feature map. This third feature map is then input into the post-processing part for post-processing to obtain the processing result of the first feature map. In this way, it can be ensured that the processing result of the first feature map can be obtained after processing the first feature map input to the neural network model.

[0113] In the embodiments of the present application, the visual task neural network for machine task processing is a feature domain task network with a feature map as input. For example, the input of the neural network model designed in the present application can be a feature map reconstructed by decoding a code stream, and the output is a task processing result corresponding to the feature map. In this way, compared with an image domain task network, the process of obtaining a reconstructed image from a reconstructed feature map is skipped, thereby reducing the computing power. Moreover, compared with an image, the computing power of the neural network model for processing a feature map is also smaller. Further, the neural network model designed in the present application includes a plurality of input adaptation branches and a post-processing part. The outputs of the plurality of input adaptation branches are the inputs of the post-processing part. Each input adaptation branch in the plurality of input adaptation branches can perform down-sampling on the feature map at different scales. For a first feature map of different sizes (resolutions) input into the neural network model, one of the plurality of input adaptation branches can be selected according to the size of the first feature map and the upper limit size of the input feature map of the post-processing part, and the first feature map is down-sampled by the selected input adaptation branch to generate a second feature map with a size not greater than the upper limit size of the input feature map of the post-processing part. In this way, the second feature map meets the size requirement of the input feature map of the post-processing part, and the processing result of the first feature map can be obtained by inputting the second feature map into the post-processing part for post-processing. Therefore, the present application can maintain high analysis accuracy with similar computing power when the input feature map of the neural network model is of different resolutions. In addition, compared with training an independent neural network model for each resolution of input feature map (each network has similar computing power), and then processing feature maps of different resolutions by the neural network models trained for different resolutions, the neural network model designed in the present application can achieve the same function through the plurality of input adaptation branches. The plurality of input adaptation branches are in front of the backbone network of the neural network model, and there is only one set of post-processing part (core part of the backbone network and the head network) in the neural network model. In this way, the network parameters can be significantly saved, and the model deployment difficulty can be reduced without switching the network model during inference.

[0114] In a possible design, the method further includes: performing N rounds of training on the neural network model according to a training sample set to obtain N first loss values, the training sample set including sample feature maps of at least N sizes, N being not less than the number of the plurality of input adaptation branches, N being an integer; obtaining a second loss value according to the N first loss values; and updating the parameters of the neural network model according to the second loss value.

[0115] For example, the training sample set of the present application includes sample feature maps of at least N sizes, for example, each batch in the training contains feature maps of N different resolutions of multiple images; wherein the multiple images can be scaled to N resolutions, and then the end-to-end image encoder is run to obtain the feature maps corresponding to the batch of images at each resolution, thereby obtaining feature maps of N different resolutions of multiple images. In this way, by using the following four steps on multiple batches, the network parameters are iteratively updated, thereby training the network parameters of the multiple input adaptation branches; each of the multiple batches contains feature maps of N different resolutions of multiple images:

[0116] (1) Divide each batch into N rounds, and input the feature maps of one resolution in each round to the neural network model to perform inference processing and obtain the loss value of this round. Wherein N is not less than the number of multiple input adaptation branches, for example, N can be equal to the number of multiple input adaptation branches, and each resolution of the N different resolution feature maps is processed by a different input adaptation branch to ensure that each branch is trained more evenly.

[0117] (2) The loss values of the above N rounds are weighted and averaged to obtain the total loss value. For example, the same weight can be used for the loss value corresponding to each input adaptation branch when weighted averaging. For another example, the distribution of the feature map size in the actual application can be analyzed, and the weight of the loss value corresponding to each input adaptation branch is set according to the distribution of the feature map size; wherein the input adaptation branch with greater distribution probability has a greater weight of the loss value corresponding thereto.

[0118] (3) According to the total loss value, the gradient of the network parameters is calculated by backward propagation.

[0119] (4) Update the network parameters according to the gradient.

[0120] In the present design, the training sample set used to train the neural network model of the present application includes sample feature maps of at least N sizes, and the value of N is not less than the number of multiple input adaptation branches. In this way, training the neural network model using the training sample set can enable each input adaptation branch in the multiple input adaptation branches to be trained, so that the N first loss values obtained by training the neural network model for N rounds include the first loss value corresponding to any multiple input adaptation branches in the multiple input adaptation branches; the second loss value is obtained according to the N first loss values, and the parameters of the neural network model are updated according to the second loss value, so that the parameters in any multiple input adaptation branches in the multiple input adaptation branches are also updated.

[0121] Figure 4A structural schematic diagram of a feature map processing apparatus provided in an embodiment of the present application; the feature map processing apparatus 400 is applied to an electronic device, and includes a processing unit 401 and a communication unit 402. The processing unit 401 is configured to perform any step in the method embodiments shown in the foregoing, and when performing data transmission such as acquisition, the communication unit 402 can be selectively invoked to complete the corresponding operation. Details are described below. Figure 3

[0122] In a possible design, the feature map processing apparatus 400 is implemented by invoking a neural network model, the neural network model includes a plurality of input adaptation branches and a post-processing part, outputs of the plurality of input adaptation branches are inputs of the post-processing part, the plurality of input adaptation branches are configured to perform down-sampling on a feature map, and each input adaptation branch in the plurality of input adaptation branches has a different down-sampling rate for the feature map. The processing unit 401 is configured to: input a first feature map into one of the plurality of input adaptation branches to obtain a second feature map, the one of the plurality of input adaptation branches is determined according to a size of the first feature map and a preset size threshold, the preset size threshold is an upper limit size of an input feature map of the post-processing part, and a size of the second feature map is not greater than the preset size threshold; and input the second feature map into the post-processing part for post-processing to obtain a processing result of the first feature map.

[0123] In a possible design, any first input adaptation branch in the plurality of input adaptation branches that has an integer down-sampling rate for the feature map includes at least one first convolutional layer, the at least one first convolutional layer is configured to perform down-sampling on the feature map, and a step length of the at least one first convolutional layer is determined according to the down-sampling rate of the any first input adaptation branch for the feature map.

[0124] In a possible design, any second input adaptation branch in the plurality of input adaptation branches that has a non-integer down-sampling rate for the feature map includes at least one up-sampling module and at least one second convolutional layer, the at least one up-sampling module is configured to perform up-sampling on the feature map, the at least one second convolutional layer is configured to perform down-sampling on the feature map obtained by the up-sampling, and a down-sampling rate of the at least one up-sampling module for the feature map and a step length of the at least one second convolutional layer are determined according to the down-sampling rate of the any second input adaptation branch for the feature map.

[0125] ​In a possible design, any third input adaptation branch of the plurality of input adaptation branches, in which the down-sampling rate of the feature map is greater than 1, includes at least one residual block structure (Resblock) configured to down-sample the feature map, and the at least one residual block structure includes a plurality of third convolutional layers, and a step length of any third convolutional layer of the plurality of third convolutional layers is determined according to the down-sampling rate of the feature map of the any third input adaptation branch.

[0126] In a possible design, the one of the input adaptation branches is determined according to a maximum value of a first ratio and a second ratio, the first ratio being a ratio between a width of the first feature map and a width of the preset size threshold, and the second ratio being a ratio between a height of the first feature map and a height of the preset size threshold.

[0127] In a possible design, if the size of the second feature map is greater than the preset size threshold, the processing unit 401 is further configured to: obtain a third feature map by cropping the second feature map, the size of the third feature map being not greater than the preset size threshold; and input the third feature map into the post-processing part for post-processing, to obtain a processing result of the first feature map.

[0128] In a possible design, the first feature map is a feature map obtained by decoding a bitstream.

[0129] In a possible design, the processing unit 401 is further configured to: perform N rounds of training on the neural network model according to a training sample set, to obtain N first loss values, the training sample set including sample feature maps of at least N sizes, the N being not less than a quantity of the plurality of input adaptation branches, and the N being an integer; obtain a second loss value according to the N first loss values; and update parameters of the neural network model according to the second loss value.

[0130] The feature map processing apparatus 400 can further include a storage unit 403 configured to store program codes and data of the electronic device. The processing unit 401 can be a processor, the communication unit 402 can be a transceiver, and the storage unit 403 can be a memory.

[0131] It should be noted that implementation of each unit can also correspond to the respective description of the method embodiments shown in Figure 3

[0132] In Figure 4 ​In the processing apparatus 400 of the feature map described, the visual task neural network for machine task processing is a feature domain task network taking the feature map as input, for example, the input of the neural network model designed in the present application can be a feature map reconstructed by decoding a code stream, and the output is a task processing result corresponding to the feature map. In this way, compared with the image domain task network, the process of obtaining the reconstructed image according to the reconstructed feature map is skipped, thereby reducing the computing power. Moreover, compared with the image, the computing power of the neural network model for processing the feature map is also smaller. Further, the neural network model designed in the present application includes a plurality of input adaptation branches and a post-processing part, the output of the plurality of input adaptation branches is the input of the post-processing part, and each input adaptation branch in the plurality of input adaptation branches can perform down-sampling on the feature map at different scales. For a first feature map of different sizes (resolutions) input into the neural network model, one of the plurality of input adaptation branches can be selected according to the size of the first feature map and the upper limit size of the input feature map of the post-processing part, the first feature map is down-sampled by the one of the plurality of input adaptation branches to generate a second feature map with a size not greater than the upper limit size of the input feature map of the post-processing part. In this way, the second feature map meets the size requirement of the input feature map of the post-processing part, and the processing result of the first feature map can be obtained by inputting the second feature map into the post-processing part for post-processing. Therefore, the present application can maintain high analysis accuracy with similar computing power when the input feature map of the neural network model is of different resolutions. In addition, compared with training an independent neural network model for each resolution of input feature map (each network has similar computing power), and then processing feature maps of different resolutions by the neural network model trained for different resolutions, the neural network model designed in the present application can achieve the same function through the plurality of input adaptation branches, and the plurality of input adaptation branches are in front of the backbone network of the neural network model, and there is only one set of post-processing part (core part of the backbone network and the head network) in the neural network model. In this way, the network parameters can be significantly saved, and the model deployment difficulty can be reduced without switching the network model during inference.

[0133] Please refer to Figure 5 , Figure 5 is a structural schematic diagram of an electronic device 510 provided by an embodiment of the present application. The electronic device 510 includes a processor 511, a memory 512 and a communication interface 513, and the processor 511, the memory 512 and the communication interface 513 are connected to each other through a bus 514.

[0134] The memory 512 includes, but is not limited to, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read only memory (EPROM), or a compact disc read-only memory (CD-ROM), which is used for storing relevant computer programs and data. The communication interface 513 is used for receiving and sending data.

[0135] The processor 511 can be one or more central processing units (CPUs). In the case where the processor 511 is a CPU, the CPU can be a single-core CPU or a multi-core CPU.

[0136] The processor 511 in the electronic device 510 is configured to read the computer program code stored in the memory 512 and execute the computer program code to implement the method shown in Figure 3

[0137] It should be noted that the implementation of each operation can also correspond to the description of the corresponding embodiment of the method shown in Figure 3

[0138] In the above embodiments, the processor 511 is configured to read the computer program code stored in the memory 512 and execute the computer program code to implement the method shown in Figure 5 ​​In the electronic device 510 described, the visual task neural network for machine task processing is a feature domain task network taking a feature map as input, for example, the input of the neural network model designed in the present application can be a feature map reconstructed by decoding a code stream, and the output is a task processing result corresponding to the feature map. In this way, compared with the image domain task network, the process of obtaining a reconstructed image from the reconstructed feature map is skipped, thereby reducing the computing power. Moreover, compared with an image, the computing power of the neural network model for processing the feature map is also smaller. Further, the neural network model designed in the present application includes a plurality of input adaptation branches and a post-processing part, the outputs of the plurality of input adaptation branches are the inputs of the post-processing part, and each input adaptation branch in the plurality of input adaptation branches can perform down-sampling on the feature map at different scales. For a first feature map of different sizes (resolutions) input into the neural network model, one of the plurality of input adaptation branches can be selected according to the size of the first feature map and the upper limit size of the input feature map of the post-processing part, the first feature map is down-sampled by using the one of the plurality of input adaptation branches to generate a second feature map with a size not greater than the upper limit size of the input feature map of the post-processing part. In this way, the second feature map meets the size requirement of the input feature map of the post-processing part, and the processing result of the first feature map can be obtained by inputting the second feature map into the post-processing part for post-processing. Therefore, the present application can maintain high analysis accuracy with similar computing power when the input feature map of the neural network model is of different resolutions. In addition, compared with training an independent neural network model for each resolution of input feature map (each network has similar computing power), and then processing feature maps of different resolutions by using the neural network models trained for different resolutions, the neural network model designed in the present application can achieve the same function through the plurality of input adaptation branches, and the plurality of input adaptation branches are in front of the backbone network of the neural network model, and there is only one set of post-processing part (core part of the backbone network and the head network) in the neural network model. In this way, the network parameters can be significantly saved, and the model deployment difficulty can be reduced without switching the network model during inference.

[0139] The embodiment of the present application also provides a chip, which comprises at least one processor, a memory and an interface circuit, the memory, the transceiver and the at least one processor are connected through lines, and the at least one memory stores a computer program; when the computer program is executed by the processor, Figure 3 The method flow shown is implemented.

[0140] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and when the computer program is executed on a computer, Figure 3 The method flow shown is implemented.

[0141] The embodiment of the present application further provides a computer program product, when the computer program product is run on a computer, Figure 3 Figure 3 The method flow shown is implemented.

[0142] It should be understood that the processor mentioned in the embodiment of the present application can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), ready programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0143] It should also be understood that the memory mentioned in the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchlink dynamic random access memory (SLDRAM) and direct memory bus random access memory (Direct Rambus RAM, DR RAM).

[0144] It should be noted that when the processor is a general processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, the memory (storage module) is integrated in the processor.

[0145] It should be noted that the memory described herein is intended to include, but not limited to, these and any other suitable types of memory.

[0146] It should also be understood that the first, second, third, fourth and various numerical references referred to herein are only for the convenience of differentiation in the description, and do not limit the scope of the application.

[0147] It should be understood that the term "and / or" herein is only a description of the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " herein generally represents an "or" relationship between the associated objects before and after it.

[0148] It should be understood that in various embodiments of the application, the size of the sequence number of each process described above does not mean the order of execution, and the execution order of each process should be determined by its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the application.

[0149] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the application.

[0150] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0151] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other manners. For example, the described device embodiments are merely schematic. The division of the units is merely logical function division. There can be another division manner for the actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0152] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, can be located in one place, or can be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0153] In addition, each functional unit in the various embodiments of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0154] If the above functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part of the prior art that contributes to the technical solutions or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The foregoing storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and various program codes that can be stored in the medium.

[0155] The steps in the method embodiments of the present application can be adjusted, combined and deleted according to actual needs.

[0156] The modules in the device embodiments of the present application can be combined, divided and deleted according to actual needs.

[0157] The above, the above examples are only used to illustrate the technical solutions of the application, rather than limit them; although the application is described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can still be modified, or some technical features therein can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the application.

Claims

1. A method of processing a feature map, characterized by, The method is implemented by calling a neural network model, the neural network model comprises a plurality of input adaptation branches and a post-processing part, outputs of the plurality of input adaptation branches are inputs of the post-processing part, the plurality of input adaptation branches are used for down-sampling feature maps, and down-sampling rates of the feature maps by each input adaptation branch in the plurality of input adaptation branches are different; and the method comprises the following steps: inputting a first feature map into one of the plurality of input adaptation branches to obtain a second feature map, the one of the plurality of input adaptation branches being determined according to a size of the first feature map and a preset size threshold, the preset size threshold being an upper limit size of an input feature map of the post-processing part, and the first feature map being a feature map obtained by decoding a bitstream; if the size of the second feature map is not greater than the preset size threshold, inputting the second feature map into the post-processing part for post-processing to obtain a processing result of the first feature map.

2. The method of claim 1, wherein, Any first input adaptation branch in the plurality of input adaptation branches, in which a down-sampling rate of a feature map is an integer multiple, comprises at least one first convolutional layer, the at least one first convolutional layer is used for down-sampling the feature map, and a step length of the at least one first convolutional layer is determined according to the down-sampling rate of the feature map by the any first input adaptation branch.

3. The method of claim 1, wherein, Any second input adaptation branch in the plurality of input adaptation branches, in which a down-sampling rate of a feature map is a non-integer multiple, comprises at least one up-sampling module and at least one second convolutional layer, the at least one up-sampling module is used for up-sampling the feature map, the at least one second convolutional layer is used for down-sampling the feature map up-sampled by the at least one up-sampling module, and an up-sampling rate of the feature map by the at least one up-sampling module and a step length of the at least one second convolutional layer are determined according to the down-sampling rate of the feature map by the any second input adaptation branch.

4. The method of claim 1, wherein, Any third input adaptation branch in the plurality of input adaptation branches, in which a down-sampling rate of a feature map is greater than 1, comprises at least one residual block structure (Resblock), the at least one residual block structure is used for down-sampling the feature map, and the at least one residual block structure comprises a plurality of third convolutional layers, a step length of any third convolutional layer in the plurality of third convolutional layers being determined according to the down-sampling rate of the feature map by the any third input adaptation branch.

5. The method according to any one of claims 1 to 4, characterized in that, The one of the plurality of input adaptation branches is determined according to a maximum value of a first ratio and a second ratio, the first ratio being a ratio between a width of the first feature map and a width of the preset size threshold, and the second ratio being a ratio between a height of the first feature map and a height of the preset size threshold.

6. The method of claim 1, wherein, If the size of the second feature map is greater than the preset size threshold, the method further comprises the following steps: cutting a third feature map from the second feature map, the size of the third feature map being not greater than the preset size threshold; inputting the third feature map into the post-processing part for post-processing to obtain the processing result of the first feature map.

7. The method of claim 1, wherein, The method further comprises the following steps: The neural network model is trained according to a training sample set for N rounds to obtain N first loss values, the training sample set includes sample feature maps of at least N sizes, N is not less than the number of the plurality of input adaptation branches, and N is an integer; a second loss value is obtained according to the N first loss values; parameters of the neural network model are updated according to the second loss value.

8. A feature map processing apparatus, characterized by comprising: The device is implemented by calling a neural network model, the neural network model includes a plurality of input adaptation branches and a post-processing part, outputs of the plurality of input adaptation branches are inputs of the post-processing part, the plurality of input adaptation branches are used for downsampling feature maps, and a downsampling rate of each input adaptation branch in the plurality of input adaptation branches is different; the device includes a processing unit, which is configured to: input a first feature map into one of the plurality of input adaptation branches to obtain a second feature map, the one of the plurality of input adaptation branches is determined according to a size of the first feature map and a preset size threshold, the preset size threshold is an upper limit size of an input feature map of the post-processing part, and the first feature map is a feature map obtained by decoding a bitstream; if the size of the second feature map is not greater than the preset size threshold, input the second feature map into the post-processing part for post-processing to obtain a processing result of the first feature map.

9. The apparatus of claim 8, wherein, Any first input adaptation branch in the plurality of input adaptation branches, which has an integer times of downsampling rate for a feature map, includes at least one first convolutional layer, the at least one first convolutional layer is used for downsampling the feature map, and a step length of the at least one first convolutional layer is determined according to the downsampling rate of the feature map of the any first input adaptation branch.

10. The apparatus of claim 8, wherein, Any second input adaptation branch in the plurality of input adaptation branches, which has a non-integer times of downsampling rate for a feature map, includes at least one upsampling module and at least one second convolutional layer, the at least one upsampling module is used for upsampling the feature map, the at least one second convolutional layer is used for downsampling the feature map obtained by the upsampling, and an upsampling rate of the feature map of the at least one upsampling module and a step length of the at least one second convolutional layer are determined according to the downsampling rate of the feature map of the any second input adaptation branch.

11. The apparatus of claim 8, wherein, Any third input adaptation branch in the plurality of input adaptation branches, which has a downsampling rate greater than 1 for a feature map, includes at least one residual block structure (Resblock), the at least one residual block structure is used for downsampling the feature map, the at least one residual block structure includes a plurality of third convolutional layers, and a step length of any third convolutional layer in the plurality of third convolutional layers is determined according to the downsampling rate of the feature map of the any third input adaptation branch.

12. The device of any one of claims 8-11, wherein, The one of the plurality of input adaptation branches is determined according to a maximum value of a first ratio and a second ratio, the first ratio is a ratio between a width of the first feature map and a width of the preset size threshold, and the second ratio is a ratio between a height of the first feature map and a height of the preset size threshold.

13. The apparatus of claim 8, wherein, If the size of the second feature map is greater than the preset size threshold, the processing unit is further configured to: cut a third feature map from the second feature map, wherein the size of the third feature map is not greater than the preset size threshold; input the third feature map into the post-processing part for post-processing to obtain a processing result of the first feature map.

14. The apparatus of claim 8, wherein, The processing unit is further configured to: perform N rounds of training on the neural network model according to a training sample set to obtain N first loss values, wherein the training sample set includes sample feature maps of at least N sizes, N is not less than the number of the plurality of input adaptation branches, and N is an integer; obtain a second loss value according to the N first loss values; update parameters of the neural network model according to the second loss value.

15. An electronic device, comprising: comprise: one or more processors; a non-transitory computer-readable storage medium coupled to the processors and storing programming for execution by the processors, wherein the programming, when executed by the processors, causes the electronic device to carry out the method of any one of claims 1-7.

16. A non-transitory computer-readable storage medium, comprising: comprise program code for carrying out the method of any one of claims 1-7 when executed by a computer device.

17. A chip, characterized by comprise: a processor configured to call and run a computer program from a memory, so that a device installed with the chip carries out the method of any one of claims 1-7.

Images