Task processing method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to a task processing method and device, electronic equipment and storage medium. The method comprises: obtaining a to-be-processed multimedia resource and feature distribution information, the feature distribution information representing the distribution of effective features in the to-be-processed multimedia resource; determining a first resource coding feature of the to-be-processed multimedia resource; inputting the feature distribution information and the first resource coding feature into a feature pruning network for feature learning to obtain a second resource coding feature, the feature pruning network being used to learn the resource coding feature of the to-be-processed multimedia resource in the case of invalid feature pruning of the first resource coding feature based on the feature distribution information; and performing a task processing operation corresponding to a target task based on the second resource coding feature to obtain a target task processing result corresponding to the to-be-processed multimedia resource. The embodiments of the present disclosure can improve resource representation accuracy, effectiveness, feature coding efficiency and network performance, thereby improving task processing effect.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, and particularly relates to a task processing method and device, electronic equipment and storage medium. BACKGROUND

[0002] In recent years, with the rapid development of artificial intelligence technology, more and more task processing such as text translation, image repair and unmanned driving can be solved by combining artificial intelligence technology. As the most important part of applying artificial intelligence technology to the task processing process, the feature encoding network has also experienced an explosive development, from traditional CNN (Convolutional Neural Network) to ResNet (Residual Neural Network), and then to the current feature extraction structure based on the Transformer network, the feature extraction is getting deeper and deeper. In the related art, although the feature encoding network can mine deep information, in some task processing scenarios, the task data to be processed itself often contains a large amount of interference information, which causes the mined features to be unable to effectively represent the corresponding task data, and further causes the problems of low feature encoding efficiency, poor network performance, poor task processing effect such as image distortion, low classification and recognition accuracy, and the like. SUMMARY

[0003] The present disclosure provides a task processing method, device, electronic equipment and storage medium to at least solve the problems in the related art that the features required in the task processing process cannot be effectively mined, the feature encoding efficiency is low, the network performance is poor, and the task processing effect is poor. The technical solutions of the present disclosure are as follows:

[0004] According to a first aspect of an embodiment of the present disclosure, a task processing method is provided, comprising:

[0005] obtaining a to-be-processed multimedia resource and feature distribution information corresponding to the to-be-processed multimedia resource, the feature distribution information representing the distribution of effective features in the to-be-processed multimedia resource;

[0006] determining a first resource encoding feature of the to-be-processed multimedia resource;

[0007] inputting the feature distribution information and the first resource encoding feature into a feature pruning network for feature learning to obtain a second resource encoding feature, the feature pruning network being used to learn the resource encoding feature of the to-be-processed multimedia resource in the case of invalid feature pruning of the first resource encoding feature based on the feature distribution information;

[0008] Based on the second resource coding feature, a task processing operation corresponding to the target task is performed to obtain a target task processing result corresponding to the to-be-processed multimedia resource.

[0009] In an optional embodiment, the first resource coding feature includes target query features, target key features, and target value features of a plurality of resource blocks corresponding to the to-be-processed multimedia resource; the feature pruning network includes a resource association analysis network, a weight learning network, a weighted association analysis network, and an aggregation processing network; and the inputting of the feature distribution information and the first resource coding feature into the feature pruning network for feature learning to obtain the second resource coding feature includes:

[0010] The target query features, the target key features, and the feature distribution information are input into the resource association analysis network for resource block association analysis to obtain resource block association information; the resource block association information represents an association degree between effective features corresponding to a target resource pair, the target resource pair including a resource block pair composed of a target resource block and the plurality of resource blocks, and the target resource block being any resource block in the plurality of resource blocks;

[0011] The feature distribution information is input into the weight learning network for feature weight learning to obtain target weight information corresponding to the target resource pair, the target weight information representing an importance degree of the target resource pair in a task processing process of the target task;

[0012] The target weight information and the resource block association information are input into the weighted association analysis network for weighted association analysis to obtain weighted association information;

[0013] The weighted association information and the target value features are input into the aggregation processing network for aggregation processing to obtain the second resource coding feature.

[0014] In an optional embodiment, the second resource coding feature includes resource coding features of the plurality of resource blocks; and the performing of a task processing operation corresponding to the target task based on the second resource coding feature to obtain a target task processing result corresponding to the to-be-processed multimedia resource includes:

[0015] The resource coding features of the plurality of resource blocks are spliced to obtain a target resource coding feature of the to-be-processed multimedia resource;

[0016] The target resource coding feature is input into a target decoding network corresponding to the target task for task decoding processing to obtain the target task processing result.

[0017] In an optional embodiment, the first resource encoding feature comprises target query features, target key features and target value features of a plurality of resource blocks corresponding to the multimedia resource to be processed; and the determining the first resource encoding feature of the multimedia resource to be processed comprises:

[0018] performing encoding processing on the multimedia resource to be processed to obtain third resource encoding features;

[0019] performing segmentation processing on the third resource encoding features to obtain block resource encoding features of each resource block in the plurality of resource blocks;

[0020] performing concatenation processing on the block resource encoding features of the plurality of resource blocks in a channel dimension corresponding to the block resource encoding features to obtain fourth resource encoding features;

[0021] generating the target query features, the target key features and the target value features based on the fourth resource encoding features.

[0022] In an optional embodiment, in a case where the multimedia resource to be processed is a plurality of video images in a video to be repaired, the generating the target query features, the target key features and the target value features based on the fourth resource encoding features comprises:

[0023] performing convolutional encoding processing on the fourth resource encoding features to obtain initial query features, initial key features and initial value features of the plurality of resource blocks;

[0024] performing morphological alignment processing on the initial query features, the initial key features and the initial value features to obtain the target key features and the target value features;

[0025] wherein the target query features are the initial query features, the target key features are key features after morphological alignment of the initial query features, and the target value features are value features after morphological alignment of the initial query features.

[0026] In an optional embodiment, the performing morphological alignment processing on the initial query features, the initial key features and the initial value features to obtain the target key features and the target value features comprises:

[0027] inputting the initial query features, the initial key features and the initial value features into a morphological alignment network to perform morphological alignment processing to obtain the target key features and the target value features.

[0028] In an optional embodiment, the morphological alignment network comprises a morphological parameter learning network and an alignment processing network.

[0029] The inputting the initial query feature, the initial key feature and the initial value feature into the morphing alignment network for morphing alignment processing to obtain the target key feature and the target value feature includes:

[0030] Inputting the initial query feature of the target resource block and the target block feature of the plurality of resource blocks into the morphing parameter learning network for morphing parameter learning to obtain a morphing parameter; the morphing parameter represents motion morphing information of the plurality of resource blocks relative to the target resource block; the target resource block is any resource block in the plurality of resource blocks, and the target block feature is the initial key feature and / or the initial value feature;

[0031] Inputting the morphing parameter, the initial key feature and the initial value feature into the alignment processing network for alignment processing to obtain the target key feature and the target value feature.

[0032] In an optional embodiment, in a case where the to-be-processed multimedia resource includes at least one to-be-processed image, the feature distribution information is acquired in the following manner:

[0033] An image mask corresponding to the at least one to-be-processed image is acquired.

[0034] Based on the image mask, the feature distribution information is generated.

[0035] According to a second aspect of the embodiments of the present disclosure, a task processing apparatus is provided, including:

[0036] An information acquisition module configured to perform acquisition of a to-be-processed multimedia resource and feature distribution information corresponding to the to-be-processed multimedia resource, the feature distribution information representing a distribution of valid features in the to-be-processed multimedia resource;

[0037] A resource encoding feature determination module configured to perform determination of a first resource encoding feature of the to-be-processed multimedia resource;

[0038] A feature learning module configured to perform inputting of the feature distribution information and the first resource encoding feature into a feature pruning network for feature learning to obtain a second resource encoding feature, the feature pruning network being used to learn resource encoding features of the to-be-processed multimedia resource in a case where invalid feature pruning is performed on the first resource encoding feature based on the feature distribution information;

[0039] A task processing module configured to perform a task processing operation corresponding to a target task based on the second resource encoding feature to obtain a target task processing result corresponding to the to-be-processed multimedia resource.

[0040] In an optional embodiment, the first resource encoding feature comprises target query features, target key features and target value features of a plurality of resource blocks corresponding to the to-be-processed multimedia resource; the feature pruning network comprises a resource association analysis network, a weight learning network, a weighted association analysis network and an aggregation processing network; the feature learning module comprises:

[0041] a resource block association analysis unit configured to perform resource block association analysis by inputting the target query features, the target key features and the feature distribution information into the resource association analysis network to obtain resource block association information; the resource block association information represents an association degree between effective features corresponding to a target resource pair, the target resource pair comprising a resource block pair composed of a target resource block and the plurality of resource blocks, the target resource block being any resource block in the plurality of resource blocks;

[0042] a feature weight learning unit configured to perform feature weight learning by inputting the feature distribution information into the weight learning network to obtain target weight information corresponding to the target resource pair, the target weight information representing an importance degree of the target resource pair in a task processing process of the target task;

[0043] a weighted processing unit configured to perform weighted association analysis by inputting the target weight information and the resource block association information into the weighted association analysis network to obtain weighted association information;

[0044] a weighted summation processing unit configured to perform aggregation processing by inputting the weighted association information and the target value features into the aggregation processing network to obtain the second resource encoding feature.

[0045] In an optional embodiment, the second resource encoding feature comprises resource encoding features of the plurality of resource blocks; the task processing module comprises:

[0046] a first splicing processing unit configured to perform splicing processing on the resource encoding features of the plurality of resource blocks to obtain target resource encoding features of the to-be-processed multimedia resource;

[0047] a task decoding processing unit configured to perform task decoding processing by inputting the target resource encoding features into a target decoding network corresponding to the target task to obtain the target task processing result.

[0048] In an optional embodiment, the first resource encoding feature comprises target query features, target key features and target value features of a plurality of resource blocks corresponding to the to-be-processed multimedia resource; the resource encoding feature determination module comprises:

[0049] The encoding processing unit is configured to perform encoding processing on the multimedia resource to be processed to obtain a third resource encoding feature;

[0050] The segmentation processing unit is configured to perform segmentation processing on the third resource encoding feature to obtain a block resource encoding feature of each resource block in the plurality of resource blocks;

[0051] The second splicing processing unit is configured to perform splicing processing on the block resource encoding features of the plurality of resource blocks in the channel dimension corresponding to the block resource encoding features to obtain a fourth resource encoding feature;

[0052] The resource encoding feature generation unit is configured to generate the target query feature, the target key feature, and the target value feature based on the fourth resource encoding feature.

[0053] In an optional embodiment, in a case where the multimedia resource to be processed is a plurality of video images in a video to be repaired, the resource encoding feature generation unit comprises:

[0054] The convolutional encoding processing unit is configured to perform convolutional encoding processing on the fourth resource encoding feature to obtain an initial query feature of the plurality of resource blocks, an initial key feature of the plurality of resource blocks, and an initial value feature of the plurality of resource blocks;

[0055] The morphological alignment processing unit is configured to perform morphological alignment processing on the initial query feature, the initial key feature, and the initial value feature to obtain the target key feature and the target value feature;

[0056] The target query feature is the initial query feature, the target key feature is a key feature after morphological alignment of the initial query feature, and the target value feature is a value feature after morphological alignment of the initial query feature.

[0057] In an optional embodiment, the morphological alignment processing unit is specifically configured to perform morphological alignment processing on the initial query feature, the initial key feature, and the initial value feature by inputting the initial query feature, the initial key feature, and the initial value feature into a morphological alignment network to obtain the target key feature and the target value feature.

[0058] In an optional embodiment, the morphological alignment network comprises a morphological parameter learning network and an alignment processing network.

[0059] The morphological alignment processing unit comprises:

[0060] The deformation parameter learning unit is configured to perform deformation parameter learning on the initial query feature of the target resource block and the target block feature of the plurality of resource blocks input into the deformation parameter learning network, to obtain a deformation parameter; the deformation parameter represents motion deformation information of the plurality of resource blocks relative to the target resource block; the target resource block is any resource block in the plurality of resource blocks, and the target block feature is the initial key feature and / or the initial value feature.

[0061] The alignment processing unit is configured to perform alignment processing on the deformation parameter, the initial key feature and the initial value feature input into the alignment processing network, to obtain the target key feature and the target value feature.

[0062] In an optional embodiment, when the to-be-processed multimedia resource includes at least one to-be-processed image, the information obtaining module includes:

[0063] The mask image obtaining unit is configured to perform obtaining a mask image corresponding to the at least one to-be-processed image.

[0064] The feature distribution information generating unit is configured to perform generating the feature distribution information based on the mask image.

[0065] According to a third aspect of embodiments of the present disclosure, an electronic device is provided, including: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the method according to any one of the first aspect.

[0066] According to a fourth aspect of embodiments of the present disclosure, a computer readable storage medium is provided, when instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the method according to any one of the first aspect of embodiments of the present disclosure.

[0067] According to a fifth aspect of embodiments of the present disclosure, a computer program product containing instructions is provided, when the computer program product is executed on a computer, the computer can execute the method according to any one of the first aspect of embodiments of the present disclosure.

[0068] The technical solutions provided by the embodiments of the present disclosure at least bring the following beneficial effects:

[0069] In the task processing process, after determining the first resource coding feature of the to-be-processed multimedia resource, in the feature pruning network, in combination with the feature distribution information that can represent the effective feature distribution situation in the to-be-processed multimedia resource, the second resource coding feature of the to-be-processed multimedia resource is learned in the case of invalid feature pruning of the first resource coding feature. The interference of invalid information in the learning of the resource coding feature can be effectively avoided, the representation accuracy and effectiveness of the learned second resource coding feature to the to-be-processed multimedia resource are greatly improved on the basis of improving the feature coding efficiency and network performance, and the effects of image repair, image classification, text translation and the like in the task processing are also improved.

[0070] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0071] The accompanying drawings incorporated in the specification and forming a part of the specification illustrate embodiments consistent with the present disclosure, and together with the description, serve to explain the principles of the present disclosure, and do not limit the present disclosure.

[0072] Figure 1 is a schematic diagram of an application environment according to an exemplary embodiment;

[0073] Figure 2 is a flowchart of a task processing method according to an exemplary embodiment;

[0074] Figure 3 is a flowchart of determining a first resource coding feature of a to-be-processed multimedia resource according to an exemplary embodiment;

[0075] Figure 4 is a flowchart of inputting feature distribution information and a first resource coding feature into a feature pruning network for feature learning to obtain a second resource coding feature according to an exemplary embodiment;

[0076] Figure 5 is a schematic diagram of a feature learning process based on a feature pruning network according to an exemplary embodiment;

[0077] Figure 6 is a schematic diagram in a task processing process according to an exemplary embodiment;

[0078] Figure 7 is an effect diagram of image repair based on the task processing method provided by the embodiments of the present disclosure according to an exemplary embodiment;

[0079] Figure 8is an effect picture of image repairing according to another task processing method provided by an example embodiment of the present disclosure;

[0080] Figure 9 is an effect picture of image repairing according to another task processing method provided by an example embodiment of the present disclosure;

[0081] Figure 10 is a block diagram of a task processing device according to an example embodiment;

[0082] Figure 11 is a block diagram of an electronic device for task processing according to an example embodiment;

[0083] Figure 12 is another block diagram of an electronic device for task processing according to an example embodiment. DETAILED DESCRIPTION

[0084] In order to make the ordinary person skilled in the art better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below in conjunction with the drawings.

[0085] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation described in the following example embodiments does not represent all implementations consistent with the present disclosure. Rather, they are only examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0086] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, analyzed data, etc.) involved in the present disclosure are all information and data authorized by the user or authorized by all parties.

[0087] Please refer to Figure 1 , Figure 1 is a schematic diagram of an application environment according to an example embodiment, which can include a terminal 100 and a server 200.

[0088] In an optional embodiment, the terminal 100 can be used to provide a task processing service for any user. Specifically, the terminal 100 can include, but is not limited to, an electronic device such as a smart phone, a desktop computer, a tablet computer, a notebook computer, a smart speaker, a digital assistant, an augmented reality (AR) / virtual reality (VR) device, a smart wearable device, etc., and can also be a software such as an application program running on the electronic device. Optionally, the operating system running on the electronic device can include, but is not limited to, an Android system, an IOS system, a Linux, a Windows, etc.

[0089] In an optional embodiment, the server 200 can provide a background service for the terminal 100. Optionally, the server can pre-train a task processing network and a feature pruning network; and correspondingly, the terminal 100 can combine the task processing network and the feature pruning network to process a task. Specifically, the server 200 can be a physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0090] In addition, it should be noted that Figure 1 The above-mentioned terminal 100 and server 200 can be directly or indirectly connected through wired or wireless communication, and the present disclosure does not limit the same.

[0091] In the embodiments of the present disclosure, the terminal 100 and the server 200 can be directly or indirectly connected through wired or wireless communication, and the present disclosure does not limit the same.

[0092] Figure 2 is a flowchart of a task processing method according to an exemplary embodiment. As shown in Figure 2 The task processing method is used in a terminal electronic device, and includes the following steps.

[0093] In step S201, a to-be-processed multimedia resource and feature distribution information corresponding to the to-be-processed multimedia resource are obtained.

[0094] In an embodiment, the multimedia resource to be processed can be a multimedia resource required for task processing of the target task; optionally, the multimedia resource can include text, image, and the like. The target task can be set in combination with actual application, and different target tasks can correspond to different multimedia resources to be processed; optionally, taking an image restoration task as an example, the multimedia resource to be processed can be at least one image to be restored; specifically, the at least one image to be restored can be at least one frame of image to be restored in a video, and the like; optionally, taking an image classification task as an example, the multimedia resource to be processed can be at least one image to be classified; specifically, the at least one image to be classified can be an environmental image around a vehicle in a driving scene; accordingly, objects in the environmental image need to be classified; optionally, taking a translation task as an example, the multimedia resource to be processed can be at least one text information to be translated.

[0095] In an embodiment, the feature distribution information can represent a distribution of effective features in the multimedia resource to be processed. In an optional embodiment, when the multimedia resource to be processed includes at least one image to be processed, the feature distribution information can be obtained in the following manner:

[0096] obtaining a mask image corresponding to the at least one image to be processed;

[0097] generating the feature distribution information based on the mask image.

[0098] In an embodiment, the at least one image to be processed can be at least one image to be restored, at least one image to be classified, and the like.

[0099] In an optional embodiment, taking the at least one image to be processed as at least one image to be restored as an example, the effective features can be features of a region to be restored in the image to be restored; accordingly, the mask image corresponding to the image to be processed can be a mask image in which 1 is a pixel value of a region to be restored in the image to be processed, and 0 is a pixel value of an effective region (a region to be restored) in the image to be processed.

[0100] In another optional embodiment, taking the at least one image to be processed as at least one image to be classified as an example, the effective features can be features of a target classification object in the image to be classified; optionally, the target classification object can correspond to different objects in combination with actual application scenarios, for example, in the driving scene, the target classification object can be a vehicle, a traffic signal, a road sign, and the like in the environmental image. Accordingly, the mask image corresponding to the image to be processed can be a mask image in which 0 is a pixel value of an object region (a region where the target classification object is located) in the image to be processed, and 1 is a pixel value of a non-object region in the image to be processed.

[0101] In a specific embodiment, the generating the feature distribution information based on the mask image can include subtracting the pixel value corresponding to each pixel point in the mask image to obtain the feature distribution information; and correspondingly, the feature distribution information can be a pixel image representing the distribution of effective features in the to-be-processed multimedia resource.

[0102] In the above embodiments, when the to-be-processed multimedia resource includes at least one to-be-processed image, the feature distribution information is generated in combination with the mask image corresponding to the at least one to-be-processed image, which can effectively ensure the accuracy and effectiveness of the feature distribution information in representing the distribution of effective features in the to-be-processed image.

[0103] In an optional embodiment, when the to-be-processed multimedia resource includes at least one to-be-processed text information (for example, to-be-translated text information), the feature distribution information can be a word feature representing the distribution of effective words in the to-be-processed text information; specifically, 1 can be set as the word feature of the effective word, and 0 can be set as the word feature of the ineffective word; optionally, the effective word can be a word having a relatively large impact on semantic recognition in the process of recognizing the semantic information of the to-be-processed text information; specifically, the feature distribution information can be set in combination with the impact of the word on semantic recognition in the to-be-processed text information.

[0104] In step S203, a first resource coding feature of the to-be-processed multimedia resource is determined.

[0105] In a specific embodiment, the first resource coding feature can be coding feature information of the to-be-processed multimedia resource. In an optional embodiment, the first resource coding feature can include target query features of a plurality of resource blocks corresponding to the to-be-processed multimedia resource, target key features of the plurality of resource blocks, and target value features of the plurality of resource blocks; correspondingly, as shown in Figure 3 The determination of the first resource coding feature of the to-be-processed multimedia resource can include:

[0106] In step S301, the to-be-processed multimedia resource is coded to obtain a third resource coding feature.

[0107] In step S303, the third resource coding feature is segmented to obtain a block resource coding feature of each resource block in a plurality of resource blocks.

[0108] In step S305, the block resource coding features of the plurality of resource blocks are spliced in a channel dimension corresponding to the block resource coding features of the plurality of resource blocks to obtain a fourth resource coding feature.

[0109] In step S307, based on the fourth resource coding feature, target query features, target key features, and target value features are generated.

[0110] In one specific embodiment, the multimedia resource to be processed can be encoded by combining the initial encoder to obtain a third resource encoding feature; the initial encoder can be a pre-trained encoder; the third resource encoding feature can be a resource feature obtained by encoding the multimedia resource to be processed by combining the initial encoder; specifically, the third resource encoding feature can be divided into a corresponding number of block resource encoding features according to the accuracy requirements of the resource encoding feature in actual application; specifically, any one block resource encoding feature can correspond to a resource block of the multimedia resource to be processed; optionally, taking at least one image to be processed as an example, the plurality of resource blocks can be a plurality of image blocks corresponding to the at least one image to be processed; accordingly, the plurality of block resource encoding features can be image encoding features corresponding to the plurality of image blocks respectively.

[0111] In one specific embodiment, it is assumed that the third resource encoding feature includes T (T is the number of multimedia resources to be processed, for example, T images to be processed) dimensions of c x h x w features, where c is the channel number of the third resource encoding feature, h is the height of the third resource encoding feature; w is the width of the third resource encoding feature; it is assumed that any third resource encoding feature is divided into Np=n*n block resource encoding features; accordingly, after division, T*Np c x h / n x w / n block resource encoding features can be obtained; further, after channel dimension splicing, a fourth resource encoding feature of (c*T*Np) x h / n x w / n can be obtained.

[0112] In one optional embodiment, determining the first resource encoding feature of the multimedia resource to be processed can be implemented by combining the target encoding network in the task processing network; optionally, in the case where the multimedia resource to be processed is a plurality of video images (a plurality of images to be repaired) in a video to be repaired, that is, the multimedia resource to be processed is a multimedia resource involving a motion scene (such as a translation, classification, etc. Scene), the target encoding network can include an initial encoder, a feature segmentation network, a feature splicing network, a convolutional encoding processing network, and a deformation alignment network; wherein the feature segmentation network is used for segmentation processing of the resource encoding feature; the feature splicing network is used for splicing processing of the plurality of block resource encoding features in the channel dimension; the convolutional encoding processing network is used for converting the spliced resource encoding feature (the fourth resource encoding feature) into the query feature, the key feature and the value feature; the deformation alignment network can be used for deformation alignment processing between resource blocks.

[0113] In one optional embodiment, in the case where the multimedia resource to be processed is a plurality of video images (a plurality of images to be repaired) in a video to be repaired, the above generating a target query feature, a target key feature and a target value feature based on the fourth resource encoding feature can include:

[0114] performing convolutional encoding processing on the fourth resource encoding feature to obtain initial query features of a plurality of resource blocks, initial key features of the plurality of resource blocks, and initial value features of the plurality of resource blocks;

[0115] performing morphing alignment processing on the initial query features, the initial key features, and the initial value features to obtain target key features and target value features;

[0116] In one specific embodiment, the target query features, the target key features, and the target value features described above can be target Query features, target Key features, and target Value features in an attention mechanism, and correspondingly, the convolutional encoding processing on the fourth resource encoding feature described above can be combined with a convolutional encoding processing network, which can include three pre-trained convolutional subnetworks; specifically, the three convolutional subnetworks can be used to learn query features (Query), key features (Key), and value features (Value), respectively. Correspondingly, the fourth resource encoding feature can be input into the three convolutional subnetworks for convolutional encoding processing to obtain the initial query features, the initial key features, and the initial value features.

[0117] In one specific embodiment, the target query features of the target resource block (any resource block in the plurality of resource blocks) can be initial query features of the target resource block, the target key features of the target resource block can be key features after morphing alignment of the initial query features (including key features after morphing alignment of the initial query features of the target resource block with the initial query features of each resource block in the plurality of resource blocks), and the target value features of the target resource block can be value features after morphing alignment of the initial query features (including value features after morphing alignment of the initial value features of the target resource block with the initial query features of each resource block in the plurality of resource blocks).

[0118] In one optional embodiment, the morphing alignment processing on the initial query features, the initial key features, and the initial value features to obtain the target key features and the target value features can include inputting the initial query features, the initial key features, and the initial value features into a morphing alignment network for morphing alignment processing to obtain the target key features and the target value features.

[0119] In one optional embodiment, the morphing alignment network can include a morphing parameter learning network and an alignment processing network; correspondingly, the inputting of the initial query features, the initial key features, and the initial value features into the morphing alignment network for morphing alignment processing to obtain the target key features and the target value features can include:

[0120] The initial query feature of the target resource block and the target block features of the plurality of resource blocks are input into a deformation parameter learning network to learn deformation parameters.

[0121] In one specific embodiment, the deformation parameters can represent motion deformation information of the plurality of resource blocks relative to the target resource block. Specifically, the motion deformation information can include movement coordinates of objects (e.g., persons in a video) in the resource blocks (image blocks), rotation angles, and other deformation information generated by the motion process. The target resource block can be any of the plurality of resource blocks, and the target block features can be the initial key features and / or the initial value features.

[0122] In an optional embodiment, the feature information difference between the key features and the value features is usually small. To reduce the amount of data processing in the deformation parameter learning process, in the process of learning the deformation parameters between the target resource block and any of the plurality of resource blocks, the initial key features or the initial value features of each of the plurality of resource blocks and the initial query feature of the target resource block can be directly input into the deformation parameter learning network to learn the deformation parameters. In the alignment processing network, the initial key features and the initial value features of each of the plurality of resource blocks are respectively aligned with the initial query feature of the target resource block in combination with the deformation parameters to obtain the target key features of each of the plurality of resource blocks and the target value features of each of the plurality of resource blocks.

[0123] In another optional embodiment, to better avoid the influence of nonlinear deformation in the motion process on the accuracy of the learned resource (image) features, in the process of learning the deformation parameters between the target resource block and each of the plurality of resource blocks, the initial key features of each of the plurality of resource blocks, the initial value features of each of the plurality of resource blocks, and the initial query feature of the target resource block can be input into the deformation parameter learning network to learn the deformation parameters. Correspondingly, the deformation parameters can include first deformation parameters for aligning the initial key features of each of the plurality of resource blocks with the initial query feature of the target resource block, and second deformation parameters for aligning the initial value features of each of the plurality of resource blocks with the initial query feature of the target resource block.

[0124] In the above embodiment, in the case where the to-be-processed multimedia resource is a plurality of video images in a to-be-repaired video, after the initial query feature, the initial key feature, and the initial value feature of the plurality of resource blocks are obtained by convolutional coding processing combined with the fourth resource coding feature, the deformation alignment network is combined to perform deformation alignment processing on the two resource blocks in the target resource pair, and in the deformation alignment processing, the deformation parameter learning is first performed on the initial query feature of the target resource block and the target block feature of the plurality of resource blocks, and then the learned deformation parameters are combined to perform alignment processing on the initial key feature and the initial value feature of the plurality of resource blocks. The influence of the non-linear deformation of the collected object in the video collection process on the image representation accuracy can be effectively reduced, and the problems such as image distortion and deformation of the repaired image in the image repair application scenario can be avoided, the accuracy and effectiveness of the learned image features are greatly improved, and the task processing efficiency and effect are also greatly improved.

[0125] In an optional embodiment, in the case where the to-be-processed multimedia resource is a multimedia resource that does not involve a motion scene (for example, a translation, classification, or the like), the initial query feature of the plurality of resource blocks can be used as the target query feature, and the initial key feature of the plurality of resource blocks can be used as the target key feature, and the initial value feature of the plurality of resource blocks can be used as the target value feature. Correspondingly, the target coding network can include an initial encoder, a feature segmentation network, a feature splicing network, and a convolutional coding processing network.

[0126] In the above embodiment, in the case where the third resource coding feature is obtained by encoding the to-be-processed multimedia resource by using the preset encoder, the target query feature, the target key feature, and the target value feature of the plurality of resource blocks used for learning the resource coding feature are generated by performing segmentation processing on the third resource coding feature and splicing in the channel dimension, which can facilitate more effective learning of the detail features of the to-be-processed multimedia resource, and the accuracy and effectiveness of the learned resource coding feature can be improved, and the processing accuracy and effect of the subsequent task processing can be improved.

[0127] In step S205, the feature distribution information and the first resource coding feature are input into the feature pruning network for feature learning to obtain the second resource coding feature.

[0128] In one specific embodiment, the feature pruning network described above can be used to learn the resource encoding features of the multimedia resource to be processed in the case of invalid feature pruning of the first resource encoding features based on the feature distribution information. Optionally, the feature pruning network described above can include a resource association analysis network, a weight learning network, a weighted association analysis network and an aggregation processing network. In one optional embodiment, the first resource encoding features described above can include target query features, target key features and target value features of a plurality of resource blocks corresponding to the multimedia resource to be processed; correspondingly, as shown in Figure 4 The input of the feature distribution information and the first resource encoding features into the feature pruning network for feature learning to obtain the second resource encoding features can include:

[0129] In step S401, the target query features, the target key features and the feature distribution information are input into the resource association analysis network for resource block association analysis to obtain resource block association information;

[0130] In step S403, the feature distribution information is input into the weight learning network for feature weight learning to obtain target weight information corresponding to the target resource pair;

[0131] In step S405, the target weight information and the resource block association information are input into the weighted association analysis network for weighted association analysis to obtain weighted association information;

[0132] In step S407, the weighted association information and the target value features are input into the aggregation processing network for aggregation processing to obtain the second resource encoding features.

[0133] In one specific embodiment, the resource block association information described above can represent the association degree between the effective features corresponding to the target resource pair. Specifically, the target resource pair includes a resource block pair composed of a target resource block and a plurality of resource blocks; the target resource block can be any resource block in the plurality of resource blocks; optionally, in the resource association analysis network, resource block association analysis can be performed in combination with the following formula to determine the resource block association information corresponding to any target resource pair:

[0134]

[0135] wherein C(q, k) represents the resource block association information corresponding to the target resource pair; f q (x, y) represents the target query features of the target resource block (any resource block in the plurality of resource blocks) in the target resource pair; (x, y) represents the position information of the corresponding resource block (for example, in the case of an image block as the resource block, the position information is the pixel coordinates; in the case of a word in the text to be processed as the resource block, the position information is the position information of the word in the text features corresponding to the text to be processed information); V q(x, y) represents distribution feature information of the target resource block (characterizes the distribution of effective features in the target resource block); V k→q (x, y) represents distribution feature information of another resource block in the target resource pair other than the target resource block (characterizes the distribution of effective features in the another resource block); f k→q (x, y) represents the key feature of the another resource block after morphological alignment with the target initial query feature of the target resource block (target key feature).

[0136] In one specific embodiment, the above-mentioned target weight information can represent the importance of the target resource pair in the task processing process of the target task. Optionally, in the weight learning network, the target weight information corresponding to the target resource pair can be learned in combination with the following formula:

[0137]

[0138] wherein S(q, k) represents the target weight information corresponding to the certain target resource pair; ∑V represents the sum of the distribution feature information of the plurality of resource blocks; (x, y) represents the position information of the corresponding resource block; V q (x, y) represents the distribution feature information of the target resource block (characterizes the distribution of effective features in the target resource block); v k→q (x, y) represents the distribution feature information of another resource block in the target resource pair other than the target resource block (characterizes the distribution of effective features in the another resource block).

[0139] In one optional embodiment, the weighted association information can be the result of weighting and normalizing the resource block association information corresponding to the certain target resource pair based on the target weight information corresponding to the certain target resource pair; specifically, in the weighted association analysis network, the weighted association analysis can be performed in combination with the following formula to obtain the weighted association information corresponding to the target resource pair:

[0140] α q，k = softmax(C(q, k) · S(q, k))

[0141] wherein α q，k represents the weighted association information corresponding to the certain target resource pair; S(q, k) represents the target weight information corresponding to the certain target resource pair; c(q, k) represents the resource block association information corresponding to the certain target resource pair.

[0142] In one specific embodiment, the second resource encoding feature of the above-mentioned second resource encoding feature can include the resource encoding feature of the above-mentioned plurality of resource blocks. In the aggregation processing network, for any resource block, the resource encoding feature of the resource block can be combined with the corresponding weighted association information of the plurality of target resources including the resource block, and the target value feature corresponding to the plurality of resource blocks (the target value feature after the morphing alignment of the initial query feature of the target resource block) is weighted and summed to obtain the second resource encoding feature of the target resource block.

[0143] In one specific embodiment, it is assumed that the multimedia resource to be processed is an image to be processed; accordingly, the plurality of resource blocks can be a plurality of image blocks, such as Figure 5 as shown in Figure 5 is a schematic diagram of a feature learning process based on a feature pruning network according to an exemplary embodiment. Specifically, for each target image block, the target query image feature of the target image block in any target image pair (target resource pair), the target key image feature of another image block in the target image pair, and the feature distribution information corresponding to the two image blocks in the target image pair can be input into the resource association analysis network to obtain the resource block association information corresponding to the target image pair. In addition, the feature distribution information corresponding to the two image blocks in each target image pair can be input into the weight learning network to obtain the target weight information corresponding to each target image pair. Then, the target weight information and the resource block association information corresponding to each target image pair are input into the weighted association analysis network for weighted association analysis to obtain the weighted association information corresponding to each target image pair. Finally, the weighted association information corresponding to each target image pair and the target value feature of another image block in each target image pair are input into the aggregation processing network for aggregation processing to obtain the second resource encoding feature (i.e., image encoding feature) of the target image block.

[0144] In the above embodiments, in the resource block association analysis process of the target query feature of the target resource pair and the target key feature of the target resource pair in the plurality of resource blocks corresponding to the to-be-processed multimedia resource, the effective features between the resource blocks in the target resource pair can be associated and analyzed in combination with the feature distribution information, and then the interference of invalid features can be effectively avoided on the basis of reducing the calculation amount. In combination with the feature weight learning of the feature distribution information, the target weight information representing the importance of the target resource pair in the task processing process of the target task can be obtained, and the weighted association information can be obtained by performing weighted association analysis on the target weight information and the resource block association information obtained by the association analysis. Then, through the aggregation processing of the weighted association information and the target value feature, the resource encoding feature of each resource block based on the attention mechanism is determined, which greatly improves the effectiveness and accuracy of the learned resource encoding feature of each resource block, and then the subsequent task processing effect and efficiency can be improved.

[0145] In step S207, based on the second resource encoding feature, a task processing operation corresponding to the target task is performed to obtain a target task processing result corresponding to the to-be-processed multimedia resource.

[0146] In one specific embodiment, the second resource encoding feature includes resource encoding features of the plurality of resource blocks. In actual application, after obtaining the resource encoding feature, it is often necessary to decode the resource encoding feature by using a target decoding network corresponding to the target task that is pre-trained to obtain a corresponding task processing result. Correspondingly, the above operation of performing a task processing operation corresponding to the target task based on the second resource encoding feature to obtain a target task processing result corresponding to the to-be-processed multimedia resource can include:

[0147] splicing the resource encoding features of the plurality of resource blocks to obtain a target resource encoding feature of the to-be-processed multimedia resource;

[0148] inputting the target resource encoding feature into a target decoding network corresponding to the target task for task decoding processing to obtain a target task processing result.

[0149] In one specific embodiment, taking at least one to-be-processed image as the to-be-processed multimedia resource, the resource encoding features of the plurality of resource blocks can include image encoding features of a plurality of image blocks corresponding to the at least one to-be-processed image. Correspondingly, the above operation of splicing the resource encoding features of the plurality of resource blocks to obtain a target resource encoding feature of the to-be-processed multimedia resource can include: splicing the image encoding features of the image blocks corresponding to the same to-be-processed image to obtain a target image encoding feature (target resource encoding feature) of each of the at least one to-be-processed image.

[0150] In a specific embodiment, taking the at least one to-be-processed text information as an example of the to-be-processed multimedia resource, the resource encoding features of the plurality of resource blocks can include word encoding features of a plurality of words corresponding to the at least one to-be-processed text information, and correspondingly, the splicing processing of the resource encoding features of the plurality of resource blocks to obtain the target resource encoding feature library of the to-be-processed multimedia resource can include splicing processing of the word encoding features of the words corresponding to the same to-be-processed text information to obtain the target text encoding features (target resource encoding features) of the at least one to-be-processed text information respectively.

[0151] In a specific embodiment, the target resource encoding features can be subjected to task decoding processing in combination with a pre-trained target decoding network (target decoding network in the task processing network) to obtain a target task processing result; optionally, taking the target task as an image inpainting task, the target task processing result can be a repaired image corresponding to the at least one to-be-repaired image (to-be-processed multimedia resource). Taking the target task as an image classification task, the target task processing result can be object category information corresponding to the at least one to-be-classified image (to-be-processed multimedia resource). Taking the target task as a translation task, the target task processing result can be a translated text information corresponding to the at least one to-be-translated text information (to-be-processed multimedia resource).

[0152] In a specific embodiment, the training data and the task processing network (the task processing network includes a target encoding network and a target decoding network) corresponding to different target tasks are often different, and correspondingly, the training data corresponding to different target tasks (including sample multimedia resources, sample feature distribution information corresponding to the sample multimedia resources, and preset task processing results corresponding to the sample multimedia resources) can be obtained in advance, and the to-be-trained task processing network of different tasks (the to-be-trained task processing network includes a preset encoding network and a preset decoding network) can be set. Optionally, the preset encoding network can include a to-be-trained initial encoder, a to-be-trained feature segmentation network, a to-be-trained feature splicing network, and a to-be-trained convolutional encoding processing network, and optionally, the preset encoding network can further include a to-be-trained deformation alignment network; the preset decoding network can include a to-be-trained decoder; correspondingly, the above method can further include:

[0153] obtaining a sample multimedia resource, sample feature distribution information corresponding to the sample multimedia resource, and a preset task processing result corresponding to the sample multimedia resource, the sample feature distribution information representing a distribution of effective features in the sample multimedia resource; determining, based on a preset encoding network, a first sample resource encoding feature of the sample multimedia resource; inputting the sample feature distribution information and the first sample resource encoding feature into a feature pruning network for feature learning to obtain a second sample resource encoding feature of the sample resource encoding feature including a plurality of sample resource blocks; performing splicing processing on the sample resource encoding feature to obtain a sample resource encoding feature of the sample multimedia resource; inputting the sample resource encoding feature into a preset decoding network corresponding to a target task for task decoding processing to obtain a sample task processing result; determining task loss information according to the sample task processing result and the preset task processing result; and training the preset encoding network and the preset decoding network based on the task loss information to obtain a target encoding network and a target decoding network.

[0154] In a specific embodiment, taking the target task as an image inpainting task as an example, the sample multimedia resource can be at least one first sample image (a sample image needing to be inpainted); the preset task processing result corresponding to the sample multimedia resource can be an already inpainted image corresponding to the at least one first sample image. In a specific embodiment, taking the target task as an image classification task as an example, the sample multimedia resource can be at least one second sample image (an image needing to be classified); the preset task processing result corresponding to the sample multimedia resource can be object category information (category information of an object in the second sample image) corresponding to the at least one second sample image. In a specific embodiment, taking the target task as a translation task as an example, the sample multimedia resource can be at least one sample text information; the preset task processing result corresponding to the sample multimedia resource can be translated text information corresponding to the at least one sample text information.

[0155] In a specific embodiment, in the process of training the target encoding network and the target decoding network, the feature pruning network can be used to learn the resource encoding feature of the sample multimedia resource based on the sample feature distribution information pruning invalid features from the first sample resource encoding feature.

[0156] In an optional embodiment, the specific refinement of obtaining the sample task processing result in the process of training the target encoding network and the target decoding network in combination with the sample multimedia resource, the sample feature distribution information corresponding to the sample multimedia resource, and the feature pruning network can refer to the specific refinement of obtaining the task processing result in combination with the to-be-processed multimedia resource, the feature distribution information corresponding to the to-be-processed multimedia resource, and the feature pruning network, which will not be described herein again.

[0157] In a specific embodiment, the task loss information can represent a difference between the sample task processing result and the preset task processing result. In the process of determining the preset loss information based on the sample task processing result and the preset task processing result, the preset loss function can be combined. The training of the preset encoding network and the preset decoding network based on the preset loss information to obtain the target encoding network and the target decoding network can include: updating the network parameters of the preset encoding network and the preset decoding network according to the task loss information, and repeatedly performing the above-mentioned process of determining the first sample resource encoding feature of the sample multimedia resource based on the preset encoding network, based on the updated preset encoding network and the preset decoding network. The training iteration operation of determining the task loss information based on the sample task processing result and the preset task processing result is performed until the preset convergence condition is met, and the preset encoding network corresponding to the time when the preset convergence condition is met is used as the target encoding network, and the preset decoding network corresponding to the time when the preset convergence condition is met is used as the target decoding network.

[0158] In an optional embodiment, the preset convergence condition can be that the number of training iteration operations reaches a preset training number. Optionally, the first preset convergence condition can also be that the task loss information is less than a specified threshold. In the embodiments of the present specification, the preset training number and the specified threshold can be set in advance according to the training speed and accuracy of the network in actual application.

[0159] In actual application, the task processing network (including the target encoding network and the target decoding network) based on the feature pruning network can be analyzed for task processing performance by combining PSNR (Peak Signal to Noise Ratio, peak signal to noise ratio) and SSIM (Structural Similarity, structural similarity) and other index data. Further, the network for task processing can also be further optimized by combining the index data obtained by analysis.

[0160] In the above embodiments, in the case of obtaining resource encoding features of multiple resource blocks, the target decoding network corresponding to the target task is used for task decoding processing after the resource encoding features of the multiple resource blocks are spliced and processed, and the target resource encoding features of the to-be-processed multimedia resource can be combined to perform corresponding task processing, which greatly improves the task processing effect and efficiency in different task scenarios.

[0161] From the technical solutions provided by the embodiments of the present specification, it can be seen that in the task processing process in the present specification, after determining the first resource encoding features of the to-be-processed multimedia resource, in the feature pruning network, the second resource encoding features of the to-be-processed multimedia resource are learned in the case of invalid feature pruning of the first resource encoding features combined with the feature distribution information that can represent the effective feature distribution in the to-be-processed multimedia resource. This can effectively avoid the interference of invalid information in the learning of resource encoding features, greatly improve the representation accuracy and effectiveness of the learned second resource encoding features for the to-be-processed multimedia resource on the basis of improving the feature encoding efficiency and network performance, and further improve the effect of image restoration, image classification, text translation and other task processing.

[0162] In one specific embodiment, taking the target task as an image restoration task and the to-be-processed multimedia resource as a plurality of frames (T frames) of video images (to-be-restored images) in a to-be-restored video as an example, as shown in Figure 6 Figure 6 is a schematic diagram in a task processing process according to an example embodiment. Specifically, the plurality of frames of video images can be input into the target encoding network. In the target encoding network, the initial encoder can be combined for encoding processing, and the resource encoding features (third resource encoding features) after the encoding processing can be sequentially subjected to splitting, channel dimension splicing processing and convolutional encoding processing. Then, the initial query features of the plurality of resource blocks, the initial key features of the plurality of resource blocks and the initial value features of the plurality of resource blocks after the convolutional encoding processing can be subjected to deformation alignment processing by combining the deformation alignment network in the target encoding network, to obtain the target query features of the target resource block in each target resource pair, and the target key features and the target value features of the other resource block in each target resource pair. Then, in the feature pruning network, the resource encoding features (second resource encoding features) of each resource block can be learned based on the invalid feature pruning of the two resource blocks in the target resource pair based on the feature distribution information. Then, the resource encoding features of the plurality of resource blocks can be subjected to splicing processing to obtain the target resource encoding features of the plurality of frames of video images respectively. Then, the target resource encoding features of the plurality of frames of video images can be subjected to decoding processing by combining the target decoding network corresponding to the image restoration task, to obtain the plurality of frames of video images after restoration (target task processing result).

[0163] In one specific embodiment, as shown in Figure 7 Figure 7 is an effect diagram of image restoration based on the task processing method provided by the embodiments of the present disclosure according to an example embodiment; wherein, Figure 7 a and Figure 7 b are two frames of images in a first to-be-restored video, and the first to-be-restored video is a video in which the foreground character moves slowly and the background does not change, combined with Figure 7 c​​Figure 7 a repaired image) and Figure 7 d( Figure 7 b repaired image) can be seen, based on the task processing method provided in the embodiments of the present disclosure, the video image with almost no action can be effectively repaired.

[0164] In another specific embodiment, as Figure 8 shown, Figure 8 is another effect diagram of image repair based on the task processing method provided in the embodiments of the present disclosure according to an exemplary embodiment; wherein, Figure 8 a and Figure 8 b two images in the second to-be-repaired video, and the second to-be-repaired video is a video with a background that does not move and a foreground that moves in place, in combination with Figure 8 c( Figure 8 a repaired image) and Figure 8 d( Figure 8 b repaired image) can be seen, based on the task processing method provided in the embodiments of the present disclosure, the video image with slow movement can be effectively repaired.

[0165] In another specific embodiment, as Figure 9 shown, Figure 9 is another effect diagram of image repair based on the task processing method provided in the embodiments of the present disclosure according to an exemplary embodiment; wherein, Figure 9 a and Figure 9 b two images in the third to-be-repaired video, and the third to-be-repaired video is a video with large changes in both foreground and background, in combination with Figure 9 c( Figure 9 a repaired image) and Figure 9 d( Figure 9 b repaired image) can be seen, based on the task processing method provided in the embodiments of the present disclosure, the video image with complex motion can be effectively repaired.

[0166] As can be seen from the above embodiments, the task processing method provided in the embodiments of the present disclosure applied to the task of video image repair can effectively reduce the influence of the nonlinear deformation of the motion of the collected object in the video collection process on the image representation accuracy, avoid the problem of image distortion and deformation corresponding to different motion forms, and after deformation alignment, introduce a feature pruning network, which can learn the image coding features of the to-be-processed image based on the mask information and prune the invalid features of the image coding features after deformation alignment, effectively avoid the interference of invalid information in the learning of image coding features, greatly improve the representation accuracy and integrity of the learned image coding features for the to-be-repaired image, and thus also improve the image repair effect.

[0167] Figure 10 is a task processing device block diagram according to an exemplary embodiment. Referring to Figure 10 , the device comprises:

[0168] The information acquisition module 1010 is configured to perform acquisition of a to-be-processed multimedia resource and feature distribution information corresponding to the to-be-processed multimedia resource, the feature distribution information representing a distribution situation of effective features in the to-be-processed multimedia resource;

[0169] The resource encoding feature determination module 1020 is configured to perform determination of a first resource encoding feature of the to-be-processed multimedia resource;

[0170] The feature learning module 1030 is configured to perform input of the feature distribution information and the first resource encoding feature into a feature pruning network for feature learning, to obtain a second resource encoding feature, the feature pruning network being used to learn a resource encoding feature of the to-be-processed multimedia resource in a case of invalid feature pruning of the first resource encoding feature based on the feature distribution information;

[0171] The task processing module 1040 is configured to perform a task processing operation corresponding to a target task based on the second resource encoding feature, to obtain a target task processing result corresponding to the to-be-processed multimedia resource.

[0172] In an optional embodiment, the first resource encoding feature comprises target query features, target key features and target value features of a plurality of resource blocks corresponding to the to-be-processed multimedia resource; the feature pruning network comprises a resource association analysis network, a weight learning network, a weighted association analysis network and an aggregation processing network; the feature learning module 1030 comprises:

[0173] The resource block association analysis unit is configured to perform input of the target query features, the target key features and the feature distribution information into the resource association analysis network for resource block association analysis, to obtain resource block association information; the resource block association information represents an association degree between effective features corresponding to a target resource pair, the target resource pair comprising a resource block pair composed of a target resource block and the plurality of resource blocks, the target resource block being any one of the plurality of resource blocks;

[0174] The feature weight learning unit is configured to perform input of the feature distribution information into the weight learning network for feature weight learning, to obtain target weight information corresponding to the target resource pair, the target weight information representing an importance degree of the target resource pair in a task processing process of the target task;

[0175] The weighted processing unit is configured to perform input of the target weight information and the resource block association information into the weighted association analysis network for weighted association analysis, to obtain weighted association information;

[0176] The weighted summation processing unit is configured to perform aggregation processing on the weighted association information and the target value feature input into an aggregation processing network to obtain a second resource encoding feature.

[0177] In an optional embodiment, the second resource encoding feature includes resource encoding features of a plurality of resource blocks; and the task processing module 1040 includes:

[0178] The first splicing processing unit is configured to perform splicing processing on the resource encoding features of the plurality of resource blocks to obtain a target resource encoding feature of the to-be-processed multimedia resource.

[0179] The task decoding processing unit is configured to perform task decoding processing on the target resource encoding feature input into a target decoding network corresponding to the target task to obtain a target task processing result.

[0180] In an optional embodiment, the first resource encoding feature includes a target query feature of a plurality of resource blocks corresponding to the to-be-processed multimedia resource, a target key feature of the plurality of resource blocks, and a target value feature of the plurality of resource blocks; and the resource encoding feature determination module 1020 includes:

[0181] The encoding processing unit is configured to perform encoding processing on the to-be-processed multimedia resource to obtain a third resource encoding feature.

[0182] The segmentation processing unit is configured to perform segmentation processing on the third resource encoding feature to obtain a block resource encoding feature of each resource block in the plurality of resource blocks.

[0183] The second splicing processing unit is configured to perform splicing processing on the block resource encoding features of the plurality of resource blocks in a channel dimension corresponding to the block resource encoding features to obtain a fourth resource encoding feature.

[0184] The resource encoding feature generation unit is configured to generate the target query feature, the target key feature, and the target value feature based on the fourth resource encoding feature.

[0185] In an optional embodiment, in a case where the to-be-processed multimedia resource is a plurality of frames of video images in a to-be-repaired video, the resource encoding feature generation unit includes:

[0186] The convolutional encoding processing unit is configured to perform convolutional encoding processing on the fourth resource encoding feature to obtain an initial query feature of the plurality of resource blocks, an initial key feature of the plurality of resource blocks, and an initial value feature of the plurality of resource blocks.

[0187] The morphing alignment processing unit is configured to perform morphing alignment processing on the initial query feature, the initial key feature, and the initial value feature to obtain the target key feature and the target value feature.

[0188] The target query feature is the initial query feature, the target key feature is the key feature after morphing alignment of the initial query feature, and the target value feature is the value feature after morphing alignment of the initial query feature.

[0189] In an optional embodiment, the morphing alignment processing unit is specifically configured to perform morphing alignment processing on the initial query feature, the initial key feature, and the initial value feature input into the morphing alignment network to obtain the target key feature and the target value feature.

[0190] In an optional embodiment, the morphing alignment network includes a morphing parameter learning network and an alignment processing network.

[0191] The morphing alignment processing unit includes:

[0192] The morphing parameter learning unit is configured to perform morphing parameter learning on the initial query feature of the target resource block and the target block feature of the plurality of resource blocks input into the morphing parameter learning network to obtain the morphing parameter. The morphing parameter represents motion morphing information of the plurality of resource blocks relative to the target resource block. The target resource block is any resource block in the plurality of resource blocks, and the target block feature is the initial key feature and / or the initial value feature.

[0193] The alignment processing unit is configured to perform alignment processing on the morphing parameter, the initial key feature, and the initial value feature input into the alignment processing network to obtain the target key feature and the target value feature.

[0194] In an optional embodiment, in the case that the multimedia resource to be processed includes at least one image to be processed, the information acquisition module 1010 includes:

[0195] The mask image acquisition unit is configured to perform acquisition of a mask image corresponding to the at least one image to be processed.

[0196] The feature distribution information generation unit is configured to perform generation of feature distribution information based on the mask image.

[0197] As to the apparatus in the above embodiments, the specific manners in which various modules perform operations have been described in detail in the embodiments of the method, and will not be described in detail here.

[0198] Figure 11 is a block diagram of an electronic device for task processing according to an exemplary embodiment. The electronic device can be a terminal, and its internal structure diagram can be as shown in Figure 11As shown, the electronic device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a task processing method. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse.

[0199] Figure 12 This is a block diagram illustrating another electronic device for task processing according to an exemplary embodiment. The electronic device may be a server, and its internal structure diagram may be as follows: Figure 12 As shown, this electronic device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a task processing method.

[0200] Those skilled in the art will understand that Figure 11 or Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present disclosure and does not constitute a limitation on the electronic device to which the present disclosure is applied. A specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0201] In an exemplary embodiment, an electronic device is also provided, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement a task processing method as described in the embodiments of this disclosure.

[0202] In an exemplary embodiment, a computer-readable storage medium is also provided, wherein when the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the task processing method of the present disclosure embodiments.

[0203] In an example embodiment, a computer program product containing instructions which, when the program is run on a computer, causes the computer to carry out the task processing method in the embodiments of the present disclosure is also provided.

[0204] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments of each method. Any reference to memory, storage, databases, or other media used to store data in embodiments provided herein can include non-volatile and / or volatile memory. Non-volatile memory can include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM), or external cache memory. As an illustration and not a limitation, RAM is available in many forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), Rambus DRAM (RDRAM), direct Rambus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0205] Other embodiments of the present disclosure will be apparent to those skilled in the art with the consideration of the specification and practice of the disclosed application. The present application is intended to cover any variations, uses, or adaptations of the present disclosure following the general principles thereof and including such departures from the present disclosure that come within known, accepted, and customary practice in the art to which the present disclosure pertains. The specification and examples are to be regarded as illustrative only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0206] It should be understood that the present disclosure is not limited to the precise structures as herein described and illustrated in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the claims that follow.

Claims

1. A task processing method characterized by, The method comprises: obtaining a to-be-processed multimedia resource and feature distribution information corresponding to the to-be-processed multimedia resource, the feature distribution information representing a distribution of effective features in the to-be-processed multimedia resource; determining a first resource encoding feature of the to-be-processed multimedia resource, the first resource encoding feature comprising a target query feature, a target key feature and a target value feature of a plurality of resource blocks corresponding to the to-be-processed multimedia resource; inputting the target query feature, the target key feature and the feature distribution information into a resource association analysis network in a feature pruning network to perform resource block association analysis, and obtaining resource block association information, the resource block association information representing an association degree between effective features corresponding to a target resource pair, the target resource pair comprising a resource block pair composed of a target resource block and the plurality of resource blocks, the target resource block being any one of the plurality of resource blocks; inputting the feature distribution information into a weight learning network in the feature pruning network to perform feature weight learning, and obtaining target weight information corresponding to the target resource pair, the target weight information representing an importance degree of the target resource pair in a task processing process of a target task; inputting the target weight information and the resource block association information into a weighted association analysis network in the feature pruning network to perform weighted association analysis, and obtaining weighted association information; inputting the weighted association information and the target value feature into an aggregation processing network in the feature pruning network to perform aggregation processing, and obtaining a second resource encoding feature, the feature pruning network being used to learn resource encoding features of the to-be-processed multimedia resource in a case where invalid feature pruning is performed on the first resource encoding feature based on the feature distribution information; based on the second resource encoding feature, performing a task processing operation corresponding to a target task to obtain a target task processing result corresponding to the to-be-processed multimedia resource.

2. The task processing method according to claim 1, characterized by, The second resource encoding feature comprises resource encoding features of the plurality of resource blocks; and the performing, based on the second resource encoding feature, of a task processing operation corresponding to a target task to obtain a target task processing result corresponding to the to-be-processed multimedia resource comprises: performing splicing processing on the resource encoding features of the plurality of resource blocks to obtain a target resource encoding feature of the to-be-processed multimedia resource; inputting the target resource encoding feature into a target decoding network corresponding to the target task to perform task decoding processing, and obtaining the target task processing result.

3. The task processing method of claim 1, wherein, The determining of the first resource encoding feature of the to-be-processed multimedia resource comprises: performing encoding processing on the to-be-processed multimedia resource to obtain a third resource encoding feature; performing segmentation processing on the third resource encoding feature to obtain a block resource encoding feature of each resource block in the plurality of resource blocks; performing splicing processing on the block resource encoding features of the plurality of resource blocks in a channel dimension corresponding to the block resource encoding features of the plurality of resource blocks to obtain a fourth resource encoding feature; generating the target query feature, the target key feature and the target value feature based on the fourth resource encoding feature.

4. The task processing method according to claim 3, characterized by, In a case where the to-be-processed multimedia resource is a plurality of video images in a to-be-repaired video, the generating, based on the fourth resource coding feature, of the target query feature, the target key feature, and the target value feature comprises: performing convolutional coding processing on the fourth resource coding feature to obtain initial query features of the plurality of resource blocks, initial key features of the plurality of resource blocks, and initial value features of the plurality of resource blocks; performing morphing alignment processing on the initial query features, the initial key features, and the initial value features to obtain the target key feature and the target value feature; wherein the target query feature is the initial query feature, the target key feature is a key feature that is morphed and aligned with the initial query feature, and the target value feature is a value feature that is morphed and aligned with the initial query feature.

5. The task processing method according to claim 4, characterized by, The morphing alignment processing on the initial query features, the initial key features, and the initial value features to obtain the target key feature and the target value feature comprises: inputting the initial query features, the initial key features, and the initial value features into a morphing alignment network to perform morphing alignment processing to obtain the target key feature and the target value feature.

6. The task processing method according to claim 5, characterized by, The morphing alignment network comprises a morphing parameter learning network and an alignment processing network; and the inputting of the initial query features, the initial key features, and the initial value features into the morphing alignment network to perform morphing alignment processing to obtain the target key feature and the target value feature comprises: inputting initial query features of a target resource block and target block features of the plurality of resource blocks into the morphing parameter learning network to perform morphing parameter learning to obtain morphing parameters; the morphing parameters represent motion morphing information of the plurality of resource blocks relative to the target resource block; the target resource block is any resource block in the plurality of resource blocks, and the target block features are the initial key features and / or the initial value features; inputting the morphing parameters, the initial key features, and the initial value features into the alignment processing network to perform alignment processing to obtain the target key feature and the target value feature.

7. The task processing method according to any one of claims 1 to 6, characterized by, In a case where the to-be-processed multimedia resource comprises at least one to-be-processed image, the feature distribution information is obtained in the following manner: obtaining a mask image corresponding to the at least one to-be-processed image; generating the feature distribution information based on the mask image.

8. A task processing apparatus characterized by comprising: The method comprises: an information obtaining module configured to perform obtaining a to-be-processed multimedia resource and feature distribution information corresponding to the to-be-processed multimedia resource, the feature distribution information representing a distribution of effective features in the to-be-processed multimedia resource; a resource coding feature determining module configured to perform determining a first resource coding feature of the to-be-processed multimedia resource; the first resource coding feature comprises target query features, target key features, and target value features of a plurality of resource blocks corresponding to the to-be-processed multimedia resource; The feature learning module comprises: a resource block association analysis unit configured to perform resource block association analysis on the target query feature, the target key feature, and the feature distribution information input into a resource association analysis network in the feature pruning network to obtain resource block association information; the resource block association information represents the association degree between the effective features corresponding to the target resource pair, and the target resource pair comprises a resource block pair composed of a target resource block and the plurality of resource blocks, and the target resource block is any resource block in the plurality of resource blocks; The feature weight learning unit is configured to perform feature weight learning on the feature distribution information input into a weight learning network in the feature pruning network to obtain target weight information corresponding to the target resource pair, and the target weight information represents the importance of the target resource pair in the task processing process of the target task; The weighted processing unit is configured to perform weighted association analysis on the target weight information and the resource block association information input into a weighted association analysis network in the feature pruning network to obtain weighted association information; The weighted sum processing unit is configured to perform aggregation processing on the weighted association information and the target value feature input into an aggregation processing network in the feature pruning network to obtain second resource encoding features, and the feature pruning network is used to learn the resource encoding features of the to-be-processed multimedia resource based on invalid feature pruning on the first resource encoding features based on the feature distribution information; The task processing module is configured to perform a task processing operation corresponding to a target task based on the second resource encoding features to obtain a target task processing result corresponding to the to-be-processed multimedia resource.

9. An electronic device, comprising: It comprises: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the task processing method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device can perform the task processing method of any one of claims 1 to 7.

11. A computer program product comprising computer instructions, characterized in that, The computer instructions are executed by the processor to implement the task processing method of any one of claims 1 to 7.

Images