Multi-task processing and related model training method and device, equipment and storage medium

Abstract

The invention discloses a multi-task processing and related model training method and device, equipment and a storage medium, and the multi-task model training method comprises the steps: obtaining a training data set, wherein the training data set comprises a plurality of training images containing targets; the plurality of branch networks at least comprise an initial target branch network and an initial entropy model; performing iterative training on the initial image encoder and the initial target branch network based on the training image to obtain an updated image encoder and an updated target branch network; inputting the training image into an updated image encoder to carry out iterative training on the initial entropy model to obtain an updated entropy model; and performing iterative training on the updated image encoder, the updated target branch network and the updated entropy model based on the training image to obtain a trained multi-task model. According to the scheme, by optimizing the modules in the multi-task model step by step, the training speed of the model and the accuracy of model application are improved.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and in particular to a multi-task processing and related model training method, device, equipment, and storage medium. Background Art

[0002] As the scale of deep neural network models gradually increases, it is difficult to carry out model reasoning on some computing-constrained platforms, so it is often necessary to compress and transmit the captured images to a remote server for processing. However, the current network model will cause the loss of image information during the compression and transmission of images, resulting in inaccurate image processing results. Summary of the invention

[0003] This application at least provides a multi-tasking processing and related model training method, device, equipment, and storage medium.

[0004] In a first aspect, the present application provides a training method for a multi-task model, wherein the multi-task model includes an initial image encoder and at least two branch networks connected to the initial image encoder; the training method includes:

[0005] Acquire a training data set, the training data set including training images containing targets; at least two branch networks include at least an initial target branch network and an initial entropy model;

[0006] Iteratively train the initial image encoder and the initial target branch network based on the training image to obtain an updated image encoder and an updated target branch network;

[0007] Input the training image into the updated image encoder to iteratively train the initial entropy model to obtain an updated entropy model;

[0008] The image encoder, target branch network and entropy model are iteratively trained based on the training images to obtain a trained multi-task model.

[0009] Therefore, in this embodiment, the initial image encoder and the initial target branch network are first optimized based on the training image, and then the entropy model is optimized based on the training image. Finally, the updated image encoder, the updated target branch network, and the updated entropy model are optimized based on the training image. By gradually optimizing the modules in the multi-task model, the training speed of the model and the accuracy of the model application are improved.

[0010] In some embodiments, the initial target branch network includes an initial classification model and / or an initial construction model;

[0011] The initial image encoder and the initial target branch network are iteratively trained based on the training image to obtain an updated image encoder and an updated target branch network, including:

[0012] The training image is encoded by an initial image encoder to obtain first encoded data of the training image; the training image has labeled category information;

[0013] Classifying the training image based on the first coded data of the training image by using an initial classification model to obtain first predicted category information of the training image; and / or; constructing an image based on the first coded data of the training image by using an initial construction model to generate a predicted image;

[0014] Based on a first error value between first predicted category information and labeled category information corresponding to the training image and / or a second error value between the predicted image and the training image, the initial image encoder and the initial target branch network are iteratively trained to obtain an updated image encoder and an updated target branch network.

[0015] Therefore, compared with the technical solution in the prior art of decoding the first encoded data and then classifying, in this embodiment, the training image is encoded by the initial image encoder, so that the initial classification model classifies the training image based on the first encoded data, thereby improving the classification speed; the image is generated based on the first encoded data with richer semantic information through the initial construction model, thereby improving the image restoration accuracy, so as to facilitate the reconstruction of the training image.

[0016] In some embodiments, the initially constructed model includes a feature fusion model and an image decoder;

[0017] The method generates a predicted image by constructing an image based on first encoded data of a training image through an initial construction model, including:

[0018] Performing feature extraction on the first coded data of the training image at least twice to obtain a plurality of feature information corresponding to the training image;

[0019] The feature fusion model is used to fuse multiple feature information to obtain the fusion features of the training image;

[0020] The fused features of the training image are decompressed through the image decoder to generate a predicted image.

[0021] Therefore, in this embodiment, feature information of multiple levels of the training image is fused through a feature fusion model to improve the semantic richness of the feature information of the training image, and the fused features with rich semantic information are decompressed through an image encoder to improve the accuracy of the predicted image.

[0022] In some embodiments, the training image has a labeled coding length; the training image is input into an updated image encoder to iteratively train the initial entropy model to obtain an updated entropy model, including:

[0023] Inputting the training image into the updated image encoder to obtain second encoded data of the training image;

[0024] Processing the second coded data of the training image through the initial entropy model to obtain a first feature tensor of the training image; the first feature tensor has a corresponding coding length;

[0025] The initial entropy model is iteratively trained based on the difference between the encoding length corresponding to the first feature tensor and the marked encoding length to obtain an updated entropy model.

[0026] Therefore, in this embodiment, the initial entropy model is optimized by the difference between the encoding length of the first feature tensor output by the initial entropy model and the encoding length of the first feature tensor output last time, so that the encoding length of the first feature tensor generated by the initial entropy model is as small as possible, so as to facilitate the storage of the training image corresponding to the first feature tensor with a smaller encoding length, so as to save storage space.

[0027] In some embodiments, the multi-task model further includes a feature quantization layer, which is disposed between the initial image encoder and the branch network;

[0028] Training methods also include:

[0029] quantizing the second coded data of the training image through the feature quantization layer to obtain third coded data of the training image;

[0030] The second encoded data of the training image is processed by the initial entropy model to obtain a first feature tensor of the training image, including:

[0031] The third encoded data of the training image is processed by the initial entropy model to obtain a first feature tensor of the training image.

[0032] Therefore, in this embodiment, the encoding data of the training image is quantized by setting a feature quantization layer, that is, the encoding data is rounded to an integer, which facilitates the encoding of the training image and reduces the computational complexity.

[0033] In some embodiments, iterative training is performed based on the training image to update the image encoder, update the target branch network, and update the entropy model to obtain a trained multi-task model, including:

[0034] The training image is encoded by updating the image encoder to obtain encoded data of the training image;

[0035] Processing the encoded data of the training image by updating the entropy model to obtain a second feature tensor of the training image; the second feature tensor has a corresponding encoding length;

[0036] The prediction information is obtained by updating the encoding data of the target branch network based on the training image;

[0037] The updated entropy model and the updated target branch network are iteratively trained based on the prediction information corresponding to the training image and the second feature tensor to obtain a trained multi-task model.

[0038] Therefore, in this embodiment, the updated entropy model and the updated target branch network are jointly trained through training images to improve the training accuracy of the multi-task model.

[0039] In some embodiments, the initial image encoder is an ELIC image encoding and decoding network, and / or the initial classification model is a ConvNeXt image classification network.

[0040] Therefore, the initial image encoder in this embodiment is set to a multi-channel ELIC image encoding and decoding network to improve the ability of the encoded data to retain more semantic information of the training image, so as to improve the application performance of the multi-task model; by setting the initial classification model to a ConvNeXt image classification network, the accuracy of the classification model in predicting the category information of the training image based on the encoded information can be improved.

[0041] A second aspect of the present application provides a multi-task processing method, the method comprising:

[0042] Get the image to be processed;

[0043] Inputting the image to be processed into the multi-task model to obtain a multi-task processing result corresponding to the image to be processed;

[0044] Among them, the multi-task model is trained using the method of the first aspect.

[0045] In a third aspect, the present application provides a training device for a multi-task model, wherein the multi-task model includes an initial image encoder and at least two branch networks connected to the initial image encoder, including:

[0046] An acquisition module is used to acquire a training data set, the training data set includes a training image containing a target; at least two branch networks include at least an initial target branch network and an initial entropy model;

[0047] A first training module is used to iteratively train the initial image encoder and the initial target branch network based on the training image to obtain an updated image encoder and an updated target branch network;

[0048] The second training module is used to input the training image into the updated image encoder to iteratively train the initial entropy model to obtain an updated entropy model;

[0049] The third training module is used to iteratively train the updated image encoder, the updated target branch network and the updated entropy model based on the training image to obtain a trained multi-task model.

[0050] A fourth aspect of the present application provides a multi-task processing device, comprising:

[0051] An acquisition module, used for acquiring images to be processed;

[0052] A processing module is used to input the image to be processed into the multi-task model to obtain a multi-task processing result corresponding to the image to be processed; wherein the multi-task model is trained using the method of the first aspect.

[0053] The fifth aspect of the present application provides an electronic device, including a memory and a processor coupled to each other, the processor being used to execute program instructions stored in the memory to implement the multi-task model training method of the first aspect, or to implement the multi-task processing method of the second aspect.

[0054] In a sixth aspect, the present application provides a computer-readable storage medium having program instructions stored thereon. When the program instructions are executed by a processor, the training method of the multi-task model of the first aspect or the multi-task processing method of the second aspect is implemented.

[0055] It can be understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0056] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0058] Figure 1 It is a flowchart of an embodiment of a training method for a multi-task model provided by the present application;

[0059] Figure 2 It is a flowchart of an embodiment of a multi-task processing method provided by the present application;

[0060] Figure 3 It is a schematic diagram of the framework of an embodiment of a multi-task model provided by the present application;

[0061] Figure 4 It is a schematic diagram of a framework of an embodiment of a training device for a multi-task model provided by the present application;

[0062] Figure 5It is a schematic diagram of a framework of an embodiment of a multi-task processing device provided by the present application;

[0063] Figure 6 It is a schematic diagram of a framework of an embodiment of an electronic device provided by the present application;

[0064] Figure 7 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0065] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0066] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0067] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0068] The present embodiment provides a training method for a multi-task model, and the training method for the multi-task model can be implemented by a server or a terminal alone, or by a server and a terminal in collaboration. In some embodiments, the terminal or server can implement the training method for the multi-task model provided by the embodiment of the present disclosure by running a computer program. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as a client that supports virtual scenes, such as a game APP; it can also be a small program, that is, a program that can be run only by downloading it to a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be an application, module or plug-in in any form.

[0069] In the disclosed embodiment, the server is used as the execution entity, and the multi-task model provided by the present application is suitable for scenarios such as the Internet of Things and smart devices. In these scenarios, a large number of images and videos will be captured by sensors and transmitted to a remote server for analysis.

[0070] See also Figure 1 , Figure 1 It is a flowchart of an embodiment of a training method for a multi-task model provided in the present application.

[0071] In one embodiment, the multi-task model includes an image encoder and at least two branch networks connected to the image encoder. The at least two branch networks include at least a target branch network and an entropy model; the image encoder is connected to the target branch network and the entropy model respectively. The target branch network includes a classification model and / or a construction model. The construction model includes a feature fusion model and an image decoder. The image encoder is an ELIC image encoding and decoding network; the classification model is a ConvNeXt image classification network.

[0072] In a disclosed embodiment, the image encoder is set to a multi-channel ELIC image encoding and decoding network, so that the encoded data can retain more semantic information of the image, so as to improve the application performance of the multi-task model; by setting the classification model to a ConvNeXt image classification network, the accuracy of the classification model in predicting the category information of the image based on the encoded information can be improved.

[0073] In one embodiment, the multi-task model further includes a feature quantization layer, which is disposed between the image encoder and the target branch network and the entropy model.

[0074] Among them, the image encoder is used to encode the image to obtain the feature tensor of the image; the feature quantization layer is used to round the feature tensor of the image to facilitate the rounding of the encoded data, facilitate the encoding of the image, and reduce the computational complexity; the entropy model is used to compress the feature tensor of the image to obtain a feature tensor with a shorter encoding length, while retaining more semantic information of the image and reducing the space occupied by the feature tensor storage; the target branch network is used to identify the target category in the image based on the feature tensor of the image; the construction model is used to reconstruct the image based on the feature tensor of the image, so as to reproduce the image corresponding to the feature tensor to meet the needs of visual tasks.

[0075] In the disclosed embodiment, the target branch network includes a classification model and a construction model.

[0076] Specifically, the training method of the multi-task model may include the following steps:

[0077] S11: Obtain a training data set, where the training data set includes training images containing targets.

[0078] S12: Iteratively train the initial image encoder and the initial target branch network based on the training image to obtain an updated image encoder and an updated target branch network.

[0079] S13: Input the training image into the updated image encoder to iteratively train the initial entropy model to obtain an updated entropy model.

[0080] S14: Based on the training image, iteratively train the updated image encoder, the updated target branch network, and the updated entropy model to obtain a trained multi-task model.

[0081] In the disclosed embodiment, the initial image encoder and the initial target branch network are first optimized based on the training image, and then the entropy model is optimized based on the training image. Finally, the updated image encoder, the updated target branch network, and the updated entropy model are optimized based on the training image. By gradually optimizing the modules in the multi-task model, the model has better compression performance, while improving the classification ability and image construction ability of the model.

[0082] In a disclosed embodiment, in step S12, the initial image encoder and the initial target branch network are iteratively trained based on the training image to obtain the updated image encoder and the updated target branch network. The specific implementation is as follows.

[0083] The training image is encoded by an initial image encoder to obtain first encoded data of the training image; the training image has labeled category information.

[0084] In one embodiment, when the initial target branch network includes an initial classification model, the training image is classified based on the first encoded data of the training image through the initial classification model to obtain the first predicted category information of the training image. Specifically, the initial classification model includes a plurality of sequentially cascaded deep convolutional layers, and the feature information of the first encoded data of the training image is sequentially extracted through the plurality of deep convolutional layers to obtain the feature information output by each deep convolutional layer. The first predicted category information of the training image is determined based on the feature information. Among them, the deep convolutional layers can be four, five, six, etc., which can be set according to actual conditions.

[0085] The initial image encoder and the initial classification model are iteratively trained based on a first error value between the first predicted category information and the labeled category information corresponding to the training image to obtain an updated image encoder and an updated classification model.

[0086] In one embodiment, when the initial target branch network includes an initial construction model, the initial construction model is used to construct an image based on the first coded data of the training image to generate a predicted image. In a specific embodiment, at least two feature extractions are performed on the first coded data of the training image to obtain a plurality of feature information corresponding to the training image; a feature fusion model is used to perform feature fusion on the plurality of feature information to obtain a fusion feature of the training image; and an image decoder is used to decompress the fusion feature of the training image to generate a predicted image.

[0087] The initial image encoder and the initial constructed model are iteratively trained based on a second error value between the predicted image corresponding to the training image and the training image to obtain an updated image encoder and an updated constructed model.

[0088] In this embodiment, feature information of multiple levels of the training image is fused through a feature fusion model to improve the semantic richness of the feature information of the training image, and the fused features with rich semantic information are decompressed through an image encoder to improve the accuracy of the predicted image.

[0089] In one embodiment, when the initial target branch network includes an initial classification model and an initial construction model, the first encoded data of the training image is firstly extracted through the initial classification network to obtain feature information output by each deep convolution layer. The first predicted category information of the training image is determined based on the feature information of the training image. Then, the feature information output by the first four deep convolution layers in the initial classification network is feature fused through the feature fusion model to obtain the fused features of the training image; the fused features of the training image are decompressed through the image decoder to generate a predicted image. Among them, the feature information output by the first four deep convolution layers retains more semantic information of the feature information of the training image. The training image can be processed based on the feature information output by the first four deep convolution layers to enrich and strengthen the data information output by the feature quantization layer, which is convenient for image reconstruction.

[0090] Based on a first error value between first predicted category information and labeled category information corresponding to a training image and a second error value between the predicted image and the training image, an initial image encoder, an initial classification model, and an initial construction model are iteratively trained to obtain an updated image encoder, an updated classification model, and an updated construction model.

[0091] Compared with the technical solution in the prior art of decoding the first encoded data and then classifying, the embodiment of the present disclosure encodes the training image through the initial image encoder, so that the initial classification model classifies the training image based on the first encoded data, thereby improving the classification speed; and generates an image based on the first encoded data with richer semantic information through the initial construction model, thereby improving the image restoration accuracy, so as to facilitate the reconstruction of the training image.

[0092] In one embodiment, in step S13, the training image is input into the updated image encoder to iteratively train the initial entropy model, and a specific implementation method of obtaining the updated entropy model is as follows.

[0093] In a specific embodiment, the training image has a label coding length. In order to reduce the storage space of the training image as much as possible, the label coding length corresponding to the feature tensor of the training image can be set to zero.

[0094] The training image is input into the updated image encoder to obtain the second encoded data of the training image; the second encoded data of the training image is processed by the initial entropy model to obtain the first feature tensor of the training image; the first feature tensor has a corresponding encoding length. The initial entropy model is iteratively trained based on the difference between the encoding length of the first feature tensor corresponding to the training image and the marked encoding length to obtain the updated entropy model.

[0095] In this embodiment, the initial entropy model is optimized by the difference between the coding length of the first feature tensor output by the initial entropy model and the annotated coding length of the corresponding training image, so that the coding length of the first feature tensor generated by the initial entropy model is as small as possible, so as to facilitate the storage of the training image corresponding to the first feature tensor with a smaller coding length, so as to save storage space.

[0096] In a specific embodiment, the multi-task model further includes a feature quantization layer, which is disposed between the initial image encoder and the branch network. Specifically, the second encoded data of the training image is quantized by the feature quantization layer to obtain the third encoded data of the training image; the third encoded data of the training image is processed by the initial entropy model to obtain the first feature tensor of the training image. The third encoded data is the data after the second encoded data is rounded off.

[0097] In this embodiment, the encoding data of the training image is quantized by setting a feature quantization layer, that is, the encoding data is rounded, which facilitates the encoding of the training image and reduces the computational complexity.

[0098] When the difference between the encoding length of the first feature tensor corresponding to the training image and the labeled encoding length is less than a preset value, the training of the initial entropy model can be stopped. The preset value can be 0.5%, 1% or 5%.

[0099] In one embodiment, in step S14, iterative training is performed based on the training image to update the image encoder, update the target branch network, and update the entropy model to obtain the trained multi-task model, which specifically includes the following steps.

[0100] The training image is encoded by updating the image encoder to obtain the encoded data of the training image; the encoded data of the training image is processed by updating the entropy model to obtain the second feature tensor of the training image; the second feature tensor has a corresponding encoding length; the prediction information is obtained by updating the target branch network based on the encoded data of the training image; the updated entropy model and the updated target branch network are iteratively trained based on the prediction information corresponding to the training image and the second feature tensor to obtain the trained multi-task model. The encoded data of the training image in this embodiment is the second encoded data in the above embodiment.

[0101] In a specific embodiment, the training image is encoded by updating the image encoder to obtain the second encoded data of the training image. The second encoded data is rounded by the feature quantization layer to obtain the third encoded data of the training image. The third encoded data of the training image is processed by updating the entropy model to obtain the second feature tensor of the training image; the second feature tensor has a corresponding encoding length. The third encoded data of the training image is extracted by updating the classification model to obtain multiple feature information of the training image; the target object category in the training image is predicted based on the multiple feature information corresponding to the training image to obtain the second predicted category information; the preset number of feature information corresponding to the training image and the third encoded data are input into the update construction model, and the feature fusion is first performed to obtain the fusion information, and then the image containing the target object is predicted based on the fusion information to obtain the second predicted image. The final loss value corresponding to the training image is determined based on the weighted sum of the first difference between the encoding length of the second feature tensor corresponding to the training image and the labeled encoding length, the second difference between the second predicted category information and the labeled category information, and the third difference between the second predicted image and the training image, and the updated image encoder, the updated entropy model, the updated classification model, and the updated construction model are optimized as a whole based on the final loss value to obtain the trained multi-task model.

[0102] In this embodiment, the updated entropy model and the updated target branch network are jointly trained through training images to improve the training accuracy of the multi-task model.

[0103] In a specific embodiment, before performing target recognition or target tracking, sample images containing target objects can be collected by multiple image acquisition devices in a preset area. The target object can be a movable target such as an animal or a vehicle. Information annotation is performed on the target object in the sample image, and the initial image encoder and the initial target branch network in the multi-task model are first iteratively trained with the collected sample images to obtain an updated image encoder and an updated target branch network; then the updated image encoder is connected to the initial entropy model, and the initial entropy model is iteratively trained based on the vehicle image to obtain an updated entropy model; finally, the updated image encoder, the updated target branch network and the updated entropy model are jointly trained based on the collected vehicle image, so that the trained multi-task model can simultaneously realize target recognition classification, image reconstruction and image compression.

[0104] In other embodiments, the target branch network in the multi-task model may also include other models for processing and analyzing images, such as an image-text conversion model, an image segmentation model, and an image clustering model.

[0105] In the embodiments of the present disclosure, the multi-task processing method can be applied to scenarios such as the Internet of Things and smart devices. The multi-task processing method can be applied to data processing centers, cloud storage image scenarios, and Internet of Things devices. In these scenarios, a large number of images and videos are captured by sensors and transmitted to a remote server for analysis. The multi-task processing method may include the following steps.

[0106] See also Figure 2 , Figure 2 1 is a flowchart of an embodiment of a multi-task processing method provided by the present application. Specifically, the multi-task processing method may include the following steps:

[0107] S21: Obtain the image to be processed.

[0108] S22: Input the image to be processed into the multi-task model to obtain a multi-task processing result corresponding to the image to be processed.

[0109] See also Figure 3 , Figure 3 It is a framework diagram of an embodiment of a multi-task model provided in this application.

[0110] In this embodiment, the multi-task model is trained by the training method of the multi-task model in the above embodiment. The multi-task model includes an image encoder, a feature quantization layer, a target branch network and an entropy model; the image encoder is connected to the target branch network and the entropy model respectively through the feature quantization layer. Among them, the target branch network includes a classification model and a construction model. The construction model includes a feature fusion model and an image decoder. Among them, the image encoder is an ELIC image encoding and decoding network; the classification model is a ConvNeXt image classification network.

[0111] In one embodiment, the execution subject of the multi-task processing method may be a server. The multi-task processing method may be applicable to data processing centers, cloud storage image scenarios, and IoT devices. The server receives an image to be processed of size H*W*3, encodes the image to be processed through an image encoder, obtains initial feature data, and the feature quantization layer rounds the initial encoded data to obtain The optimized feature data of . Among them, the number of channels C can be set to 320.

[0112] Through the entropy model The optimized feature data is compressed to obtain a feature tensor of the image to be processed. The compressed feature tensor of the image to be processed is stored so that the image to be processed can be stored by storing the feature tensor.

[0113] Through multiple deep convolutional layers in the classification model, The optimized feature data is used for feature extraction, and each deep convolutional layer outputs feature information. The size of the feature information is The image to be processed is classified through the feature information output by the deep convolutional layer to obtain the classification result of the image to be processed.

[0114] The feature fusion model in the construction module is used to fuse the feature information output by the first preset number of deep convolutional layers in the classification model to obtain the fusion features of the image to be processed. Specifically, the feature information output by the first preset number of deep convolutional layers in the classification model is input into the feature fusion model, the feature information output by the deep convolutional layer is linearly mapped, and then the feature information output by the first preset number of deep convolutional layers is stacked according to the channel dimension to form a layer of size The fusion feature is obtained by linearly mapping the fusion feature and performing dimensionality reduction processing, so that the size of the obtained fusion feature is adjusted to Through the image decoder The fused features are decoded and restored to obtain the image to be processed.

[0115] In one embodiment, the multi-task processing method can also be implemented collaboratively by the terminal and the server. Specifically, the image encoder and feature quantization layer in the multi-task model are deployed on the terminal, and the entropy model, classification model and construction model are deployed on the server. The specific implementation method is as follows.

[0116] In a specific embodiment, the terminal collects the image to be processed, and the image encoder and feature quantization layer deployed on the terminal perform image encoding and blurring processing on the image to be processed to obtain optimized feature data of the image to be processed. The terminal compresses the optimized feature data of the image to be processed to obtain a code stream; the terminal transmits the code stream to the server.

[0117] The server receives the code stream transmitted by the terminal, and decompresses the received code stream to obtain the optimized feature data of the image to be processed. The optimized feature data is compressed by the entropy model to obtain the feature tensor of the image to be processed. The compressed feature tensor of the image to be processed is stored, so that the image to be processed can be stored by storing the feature tensor.

[0118] The optimized feature data is extracted in turn through multiple deep convolutional layers in the classification model, and each deep convolutional layer outputs feature information. The image to be processed is classified through the feature information output by the deep convolutional layer to obtain the classification result of the image to be processed.

[0119] The feature fusion model in the construction module is used to fuse the feature information output by the first preset number of deep convolutional layers in the classification model to obtain the fusion features of the image to be processed. The image decoder is used to decode and restore the fusion features to obtain the image to be processed.

[0120] In this embodiment, a method directly in the image compression domain is used to perform reasoning on the machine vision task, which can improve the reasoning speed; at the same time, it also has good image reconstruction quality, which is convenient for relevant personnel to supervise the implementation of the machine vision task.

[0121] In a specific embodiment, when performing target recognition or target tracking, an image to be processed containing the target object is collected in real time. The target object can be a movable target such as an animal or a vehicle. The image to be processed containing the target object is input into the multi-task model obtained by the above training. The multi-task model performs image encoding on the image to be processed to obtain the initial feature data corresponding to the target object in the image to be processed. The feature quantization layer rounds the initial encoded data to obtain The optimized feature data of is the optimized feature data corresponding to the target object in the image to be processed. The optimized feature data is compressed to obtain the feature tensor of the target object in the image to be processed, so as to save the image to be processed. The compressed feature tensor of the target object in the image to be processed is stored. The optimized feature data is used for feature extraction, and each deep convolution layer outputs feature information. The target object in the image to be processed is classified through the feature information output by the deep convolution layer, and the classification result of the target object in the image to be processed is obtained, so that the image to be processed is attributed to the image set of the target object and the trajectory of the target object is generated. The feature fusion model in the construction module is used to fuse the feature information output by the front preset number of deep convolution layers in the classification model to obtain the fusion feature of the target object in the image to be processed. The fusion feature of the target object in the image to be processed contains more semantic information of the target object. The fusion feature is linearly mapped and dimensionally reduced to adjust the size of the obtained fusion feature to Through the image decoder The fused features are decoded and restored to obtain an image to be processed containing the target object, so that the image to be processed can be displayed on other displays.

[0122] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0123] See also Figure 4, Figure 4 It is a schematic diagram of the framework of an embodiment of a training device for a multi-task model provided in the present application.

[0124] The embodiment of the present disclosure provides a multi-task model training device 60, which includes an image acquisition module 61, a first training module 62, a second training module 63 and a third training module 64. The multi-task model includes an initial image encoder and at least two branch networks connected to the initial image encoder.

[0125] The acquisition module 61 is used to acquire a training data set, where the training data set includes a training image containing a target; the at least two branch networks include at least an initial target branch network and an initial entropy model.

[0126] The first training module 62 is used to iteratively train the initial image encoder and the initial target branch network based on the training image to obtain an updated image encoder and an updated target branch network.

[0127] The second training module 63 is used to input the training image into the updated image encoder to iteratively train the initial entropy model to obtain an updated entropy model.

[0128] The third training module 64 is used to iteratively train the updated image encoder, the updated target branch network and the updated entropy model based on the training image to obtain a trained multi-task model.

[0129] In this embodiment, the initial image encoder and the initial target branch network are first optimized based on the training image, and then the entropy model is optimized based on the training image. Finally, the updated image encoder, the updated target branch network, and the updated entropy model are optimized based on the training image. By gradually optimizing the modules in the multi-task model, the training speed of the model and the accuracy of the model application are improved.

[0130] See also Figure 5 , Figure 5 It is a schematic diagram of a framework of an embodiment of a multi-task processing device provided by the present application.

[0131] This embodiment provides a multi-task processing device 70 , which includes an image acquisition module 71 and a processing module 72 .

[0132] The acquisition module 71 is used to acquire the image to be processed.

[0133] The processing module 72 is used to input the image to be processed into the multi-task model to obtain a multi-task processing result corresponding to the image to be processed; wherein the multi-task model is trained using the multi-task model training device in any of the above-mentioned multi-task model training device embodiments.

[0134] The above scheme uses the multi-task model trained by the multi-task model training device in any of the above-mentioned multi-task model training device embodiments to process the image to be processed, and improves the model training speed and the accuracy of model application by gradually optimizing the modules in the multi-task model.

[0135] See also Figure 6 , Figure 6 It is a schematic diagram of the framework of an embodiment of an electronic device provided by the present application.

[0136] The electronic device 80 includes a memory 81 and a processor 82 coupled to each other, and the processor 82 is used to execute the program instructions stored in the memory 81 to implement the steps of any of the above-mentioned multi-task model training method embodiments, or to implement the steps of any of the above-mentioned multi-task processing method embodiments. In a specific implementation scenario, the electronic device 80 may include, but is not limited to: a microcomputer, a server, and in addition, the electronic device 80 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0137] Specifically, the processor 82 is used to control itself and the memory 81 to implement the steps of the training method embodiment of any of the multi-task models described above, or to implement the steps of any of the multi-task processing method embodiments described above. The processor 82 may also be referred to as a CPU (Central Processing Unit). The processor 82 may be an integrated circuit chip having signal processing capabilities. The processor 82 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. In addition, the processor 82 may be implemented by an integrated circuit chip.

[0138] See also Figure 7 , Figure 7 1 is a schematic diagram of a framework of an embodiment of a computer-readable storage medium provided by the present application. The computer-readable storage medium 90 stores program instructions 901 that can be executed by a processor, and the program instructions 901 are used to implement the steps of any of the above-mentioned multi-task model training method embodiments, or to implement the steps of any of the above-mentioned multi-task processing method embodiments.

[0139] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0140] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0141] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0142] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0143] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A training method for a multi-task model, characterized in that: The multi-task model includes an initial image encoder and at least two branch networks connected to the initial image encoder; The training method comprises: Acquire a training data set, wherein the training data set includes a training image containing a target; the at least two branch networks include at least an initial target branch network and an initial entropy model; Iteratively training the initial image encoder and the initial target branch network based on the training image to obtain an updated image encoder and an updated target branch network; Inputting the training image into the updated image encoder to iteratively train the initial entropy model to obtain an updated entropy model; The updated image encoder, the updated target branch network and the updated entropy model are iteratively trained based on the training image to obtain a trained multi-task model.

2. The multi-task model training method according to claim 1, characterized in that: The initial target branch network includes an initial classification model and / or an initial construction model; The iterative training of the initial image encoder and the initial target branch network based on the training image to obtain an updated image encoder and an updated target branch network includes: The training image is encoded by the initial image encoder to obtain first encoded data of the training image; the training image has labeled category information; Classifying the training image based on the first coded data of the training image by the initial classification model to obtain first predicted category information of the training image; and / or; generating a predicted image by performing image construction based on the first coded data of the training image by the initial construction model; The initial image encoder and the initial target branch network are iteratively trained based on a first error value between the first predicted category information and the labeled category information corresponding to the training image and / or a second error value between the predicted image and the training image to obtain the updated image encoder and the updated target branch network.

3. The training method of the multi-task model according to claim 2, characterized in that: The initial construction model includes a feature fusion model and an image decoder; The generating a predicted image by performing image construction based on the first coded data of the training image through the initial construction model comprises: Performing feature extraction on the first coded data of the training image to obtain a plurality of feature information corresponding to the training image; Performing feature fusion on the plurality of feature information through the feature fusion model to obtain fusion features of the training image; The fused features of the training image are decompressed by the image decoder to generate the predicted image.

4. The training method of the multi-task model according to any one of claims 1 to 3, characterized in that: The training image has a labeled coding length; The step of inputting the training image into the updated image encoder to iteratively train the initial entropy model to obtain an updated entropy model comprises: Inputting the training image into the update image encoder to obtain second encoded data of the training image; Processing the second coded data of the training image by the initial entropy model to obtain a first feature tensor of the training image; the first feature tensor has a corresponding coding length; The initial entropy model is iteratively trained based on the difference between the coding length corresponding to the first feature tensor and the marked coding length to obtain the updated entropy model.

5. The multi-task model training method according to claim 4, characterized in that: The multi-task model further includes a feature quantization layer, which is arranged between the initial image encoder and the branch network; The training method further comprises: quantizing the second coded data of the training image through the feature quantization layer to obtain third coded data of the training image; The step of processing the second encoded data of the training image by using the initial entropy model to obtain a first feature tensor of the training image includes: The third encoded data of the training image is processed by the initial entropy model to obtain a first feature tensor of the training image.

6. The multi-task model training method according to any one of claims 1 to 5, characterized in that: The iterative training of the updated image encoder, the updated target branch network and the updated entropy model based on the training image to obtain a trained multi-task model includes: encoding the training image by using the updated image encoder to obtain encoded data of the training image; Processing the encoded data of the training image by the updated entropy model to obtain a second feature tensor of the training image; the second feature tensor has a corresponding encoding length; Obtaining prediction information based on the encoded data of the training image by the updated target branch network; The updated entropy model and the updated target branch network are iteratively trained based on the prediction information corresponding to the training image and the second feature tensor to obtain a trained multi-task model.

7. The multi-task model training method according to claim 2, characterized in that: The initial image encoder is an ELIC image encoding and decoding network, and / or the initial classification model is a ConvNeXt image classification network.

8. A multitasking method, characterized in that: The method comprises: Get the image to be processed; Inputting the image to be processed into a multi-task model to obtain a multi-task processing result corresponding to the image to be processed; Wherein, the multi-task model is trained using the method described in any one of claims 1 to 7.

9. A multi-task model training device, characterized in that: The multi-task model includes an initial image encoder and at least two branch networks connected to the initial image encoder, including: An acquisition module is used to acquire a training data set, wherein the training data set includes a training image containing a target; the at least two branch networks include at least an initial target branch network and an initial entropy model; A first training module is used to iteratively train the initial image encoder and the initial target branch network based on the training image to obtain an updated image encoder and an updated target branch network; A second training module is used to input the training image into the updated image encoder to iteratively train the initial entropy model to obtain an updated entropy model; The third training module is used to iteratively train the updated image encoder, the updated target branch network and the updated entropy model based on the training image to obtain a trained multi-task model.

10. A multi-tasking processing device, characterized in that: include: An acquisition module, used for acquiring images to be processed; A processing module, used for inputting the image to be processed into a multi-task model to obtain a multi-task processing result corresponding to the image to be processed; wherein the multi-task model is trained using the method described in any one of claims 1 to 7.

11. An electronic device, characterized in that: It includes a memory and a processor coupled to each other, and the processor is used to execute program instructions stored in the memory to implement the training method of the multi-task model described in any one of claims 1 to 7, or to implement the multi-task processing method described in claim 8.

12. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by the processor, the training method of the multi-task model described in any one of claims 1 to 7 is implemented, or the multi-task processing method described in claim 8 is implemented.

Images