A two-stage image multi-task processing method based on functional modules

Abstract

The present invention belongs to the field of computer vision application technology, and in particular, relates to a two-stage image multi-task processing method based on functional modules, including first data acquisition, obtaining image-text pair data sets, image classification data sets, target detection data sets, and image segmentation data sets; then building a one-stage image-text latent space mapping model; then training and testing the one-stage model, verifying the accuracy of the image-text latent space mapping, and obtaining a universal encoder; then building a two-stage universal encoder, a decoding part, constructing different decoder modules according to different tasks, and obtaining a multi-task image processing model; finally training and testing the two-stage model, and respectively verifying the effectiveness of tasks such as image classification, target detection, and image segmentation through simulation experiments. The present invention guides the model encoder to learn through natural language, builds a universal encoder that is more in line with text semantics, and finally realizes the universality and reusability of the model.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision application, and in particular relates to a two-stage image multi-task processing method based on functional modules. Background Art

[0002] In today's society, computer vision has made great progress, mainly due to the improvement of computing power and the widespread application of deep learning technology. At present, deep learning technology, especially convolutional neural networks, has become the mainstream method in the field of computer vision, bringing significant performance improvements to various image understanding tasks. These tasks include image classification, object detection, image segmentation, etc. Although deep learning technology has achieved great success in the field of computer vision, current research mainly focuses on solving single image understanding tasks. In order to achieve more comprehensive and integrated image understanding capabilities, it is necessary to develop multi-task deep learning models that can handle multiple related tasks at the same time, thereby improving the comprehensive understanding and analysis capabilities of images.

[0003] However, the existing models for various image processing tasks have these functions separately and are single-task models, lacking a multi-task model with a common structure. For example, models such as ResNet, MobilNet, and ShuffleNet can only implement classification tasks; models such as YOLO, RCNN, and SSD can only implement detection tasks; models such as UNET, HRNet, and DeepLab can only implement segmentation tasks. Moreover, these models are trained with a series of fixed object and label pairs. This restricted supervision mode limits their versatility and reusability because additional labeled data is required to train different encoding networks. Summary of the invention

[0004] In response to the technical problems existing in the current image multi-task processing, the present invention proposes a two-stage image multi-task processing method based on functional modules, which has a reasonable design, simple method, strong theoretical basis and can adopt various task decoding modules to realize multi-task processing of images such as image classification, target detection, and image segmentation.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is: a two-stage image multi-task processing method based on functional modules, comprising the following steps:

[0006] S1. Data acquisition: obtain image-text pair datasets, image classification datasets, object detection datasets, and image segmentation datasets;

[0007] S2. Build a one-stage model, image-text latent space mapping model;

[0008] S3, training and testing the one-stage model to verify the accuracy of image-text latent space mapping and obtain a universal encoder;

[0009] S4, build a two-stage model;

[0010] S5, training and testing two-stage models, simulation experiments respectively verify the effectiveness of tasks such as image classification, target detection, and image segmentation;

[0011] The specific operation method of the S4 step is:

[0012] S4-1, build a multi-task image processing model, superimpose the 4-layer Transformer encode output in the general image encoder obtained in S3-2 through the Concat method, and then use 1×1 convolution to reduce its dimension to obtain a feature vector V containing multi-layer semantics out , and finally sent to the multi-task module for corresponding task analysis;

[0013] S4-2. Constructing classification decoding module: feature vector V out Implement classification tasks through Feed Forward Network (FFN);

[0014] S4-3. Construct detection and decoding module: feature vector V out Two feedforward networks are used respectively, one for classification to determine the category of the detected target, and the other for determining the target position to give the coordinates of the upper left corner and lower right corner of the target;

[0015] S4-4. Constructing segmentation decoding module: feature vector V out Two feedforward networks are used respectively, one for classification to determine the category of the segmented target, and the other for giving the target mask to separate the target and the background at the pixel level;

[0016] S4-5. Finally, the general image codec obtained in S4-1 is combined with the multi-task decoders of S4-1, S4-2, and S4-3 to obtain a multi-task image processing model.

[0017] Preferably, the specific operation method of step S1 is:

[0018] S1-1. Build an image-text pair dataset. Extract images from public datasets Imagenet, PASCAL-VOC, COCO, and PASCAL-Context that cover various image tasks, and describe these images. The text description template is "This is a picture of [object]", where [object] refers to the target object contained in the picture, so as to build image-text pairs.

[0019] S1-2. Constructing image classification dataset D Class =[Img, LabelClass ], based on the public classification dataset Imagenet, more categories of pictures are obtained through web crawlers, and the data is stored in the same folder for objects of the same category;

[0020] S1-3. Constructing target detection dataset D Object =[Img, Label Class , Label Object ], based on the public detection datasets PASCAL-VOC and COCO, the images crawled from the S1-2 network are annotated using labelimg or labelme annotation tools, and the data is finally unified into the format of json key-value pairs. The data content includes the image path, target category, upper left corner coordinates, and lower right corner coordinates;

[0021] S1-4, build an image segmentation dataset. Based on the public detection dataset PASCAL-Context, use the labelme annotation tool to annotate the images crawled from the S1-2 network, and finally unify the data into the format of json key-value pairs. The data content includes the image path, target category, and the coordinates of multiple contour points.

[0022] S1-5. Data set D I_T , D Class , D Object , D Seg It is divided into training set and test set in an 8:2 ratio.

[0023] Preferably, the specific operation method of step S2 is:

[0024] S2-1. The image-text latent space mapping model draws on the idea of ​​CLIP. First, the image is segmented into patches, then patch encoding and position encoding are performed. Four Transformer Encoders are used to build the image encoder core, and finally the latent space feature vector V of the image is obtained. i , i represents the vector dimension;

[0025] S2-2. Tokenize the text description, then encode it with tokens and positions, and use 4 Transformer Encoders to build the text encoder core to obtain the latent space feature vector V of the text. t , t represents the vector dimension;

[0026] S2-3. At the same time, it is ensured that the dimensions of the latent space feature vectors and output by the image and text encoders are the same, i.e., i = t;

[0027] S2-4, regularize the latent space feature vectors output by the image and text encoders respectively, defined as

[0028]

[0029]

[0030] The logarithmically transformed cosine similarity is used as the loss function, defined as

[0031] Loss = log(LV i ×LV t )

[0032] Preferably, the specific operation method of step S3 is:

[0033] S3-1, send the training set in the data into the model, use warmup as the training method and learning rate change strategy, train for 100 rounds, and finally obtain the mapping relationship between image and text, and obtain the trained image-text latent space mapping model;

[0034] S3-2. Send the test set in the data into the model to verify the accuracy of the mapping relationship between image and text, save the image encoding weights that meet the requirements, and obtain a universal image encoder.

[0035] Preferably, the specific operation method of step S5 is:

[0036] S5-1. Send the training set in the data into the model, freeze the image encoder weights, use warmup as the training method and learning rate change strategy, and use the category balanced cross entropy loss function as the loss function, whose formula is:

[0037]

[0038] where w l represents the weight of a single category, l represents the category, n represents the amount of data in the l category, and m represents the amount of data in all categories. represents the true classification label, Represents the predicted classification label. After 30 rounds of training, the classification decoder is finally obtained, and then the test set in the data is sent to the model to verify the accuracy of the classification task;

[0039] S5-2, send the training set in the data into the model, freeze the image encoder weights, use warmup as the training method learning rate change strategy, use the category balanced cross entropy loss function for classification loss, and use the weighted IOU+Dice loss function for detection loss function. The formula is:

[0040]

[0041] , train for 30 rounds, and finally obtain the detection decoder. Then, send the test set in the data into the model to verify the accuracy of the detection task;

[0042] S5-3. Send the training set in the data into the model, freeze the image encoder weights, use warmup as the learning rate change strategy for training, use the category-balanced cross entropy loss function for classification loss, and use the weighted IOU+Dice loss function for segmentation loss function. Train for 30 rounds and finally obtain the segmentation decoder. Then send the test set in the data into the model to verify the accuracy of the segmentation task.

[0043] Compared with the prior art, the advantages and positive effects of the present invention are:

[0044] The present invention provides a two-stage functional module-based image multi-task processing method, constructs a multi-task model with a universal structure, guides the visual concepts learned by the model encoder through natural language, and constructs a universal encoder that is more in line with the text semantics. At the same time, the universal encoder is frozen in the second stage, and only the decoder part is trained, thereby reducing the training cost of various task modules. Finally, the present invention realizes the universality and reusability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0046] Figure 1 A schematic diagram of the structure of the image-text latent space mapping model provided by the present invention;

[0047] Figure 2 A schematic diagram of the structure of a general image encoder provided by an example of the present invention;

[0048] Figure 3 Schematic diagram of the multi-task image processing model structure provided for this method example;

[0049] Figure 4 A schematic diagram of the classification module structure provided for the purposes of this law;

[0050] Figure 5 A schematic diagram of the structure of a detection module provided by an example of the present invention;

[0051] Figure 6 A schematic diagram of the structure of a segmentation module provided by an example of the present invention;

[0052] Figure 7This is a diagram showing the effect of the multi-task image processing model provided by the example of the present invention. DETAILED DESCRIPTION

[0053] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described below in conjunction with the accompanying drawings and embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0054] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways than those described herein. Therefore, the present invention is not limited to the specific embodiments of the following disclosure.

[0055] Considering that the existing models of various image processing tasks have separate functions, they are all single-task models, lacking a multi-task model with a general structure. For example, models such as ResNet, MobilNet, ShuffleNet, etc. can only implement classification tasks; models such as YOLO, RCNN, SSD, etc. can only implement detection tasks; models such as UNET, HRNet, DeepLab, etc. can only implement segmentation tasks. And these models are trained through a series of fixed objects and label pairs. This restricted supervision mode limits their versatility and reusability because additional labeled data is required to train different encoding networks. In view of the above problems, the present invention proposes a two-stage multi-task image multi-task processing method based on functional modules.

[0056] First, data acquisition, obtain image-text pair dataset, image classification dataset, object detection dataset, and image segmentation dataset.

[0057] Specifically, we construct an image-text pair dataset, extract images from public datasets covering various image tasks, such as Imagenet, PASCAL-VOC, COCO, and PASCAL-Context, and describe these images. The text description template is "This is a picture of [object]", where [object] refers to the target object contained in the picture, so as to construct image-text pairs; we construct an image classification dataset D Class =[Img, Label Class ]: Based on the public classification dataset Imagenet, more categories of pictures are obtained through web crawlers. The data is stored in the same folder for objects of the same category. Object =[Img, Label Class , Label Object]: Based on the public detection datasets PASCAL-VOC, COCO, etc., the images crawled from the network are annotated by the labelimg or labelme annotation tool, and the data is finally unified into the format of json key-value pairs. The data content includes the image path, target category, upper left corner point coordinates, and lower right corner point coordinates; build an image segmentation dataset. Based on the public detection datasets PASCAL-Context, etc., the images crawled from the network are annotated by the labelme annotation tool, and the data is finally unified into the format of json key-value pairs. The data content includes the image path, target category, and the coordinates of multiple contour points; the dataset D I_T , D Class , D Object , D Seg It is divided into training set and test set in an 8:2 ratio.

[0058] Secondly, build a first-stage model, the image-text latent space mapping model.

[0059] Specifically, the image-text latent space mapping model draws on the idea of ​​CLIP, such as Figure 1 The structure of the image-text latent space mapping model first divides the image into patches, then performs patch encoding and position encoding on it, and uses 4 Transformer Encoders to build the image encoder core, and finally obtains the latent space feature vector V of the image. i , i represents the vector dimension; then the text description is tokenized, and then token encoded and position encoded. Four Transformer Encoders are used to build the text encoder core to obtain the latent space feature vector V of the text t , t represents the vector dimension; at the same time, the dimensions of the latent space feature vectors and output by the image and text encoders are guaranteed to be the same, that is, i = t; finally, the latent space feature vectors output by the image and text encoders are regularized respectively, defined as

[0060]

[0061] The logarithmically transformed cosine similarity is used as the loss function, defined as

[0062] Loss = log(LV i ×LV t )

[0063] Next, the one-stage model is trained and tested to verify the accuracy of the image-text latent space mapping and obtain a universal encoder.

[0064] Specifically, the training set in the data is sent to the model, and the learning rate change strategy is warmup. After 100 rounds of training, the mapping relationship between image and text is finally obtained, and the trained image-text latent space mapping model is obtained. Then Figure 2 As shown, the test set in the data is sent to the model to verify the accuracy of the mapping relationship between the image and the text, and the image encoding weights that meet the requirements are saved to obtain a universal image encoder.

[0065] Then, a two-stage model is built. The encoding part uses the obtained general encoder, and the decoding part builds different decoder modules according to different tasks, and finally a multi-task image processing model is obtained.

[0066] Specifically, we build a multi-task image processing model, such as Figure 3 As shown in the figure, the four-layer Transformer encoder outputs in the obtained general image encoder are superimposed by Concat, and then 1×1 convolution is used to reduce its dimension to obtain a feature vector V containing multiple layers of semantics. out , and finally sent to the multi-task module for corresponding task analysis; then Figure 4 As shown, the classification decoding module is constructed, and the feature vector V out The classification task is implemented through the Feed Forward Network (FFN); then Figure 5 As shown, the detection decoding module is constructed, and the feature vector V out Two feedforward networks are used respectively, one for classification to determine the category of the detected target, and the other for determining the target position to give the coordinates of the upper left corner and lower right corner of the target; then, Figure 6 As shown, the segmentation decoding module is constructed, and the feature vector V out Two feedforward networks are used respectively, one for classification to determine the category of the segmented target, and the other for giving the target mask to divide the target and the background at the pixel level; finally, the general image codec is combined with the multi-task decoder to obtain a multi-task image processing model.

[0067] Finally, the two-stage model is trained and tested, and simulation experiments verify the effectiveness of tasks such as image classification, object detection, and image segmentation.

[0068] Specifically, the training set in the data is fed into the model, the image encoder weights are frozen, the learning rate change strategy of the training method adopts warmup, and the loss function adopts the category balanced cross entropy loss function, whose formula is:

[0069]

[0070] where w lrepresents the weight of a single category, l represents the category, n represents the amount of data in the l category, and m represents the amount of data in all categories. represents the true classification label, Represents the predicted classification label. After 30 rounds of training, the classification decoder is finally obtained, and then the test set in the data is sent to the model to verify the accuracy of the classification task; then the training set in the data is sent to the model, the image encoder weight is frozen, the training method learning rate change strategy adopts warmup, the classification loss adopts the category balanced cross entropy loss function, and the detection loss function adopts the weighted IOU+Dice loss function, the formula is:

[0071]

[0072] After 30 rounds of training, the detection decoder is finally obtained, and the test set in the data is then fed into the model to verify the accuracy of the detection task; then the training set in the data is fed into the model, the image encoder weights are frozen, the learning rate change strategy for training adopts warmup, the classification loss adopts the category-balanced cross entropy loss function, and the segmentation loss function adopts the weighted IOU+Dice loss function. After 30 rounds of training, the segmentation decoder is finally obtained, and the test set in the data is then fed into the model to verify the accuracy of the segmentation task. Figure 7 As shown, the effect diagram of the multi-task image processing model is shown.

[0073] The above description is only a preferred embodiment of the present invention and does not limit the present invention in other forms. Any technician familiar with the profession may use the technical content disclosed above to change or modify it into an equivalent embodiment with equivalent changes and apply it to other fields. However, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A two-stage functional module-based image multi-task processing method, characterized in that: The steps include: S1. Data acquisition: obtain image-text pair datasets, image classification datasets, object detection datasets, and image segmentation datasets; S2. Build a one-stage model, image-text latent space mapping model; S3, training and testing the one-stage model to verify the accuracy of image-text latent space mapping and obtain a universal encoder; S4, build a two-stage model; S5, training and testing the two-stage model, and simulation experiments verify the effectiveness of image classification, object detection, and image segmentation tasks respectively; The specific operation method of the S3 step is: S3-1, send the training set in the data into the model, use warmup as the training method and learning rate change strategy, train for 100 rounds, and finally obtain the mapping relationship between image and text, and obtain the trained image-text latent space mapping model; S3-2, send the test set in the data into the model to verify the accuracy of the mapping relationship between the image and the text, save the image encoding weights that meet the requirements, and obtain a universal image encoder; The specific operation method of the S4 step is: S4-1, build a multi-task image processing model, superimpose the 4-layer Transformerencode output of the general image encoder obtained in S3-2 through the Concat method, and then use 1×1 convolution to reduce its dimension to obtain a feature vector V containing multi-layer semantics out , and finally sent to the multi-task module for corresponding task analysis; S4-2, build classification decoding module, feature vector V out Implement classification tasks through feed-forward networks; S4-3, build detection decoding module, feature vector V out Two feedforward networks are used respectively, one for classification to determine the category of the detected target, and the other for determining the target position to give the coordinates of the upper left corner and lower right corner of the target; S4-4, construct segmentation decoding module, feature vector V out Two feedforward networks are used respectively, one for classification to determine the category of the segmented target, and the other for giving the target mask to separate the target and the background at the pixel level; S4-5. Finally, the general image codec obtained in S4-1 is combined with the multi-task decoders of S4-1, S4-2, and S4-3 to obtain a multi-task image processing model.

2. The two-stage functional module-based image multi-task processing method according to claim 1, characterized in that: The specific operation method of step S1 is as follows: S1-1. Build an image-text pair dataset. Extract images from public datasets Imagenet, PASCAL-VOC, COCO, and PASCAL-Context that cover various image tasks, and describe these images. The text description template is "This is a picture of [object]", where [object] refers to the target object contained in the picture, so as to build an image-text pairing method. S1-2. Constructing image classification dataset D Class＝ [Img, Label Class ], based on the public classification dataset Imagenet, more categories of pictures are obtained through web crawlers, and the data is stored in the same folder for objects of the same category; S1-3. Constructing target detection dataset D Object =[Img, Label Class , Label Object ], based on the public detection datasets PASCAL-VOC and COCO, the images crawled from the S1-2 network are annotated using labelimg or labelme annotation tools, and the data is finally unified into the format of json key-value pairs. The data content includes the image path, target category, upper left corner coordinates, and lower right corner coordinates; S1-4, build an image segmentation dataset. Based on the public detection dataset PASCAL-Context, use the labelme annotation tool to annotate the images crawled from the S1-2 network, and finally unify the data into the format of json key-value pairs. The data content includes the image path, target category, and the coordinates of multiple contour points. S1-5. Data set D I_T , D Class， D Object , D Seg It is divided into training set and test set in an 8:2 ratio.

3. The two-stage functional module-based image multi-task processing method according to claim 1, characterized in that: The specific operation method of the S2 step is: S2-1. The image-text latent space mapping model draws on the idea of ​​CLIP. First, the image is segmented into patches, then patch encoding and position encoding are performed. Four Transformer Encoders are used to build the image encoder core, and finally the latent space feature vector V of the image is obtained. i , i represents the vector dimension; S2-2. Tokenize the text description, then encode it with tokens and positions, and use 4 Transformer Encoders to build the text encoder core to obtain the latent space feature vector V of the text. t , t represents the vector dimension; S2-3. At the same time, it is ensured that the dimensions of the latent space feature vectors and output by the image and text encoders are the same, i.e., i = t; S2-4, regularize the latent space feature vectors output by the image and text encoders respectively, defined as The logarithmically transformed cosine similarity is used as the loss function, defined as Loss=log(LV i ×LV t )。 4. The two-stage functional module-based image multi-task processing method according to claim 1, characterized in that: The specific operation method of step S5 is as follows: S5-1. Send the training set in the data into the model, freeze the image encoder weights, use warmup as the training method and learning rate change strategy, and use the category balanced cross entropy loss function as the loss function, whose formula is: where w l represents the weight of a single category, l represents the category, n represents the amount of data in the l category, and m represents the amount of data in all categories. represents the true classification label, Represents the predicted classification label, trained for 30 rounds, and finally obtained the classification decoder. Then, the test set in the data is sent to the model to verify the accuracy of the classification task; S5-2, send the training set in the data into the model, freeze the image encoder weights, use warmup as the training method learning rate change strategy, use the category balanced cross entropy loss function for classification loss, and use the weighted IOU+Dice loss function for detection loss function. The formula is: After 30 rounds of training, the detection decoder is finally obtained, and then the test set in the data is fed into the model to verify the accuracy of the detection task; S5-3. Send the training set in the data into the model, freeze the image encoder weights, use warmup as the learning rate change strategy for training, use the category-balanced cross entropy loss function for classification loss, and use the weighted IOU+Dice loss function for segmentation loss function. Train for 30 rounds and finally obtain the segmentation decoder. Then send the test set in the data into the model to verify the accuracy of the segmentation task.

Images