Image processing method based on multi-task learning for fusion stitching and object detection

Abstract

The present invention discloses an image processing method of fusion splicing and target detection based on multi-task learning, and the multi-task learning processing model used includes a feature extraction module, an image splicing module, a target detection module and a fusion module. The feature extraction module is used to extract features from each original image respectively, and transmit the extracted feature map to the image splicing module and the target detection module. The image splicing module splices the original images into a complete image based on the feature maps of each original image. The target detection module detects the target in each original image and obtains the initial annotation information corresponding to each original image. The fusion module is used to map the initial annotation information of each original image to the complete image to obtain the final annotation information. The model of the present invention is constructed based on multi-task learning, which reduces the requirements for samples and improves the training efficiency, generalization ability and processing speed.

Description

Technical Field

[0001] The invention belongs to the field of specific computing models, and in particular relates to an image processing method. Background Art

[0002] In many computer vision application scenarios, such as multi-drone collaborative inspections and multi-camera monitoring, it is often necessary to stitch multiple sets of images together and perform target detection on the stitched images. The existing solution is to stitch multiple photos together first, and then perform target detection on the stitched complete image. However, this "serial" processing flow has many defects:

[0003] 1. The model cannot be trained end-to-end and requires a large number of actual samples for image stitching and object detection, which is costly.

[0004] 2. The loss functions of image stitching and object detection are not unified, resulting in weak generalization ability of the overall model.

[0005] 3. When processing, image stitching must be completed first, and then target detection is performed. The processing speed is slow and it is difficult to improve efficiency through parallel means. Summary of the invention

[0006] The present invention proposes an image processing method of fusion stitching and target detection based on multi-task learning, the purpose of which is to solve the problems of high training cost, weak generalization ability and slow processing speed.

[0007] The technical solution of the present invention is as follows:

[0008] An image processing method based on multi-task learning fusion stitching and target detection, wherein multiple original images are input into a processing model to obtain a stitched complete image and one or more annotation information corresponding to the complete image, each annotation information includes a bounding box, and a category corresponding to a target in the bounding box and a probability of belonging to the category, wherein the processing model is a multi-task learning processing model, which includes a feature extraction module, an image stitching module, a target detection module and a fusion module;

[0009] The feature extraction module is used to extract features from each original image and transmit the extracted feature map to the image stitching module and the target detection module;

[0010] The image stitching module stitches the original images into a complete image based on the feature maps of the original images;

[0011] The target detection module detects the target in each original image and obtains the initial annotation information corresponding to each original image;

[0012] The fusion module is used to map the initial annotation information of each original image to the complete image to obtain the final annotation information.

[0013] As a further improvement of the image processing method for fusion stitching and target detection based on multi-task learning: the feature extraction module generates multi-level and multi-scale feature maps, and these feature maps are simultaneously output to the image stitching module and the target detection module;

[0014] For Original image , the obtained feature map is recorded as .

[0015] As a further improvement of the image processing method based on multi-task learning for fusion stitching and target detection: the specific processing process of the image stitching module is as follows:

[0016] First, use the feature point matching algorithm Extract the feature point set from each feature map: , For the original image The corresponding feature map set;

[0017] Then, let the total original images, select one of the original images As the reference image, calculate the other original images Transformation matrix to base image , :

[0018] , ;

[0019] , ;

[0020] in, It is Original image The first image matched with the reference image For feature points, Indicates the use of Euclidean distance to the feature map set and Perform feature point matching. represents the homography matrix method;

[0021] Then perform geometric transformation on the original images except the reference image to obtain the corresponding transformed images:

[0022] , ;

[0023] Finally, the reference image is fused with each transformed image to generate a complete image:

[0024] ;

[0025] in, The obtained complete image.

[0026] As a further improvement of the image processing method based on multi-task learning fusion stitching and target detection: in the target detection module, for the Original image , the processing process is:

[0027] ;

[0028] in, is the original image The corresponding initial annotation information; For the A bounding box, which includes the following parameters: the horizontal coordinate position of the bounding box , the vertical coordinate position of the bounding box , Bounding box width and the bounding box height ; For the The predicted category of the object in the bounding box, For the The predicted probability distribution of objects in each bounding box belonging to each category; For the original image The number of bounding boxes predicted in ; Indicates the process of detecting the head.

[0029] As a further improvement of the image processing method based on multi-task learning for fusion stitching and target detection, the processing process of the fusion module is as follows:

[0030] First, geometric transformation is performed on each bounding box corresponding to the original image other than the reference image to obtain the transformed bounding box:

[0031] , ;

[0032] in, For the target detection module to detect the original image The number of predicted bounding boxes, For the The parameter vector of the bounding box, For the original image The transformation matrix to the reference image, for The corresponding transformed bounding box;

[0033] Then, the initial annotation information of the reference image is superimposed on the complete image:

[0034] ;

[0035] in, Indicates drawing a bounding box and adding annotation information;

[0036] Then add the transformed bounding boxes of the original images other than the reference image, as well as the predicted categories and predicted probability distributions in the initial annotation information to the complete image:

[0037] , .

[0038] As a further improvement of the image processing method based on multi-task learning for fusion stitching and target detection, the training process of the multi-task learning processing model is as follows:

[0039] Step 1: Prepare the data set; each sample in the data set includes The original images correspond to each other, and also include a real complete image, and the real initial annotation information manually annotated in each original image;

[0040] Step 2: Input each training sample into the multi-task learning processing model, calculate the comprehensive loss function, and update the parameters of the multi-task learning processing model according to the comprehensive loss function; the comprehensive loss function is composed of a weighted splicing loss function and a detection loss function.

[0041] As a further improvement of the image processing method based on multi-task learning for fusion stitching and object detection, the comprehensive loss function for:

[0042] ;

[0043] in, is the concatenation loss function, To detect the loss function, and is the weighting coefficient, and are both greater than or equal to zero and + =1.

[0044] As a further improvement of the image processing method for fusion stitching and target detection based on multi-task learning, the stitching loss function is calculated as follows:

[0045] ;

[0046] in, The complete image is obtained The total number of pixels, For the complete image Middle pixel values, is the first real complete image in the current training sample pixel values.

[0047] As a further improvement of the image processing method based on multi-task learning for fusion stitching and target detection, the calculation method of the detection loss function is:

[0048] ;

[0049] in, For the current sample The classification loss of the original image, For the current sample The regression loss of the original image;

[0050] The classification loss is calculated as:

[0051] ;

[0052] in, For the current sample The number of bounding boxes predicted by the model for the original image, is the number of categories, For the current sample The true initial annotation information of the original image The bounding box for An indicator variable with categorical Indicates that it belongs to categories, Indicates that it does not belong to categories; For the current sample The original image is predicted by the model through the initial annotation information The predicted probability distribution corresponding to the bounding box The predicted probability of each category;

[0053] The regression loss is calculated as:

[0054] ;

[0055] in, For the current sample The number of bounding boxes predicted by the model for the original image, The model predicts The parameters of the bounding box, is the first The parameters of the bounding box include: the horizontal coordinate position of the bounding box , the vertical coordinate position of the bounding box , Bounding box width and the bounding box height , the four groups of parameters are calculated separately and then summed; For a custom function:

[0056] .

[0057] As a further improvement of the image processing method based on multi-task learning for fusion stitching and target detection: the feature extraction module is the backbone network of YOLOv8, and the target detection module is the detection module of YOLOv8.

[0058] Compared with the prior art, the present invention has the following beneficial effects:

[0059] 1. The model of the present invention is built based on multi-task learning and supports an end-to-end training process. Training no longer relies solely on the data of a single task, but can use the data of image stitching and object detection tasks for joint training, thereby reducing the requirement for the number of samples, improving training efficiency, reducing costs and required computing resources, and enhancing generalization capabilities.

[0060] 2. The two branches of the model of the present invention can be carried out simultaneously, which significantly improves the processing speed.

[0061] 3. Image stitching and target detection can guide each other. The panoramic perspective provided by image stitching can be used to improve the accuracy of target detection, and the semantic information provided by target detection can be used to enhance the accuracy of image stitching. The model can learn the semantic features common to the two tasks, thus enhancing the model's capabilities.

[0062] 4. Image stitching and object detection share a feature extraction module, which reduces the redundancy of model parameters, reduces the overall computational complexity, and further improves the operating efficiency of the model, so that the model can be effectively applied on resource-constrained devices.

[0063] 5. The fusion module maps the detection results to the complete image through an innovative mapping mechanism, ensuring the accurate presentation of the detection results and making the model architecture more compact and efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 This is a structural diagram of the multi-task learning processing model in the present invention. DETAILED DESCRIPTION

[0065] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments.

[0066] An image processing method based on multi-task learning for fusion stitching and target detection, which inputs multiple original images into a multi-task learning processing model to obtain a stitched complete image and one or more annotation information corresponding to the complete image. Each annotation information includes a bounding box, the category corresponding to the target in the bounding box, and the probability of belonging to the category.

[0067] The multi-task learning processing model includes a feature extraction module, an image stitching module, a target detection module and a fusion module. The feature extraction module is used to extract features from each original image and transfer the extracted feature map to the image stitching module and the target detection module. The image stitching module stitches the original images into a complete image based on the feature maps of each original image. The target detection module detects the target in each original image and obtains the initial annotation information corresponding to each original image. The fusion module is used to map the initial annotation information of each original image to the complete image to obtain the final annotation information.

[0068] The specific structure of each module is as follows:

[0069] 1. Feature extraction module.

[0070] In this embodiment, the backbone network of YOLOv8 is used as the feature extraction module. The backbone network can generate multi-level and multi-scale feature maps, and these feature maps will be output to the image stitching module and the target detection module at the same time to achieve feature reuse and provide rich feature information for subsequent image stitching and target detection tasks.

[0071] For Original image , the obtained feature map is recorded as , the extraction process can be expressed as: ,in Represents the backbone network of YOLOv8.

[0072] 2. Image stitching module.

[0073] This module stitches multiple original images into a complete image by matching feature points, estimating transformation matrices, etc. During the stitching process, feature point matching utilizes the extracted deep features, improving the accuracy and robustness of matching, especially when dealing with complex scenes and large-scale data.

[0074] The specific process is:

[0075] First, use the feature point matching algorithm Extract the feature point set from each feature map: , For the original image The corresponding feature map set.

[0076] Then, let the total original images, select one of the original images As the reference image, calculate the other original images Transformation matrix to base image , :

[0077] , ;

[0078] , ;

[0079] in, It is Original image The first image matched with the reference image For feature points, Indicates the use of Euclidean distance to the feature map set and Perform feature point matching. Represents the homography matrix method, and you can choose RANSAC, etc.

[0080] Then perform geometric transformation on the original images except the reference image to obtain the corresponding transformed images:

[0081] , ;

[0082] Finally, the reference image is fused with each transformed image to generate a complete image:

[0083] ;

[0084] in, The obtained complete image.

[0085] During this process, the geometric and color consistency between images can be further adjusted through other optimization algorithms as needed.

[0086] 3. Object detection module.

[0087] The target detection module is a detection head including multiple convolutional layers and pooling layers. In this embodiment, the detection module of YOLOv8 is used as the target detection module.

[0088] For Original image , the processing of the target detection module is expressed as:

[0089] ;

[0090] in, is the original image The corresponding initial annotation information; For the A bounding box, which includes the following parameters: the horizontal coordinate position of the bounding box , the vertical coordinate position of the bounding box , Bounding box width and the bounding box height ; For the The predicted category of the object in the bounding box, For the The predicted probability distribution of objects in each bounding box belonging to each category; For the original image The number of bounding boxes predicted in ; Indicates the process of detecting the head.

[0091] In the detection head, in order to eliminate overlapping detection results, non-maximum suppression technology can be used to retain the bounding box with the highest confidence and suppress other bounding boxes that overlap with it. The predicted bounding box can also be further optimized through regression analysis to more accurately locate the position and size of the target.

[0092] 4. Fusion module.

[0093] During the mapping process, a series of meticulous operations are performed to ensure seamless integration of the annotation information with the full image. First, the bounding box output by the object detection module is transformed to fit the coordinate system of the full image, taking into account geometric transformations such as translation, rotation, and scaling that may occur during the mapping process. Then, a spatial alignment operation is performed to adjust the size and shape of the bounding box and correct for differences in viewing angles to ensure that the annotation information is visually consistent with the full image.

[0094] The specific process is:

[0095] First, geometric transformation is performed on each bounding box corresponding to the original image other than the reference image to obtain the transformed bounding box:

[0096] , ;

[0097] in, For the target detection module to detect the original image The number of predicted bounding boxes, For the The parameter vector of the bounding box, For the original image The transformation matrix to the reference image, for The corresponding transformed bounding box.

[0098] Then, the initial annotation information of the reference image is superimposed on the complete image:

[0099] ;

[0100] in, Indicates drawing a bounding box and adding annotation information.

[0101] Then add the transformed bounding boxes of the original images other than the reference image, as well as the predicted categories and predicted probability distributions in the initial annotation information to the complete image:

[0102] , .

[0103] Finally, a visual consistency check is performed to ensure that the superimposed annotation information is coordinated with the complete image content without occlusion or overlap. Through these steps, the visual continuity, integrity and accuracy of the annotation information of the final output image are ensured, which can be directly used for visual display and subsequent analysis.

[0104] Furthermore, the training process of the multi-task learning processing model is as follows:

[0105] Step 1: Prepare the data set. Each sample in the data set includes The image contains three corresponding original images, a real complete image, and the real initial annotation information manually annotated in each original image.

[0106] Step 2: Input each training sample into the multi-task learning processing model, calculate the comprehensive loss function, and update the parameters of the multi-task learning processing model according to the comprehensive loss function. The comprehensive loss function is composed of a weighted combination of a splicing loss function and a detection loss function.

[0107] Specifically, the comprehensive loss function for:

[0108] ;

[0109] in, is the concatenation loss function, To detect the loss function, and is the weighting coefficient, and are both greater than or equal to zero and + =1.

[0110] Among them, the calculation method of the splicing loss function is:

[0111] ;

[0112] in, The complete image is obtained The total number of pixels, For the complete image Middle pixel values, is the first real complete image in the current training sample pixel values.

[0113] The detection loss function is calculated as:

[0114] ;

[0115] in, For the current sample The classification loss of the original image, For the current sample The regression loss of the original image.

[0116] The classification loss is calculated as:

[0117] ;

[0118] in, For the current sample The number of bounding boxes predicted by the model for the original image, is the number of categories, For the current sample The true initial annotation information of the original image The bounding box for An indicator variable with categorical Indicates that it belongs to categories, Indicates that it does not belong to categories; For the current sample The original image is predicted by the model through the initial annotation information The predicted probability distribution corresponding to the bounding box The predicted probability of each class.

[0119] The regression loss is calculated as:

[0120] ;

[0121] in, For the current sample The number of bounding boxes predicted by the model for the original image, The model predicts The parameters of the bounding box, is the first The parameters of the bounding box include: the horizontal coordinate position of the bounding box , the vertical coordinate position of the bounding box , Bounding box width and the bounding box height , the four groups of parameters are calculated separately and then summed; For a custom function:

[0122] .

[0123] The multi-task learning method enables the backbone network to be trained jointly on the datasets of two tasks, which significantly improves the generalization ability of feature extraction and the overall performance of the model.

[0124] This model also supports training on single image stitching or object detection tasks, showing adaptability to training images, enabling it to maintain excellent performance and stability in the face of complex and changing application scenarios. Specifically, when training on single object detection: , ; When training single image stitching: , .

[0125] This model can be used in multi-task application scenarios, such as autonomous driving assistance systems, augmented reality (AR), medical image analysis, etc., as well as in single-task scenarios such as image stitching and target detection.

[0126] It should be noted that it is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. The scope of the present invention is defined by the claims rather than the above description.

Claims

1. An image processing method based on multi-task learning for fusion stitching and target detection, wherein multiple original images are input into a processing model to obtain a stitched complete image and one or more annotation information corresponding to the complete image, wherein each annotation information includes a bounding box, a category corresponding to a target in the bounding box and a probability of belonging to the category, and wherein: The processing model is a multi-task learning processing model, which includes a feature extraction module, an image stitching module, a target detection module and a fusion module; The feature extraction module is used to extract features from each original image and transmit the extracted feature map to the image stitching module and the target detection module; The image stitching module stitches the original images into a complete image based on the feature maps of the original images; The target detection module detects the target in each original image and obtains the initial annotation information corresponding to each original image; The fusion module is used to map the initial annotation information of each original image to the complete image to obtain the final annotation information.

2. The image processing method based on multi-task learning for fusion stitching and target detection as claimed in claim 1, characterized in that: The feature extraction module generates multi-level and multi-scale feature maps, and these feature maps are simultaneously output to the image stitching module and the object detection module; For Original image , the obtained feature map is recorded as .

3. The image processing method based on multi-task learning for fusion stitching and target detection as claimed in claim 2, characterized in that: The specific processing process of the image stitching module is: First, use the feature point matching algorithm Extract the feature point set from each feature map: , For the original image The corresponding feature map set; Then, let the total original images, select one of the original images As the reference image, calculate the other original images Transformation matrix to base image , : ， ； ， ； in, It is Original image The first image matched with the reference image For feature points, Indicates the use of Euclidean distance to the feature map set and Perform feature point matching. represents the homography matrix method; Then perform geometric transformation on the original images except the reference image to obtain the corresponding transformed images: ， ； Finally, the reference image is fused with each transformed image to generate a complete image: ； in, The obtained complete image.

4. The image processing method based on multi-task learning fusion stitching and target detection as claimed in claim 3, characterized in that: In the target detection module, for the Original image , the processing process is: ； in, is the original image The corresponding initial annotation information; For the A bounding box, which includes the following parameters: the horizontal coordinate position of the bounding box , the vertical coordinate position of the bounding box , Bounding box width and the bounding box height ; For the The predicted category of the object in the bounding box, For the The predicted probability distribution of objects in each bounding box belonging to each category; For the original image The number of bounding boxes predicted in ; Indicates the process of detecting the head.

5. The image processing method based on multi-task learning for fusion stitching and target detection as claimed in claim 4, characterized in that: The processing process of the fusion module is: First, geometric transformation is performed on each bounding box corresponding to the original image other than the reference image to obtain the transformed bounding box: ， ； in, For the target detection module to detect the original image The number of predicted bounding boxes, For the The parameter vector of the bounding box, For the original image The transformation matrix to the reference image, for The corresponding transformed bounding box; Then, the initial annotation information of the reference image is superimposed on the complete image: ； in, Indicates drawing a bounding box and adding annotation information; Then add the transformed bounding boxes of the original images other than the reference image, as well as the predicted categories and predicted probability distributions in the initial annotation information to the complete image: ， 。 6. The image processing method based on multi-task learning fusion stitching and target detection as claimed in claim 1, characterized in that: The training process of the multi-task learning processing model is as follows: Step 1: Prepare the data set; each sample in the data set includes The original images correspond to each other, and also include a real complete image, and the real initial annotation information manually annotated in each original image; Step 2: Input each training sample into the multi-task learning processing model, calculate the comprehensive loss function, and update the parameters of the multi-task learning processing model according to the comprehensive loss function; the comprehensive loss function is composed of a weighted splicing loss function and a detection loss function.

7. The image processing method based on multi-task learning fusion stitching and target detection as claimed in claim 6, characterized in that: The comprehensive loss function for: ； in, is the concatenation loss function, To detect the loss function, and is the weighting coefficient, and are both greater than or equal to zero and + =1.

8. The image processing method based on multi-task learning fusion stitching and target detection as claimed in claim 7, characterized in that: The calculation method of the splicing loss function is: ； in, The complete image is obtained The total number of pixels, For the complete image Middle pixel values, is the first real complete image in the current training sample pixel values.

9. The image processing method based on multi-task learning fusion stitching and target detection as claimed in claim 7, characterized in that: The calculation method of the detection loss function is: ； in, For the current sample The classification loss of the original image, For the current sample The regression loss of the original image; The classification loss is calculated as: ； in, For the current sample The number of bounding boxes predicted by the model for the original image, is the number of categories, For the current sample The true initial annotation information of the original image The bounding box for An indicator variable with categorical Indicates that it belongs to categories, Indicates that it does not belong to categories; For the current sample The original image is predicted by the model through the initial annotation information The predicted probability distribution corresponding to the bounding box The predicted probability of each category; The regression loss is calculated as: ； in, For the current sample The number of bounding boxes predicted by the model for the original image, The model predicts The parameters of the bounding box, is the first The parameters of the bounding box include: the horizontal coordinate position of the bounding box , the vertical coordinate position of the bounding box , Bounding box width and the bounding box height , the four groups of parameters are calculated separately and then summed; For a custom function: 。 10. The image processing method based on multi-task learning fusion stitching and target detection according to any one of claims 1 to 9, characterized in that: The feature extraction module is the backbone network of YOLOv8, and the target detection module is the detection module of YOLOv8.

Images