A method and system for extracting image interest regions based on weak label data

Through the image interest region extraction method based on weak label data, the minimum external rotating rectangle box and deep learning segmentation network model is used to solve the problem of strong label dependence in the existing technology, and low-cost and efficient image interest region extraction is achieved, and the expansion of the image interest region It marks applicable scenarios and improves extraction accuracy.

CN118628724BActive Publication Date: 2024-11-22SHAOXING UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411109960.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-14
Publication Date
2024-11-22
Estimated Expiration
2044-08-14

AI Technical Summary

Technical Problem

In the prior art, image area of ​​interest extraction algorithms rely mostly on strong label data, resulting in high labeling costs and difficult to judge in some applications, resulting in difficulty in precise labeling, and the image area of ​​interest cannot be accurately extracted.

Method used

Using a method based on weak label data, the image samples are marked with low precision through the minimum external rotating rectangle box, and the image samples are generated by multi-scale scaling, input into the deep learning segmentation network model for training, and output interest Regional prediction map, combining rotating rectangle mapping estimation feature map to calculate loss value, adjust model hyperparameters, and improve extraction accuracy.

Benefits of technology

It realizes low-cost and efficient image interest area extraction, makes up for the insufficient labeling accuracy caused by low-precision labeling, expands the applicable annotation scenarios, can fully and accurately mark difficult-to-judge area boundaries, and improves extraction accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118628724B_ABST
    Figure CN118628724B_ABST
Patent Text Reader

Abstract

The present invention provides a method and system for extracting regions of interest in images based on weak label data, and relates to the technical field of image processing. The method comprises: obtaining a data set; annotating the region of interest labels in image samples by a minimum circumscribed rotated rectangular frame to obtain a rotated rectangular mask map; performing multi-scale scaling on the image samples to obtain image samples of different scales; inputting the image samples of different scales into an image segmentation model for training, and outputting a region of interest prediction map; mapping the region of interest prediction map to a rectangular region in the image sample to which the long and short sides of the rotated rectangular frame are consistent, and obtaining a rotated rectangular mapping estimated feature map; calculating the loss value of the image sample; when the loss value is greater than a preset loss value, adjusting the hyperparameters of the image segmentation model, and repeating the training; and extracting the image region of interest of the image to be detected by using the trained image segmentation model. While not reducing the recognition accuracy, the annotation cost is effectively reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method and system for extracting image interest regions based on weak label data. Background Art

[0002] An image region of interest is a portion of an image that is important or relevant for a specific analysis task. In image processing and computer vision, an ROI can be an object, a specific shape, or any area in an image that contains critical information needed for further analysis. For example, in medical image analysis, the region of interest may be a tumor; in satellite images, the region of interest may be a specific terrain feature; in traffic monitoring systems, the region of interest may be a vehicle or a pedestrian. By focusing on the most critical parts of the image, the speed and accuracy of the analysis can be improved, the consumption of computing resources can be reduced, and the efficiency of the overall system can be improved. In addition, analyzing only the region of interest helps reduce the interference of background noise and irrelevant information, making the results more reliable and accurate.

[0003] Image region of interest extraction is a key support in many visual analysis tasks and is the core technical support for application fields such as fun interaction, target retrieval, and medical-assisted detection. However, existing algorithms are mostly based on strong label data, which means that the region of interest in the image needs to be accurately annotated at the pixel level. This type of strong labeling is extremely expensive, and in some applications, there are difficult-to-determine regional boundaries, making accurate labeling extremely difficult and limiting application scenarios, which makes it impossible to accurately extract the image region of interest. Summary of the invention

[0004] In order to solve the technical problem that the existing algorithms in the prior art are mostly based on strong label data, that is, the interest region in the image needs to be accurately marked at the pixel level, such strong labeling is extremely costly, and in some applications there are region boundaries that are difficult to judge, making accurate labeling extremely difficult and having application scenario limitations, and thus failing to accurately extract the image interest region, the present invention provides an image interest region extraction method and system based on weak label data.

[0005] The technical solution provided by the embodiment of the present invention is as follows:

[0006] First aspect

[0007] An embodiment of the present invention provides a method for extracting an image region of interest based on weakly labeled data, comprising:

[0008] S1: Get a dataset with multiple image samples;

[0009] S2: labeling the region of interest in the image sample by a minimum circumscribed rotated rectangular frame to obtain a rotated rectangular mask image;

[0010] S3: Perform multi-scale scaling on the same image sample to obtain image samples of the same image sample at different scales;

[0011] S4: inputting image samples of different scales into the image segmentation model for training respectively, and outputting a prediction map of the region of interest, wherein each scale of the image sample corresponds to a prediction map of the region of interest;

[0012] S5: Map the predicted region of interest map to a rectangular region in the image sample to which the long and short sides of the rotated rectangular frame are consistent, to obtain a rotated rectangular mapping estimated feature map;

[0013] S6: Calculate the loss value of the image sample by combining the region of interest prediction map, the rotated rectangular mask map and the rotated rectangular mapping estimated feature map;

[0014] S7: if the loss value is greater than a preset loss value, adjust the hyperparameters of the image segmentation model and return to step S4; otherwise, proceed to step S8;

[0015] S8: Use the trained image segmentation model to extract the image region of interest of the image to be detected.

[0016] Second aspect

[0017] An embodiment of the present invention provides an image interest region extraction system based on weak label data, comprising:

[0018] processor;

[0019] A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method for extracting an image region of interest based on weak label data according to the first aspect is implemented.

[0020] The third aspect

[0021] An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the method for extracting an image region of interest based on weak label data according to the first aspect is implemented.

[0022] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0023] In the present invention, a minimum circumscribed rotated rectangular frame is introduced to perform low-precision annotation of the region of interest to obtain a rotated rectangular mask map, and then the image sample is scaled at multiple scales and input into the existing deep learning segmentation network model for result prediction, accurately capturing the region of interest features, and mapping the results back to the corresponding minimum circumscribed rotated rectangular frame to obtain a high-precision rotated rectangular mapping estimated feature map that can accurately characterize the region of interest. Then, under its supervision, the model is trained to narrow the characterization gap between the prediction results and the region of interest labels, and to make up for the problem of insufficient annotation accuracy caused by the weak region of interest labels with low-precision annotation. This method has low cost and high efficiency, thereby improving the accuracy of image region of interest extraction. This method is not limited to specific annotation tasks, and can also completely and accurately annotate regions with difficult-to-judge region boundaries. While not reducing the accuracy of region of interest extraction, it greatly expands the applicable annotation scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 A flowchart of a method for extracting an image region of interest based on weakly labeled data provided by an embodiment of the present invention;

[0026] Figure 2 A schematic diagram of the structure of a system for extracting image regions of interest based on weakly labeled data provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0028] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0029] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0030] In the embodiments of the present invention, sometimes a subscript such as W1 may be mistakenly written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are consistent.

[0031] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0032] Reference Manual Attached Figure 1 , shows a flow chart of a method for extracting image interest regions based on weakly labeled data provided by an embodiment of the present invention.

[0033] The embodiment of the present invention provides a method for extracting an image region of interest based on weak label data, which can be implemented by an image region of interest extraction device based on weak label data, and the image region of interest extraction device based on weak label data can be a terminal or a server. The processing flow of the method for extracting an image region of interest based on weak label data may include the following steps:

[0034] S1: Get a dataset with multiple image samples;

[0035] It should be noted that the dataset containing multiple image samples is the starting point of the entire image analysis and processing process. These image samples will be used for subsequent annotation, training, and testing to ensure that the model can accurately identify and extract the region of interest in the image in practical applications. Choosing a suitable dataset is crucial for training an effective image segmentation model, because the quality and diversity of the data directly affect the performance and generalization ability of the model.

[0036] S2: labeling the region of interest in the image sample by a minimum circumscribed rotated rectangular frame to obtain a rotated rectangular mask image;

[0037] Among them, the minimum circumscribed rotated rectangle is the minimum rectangular box around a specific object or area in the image. It can be rotated at any angle to tightly surround the target area. It is not restricted by the direction of the image boundary to ensure that the coverage area is minimized. The region of interest label refers to the important parts of the image that are marked. These labels guide the model to learn to identify and analyze these areas. The rotated rectangular mask map is a mask map created based on the minimum circumscribed rotated rectangle, where the pixel value in the region of interest is set to 255 (indicating the area of ​​concern) and other areas are set to 0 (background). This mask map helps the model accurately identify and learn the specific location and shape of the region of interest.

[0038] It should be noted that the minimum circumscribed rotated rectangle is used to accurately mark the region of interest in the image sample, and then the corresponding rotated rectangular mask is generated. This step is to create an image form that can intuitively represent the boundary and shape of the region of interest, which helps to better learn the characteristics of the region in subsequent model training, especially when the direction and shape of the region of interest in the image are diverse, to ensure that the model can accurately capture these details.

[0039] In a possible implementation manner, the S2 specifically includes:

[0040] S201: When the image sample has a strong label of the region of interest, the pixel values ​​in the strong label of the region of interest are set to 255, and the remaining pixel values ​​are set to 0, to obtain the rotated rectangular mask image, and obtain the minimum circumscribed rotated rectangular frame and the minimum circumscribed rotated rectangular frame rotation angle by a minimum circumscribed rotated rectangular frame estimation method, otherwise, enter S202;

[0041] S202: manually label the region of interest in the image sample according to the minimum circumscribed principle to obtain the minimum circumscribed rotated rectangular frame, set the pixel values ​​in the minimum circumscribed rotated rectangular frame to 255, and set the remaining pixel values ​​to 0 to obtain the rotated rectangular mask image, the minimum circumscribed rotated rectangular frame rotation angle, and obtain the minimum circumscribed rotated rectangular frame rotation angle through the minimum circumscribed rotated rectangular frame estimation method.

[0042] Specifically, first, S201 describes that if the image sample already has a strong label for the region of interest, that is, the region of interest is clearly marked, then the pixel values ​​in these regions will be set to 255 (white), and the rest will be set to 0 (black), forming a rotated rectangular mask map, and calculating the minimum circumscribed rotated rectangular box and its rotation angle. If there is no strong label, step S202 requires manual annotation of the region of interest, and also generates a rotated rectangular mask map and estimates the rotation angle of the box. Both sub-steps are intended to prepare accurately labeled data for subsequent image processing, and ensure that the region of interest can be accurately captured and expressed regardless of its orientation by means of a minimum circumscribed rotated rectangular box, which is crucial to improving the accuracy and efficiency of subsequent model training.

[0043] In a possible implementation manner, the minimum circumscribed rotated rectangular frame estimation method is specifically as follows:

[0044] Based on the region of interest label, an edge map of the region of interest is obtained through the findContours function of the opencv library;

[0045] The minAreaRect function is called to extract the minimum circumscribed rotation rectangle and the rotation angle of the minimum circumscribed rotation rectangle in the edge map of the region of interest.

[0046] Among them, OpenCV (Open Source Computer Vision Library) is an open source computer vision and machine learning software library. findContours is a function in the OpenCV library used to detect contours in an image. It extracts contours from a binary image by finding the boundary between a white object (foreground) and a black background. The function returns a list containing a point set of all contours in the image. Contours can be regarded as curves composed of continuous points, which is very useful for shape analysis and object detection and recognition. minAreaRect is a function used in OpenCV to calculate and return the minimum area rectangle that can be enclosed by a given set of points. This rectangle may have a rotation angle. Even if the rectangle is rotated, it will ensure that all given points are covered. The function returns a Box2D structure, which includes the center point coordinates, width, height, and rotation angle of the rectangle. This function is very useful in calculating the minimum enclosing rectangle of a given shape or object, and is often used in applications such as image segmentation and object recognition.

[0047] Specifically, the findContours function of the OpenCV library is first used to detect edges from the marked region of interest mask map to generate an edge map of the region of interest. Subsequently, by calling the minAreaRect function, the rotated rectangular box that can cover each detected edge to the minimum extent is calculated and extracted from these edge maps, and the rotation angle of the rectangular box is obtained. This method can accurately define the position and orientation of the region of interest, which is very important for subsequent image analysis and processing, because it ensures that the region of interest can be accurately identified and marked no matter how it is positioned or rotated in the image.

[0048] S3: Perform multi-scale scaling on the same image sample to obtain image samples of the same image sample at different scales;

[0049] The same image sample refers to any image sample in the data set, and the image sample is scaled to multiple scales to obtain corresponding image samples of different scales.

[0050] It should be noted that the same image sample is scaled to multiple scales to generate image sample sets of different scales. The purpose of this process is to capture the features of images at different scales and improve the adaptability and robustness of the model to image scale changes. Through this multi-scale processing, the model can learn image features more comprehensively, so as to effectively identify and analyze objects of various sizes in practical applications, especially when the image resolution and target size vary. Doing so helps improve the generalization ability and performance of the model.

[0051] S4: Input image samples of different scales into the image segmentation model for training, and output the prediction map of the region of interest.

[0052] Among them, each scale image sample corresponds to a region of interest prediction map;

[0053] It should be noted that the image samples of different scales obtained through multi-scale processing are input into the image segmentation model for training. The purpose of this step is to let the model learn how to identify and predict the region of interest at various scales. The image sample of each scale will generate a corresponding region of interest prediction map, which helps the model to effectively identify objects of different sizes in practical applications, thereby improving the accuracy of the prediction and the adaptability of the model. Through this multi-scale training, the model can better understand and process the size changes in the image.

[0054] In a possible implementation, the image segmentation model is a deep learning model based on a deep segmentation network.

[0055] Among them, the image segmentation model is based on a deep segmentation network, which is a deep learning model specially designed to identify and separate different regions or objects in an image. This model uses a multi-layer neural network to learn complex patterns and features in images, and can accurately classify images at the pixel level, thereby achieving accurate prediction of the region of interest to which each pixel in the image belongs.

[0056] S5: Map the predicted region of interest map to a rectangular region in the image sample to which the long and short sides of the rotated rectangular frame are consistent, to obtain a rotated rectangular mapping estimated feature map;

[0057] Among them, the size information includes the long side direction and the short side direction. Specifically, the prediction results of the interest region of each scale image, that is, the interest region prediction map, are mapped to a rectangular area consistent with the long and short side directions of the rotated rectangular frame annotated in the image to obtain a rotated rectangular mapping estimated feature map.

[0058] It should be noted that the predicted region of interest obtained from training at different scales is mapped back to the corresponding size and position of the original image. The specific operation is to align the predicted map to the smallest circumscribed rotated rectangular box annotated in the image sample. This process helps to generate a rotated rectangular mapping estimated feature map that accurately corresponds to the actual region of interest. This mapping ensures the accurate correspondence between the prediction result and the actual image area, so that the spatial position predicted by the model accurately matches the actual important area, improving the accuracy and practicality of the prediction.

[0059] In a possible implementation manner, the S5 specifically includes:

[0060] S501: Generate a first basic grayscale image with a pixel value of 0, wherein the size of the first basic grayscale image is larger than the size of the region of interest prediction image;

[0061] In a possible implementation manner, the size of the first basic grayscale image is 1 / 2 of the size of the region of interest prediction image. times.

[0062] S502: copying the region of interest prediction map to the basic grayscale image to obtain a second basic grayscale image;

[0063] S503: rotating the second basic grayscale image around its own center point by a rotation angle of the minimum circumscribed rotation rectangular frame of the minimum circumscribed rotation rectangular frame corresponding to the prediction image of the region of interest, to obtain a rotated second basic grayscale image;

[0064] S504: cutting off the redundant part of the second basic grayscale image after rotation that exceeds the second basic grayscale image before rotation to obtain a cropped second grayscale image, wherein the uncopied area in the cropped second grayscale image is filled with the minimum pixel value of the second basic grayscale image;

[0065] S505: Map the long side and the short side of the second basic grayscale image to the rectangular area of ​​the image sample to which the long and short sides of the rotated rectangular frame have the same direction:

[0066]

[0067] in, Represents the second basic grayscale image after mapping In coordinates The pixel value at H and W Respectively represent the height and width of the second basic grayscale image after filling, max and min represent the maximum value and minimum value respectively, Represents the coordinates to be mapped;

[0068] Specifically, first, at each fixed x-coordinate (along the long side or short side), all y-coordinates are traversed to calculate the maximum pixel value at these positions. Similarly, at each fixed y-coordinate, all x-coordinates are traversed to obtain the maximum pixel value at these positions. Then, for each pixel position (x, y) in the image, the smaller value of the above two calculation results is selected as the final pixel value of the position. The advantages of this mapping method include the ability to significantly enhance the main structures and edges in the image and improve the recognition rate of image features. At the same time, by selecting the smallest maximum value, the interference of noise and non-critical information can be effectively suppressed, making the image more stable and consistent. In addition, this processing method is particularly beneficial for subsequent image analysis and machine learning model training because it strengthens the key visual features in the image and helps the model learn and predict more accurately.

[0069] S506: reversely rotating the mapped second basic grayscale image around its own center point by a rotation angle of the minimum circumscribed rotation rectangle of the minimum circumscribed rotation rectangle corresponding to the prediction image of the region of interest;

[0070] S507: Taking a preset size central area of ​​the second basic grayscale image after the reverse rotation to obtain the rotated rectangular mapping estimated feature map.

[0071] It should be noted that those skilled in the art can set the size of the preset size central area according to actual needs, and the present invention is not limited here. Optionally, the preset size central area can be a 352 x 352 size area at the center of the second basic grayscale image after reverse rotation.

[0072] Specifically, first, create a base grayscale image larger than the size of the prediction image, and then copy the prediction image into this grayscale image. Next, rotate the entire grayscale image according to the rotation angle of the rotation rectangle in the prediction image, crop the part that exceeds the original size, and fill the uncovered area with the minimum pixel value. The processed image is then mapped according to the long and short sides. The smaller of the maximum pixel values ​​of each coordinate point on the long and short sides is taken as the final value, which helps to highlight the main features of the image and suppress noise. Finally, the mapped image is reversely rotated back to the initial angle, and the central area of ​​the preset size is intercepted from it to obtain the final rotated rectangle mapping estimated feature map. This estimated feature map can accurately reflect the model's recognition results of the region of interest, which is helpful for further analysis and application, such as machine learning model training and image recognition tasks, to ensure the accuracy and applicability of the region of interest prediction.

[0073] S6: Calculate the loss value of the image sample by combining the region of interest prediction map, the rotated rectangular mask map and the rotated rectangular mapping estimated feature map;

[0074] It should be noted that the loss value of each image sample is calculated by combining the predicted region of interest map, the rotated rectangular mask map, and the rotated rectangular mapping estimated feature map. This loss value is a key indicator for model optimization and is used to evaluate the deviation between the region of interest predicted by the model and the actual annotation (mask map). Calculating the loss value is an indispensable step in the training process. It guides the model to reduce the prediction error by adjusting parameters, thereby improving the prediction accuracy and model performance. This method ensures that the output of the model is as consistent as possible with the actual image features and optimizes the model's ability to recognize the region of interest in the image.

[0075] In a possible implementation, the loss value includes a joint scale equalization loss value between the region of interest prediction images obtained based on image samples of different scales and a BCE loss value and a DICE loss value between the rotated rectangular mask image and the rotated rectangular mapping estimated feature image.

[0076] Among them, the joint scale-equalized loss value: This loss calculation is based on the difference between the predicted regions of interest obtained from samples of different scales. This loss value helps ensure that the model can maintain consistency in multi-scale processing, that is, regardless of the scale of the input image, the model can stably predict accurate regions of interest. The joint scale-equalized loss encourages the model to give similar predictions at all scales, reducing the impact of scale changes on the prediction results, thereby improving the model's robustness to scale changes. BCE loss value (Binary Cross-Entropy Loss): This is a loss function commonly used in binary classification tasks, suitable for evaluating the difference between the model output (rotated rectangular mapping estimated feature map) and the true label (rotated rectangular mask map). BCE loss calculates the cross entropy between the predicted probability and the actual label of each pixel, effectively measuring the accuracy of the prediction. DICE loss value: DICE loss (also known as F1 Score or Sørensen–Dice coefficient) is another loss function commonly used in image segmentation, especially in the case of unbalanced samples. It measures the overlap area of ​​two samples, that is, the overlap between the region of interest predicted by the model and the actual region of interest. DICE loss helps improve the prediction accuracy of the model in overlapping areas, especially in boundary areas. These three loss values ​​work together to provide comprehensive feedback to the model, guiding it to not only improve prediction accuracy during learning, but also to ensure good consistency and robustness across different image scales and complex shape changes. In this way, the model can achieve higher performance and reliability in practical applications.

[0077] In a possible implementation manner, the joint scale balancing loss value is calculated as follows:

[0078]

[0079] in, represents the joint scale equalization loss value, N represents the total number of zoom scales, and Respectively represent the i Scale and i The predicted region of interest obtained by the image sample of +1 scale is at the same coordinate x The predicted values ​​at i Scale and i +1 scale image sample obtained by the prediction map of the region of interest, represents the first weight parameter, Indicates taking the absolute value, B represents the set of pixels within the minimum circumscribed rotated rectangular frame corresponding to the image sample, Represents a set of statistical pixels B The number of pixels in P represents the predicted region of interest map, and Respectively represent the width and height of the prediction map of the region of interest;

[0080] The formula given here is only one possible method for calculating the joint scale equalization loss. This method only considers the differences in the prediction results of adjacent scales. In addition to this method, different mechanisms can also be used to statistically calculate the differences, such as comparing all scales with the prediction results of the first scale.

[0081] The BCE loss value and the DICE loss value between the rotated rectangular mask image and the rotated rectangular mapping estimated feature map are specifically:

[0082]

[0083] in, Indicates the BCE loss value and DICE loss value The sum of Represents the estimated feature map of the rotated rectangle map The corresponding rotated rectangular mask image, Indicates that based on i The rotated rectangular mapping feature map obtained by the image sample of the scale is estimated, represents the second weight parameter;

[0084] The loss value of the image sample is calculated as follows:

[0085]

[0086] in, Represents the loss value of the image sample.

[0087] It should be noted that the joint scale equalization loss value is used to measure the consistency of the predicted maps of the region of interest between different scales. Specifically, the loss calculates the difference between the predicted map of each scale and the predicted map of its adjacent scale, which is achieved by taking the absolute value and weighted average. The purpose is to ensure that the model can give similar prediction results at different scales and reduce the scale dependence of the prediction. In addition, BCE loss and DICE loss are used to evaluate the degree of match between the estimated feature map of the rotated rectangular mapping and the rotated rectangular mask map. Through the combination of these two loss values, the performance of the model can be comprehensively optimized to ensure the accuracy of the prediction and the accuracy of edge positioning. The combination of these loss values ​​provides a multi-faceted feedback mechanism to help adjust the model to improve the recognition and positioning of the region of interest.

[0088] S7: if the loss value is greater than a preset loss value, adjust the hyperparameters of the image segmentation model and return to step S4; otherwise, proceed to S8;

[0089] It should be noted that if the loss value is high, it indicates that there is room for improvement in the prediction performance of the model, so the hyperparameters need to be adjusted to improve the model performance. After the adjustment, the model enters the training phase (S4) again and continues to optimize until the loss value is lower than the preset standard, and then turns to step S8 for final application and verification. This method ensures that the model achieves the best performance in practical applications.

[0090] It should be noted that those skilled in the art can set the size of the preset loss value according to actual needs, and the present invention is not limited here.

[0091] S8: Use the trained image segmentation model to extract the image region of interest of the image to be detected.

[0092] The image to be detected is any image that needs to have its region of interest extracted.

[0093] The present invention relates to low-cost data annotation, and can obtain a more accurate representation of the region of interest at a low cost compared to a regular rectangular frame. In addition, the existing segmentation network can be directly referenced, the architecture is simple, and it can adapt well to the multi-scale features of the image. By directly mapping the prediction results with reference to the rotated rectangular morphology, the mapping method is efficient and has a low computational cost, and can well solve the problem of regional interference caused by the loss of the network training directly based on the prediction results and the rotated rectangular mask map calculation, and effectively improve the model performance. In the training process, multi-scale image sample input is used, and network parameters are shared. Combined with the joint scale equalization loss and the BCE and DICE losses of each scale, the adaptability of the network to different scales is jointly constrained, and the analysis robustness of the network model to different scales is improved.

[0094] For example, a certain scale of image data is collected to construct a data set, and the minimum circumscribed rotated rectangular box is marked for the region of interest to form a rotated rectangular mask map; the same image is scaled to N scales and used as the segmentation network input in turn, and the existing deep segmentation network is used as the basic unit to calculate the region of interest prediction results of images of different scales; the region of interest prediction results of images of each scale are mapped to a rectangular area with the same long and short sides as the rotated rectangular box marked on the image, and a rotated rectangular mapping estimated feature map is obtained; the joint scale balance loss between the region of interest prediction results of images of different scales, as well as the BCE and DICE losses between the rotated rectangular mapping estimated feature map and the rotated rectangular mask map are calculated; the image region of interest extraction model training is completed to obtain the deep segmentation network weights adapted to the corresponding image analysis needs; the image to be detected is read and input into the deep segmentation network to obtain the region of interest prediction result of the image.

[0095] Optionally, taking colonoscopy images as an example, first: 1450 colonoscopy images are collected to construct a colonoscopy image dataset, in which the polyp region is the region of interest and the region of interest has a strong label.

[0096] Secondly: Based on the strong label of the region of interest, the opencv library function findContours() is used to obtain the edge map of the region of interest of each image, and then the minAreaRect() function is called to obtain the minimum external rotated rectangle, and the rotated rectangle mask map and the tilt angle of the rotated rectangle are obtained at the same time. The pixel intensity value of the covered area inside the rotated rectangle is set to 255, and the pixel intensity value of the remaining area is set to 0 to obtain the rotated rectangle mask map.

[0097] Then, the selected deep segmentation network uses res2net50 as the backbone. The output features of the last three layers of the res2net50 backbone are respectively convolved by 1x1, and the corresponding feature channels are unified to 64. The feature scales of the last two layers are aligned to the second layer features, and then the three features after convolution and scale alignment are multiplied to obtain fusion features. Finally, the fusion features are convolved by 1x1 to obtain the prediction results. The same image is scaled to two scales, 352x352 and 320x320, and used as the input of the deep segmentation network at the same time. The prediction result images of the regions of interest of images of different scales are calculated, and all prediction result images are aligned to 352x352.

[0098] After that: Calculate the corresponding rotated rectangular mapping estimated feature map as follows: Generate a The size of the all-zero grayscale image I, where the value of K is 352, the width and height of the image I can be slightly larger than ; Copy the predicted result image to the center of the all-zero grayscale image I to obtain image I1; rotate image I1 around its center point The part beyond the range of the canvas (image I1 area) after rotation is cut off, and the uncovered part of the canvas is filled with the minimum pixel intensity value in the pixel point set of image I1 to obtain image I2; image I2 is mapped along the long and short sides to obtain image I3. The calculation method is further defined as:

[0099] Among them, H and W are the height and width of image I3, and the sizes of both are , which is 528x528; rotate image I3 around its center point The 352 x 352 area at the center is taken as the estimated feature map P of the rotated rectangular mapping. r. Calculate the joint scale balance loss between the prediction results of the interest regions of images of different scales; calculate the BCE and DICE losses between the estimated feature maps of the rotated rectangular mapping of images of different scales and the rotated rectangular mask map, and the value can be 0.5. Then calculate the overall loss of the network model ; The optimizer AdamW is selected to set the training segmentation network model, the initial learning rate is set to 1e-4, the bachsize is set to 16, and a total of 20 rounds of training are performed to finally obtain the segmentation network weights. Read the image to be detected and input it into the trained segmentation network to obtain the region of interest prediction result of the image. The architecture is simple, the data annotation cost is low, and the robustness is high, which can effectively adapt to the image region of interest extraction task.

[0100] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0101] In the present invention, a minimum circumscribed rotated rectangular frame is introduced to perform low-precision annotation of the region of interest to obtain a rotated rectangular mask map, and then the image sample is scaled at multiple scales and input into the existing deep learning segmentation network model for result prediction, accurately capturing the region of interest features, and mapping the results back to the corresponding minimum circumscribed rotated rectangular frame to obtain a high-precision rotated rectangular mapping estimated feature map that can accurately characterize the region of interest. Then, under its supervision, the model is trained to narrow the characterization gap between the prediction results and the region of interest labels, and to make up for the problem of insufficient annotation accuracy caused by the weak region of interest labels with low-precision annotation. This method has low cost and high efficiency, thereby improving the accuracy of image region of interest extraction. This method is not limited to specific annotation tasks, and can also completely and accurately annotate regions with difficult-to-judge region boundaries. While not reducing the accuracy of region of interest extraction, it greatly expands the applicable annotation scenarios.

[0102] Reference Manual Attached Figure 2 , showing a structural schematic diagram of an image interest region extraction system based on weak label data provided by the present invention.

[0103] The present invention further provides an image region of interest extraction system 20 based on weak label data, which is applied to the above-mentioned image region of interest extraction method based on weak label data, and comprises:

[0104] Processor 201.

[0105] The memory 202 stores computer-readable instructions. When the computer-readable instructions are executed by the processor 201 , the method for extracting an image region of interest based on weak label data in the method embodiment is implemented.

[0106] The image region of interest extraction system 20 based on weak label data provided by the present invention can execute the above-mentioned image region of interest extraction method based on weak label data and achieve the same or similar technical effects. To avoid repetition, the present invention will not go into details.

[0107] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0108] In the present invention, a minimum circumscribed rotated rectangular frame is introduced to perform low-precision annotation of the region of interest to obtain a rotated rectangular mask map, and then the image sample is scaled at multiple scales and input into the existing deep learning segmentation network model for result prediction, accurately capturing the region of interest features, and mapping the results back to the corresponding minimum circumscribed rotated rectangular frame to obtain a high-precision rotated rectangular mapping estimated feature map that can accurately characterize the region of interest. Then, under its supervision, the model is trained to narrow the characterization gap between the prediction results and the region of interest labels, and to make up for the problem of insufficient annotation accuracy caused by the weak region of interest labels with low-precision annotation. This method has low cost and high efficiency, thereby improving the accuracy of image region of interest extraction. This method is not limited to specific annotation tasks, and can also completely and accurately annotate regions with difficult-to-judge region boundaries. While not reducing the accuracy of region of interest extraction, it greatly expands the applicable annotation scenarios.

[0109] It should be understood that the processor in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0110] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0111] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When a computer instruction or computer program is loaded or executed on a computer, a process or function according to an embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0112] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0113] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0114] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0115] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0116] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0117] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0118] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0119] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0120] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0121] An embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, a method for extracting an image region of interest based on weak label data as described in a method embodiment is implemented.

[0122] A computer-readable storage medium provided by the present invention can implement the steps and effects of the method for extracting image interest regions based on weakly labeled data in the above method embodiment. To avoid repetition, the present invention will not go into details.

[0123] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0124] In the present invention, a minimum circumscribed rotated rectangular frame is introduced to perform low-precision annotation of the region of interest to obtain a rotated rectangular mask map, and then the image sample is scaled at multiple scales and input into the existing deep learning segmentation network model for result prediction, accurately capturing the region of interest features, and mapping the results back to the corresponding minimum circumscribed rotated rectangular frame to obtain a high-precision rotated rectangular mapping estimated feature map that can accurately characterize the region of interest. Then, under its supervision, the model is trained to narrow the characterization gap between the prediction results and the region of interest labels, and to make up for the problem of insufficient annotation accuracy caused by the weak region of interest labels with low-precision annotation. This method has low cost and high efficiency, thereby improving the accuracy of image region of interest extraction. This method is not limited to specific annotation tasks, and can also completely and accurately annotate regions with difficult-to-judge region boundaries. While not reducing the accuracy of region of interest extraction, it greatly expands the applicable annotation scenarios.

[0125] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

[0126] There are a few points to note:

[0127] (1) The drawings of the embodiments of the present invention only involve structures related to the embodiments of the present invention. Other structures may refer to conventional designs.

[0128] (2) For the sake of clarity, in the drawings used to describe the embodiments of the present invention, the thickness of layers or regions is exaggerated or reduced, that is, these drawings are not drawn according to the actual scale. It is understood that when an element such as a layer, film, region or substrate is referred to as being "on" or "under" another element, the element may be "directly" "on" or "under" the other element or there may be intermediate elements.

[0129] (3) In the absence of conflict, the embodiments of the present invention and the features therein may be combined with each other to obtain new embodiments.

[0130] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. The protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for extracting image interest regions based on weakly labeled data, characterized in that: include: S1: Get a dataset with multiple image samples; S2: labeling the region of interest in the image sample by a minimum circumscribed rotated rectangular frame to obtain a rotated rectangular mask image; S3: Perform multi-scale scaling on the same image sample to obtain image samples of the same image sample at different scales; S4: inputting image samples of different scales into the image segmentation model for training respectively, and outputting a prediction map of the region of interest, wherein each scale of the image sample corresponds to a prediction map of the region of interest; S5: Mapping the predicted region of interest map to a rectangular region in the image sample to which the long and short sides of the rotated rectangular frame are in the same direction, to obtain a rotated rectangular mapping estimated feature map; S6: Calculate the loss value of the image sample by combining the region of interest prediction map, the rotated rectangular mask map and the rotated rectangular mapping estimated feature map; S7: if the loss value is greater than a preset loss value, adjust the hyperparameters of the image segmentation model and return to step S4; otherwise, proceed to step S8; S8: extracting the image region of interest of the image to be detected using the trained image segmentation model; The loss value includes a joint scale equalization loss value between the region of interest prediction maps obtained based on image samples of different scales, and a BCE loss value and a DICE loss value between the rotated rectangular mask map and the rotated rectangular mapping estimated feature map; The calculation method of the joint scale balance loss value is specifically as follows: in, represents the joint scale equalization loss value, N represents the total number of zoom scales, and Respectively represent the i Scale and i The predicted region of interest obtained by the image sample of +1 scale is at the same coordinate x The predicted values ​​at i Scale and i +1 scale image sample obtained by the prediction map of the region of interest, represents the first weight parameter, Indicates taking the absolute value, B represents the set of pixels within the minimum circumscribed rotated rectangular frame corresponding to the image sample, Represents a set of statistical pixels B The number of pixels in P represents the predicted region of interest map, and Respectively represent the width and height of the prediction map of the region of interest; The BCE loss value and the DICE loss value between the rotated rectangular mask image and the rotated rectangular mapping estimated feature map are specifically: in, Indicates the BCE loss value and DICE loss value The sum of Represents the estimated feature map of the rotated rectangle map The corresponding rotated rectangular mask image, Indicates that based on i The rotated rectangular mapping feature map obtained by the image sample of the scale is estimated, represents the second weight parameter; The loss value of the image sample is calculated as follows: in, Represents the loss value of the image sample.

2. The method for extracting image regions of interest based on weakly labeled data according to claim 1, characterized in that: The S2 specifically includes: S201: When the image sample has a strong label of the region of interest, the pixel values ​​in the strong label of the region of interest are set to 255, and the remaining pixel values ​​are set to 0, to obtain the rotated rectangular mask image, and obtain the minimum circumscribed rotated rectangular frame and the minimum circumscribed rotated rectangular frame rotation angle by a minimum circumscribed rotated rectangular frame estimation method, otherwise, enter S202; S202: manually label the region of interest in the image sample according to the minimum circumscribed principle to obtain the minimum circumscribed rotated rectangular frame, set the pixel values ​​in the minimum circumscribed rotated rectangular frame to 255, and set the remaining pixel values ​​to 0 to obtain the rotated rectangular mask image, the minimum circumscribed rotated rectangular frame rotation angle, and obtain the minimum circumscribed rotated rectangular frame rotation angle through the minimum circumscribed rotated rectangular frame estimation method.

3. The method for extracting image regions of interest based on weakly labeled data according to claim 2, characterized in that: The minimum circumscribed rotated rectangular frame estimation method is specifically as follows: Based on the region of interest label, an edge map of the region of interest is obtained through the findContours function of the opencv library; The minAreaRect function is called to extract the minimum circumscribed rotation rectangle and the rotation angle of the minimum circumscribed rotation rectangle in the edge map of the region of interest.

4. The method for extracting image regions of interest based on weakly labeled data according to claim 1, characterized in that: The image segmentation model is a deep learning model with a deep segmentation network as a basic unit.

5. According to claim 1, the method for extracting image regions of interest based on weak label data is characterized in that: The S5 specifically includes: S501: Generate a first basic grayscale image with a pixel value of 0, wherein the size of the first basic grayscale image is larger than the size of the region of interest prediction image; S502: copying the region of interest prediction map to the basic grayscale image to obtain a second basic grayscale image; S503: rotating the second basic grayscale image around its own center point by a rotation angle of the minimum circumscribed rotation rectangular frame of the minimum circumscribed rotation rectangular frame corresponding to the prediction image of the region of interest, to obtain a rotated second basic grayscale image; S504: cutting off the redundant part of the second basic grayscale image after rotation that exceeds the second basic grayscale image before rotation to obtain a cropped second grayscale image, wherein the uncopied area in the cropped second grayscale image is filled with the minimum pixel value of the second basic grayscale image; S505: Mapping the second basic grayscale image to a rectangular area in the corresponding image sample where the long and short sides of the rotated rectangular frame are in the same direction: in, Represents the second basic grayscale image after mapping In coordinates The pixel value at H and W Respectively represent the height and width of the second basic grayscale image after filling, max and min represent the maximum value and minimum value respectively, Represents the coordinates to be mapped; S506: reversely rotating the mapped second basic grayscale image around its own center point by a rotation angle of the minimum circumscribed rotation rectangle of the minimum circumscribed rotation rectangle corresponding to the prediction image of the region of interest; S507: Taking a preset size central area of ​​the second basic grayscale image after the reverse rotation to obtain the rotated rectangular mapping estimated feature map.

6. The method for extracting image regions of interest based on weakly labeled data according to claim 5, characterized in that: The size of the first basic grayscale image is the size of the region of interest prediction map. times.

7. A system for extracting image regions of interest based on weakly labeled data, characterized in that: include: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method for extracting an image region of interest based on weakly labeled data according to any one of claims 1 to 6 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for extracting an image region of interest based on weak label data as described in any one of claims 1 to 6 is implemented.