Real-time detection method for parathyroid glands based on anchor-free and spatiotemporal consistency

Abstract

The application relates to a parathyroid real-time detection method based on Anchor-Free and space-time consistency. An Anchor-Free base convolutional neural network and a feature pyramid model are used to perform feature extraction on a detection image to generate a pyramid multi-scale feature map group, then a shared detection head is used to obtain a plurality of scale corresponding classification confidence maps, a center confidence map and a target frame regression map, and then the classification confidence map and the center confidence map are multiplied to obtain a center classification confidence map. A space-time classification memory map is constructed, information is extracted by using the center confidence map, a time sequence classification result map is generated, and the time sequence classification result map and the center confidence map are fused to generate a final classification confidence map. The application enhances the detection effect and the adaptability of the detection network, and alleviates the target drift problem of the detector under complex conditions such as shielding, fog and illumination when the parathyroid is encountered.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and specifically to a parathyroid gland detection method based on Anchor-Free and spatiotemporal consistency. Background Technology

[0002] Object detection has garnered significant attention due to its widespread application in autonomous driving, traffic flow monitoring, surveillance, robotics, human-computer interaction, medical diagnostic systems, and activity recognition. Object detection is a fundamental yet challenging task in computer vision, requiring algorithms to predict a bounding box with a category label for each specific object in an image. In recent years, a series of anchor-box-based detection algorithms, such as the R-CNN series, have been proposed. These algorithms have achieved breakthroughs in detection accuracy by introducing pre-selected anchor boxes, but this also incurs a computational burden. Subsequently, a series of detection algorithms, such as the YOLO series, have been proposed that directly regress the target box without using an anchor-box strategy. These algorithms achieve a better balance between detection accuracy and speed. Because the parathyroid glands in laparoscopic surgery are easily affected by occlusion, lighting, and fog, existing algorithms have relatively low detection and recognition accuracy. Current detection methods typically use a single image as input for object detection, generating bounding boxes by extracting visual features from a single image. This approach ignores the implicit correlation information between sequential images, i.e., it relies on temporal and spatial consistency to assist object detection. Summary of the Invention

[0003] In view of this, the purpose of this invention is to provide a parathyroid gland detection method based on Anchor-Free and spatiotemporal consistency, which effectively improves detection efficiency and quality.

[0004] To achieve the above objectives, the present invention adopts the following technical solution:

[0005] An anchor-free and spatiotemporally consistent method for parathyroid gland detection includes the following steps:

[0006] Step S1: Use the basis convolutional neural network model in the Anchor-Free framework to extract features from the detection image and obtain a basis multi-scale feature map set;

[0007] Step S2: Based on the basis multi-scale feature map set, use the feature pyramid method to obtain the pyramid multi-scale feature map set;

[0008] Step S3: For the members in the pyramid multi-scale feature map group, use the parameter-sharing detection head to process them and obtain the classification confidence map and the target box regression map;

[0009] Step S4: Add a parallel convolutional layer to calculate and obtain the center confidence map;

[0010] Step S5: Multiply the center confidence score with the corresponding classification score to obtain the center classification confidence map. ;

[0011] Step S6: At scale Upper construction of spatiotemporal classification memory map , From the beginning Frame center classification confidence map Composition, if the current detection image is ,but Depend on Composition; when When, set First, calculate the spatiotemporal classification memory map. confidence plot of the current central classification Similarity matrix Similarity matrix ;

[0012] Step S7: As a soft selection weight map, and Spatiotemporal information maps are obtained by performing matrix multiplication. Convolution kernel is provided. ,use As training data for convolution kernels Perform parameter updates; use convolution kernels right Temporal classification results obtained by performing convolution operations ; Obtain the final classification confidence map ;

[0013] Step S8: Perform spatiotemporal classification memory map update operation: save the current final classification confidence map. to The specific method is to use the current final classification confidence map. replace The earliest classification confidence plot in the middle;

[0014] Step S9: Select the final classification confidence map Positions with a classification confidence score greater than a preset value are considered positive samples, and simultaneously, [the following is obtained]... The corresponding regression target box is the candidate box for detection results, and the final detection result box is obtained by using the non-maximum suppression algorithm.

[0015] Furthermore, step S2 specifically involves: based on the basis multi-scale feature map set obtained in step S1... ,in Representing the Scale on the map The feature map above is used to obtain a pyramid multi-scale feature map set using the feature pyramid method. ,in represent Upper scale is The corresponding feature map is generated.

[0016] Furthermore, the method of obtaining a pyramid multi-scale feature map group using the feature pyramid method specifically involves: using a size of... The convolution kernel performs convolution operations on the basis multi-scale feature maps and connects the upper and lower features to generate... , and Through and It is generated by applying a convolutional layer with a stride of 2.

[0017] Furthermore, step S3 specifically involves: for the pyramid multi-scale feature map group Members The parameter-sharing detection head is used for processing. The detection head consists of two parts: a classification branch and a regression branch. Each branch consists of 5 convolutional layers.

[0018] Using parameter sharing detection head pairs After processing, a classification confidence map is obtained. Regression plot of target boxes ,in and These represent the length and width of the graph, respectively, and are scaled. The number of multi-scale feature maps corresponding to the pyramid;

[0019] in, Representing the Feature maps at different scales on the image The generated classification probability values, is the predicted target bounding box position value for any point on the feature map, and its 4-dimensional values ​​represent the straight-line distance from the predicted position to the predicted bounding box.

[0020] Furthermore, step S4 specifically involves adding a parallel convolutional layer after the fourth convolutional layer in the classification branch to calculate and obtain the center confidence map. , which represents the confidence level that a certain position on the feature map is the center position of the target box.

[0021] Furthermore, step S6 specifically includes:

[0022] Step S61: For the spatiotemporal classification memory map And center classification confidence plot ,Will scale transformation ,Will scale transformation ,in , and The number of channels, length, and width of the feature map. Represents the number of memories;

[0023] Step S62: Spatiotemporal classification memory map confidence plot of the center classification Obtaining similar matrices by performing matrix multiplication , ;

[0024] Step S63: Use Functions on similar matrices Standardize it.

[0025] Furthermore, step S7 specifically includes:

[0026] Step S71: Conversion The scale is ,Will As a soft-selection weighting graph, for and Spatiotemporal information classification diagrams are obtained by performing matrix multiplication. ;

[0027] Step S72: Set the convolution kernel ,Will As input data, As training labels, convolutional kernels The update process is represented as follows: ,in Represents the number of iterations. The representative conducted the first Backpropagation parameters, Represents the learning rate. This represents the calculation of the cross-entropy loss function. For the gradient operator; for the current convolution kernel The loss, based on the Taylor expansion method, has the following approximate loss function:

[0028]

[0029] in It is a positive definite matrix;

[0030] For approximate loss function ,pass For step size have:

[0031]

[0032] During the steepest descent, the above formula is used to calculate the scalar stride in each iteration of the convolution kernel update. ; It is a positive definite matrix;

[0033] Step S73: For Using convolution kernels Perform convolution operations and output the temporal classification result image. , Size and Similarly, further obtain foreground and background classification images of the target. .

[0034] Compared with the prior art, the present invention has the following advantages:

[0035] 1. This invention introduces rich spatiotemporal information, and by using spatiotemporal information to predict the classification status of the target, it effectively mitigates the target drift problem when encountering target occlusion and the appearance of multiple similar targets;

[0036] 2. This invention uses the steepest gradient descent method to calculate the optimal step size for each iteration. This method significantly accelerates the online model update speed, making our model more effective in practical applications. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the parathyroid gland target in an embodiment of the present invention.

[0038] Figure 2 This is a diagram showing the detection effect of parathyroid glands in an embodiment of the present invention.

[0039] Figure 3 This is a flowchart of the parathyroid gland detection method based on spatiotemporal consistency in this invention. Detailed Implementation

[0040] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0041] Please refer to Figure 1 This invention provides a method for parathyroid gland detection based on spatiotemporal consistency, implemented according to the following steps:

[0042] Step S1: For video image sequences with temporal relationships... ,in Representing the image sequence number, the detection image is processed using a basis convolutional neural network model. Feature extraction is performed to obtain a set of basis multi-scale feature maps. ,in Representing the Scale on the map Feature map on;

[0043] Step S2: Utilize the feature pyramid method based on Obtain multi-scale feature maps of the pyramid ,in represent Upper scale is The corresponding feature map is generated. Specifically, this method utilizes a feature map of size [size missing]. The convolution kernel performs convolution operations on the basis multi-scale feature maps and connects the upper and lower features to generate... . and Through in and This is produced by applying a convolutional layer with a stride of 2.

[0044] Step S3: For the pyramid multi-scale feature map group Members It uses a shared head for processing;

[0045] Preferably, the detection head consists of two parts: a classification branch and a regression branch, with each branch consisting of 5 convolutional layers. The parameter-sharing detection head is used to... After processing, a classification confidence map is obtained. Regression plot of target boxes ,in and These represent the length and width of the graph, respectively, and are scaled. This corresponds to the number of multi-scale feature maps in the pyramid. Representing the Feature maps at different scales on the image The generated classification probability values, is the predicted target bounding box position value for any point on the feature map, and its 4-dimensional values ​​represent the straight-line distance from the predicted position to the predicted bounding box.

[0046] Step S4: Add a parallel convolutional layer after the 4th convolutional layer in the classification branch to calculate the center confidence map. , which represents the confidence level that a certain position on the feature map is the center position of the target box;

[0047] Step S5: Multiply the predicted center confidence score with the corresponding classification score to obtain the center classification confidence map. ;

[0048] Step S6: At scale Upper construction of spatiotemporal classification memory map , From the beginning Frame center classification confidence map Composition, if the current detection image is ,but Depend on Composition. When When, set First, calculate the spatiotemporal classification memory map. confidence plot of the current central classification Similarity matrix Similarity matrix ;

[0049] Step S7: ... As a soft selection weight map, and Spatiotemporal information maps are obtained by performing matrix multiplication. There is a convolution kernel. ,use As training data for convolution kernels Perform parameter updates. Use convolution kernels. right Temporal classification results obtained by performing convolution operations The final classification confidence plot. ;

[0050] Step S8: Perform spatiotemporal classification memory map update operation: save the current final classification confidence map. to The specific method is to use the current final classification confidence map. replace The earliest classification confidence plot in the middle;

[0051] Step S9: Select the final classification confidence map Positions with a classification confidence score greater than 0.05 are considered positive samples, and at the same time, [the following information is obtained]. The corresponding regression target box is the candidate box for detection results, and the final detection result box is obtained by using the non-maximum suppression algorithm.

[0052] Preferably, in this embodiment, step S6 further includes the following step:

[0053] Step S61: For the spatiotemporal classification memory map And center classification confidence plot ,Will scale transformation ,Will scale transformation ,in , and The number of channels, length, and width of the feature map. Represents the number of memories;

[0054] Step S62: Spatiotemporal classification memory map confidence plot of the center classification Obtaining similar matrices by performing matrix multiplication ,Right now .

[0055] Step S63: Use Functions on similar matrices Standardize;

[0056] Preferably, in this embodiment, step S7 further includes the following step:

[0057] Step S71: Conversion The scale is ,Will As a soft-selection weighting graph, for and Spatiotemporal information classification diagrams are obtained by performing matrix multiplication. .

[0058] Step S72: Set the convolution kernel ,Will As input data, As training labels, convolutional kernels The update process can be represented as: ,in Represents the number of iterations. The representative conducted the first Backpropagation parameters, Represents the learning rate. This represents the calculation of the cross-entropy loss function. For the current convolution kernel, the gradient operator is used. The loss, based on the Taylor expansion method, has the following approximate loss function:

[0059]

[0060] in It is a positive definite matrix. For the approximate loss function... ,pass For step size have:

[0061]

[0062] During the steepest descent, the above formula is used to calculate the scalar stride in each iteration of the convolution kernel update. For positive definite matrices There are multiple options available in this method. ,in It is a Jacobian determinant. In fact, a matrix or Jacobi matrix They do not need to be explicitly constructed, but are implemented as a series of neural network operations.

[0063] Step S73: For Using convolution kernels Perform convolution operations and output the temporal classification result image. , Size and Same. Further obtain foreground and background classification images of the target. ;

[0064] Example 1:

[0065] The specific steps of applying the method proposed in this invention to the detection of parathyroid gland targets are as follows:

[0066] 1. Establish a priori parathyroid gland recognition dataset. ;

[0067] 2. Data in batches of images is transmitted into the network model for forward propagation, and the center classification confidence map and target bounding box regression map are output.

[0068] 3. Calculate the loss function, where the regression branch loss function is: The classification branch loss function is , represent The loss function, the center confidence loss function is: , Represents the binary cross-entropy loss function;

[0069] 4. Use the SGD method for backpropagation to update the network model parameters;

[0070] 5. Repeat steps 2)-4) several times to train the network model and obtain the network parameters after training.

[0071] 6. Input the image sequence into the detector in chronological order;

[0072] 7. Detector output center classification confidence map and target bounding box regression map;

[0073] 8. Construct a spatiotemporal memory map, which is built using a central classification confidence map at the beginning of the detection phase;

[0074] 9. Generate a spatiotemporal information map using the spatiotemporal memory map and the center classification confidence map, perform online model updates, and generate a time-series classification confidence map at the same time;

[0075] 10. Merge the time-series classification confidence map with the center classification confidence map to generate the final classification confidence map;

[0076] 11. Based on the final classification confidence map, select the corresponding target bounding box regression map data to generate candidate boxes, use the non-maximum suppression algorithm to filter the candidate boxes, and output the final candidate box results.

[0077] 12. Perform a spatiotemporal memory map update operation and save the final classification confidence map as a spatiotemporal memory map.

[0078] Figure 1 This is a schematic diagram of the parathyroid gland. Figure 2 This is a screenshot of the parathyroid gland detection method example described above. This method effectively mitigates the drift in detection results when encountering target occlusion or multiple similar targets by introducing rich spatiotemporal information and predicting the target's classification state using this information. For online model updates, we use a steepest gradient descent-based method to calculate the optimal step size for each iteration. This method significantly accelerates the online model update speed, making our model more efficient in practical applications.

[0079] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made within the scope of the claims of the present invention should be included in the scope of the present invention.

Claims

1. A parathyroid gland detection method based on Anchor-Free and Spatiotemporal Consistency, characterized in that, Includes the following steps: Step S1: Use the basis convolutional neural network model in the Anchor-Free framework to extract features from the detection image and obtain a basis multi-scale feature map set; Step S2: Based on the basis multi-scale feature map set, use the feature pyramid method to obtain the pyramid multi-scale feature map set; Step S3: For the members in the pyramid multi-scale feature map group, use the parameter-sharing detection head to process them and obtain the classification confidence map and the target box regression map; Step S4: Add a parallel convolutional layer to calculate and obtain the center confidence map; Step S5: Multiply the center confidence score with the corresponding classification score to obtain the center classification confidence map. , Representing the Feature maps at different scales on the image The generated classification probability values, Represents the center confidence plot; Step S6: At scale Upper construction of spatiotemporal classification memory map , From the beginning Frame center classification confidence map Composition, if the current detection image is ,but Depend on Composition; when When, set First, calculate the spatiotemporal classification memory map. confidence plot of the current central classification Similarity matrix Similarity matrix ; Step S7: As a soft selection weight map, and Spatiotemporal information classification diagrams are obtained by performing matrix multiplication. ; Convolution kernel is provided ,use As training data for convolution kernels Perform parameter updates; use convolution kernels right Temporal classification results obtained by performing convolution operations ; Obtain the final classification confidence map ; Step S8: Perform spatiotemporal classification memory map update operation: save the current final classification confidence map. to The specific method is to use the current final classification confidence map. replace The earliest time-center classification confidence plot; Step S9: Select the final classification confidence map Positions with a classification confidence score greater than a preset value are considered positive samples, and a target bounding box regression map is obtained simultaneously. The corresponding regression target box is the candidate box for detection results, and the final detection result box is obtained by using the non-maximum suppression algorithm.

2. The parathyroid gland detection method based on Anchor-Free and Spatiotemporal Consistency according to claim 1, characterized in that, Step S2 specifically involves: using the basis multi-scale feature map set obtained in step S1... ,in Representing the Scale on the map The feature map above is used to obtain a pyramid multi-scale feature map set using the feature pyramid method. ,in represent Upper scale is The corresponding feature map is generated.

3. The parathyroid gland detection method based on Anchor-Free and Spatiotemporal Consistency according to claim 2, characterized in that, The method of obtaining a pyramid multi-scale feature map set using the feature pyramid method specifically involves: using a size of The convolution kernel performs convolution operations on the basis multi-scale feature maps and connects the upper and lower features to generate... , and Through and It is generated by applying a convolutional layer with a stride of 2.

4. The parathyroid gland detection method based on Anchor-Free and Spatiotemporal Consistency according to claim 1, characterized in that, Step S3 specifically involves: for the pyramid multi-scale feature map group Members The parameter-sharing detection head is used for processing. The detection head consists of two parts: a classification branch and a regression branch. Each branch consists of 5 convolutional layers. Using parameter sharing detection head pairs After processing, a classification confidence map is obtained. Regression plot of target boxes ,in and These represent the length and width of the graph, respectively, and are scaled. The number of multi-scale feature maps corresponding to the pyramid; in, is the predicted target bounding box position value for any point on the feature map, and its 4-dimensional values ​​represent the straight-line distance from the predicted position to the predicted bounding box.

5. The parathyroid gland detection method based on Anchor-Free and Spatiotemporal Consistency according to claim 1, characterized in that, Step S4 specifically involves: adding a parallel convolutional layer after the fourth convolutional layer in the classification branch to calculate and obtain the center confidence map. , which represents the confidence level that a certain position on the feature map is the center position of the target box.

6. The parathyroid gland detection method based on Anchor-Free and Spatiotemporal Consistency according to claim 1, characterized in that, Step S6 specifically involves: Step S61: For the spatiotemporal classification memory map And center classification confidence plot ,Will The scale is transformed ,Will scale transformation ,in , and The number of channels, length, and width of the feature map. Represents the number of memories; Step S62: Spatiotemporal classification memory map confidence plot of the center classification Obtaining similar matrices by performing matrix multiplication , ; Step S63: Use Functions on similar matrices Standardize it.

7. The parathyroid gland detection method based on Anchor-Free and Spatiotemporal Consistency according to claim 1, characterized in that, Step S7 specifically involves: Step S71: Conversion The scale is ,Will As a soft-selection weighting graph, for and Spatiotemporal information classification diagrams are obtained by performing matrix multiplication. ; Step S72: Set the convolution kernel ,Will As input data, As training labels, convolutional kernels The update process is represented as follows: ,in Represents the number of iterations. The representative conducted the first Backpropagation parameters, Represents the learning rate. This represents the calculation of the cross-entropy loss function. For the gradient operator; for the current convolution kernel The loss, based on the Taylor expansion method, has the following approximate loss function: in It is a positive definite matrix; For approximate loss function ,pass For step size have: During the steepest descent, the above formula is used to calculate the scalar stride in each iteration of the convolution kernel update. ; It is a positive definite matrix; Step S73: For Using convolution kernels Perform convolution operations and output the temporal classification result image. , Size and Similarly, the foreground and background classification images of the target are obtained as the final classification confidence map. .

Images