A method and system for detecting elements of a thangka based on improved RT-DETR

Abstract

The application discloses a kind of Tangka element detection method and system based on improved RT-DETR, method includes: obtaining Tangka image data;Generate multiple Tangka image samples;The features of each element of Tangka image sample are extracted by multi-scale convolution kernel, and form feature coding;Fusion features are sequentially input into the query selector and decoder of RT-DETR;According to the output result of decoder, scale-based dynamic loss calculation is carried out, and the parameters are iteratively updated according to the calculation result until the dynamic loss meets the preset value to form a detection model;Tangka element detection is carried out by detection model.The application does not need complex non-maximum suppression redundant frame process, under the condition of reducing calculation complexity, realizes the fast and accurate detection of Tangka image cultural elements, solves the problem of high calculation complexity and low detection accuracy of current Tangka element detection model, meets the application demand of mobile terminal and edge computing lightweight deployment.

Description

Technical Field

[0001] This invention relates to image processing technology, specifically to a method and system for detecting Thangka elements based on an improved RT-DETR. Background Technology

[0002] Due to the complex color textures, varying scales and shapes of cultural elements, and partial occlusion of cultural elements in Thangka images, traditional YOLO-based object detection algorithms suffer from poor real-time performance and rely on complex post-processing. They are ineffective at detecting complex color backgrounds and multi-scale cultural elements in Thangka images. Therefore, DETR object detection, based on a Transformer structure, can effectively achieve real-time and accurate detection of multi-class objects of different scales and shapes. Especially under varying lighting intensities, viewing angles, and complex color textures, it can adaptively adjust to detect objects at different scales, achieving high-precision real-time detection. However, existing DETR networks still have limitations in capturing elements in Thangka images, primarily because the composition of elements in Thangka images is complex, and different types of elements may exhibit high similarity. Summary of the Invention

[0003] In order to at least overcome the above-mentioned shortcomings in the prior art, the purpose of this application is to provide a method and system for detecting Thangka elements based on an improved RT-DETR.

[0004] In a first aspect, embodiments of this application provide a method for detecting Thangka elements based on an improved RT-DETR, including:

[0005] Multiple Thangka images are acquired, and the Thangka images are preprocessed to form multiple Thangka image data;

[0006] Multiple Thangka image samples are generated by analyzing and labeling various elements of multiple Thangka image data; the various elements include headdress elements, base elements, gesture elements, hand-held elements, and other elements;

[0007] Features of each element of the Thangka image sample are extracted by multi-scale convolution kernels, and the features are then subjected to multi-scale interactive encoding to form feature codes.

[0008] The feature encoding is compared and driven to form a fused feature, and the fused feature is then sequentially input into the query selector and decoder of RT-DETR;

[0009] Based on the output of the decoder, scale-based dynamic loss is calculated, and the parameters are iteratively updated according to the calculation results until the dynamic loss meets the preset value to form a detection model.

[0010] The detection model is used to detect Thangka elements.

[0011] In one possible implementation, analyzing multiple elements of the various Thangka image data includes:

[0012] The Thangka image data is subjected to limb key point recognition to obtain limb key points; the limb key points include hand key points, leg key points and head key points;

[0013] Calculate the distance from the marked element to the nearest limb key point, and calculate the ratio of the distance to the diagonal distance of the Thangka image data as the feature distance;

[0014] A verification model is constructed based on the feature distance.

[0015] In one possible implementation, constructing the verification model based on the feature distance includes:

[0016] Based on the feature distance, a detection interval is constructed for each element as the verification model;

[0017] When the feature distance corresponding to the element detected by the detection model falls into the corresponding detection interval, the detection result is determined to be true;

[0018] When the feature distance corresponding to the element detected by the detection model does not fall into the corresponding detection interval, the detection result is determined to be false.

[0019] In one possible implementation, the detection of Thangka elements using the detection model includes:

[0020] The image to be detected is subjected to limb key point recognition to obtain the key points of the limb to be detected;

[0021] The image to be detected is input into the detection model, and the element information detected by the detection model is obtained;

[0022] The distance to be verified is calculated using the key points of the limb to be detected and the element information, and the distance to be verified and the type of the corresponding element information are input into the verification model for verification.

[0023] In one possible implementation, the extraction of features from each element includes:

[0024] The Thangka image samples are processed sequentially through multiple consecutive levels to generate multiple feature images corresponding to different resolutions; each level corresponds to a feature image of one resolution.

[0025] The feature image is used as the feature of the corresponding element.

[0026] In one possible implementation, the process of calculating and generating feature images at each level includes:

[0027] The image received at each level is reduced in dimensionality through 1×1 convolution, and then multi-path convolutional data is generated by multi-path parallel processing of multiple lightweight convolutions of different sizes on the reduced image.

[0028] The multi-channel convolutional data is upscaled to the corresponding resolution using 1×1 convolution, and residuals are added to form a feature image.

[0029] In one possible implementation, performing multi-scale interactive encoding on the features to form feature codes includes:

[0030] All feature images of the same Thangka image sample are mapped to the same dimension, and all mapped feature images are expanded to form a feature sequence; the position corresponding to each pixel in the feature sequence is configured with pixel coordinate encoding;

[0031] After cross-scale information interaction based on the pixel coordinate encoding, the feature sequence is re-encoded according to the resolution of all the feature images to form multiple feature codes corresponding to different resolutions of the feature images.

[0032] In one possible implementation, performing contrast-driven feature aggregation on the feature encoding to form fused features includes:

[0033] The feature encoding is input into the foreground feature convolution branch to obtain foreground features, and the feature encoding is input into the background feature convolution branch to obtain background features;

[0034] The system simultaneously slides within the foreground and background features using a preset window of a preset size and a preset step size, capturing the differences between the foreground and background features at different locations, and then normalizes these differences to serve as the attention weight at that location.

[0035] After multiplying the attention weight by the feature encoding at the corresponding position, the feature encodings at all resolutions after the attention weight is weighted are fused to form the fused feature.

[0036] In one possible implementation, performing scale-based dynamic loss calculation based on the decoder's output includes:

[0037] Calculate the size of the image corresponding to the output result based on the output result of the decoder;

[0038] The weights corresponding to the image are calculated based on the size, and the dynamic loss is obtained by weighted regression using these weights.

[0039] Secondly, embodiments of this application provide a Thangka element detection system based on an improved RT-DETR, comprising:

[0040] The acquisition unit is configured to acquire multiple Thangka images and preprocess the Thangka images to form multiple Thangka image data.

[0041] The sample unit is configured to generate multiple Thangka image samples by analyzing and labeling multiple elements of the multiple Thangka image data; the multiple elements include headdress elements, base elements, gesture elements, hand-held elements, and other elements;

[0042] The encoding unit is configured to extract features of each element of the Thangka image sample through multi-scale convolutional kernels, and to perform multi-scale interactive encoding of the features to form feature codes;

[0043] The fusion unit is configured to perform contrast-driven feature aggregation on the feature encoding to form a fused feature, and to sequentially input the fused feature into the query selector and decoder of RT-DETR;

[0044] The training unit is configured to perform scale-based dynamic loss calculation based on the output of the decoder, and to iteratively update the parameters according to the calculation results until the dynamic loss meets the preset value to form a detection model.

[0045] The detection unit is configured to detect Thangka elements using the detection model.

[0046] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0047] This invention presents a Thangka element detection method and system based on an improved RT-DETR, which eliminates the need for a complex non-maximum suppression redundant bounding box process. It achieves rapid and accurate detection of cultural elements in Thangka images while reducing computational complexity, solving the problems of high computational complexity and low detection accuracy in current Thangka element detection models. It also meets the lightweight deployment application requirements of mobile terminals and edge computing. Attached Figure Description

[0048] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and form part of this application, do not constitute a limitation thereof. In the drawings:

[0049] Figure 1 This is a schematic diagram of the method steps in an embodiment of this application;

[0050] Figure 2 This is a schematic diagram of the model architecture of an embodiment of this application. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0052] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0053] Please refer to the following: Figure 1 The above is a flowchart illustrating a method for detecting Thangka elements based on an improved RT-DETR, provided by an embodiment of the present invention. Further, the method for detecting Thangka elements based on an improved RT-DETR may specifically include the contents described in steps S1-S6.

[0054] S1: Acquire multiple Thangka images and preprocess the Thangka images to form multiple Thangka image data;

[0055] S2: After analyzing and labeling various elements of the multiple Thangka image data, multiple Thangka image samples are generated; the various elements include headdress elements, base elements, gesture elements, hand-held elements, and other elements;

[0056] S3: Extract the features of each element of the Thangka image sample through multi-scale convolution kernels, and perform multi-scale interactive encoding on the features to form feature codes;

[0057] S4: Perform comparison-driven feature aggregation on the feature encoding to form a fused feature, and input the fused feature into the query selector and decoder of RT-DETR in sequence;

[0058] S5: Calculate the scale-based dynamic loss based on the output of the decoder, and iteratively update the parameters according to the calculation results until the dynamic loss meets the preset value to form a detection model;

[0059] S6: Detect Thangka elements using the aforementioned detection model.

[0060] When implementing the embodiments of this application, it is necessary to first obtain Thangka images, which can be obtained through websites or on-site photography; the preprocessing process generally involves completing the incomplete Thangka images, screening them, and performing corresponding image enhancement to expand the samples to form Thangka image data; through the analysis and annotation of various elements of the Thangka image data, Thangka image samples can be further formed for model training.

[0061] In this embodiment, an improved RT-DETR model for Thangka element detection was used for model training. Due to the scarcity of Thangka image data sources, the number of Thangka image samples is relatively small, and each enhanced Thangka image needs to be utilized as much as possible for training. Furthermore, the size of Thangka images varies greatly, ranging from 20cm×15cm to 3.5m×1m, and even includes extremely large images of 55.8m×46.8m. Therefore, this embodiment made some improvements to RT-DETR to accommodate images with such a wide size range.

[0062] In this embodiment, the backbone network, encoder stage, and decoder stage of the RT-DETR model have been optimized and improved. Specifically, in the backbone network, this embodiment employs multi-scale convolutional kernels for feature extraction, which differs from the Basic Block module of the RT-DETR model. Under the feature extraction of multi-scale convolutional kernels, different color textures in Thangka images at different scales can be extracted better. In the encoding stage, multi-scale interactive coding replaces AIFI in the original encoder, enabling the acquisition of visual features at different scales. Furthermore, contrast-driven feature aggregation effectively captures the differences between foreground and background images of Thangka images at different sizes, thereby improving the discriminability of elements at different scales in Thangka images and effectively enhancing the performance of Thangka element detection. During loss calculation, different loss values ​​can be calculated for images of different sizes, and parameters are iterated based on these values. It should be understood that the parameter iteration mentioned here generally refers to the parameter iteration in the backbone network, query selector, and decoder, which can be performed using methods such as gradient descent; this embodiment does not impose further limitations. The trained detection model can effectively achieve the detection of cultural elements in Thangka images without the need for a complex non-maximum suppression redundant bounding box process. While reducing computational complexity, it improves the accuracy and inference speed of Thangka element detection, thus meeting the application deployment requirements of mobile edge devices and enabling real-time and accurate detection of Thangka elements.

[0063] In one possible implementation, analyzing multiple elements of the various Thangka image data includes:

[0064] The Thangka image data is subjected to limb key point recognition to obtain limb key points; the limb key points include hand key points, leg key points and head key points;

[0065] Calculate the distance from the marked element to the nearest limb key point, and calculate the ratio of the distance to the diagonal distance of the Thangka image data as the feature distance;

[0066] A verification model is constructed based on the feature distance.

[0067] In the implementation of this application embodiment, the recognition of limb key points can be achieved using the prior art of Chinese Patent 202410708784.7, filed by the inventor in 2024. For example, YOLO is used to train the limb recognition model. After obtaining samples, sample augmentation is performed using the method described in the aforementioned patent, and the hand, leg, and head key points of the samples are marked. The hand key point uses the center point of the wrist, the leg key point uses the center points of both feet, and the head key point uses the center points of both eyebrows. The limb key point recognition model can be obtained by training the marked samples in YOLO. Its construction method is consistent with that in the aforementioned patent. It should be understood that those skilled in the art can also construct limb key point recognition models in other ways, and this application embodiment does not impose any limitations.

[0068] In this embodiment of the application, it is necessary to calculate the distance from the element to the nearest limb key point. Generally, the shortest distance from the nearest limb key point to the element to the boundary of the element is selected. In order to reduce the data difference caused by size, the ratio of the distance to the diagonal distance of the Thangka image data is used as the feature distance.

[0069] In one possible implementation, constructing the verification model based on the feature distance includes:

[0070] Based on the feature distance, a detection interval is constructed for each element as the verification model;

[0071] When the feature distance corresponding to the element detected by the detection model falls into the corresponding detection interval, the detection result is determined to be true;

[0072] When the feature distance corresponding to an element detected by the detection model does not fall within the corresponding detection interval, the detection result is determined to be false.

[0073] In this embodiment, a model for verifying the accuracy of element detection can be constructed based on feature distance. This is because different elements have varying distances to limb keypoints. For example, the closest limb keypoint to a headdress element is the head keypoint, with a detection range of [0, 0.05]; the closest limb keypoint to a base element is the leg keypoint, with a detection range of [0, 0.13]; the closest limb keypoint to a gesture element is the hand keypoint, with a detection range of [0, 0.01]; and the closest limb keypoint to a handheld element is the hand keypoint, with a detection range of [0, 0.03]. Other elements are generally distributed in different external regions, so their detection range is [0.14, ∞]. In actual research, the inventors found that for the trained detection model, since other elements and handheld elements are often similar in image and color, the detection error of the model often occurs between these two elements. However, after constructing interval detection, the false detection rate can be reduced to almost zero.

[0074] In one possible implementation, the detection of Thangka elements using the detection model includes:

[0075] The image to be detected is subjected to limb key point recognition to obtain the key points of the limb to be detected;

[0076] The image to be detected is input into the detection model, and the element information detected by the detection model is obtained;

[0077] The distance to be verified is calculated using the key points of the limb to be detected and the element information, and the distance to be verified and the type of the corresponding element information are input into the verification model for verification.

[0078] In one possible implementation, the extraction of features from each element includes:

[0079] The Thangka image samples are processed sequentially through multiple consecutive levels to generate multiple feature images corresponding to different resolutions; each level corresponds to a feature image of one resolution.

[0080] The feature image is used as the feature of the corresponding element.

[0081] In one possible implementation, the process of calculating and generating feature images at each level includes:

[0082] The image received at each level is reduced in dimensionality through 1×1 convolution, and then multi-path convolutional data is generated by multi-path parallel processing of multiple lightweight convolutions of different sizes on the reduced image.

[0083] The multi-channel convolutional data is upscaled to the corresponding resolution using 1×1 convolution, and residuals are added to form a feature image.

[0084] In one possible implementation, performing multi-scale interactive encoding on the features to form feature codes includes:

[0085] All feature images of the same Thangka image sample are mapped to the same dimension, and all mapped feature images are expanded to form a feature sequence; the position corresponding to each pixel in the feature sequence is configured with pixel coordinate encoding;

[0086] After cross-scale information interaction based on the pixel coordinate encoding, the feature sequence is re-encoded according to the resolution of all the feature images to form multiple feature codes corresponding to different resolutions of the feature images.

[0087] In one possible implementation, performing contrast-driven feature aggregation on the feature encoding to form fused features includes:

[0088] The feature encoding is input into the foreground feature convolution branch to obtain foreground features, and the feature encoding is input into the background feature convolution branch to obtain background features;

[0089] The system simultaneously slides within the foreground and background features using a preset window of a preset size and a preset step size, capturing the differences between the foreground and background features at different locations, and then normalizes these differences to serve as the attention weight at that location.

[0090] After multiplying the attention weight by the feature encoding at the corresponding position, the feature encodings at all resolutions after the attention weight is weighted are fused to form the fused feature.

[0091] In one possible implementation, performing scale-based dynamic loss calculation based on the decoder's output includes:

[0092] Calculate the size of the image corresponding to the output result based on the output result of the decoder;

[0093] The weights corresponding to the image are calculated based on the size, and the dynamic loss is obtained by weighted regression using these weights.

[0094] In the implementation of this application, a specific detection model structure is provided. Please refer to [link / reference]. Figure 2In the implementation of this application embodiment, the Thangka image sample is first downsampled to 1 / 4 resolution, and the number of channels is increased to 64. Then it is input into a four-layer network. The layers are connected sequentially, and the structure of each layer is the same. When each layer receives the input data, it first passes through a 1×1 convolution to reduce the dimensionality, and then through three parallel lightweight convolutions: 3×3, 5×5 and 7×7. The output data of the three convolutions is then increased to the required resolution by a 1×1 convolution and residuals are added to form a complete image. When the downsampled image is input to the first layer, the final output of the first layer is also 64 channels, corresponding to 1 / 4 resolution. The data output from the first layer is then provided to the second layer, whose final output is 128 channels, corresponding to 1 / 8 resolution. The data output from the second layer is then provided to the third layer, whose final output is 256 channels, corresponding to 1 / 16 resolution. The data output from the third layer is then provided to the fourth layer, whose final output is 512 channels, corresponding to 1 / 32 resolution. This process demonstrates that data from different dimensions can be effectively extracted, and the extraction process is insensitive to the size of the original image, allowing for better display of relevant features. Finally, the data output from the second, third, and fourth layers at 1 / 8, 1 / 16, and 1 / 32 resolutions, with 128, 256, and 512 channels respectively, are encoded.

[0095] During the encoding process, a 1×1 convolutional kernel is used to simultaneously map data from these three scales into 256 channels, followed by scale-shifting information interaction. The data at 1 / 8 resolution is called the high-resolution low-semantic layer, the data at 1 / 16 resolution is called the medium-resolution medium-semantic layer, and the data at 1 / 32 resolution is called the low-resolution high-semantic layer. These three images differ in spatial size and number of channels, but together they carry complete information of local details and global semantics, which can effectively express the characteristics of Thangka images at different sizes.

[0096] During the mapping process, a token is generated at each mapped pixel location. This token records its initial data set and image position. The three mapped data sets are then arranged spatially into a long sequence: high-resolution low-semantic layer + medium-resolution medium-semantic layer + low-resolution high-semantic layer. Cross-scale information interaction then occurs: each query token interacts with tokens at the same layer, and also reads information from tokens at a small number of keypoints at other scales. After the interaction is complete, the data needs to be re-encoded into images at three different scales.

[0097] For data at each scale, a contrast-driven feature aggregation method can be used to form fused features. This method obtains foreground and background features separately through foreground feature convolution and background feature convolution branches. Then, at each spatial location, a fixed-size window is taken, and the foreground features and background features within the window are compared to calculate a difference score. The greater the difference, the more likely the location is to be a target edge or key region. This score is normalized to an attention weight in the [0,1] interval. This attention weight is multiplied by the feature encoding at the corresponding location to achieve weighting, effectively highlighting the parts with large differences between the foreground and background, thereby improving the accuracy of recognition. Features at locations with high weights are amplified, while features at locations with low weights are suppressed.

[0098] Then, the attention-weighted data is fed into the query selector, and the result calculated by the query selector is fed into the decoder. The query selector can be constructed in the same way as the RT-DETR model, and this embodiment does not impose any limitations. When calculating the loss of the decoder's output, a dynamic loss evaluation is performed using a weighted method: the center point (x, y), width w, and height h of the ground truth bounding boxes are taken, and the area of ​​the ground truth bounding boxes, Area = w × h, is calculated. At this point, the maximum value of the area of ​​all ground truth bounding boxes, Area, is obtained. max The scaling factor of the current ground truth bounding box is calculated as Scale = Area / Area. max Then, weight splitting is performed. In this embodiment, two evaluation metrics, GioU and L1, are selected. The evaluation weight of GioU is α=Scale, which represents the emphasis on bounding box matching for large targets; the evaluation weight of L1 is β=1-α, which represents the emphasis on localization accuracy for small targets. Then, the GioU value between the predicted bounding box and the ground truth bounding box is calculated, and the L1 error value Loss is calculated. L1 The dynamic loss is obtained by weighting the loss using the following formula: Loss = α(1-GioU) + β×Loss L1 At this point, the decoder, query selector, encoder, and backbone network are updated sequentially through a backpropagation.

[0099] Secondly, embodiments of this application provide a Thangka element detection system based on an improved RT-DETR, comprising:

[0100] The acquisition unit is configured to acquire multiple Thangka images and preprocess the Thangka images to form multiple Thangka image data.

[0101] The sample unit is configured to generate multiple Thangka image samples by analyzing and labeling multiple elements of the multiple Thangka image data; the multiple elements include headdress elements, base elements, gesture elements, hand-held elements, and other elements;

[0102] The encoding unit is configured to extract features of each element of the Thangka image sample through multi-scale convolutional kernels, and to perform multi-scale interactive encoding of the features to form feature codes;

[0103] The fusion unit is configured to perform contrast-driven feature aggregation on the feature encoding to form a fused feature, and to sequentially input the fused feature into the query selector and decoder of RT-DETR;

[0104] The training unit is configured to perform scale-based dynamic loss calculation based on the output of the decoder, and to iteratively update the parameters according to the calculation results until the dynamic loss meets the preset value to form a detection model.

[0105] The detection unit is configured to detect Thangka elements using the detection model.

[0106] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0107] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices or units, or may be electrical, mechanical or other forms of connection.

[0108] The units described as separate components may or may not be physically separate. As will be apparent to those skilled in the art, the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0109] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0110] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or grid device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0111] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for detecting Thangka elements based on an improved RT-DETR, characterized in that, include: Multiple Thangka images are acquired, and the Thangka images are preprocessed to form multiple Thangka image data; Multiple Thangka image samples are generated by analyzing and labeling various elements of multiple Thangka image data; the various elements include headdress elements, base elements, gesture elements, hand-held elements, and other elements; Features of each element of the Thangka image sample are extracted by multi-scale convolution kernels, and the features are then subjected to multi-scale interactive encoding to form feature codes. The feature encoding is compared and driven to form a fused feature, and the fused feature is then sequentially input into the query selector and decoder of RT-DETR; Based on the output of the decoder, scale-based dynamic loss is calculated, and the parameters are iteratively updated according to the calculation results until the dynamic loss meets the preset value to form a detection model. Detection of Thangka elements using the aforementioned detection model; The extraction of features for each element includes: The Thangka image samples are processed sequentially through multiple consecutive levels to generate multiple feature images corresponding to different resolutions; each level corresponds to a feature image of one resolution. The feature image is used as the feature of the corresponding element; The process of performing multi-scale interactive encoding on the features to form feature codes includes: All feature images of the same Thangka image sample are mapped to the same dimension, and all mapped feature images are expanded to form a feature sequence; the position corresponding to each pixel in the feature sequence is configured with pixel coordinate encoding; After performing cross-scale information interaction on the feature sequence based on the pixel coordinate encoding, the feature sequence is re-encoded according to the resolution of all the feature images to form multiple feature codes corresponding to different resolutions of the feature images; The process of comparing and driving feature aggregation to form fused features by the feature encoding includes: The feature encoding is input into the foreground feature convolution branch to obtain foreground features, and the feature encoding is input into the background feature convolution branch to obtain background features; The system simultaneously slides within the foreground and background features using a preset window of a preset size and a preset step size, capturing the differences between the foreground and background features at different locations, and then normalizes these differences to serve as the attention weight at that location. After multiplying the attention weight by the feature encoding at the corresponding position, the feature encodings at all resolutions after the attention weight is weighted are fused to form the fused feature.

2. The method for detecting Thangka elements based on improved RT-DETR according to claim 1, characterized in that, Analysis of multiple elements of the various Thangka image data includes: The Thangka image data is subjected to limb key point recognition to obtain limb key points; the limb key points include hand key points, leg key points and head key points; Calculate the distance from the marked element to the nearest limb key point, and calculate the ratio of the distance to the diagonal distance of the Thangka image data as the feature distance; A verification model is constructed based on the feature distance.

3. The method for detecting Thangka elements based on an improved RT-DETR according to claim 2, characterized in that, Constructing a verification model based on the feature distance includes: Based on the feature distance, a detection interval is constructed for each element as the verification model; When the feature distance corresponding to the element detected by the detection model falls into the corresponding detection interval, the detection result is determined to be true; When the feature distance corresponding to the element detected by the detection model does not fall into the corresponding detection interval, the detection result is determined to be false.

4. The method for detecting Thangka elements based on improved RT-DETR according to claim 3, characterized in that, Detecting Thangka elements using the aforementioned detection model includes: Perform limb key point recognition on the image to be detected to obtain the key points of the limb to be detected; The image to be detected is input into the detection model, and the element information detected by the detection model is obtained; The distance to be verified is calculated using the key points of the limb to be detected and the element information, and the distance to be verified and the type of the corresponding element information are input into the verification model for verification.

5. The method for detecting Thangka elements based on improved RT-DETR according to claim 1, characterized in that, The process of generating feature images at each level includes: The image received at each level is reduced in dimensionality through 1×1 convolution, and then multi-path convolutional data is generated by multi-path parallel processing of multiple lightweight convolutions of different sizes on the reduced image. The multi-channel convolutional data is upscaled to the corresponding resolution using 1×1 convolution, and residuals are added to form a feature image.

6. The method for detecting Thangka elements based on improved RT-DETR according to claim 1, characterized in that, The scale-based dynamic loss calculation based on the decoder output includes: Calculate the size of the image corresponding to the output result based on the output result of the decoder; The weights corresponding to the image are calculated based on the size, and the dynamic loss is obtained by weighted regression using these weights.

7. A Thangka element detection system based on an improved RT-DETR using the method described in any one of claims 1 to 6, characterized in that, include: The acquisition unit is configured to acquire multiple Thangka images and preprocess the Thangka images to form multiple Thangka image data. The sample unit is configured to generate multiple Thangka image samples by analyzing and labeling multiple elements of the multiple Thangka image data; the multiple elements include headdress elements, base elements, gesture elements, hand-held elements, and other elements; The encoding unit is configured to extract features of each element of the Thangka image sample through multi-scale convolutional kernels, and to perform multi-scale interactive encoding of the features to form feature codes; The fusion unit is configured to perform contrast-driven feature aggregation on the feature encoding to form a fused feature, and to sequentially input the fused feature into the query selector and decoder of RT-DETR; The training unit is configured to perform scale-based dynamic loss calculation based on the output of the decoder, and to iteratively update the parameters according to the calculation results until the dynamic loss meets the preset value to form a detection model. The detection unit is configured to detect Thangka elements using the detection model.

Images