An Image Processing Method Based on Non-Uniform Sampling and Scale-Adaptive Supervision

Abstract

This invention discloses an image processing method based on non-uniform sampling and scale-adaptive supervision, belonging to the field of image processing technology. The method includes sequentially inputting a super-resolution image into an adaptive instance-level mesh generator, a warped sampling network, and a first sampler. The first sampler obtains a non-uniform image. An anti-warped sampling network performs multi-scale feature anti-warping on each block set through an inverse bilinear mapping. The output of the anti-warped sampling network is sent to a second sampler. The non-uniform image is then input into the backbone layer to obtain a non-uniform feature map, which is then sent to the second sampler. The second sampler obtains the recovered feature map and inputs it into the model to obtain the image processing result. This invention effectively preserves detailed features. On oilfield datasets, it significantly improves the processing accuracy of small targets without affecting the processing performance of large and medium-sized targets. The dynamic token filtering strategy reduces computational redundancy, balancing efficiency and accuracy.

Description

Technical Field

[0001] This invention discloses an image processing method based on non-uniform sampling and scale adaptive supervision, belonging to the field of image processing technology. Background Technology

[0002] In the field of computer vision, small object processing has always been a highly challenging research area, especially in practical applications in industrial settings such as oil fields. While traditional object processing algorithms have made significant progress in general scenarios, they still face numerous technical bottlenecks when dealing with small objects. Oil field operations are typically characterized by complex environments, long monitoring distances, and variable lighting conditions, resulting in a generally low pixel proportion for targets in images, leading to blurred target features and a lack of texture information. Simultaneously, the complex industrial background generates a large amount of interference, further increasing the difficulty of distinguishing small objects from the background. Most existing processing methods employ uniform downsampling strategies to process input images. While this approach reduces computational complexity, it inevitably leads to the loss of key details of small objects and lacks the ability to adaptively process multi-scale targets.

[0003] In recent years, deep learning-based target processing algorithms have made significant progress. Two-stage processors such as Faster R-CNN generate candidate boxes through region proposal networks, while single-stage processors such as the YOLO series directly predict target locations. Transformer-based processors such as DETR utilize self-attention mechanisms to model global relationships. However, these methods all have significant limitations when dealing with small targets: CNN-based methods are limited by the size of the receptive field, making it difficult to capture the global contextual information of small targets; while Transformer-based methods have global modeling capabilities, they are insufficient in feature extraction for small targets and have high computational complexity. Furthermore, existing methods have poor adaptability to scale changes in small targets when dealing with oilfield scenarios, making it difficult to effectively extract key features of small targets. Therefore, there is an urgent need for a feature enhancement and efficient processing method for small targets. Summary of the Invention

[0004] The purpose of this invention is to provide an image processing method based on non-uniform sampling and scale adaptive supervision to solve the problem of poor accuracy in small target image processing in the prior art.

[0005] An image processing method based on non-uniform sampling and scale-adaptive supervision includes:

[0006] S1. Obtain the image to be processed, and feed it into the super-resolution reconstruction network CAMixerSR to restore the high-frequency details of the image and generate a super-resolution image;

[0007] S2. Input the super-resolution image sequentially into the adaptive instance-level mesh generator, the warped sampling network, and the first sampler;

[0008] S3. The first sampler performs non-uniform distortion sampling on the super-resolution image based on the saliency weight map to obtain a non-uniform image;

[0009] S4. Input the non-uniform image into the backbone layer to obtain a non-uniform feature map and send it to the second sampler;

[0010] The output of the adaptive instance-level mesh generator is divided into several rectangular block sets, which are then fed into an anti-distortion sampling network to extract features. The anti-distortion sampling network performs multi-scale feature anti-distortion on each block set through the inverse transformation of bilinear mapping, aligning the features to the original image coordinate system.

[0011] S5. The second sampler obtains the recovered feature map and inputs it into the Transformer-based SAS DETR model to obtain the image processing result.

[0012] CAMixerSR includes a content-aware mixer and a residual connection module. The content-aware mixer uses convolution operations on simple regions of the image area and a self-attention mechanism to process regions with rich details, depending on the complexity of the image area. The residual connection module fuses multi-scale features and outputs a super-resolution image.

[0013] The adaptive instance-level mesh generator models the target bounding box distribution of the super-resolution image using a Gaussian kernel function, adaptively calculates the saliency coefficient, and dynamically generates a saliency weight map.

[0014] Adaptive calculation of significance coefficients includes:

[0015] ;

[0016] In the formula, For the first The significance coefficient of each bounding box. This is a scale control factor. For the first The width of the bounding box, For the first The height of the bounding box.

[0017] S3 includes constructing separable one-dimensional sampling grids along the x-axis and y-axis of the super-resolution image, adjusting the grid density through a saliency weight map, oversampling high saliency regions, undersampling low saliency regions, and calculating pixel values ​​for non-integer coordinates through bilinear interpolation.

[0018] The SAS DETR model based on Transformer includes inputting the recovered feature map into the backbone layer to obtain multi-level features, and inputting the multi-level features sequentially into the top-down score modulation layer, the encoder, the query refinement layer, and the decoder. The encoder has multiple cascaded encoder layers.

[0019] The input to the encoder layer is divided into three branches: one branch directly inputs to the position update layer (Update CorrespondingPositions), one branch inputs to the global score predictor (GSP), and one branch inputs to the global threshold predictor (GTP).

[0020] The results from GSP are input into the Foreground Confidence layer to obtain the foreground confidence score. The foreground confidence score and the filtering threshold of each feature map calculated by GTP are then input into the Dynamic Token Filtering layer. The Dynamic Token Filtering layer is followed by the Layer Filtering layer. The output of the Layer Filtering layer includes background features and salient features. The background features are filtered through the Self Attention mechanism. The obtained salient features are then input into the Position Update layer. The Position Update layer outputs the processing result of the encoder layer.

[0021] Scale-Adaptive Salience Supervision is added to the dynamic token filtering layer and the layer filtering layer.

[0022] The global threshold predictor processes multi-scale features. The global threshold predictor includes four parallel paths. The first path adds the input features to itself and then inputs them into the convolutional layer CONV to obtain the output features of the first path.

[0023] The other three paths have the same structure: the output features of the previous path are upsampled, then multiplied with the input features of this path, then added with the input features of this path, and then input into the convolutional layer to obtain the output features of this path.

[0024] The output features of each path are fused with the output features of the previous path. Finally, the fourth path outputs the total output features, which are then sequentially input into the layer normalization module LN, the activation function layer GELU, the linear layer, and the global average pooling layer GAP to obtain the filtering threshold.

[0025] We introduce the scale-sensitive intersection-union ratio (SIoU) as the loss function for bounding box regression and use the Hungarian algorithm to construct the loss function for the SAS DETR model.

[0026] The loss function of the SAS DETR model includes:

[0027] ;

[0028] In the formula, For loss function, Match the loss for Hungary. For noise reduction loss, To provide auxiliary loss for the encoder, For the focus loss of significance monitoring, , , , They are , , , The corresponding weights.

[0029] Compared with existing technologies, the present invention has the following advantages: The present invention effectively preserves detailed features, and significantly improves the processing accuracy of small targets on oilfield datasets; it significantly improves the processing effect of small targets without affecting the processing performance of large and medium targets; the dynamic token filtering strategy reduces computational redundancy, balancing efficiency and accuracy. Attached Figure Description

[0030] Figure 1 This is an overall flowchart of the present invention;

[0031] Figure 2 This is a schematic diagram of the SAS DETR model structure based on Transformer;

[0032] Figure 3 This is a schematic diagram of the global threshold predictor structure. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention are described clearly and completely below. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0034] An image processing method based on non-uniform sampling and scale-adaptive supervision, such as Figure 1 ,include:

[0035] S1. Obtain the image to be processed, and feed it into the super-resolution reconstruction network CAMixerSR to restore the high-frequency details of the image and generate a super-resolution image;

[0036] S2. Input the super-resolution image sequentially into the adaptive instance-level mesh generator, the warped sampling network, and the first sampler;

[0037] S3. The first sampler performs non-uniform distortion sampling on the super-resolution image based on the saliency weight map to obtain a non-uniform image;

[0038] S4. Input the non-uniform image into the backbone layer to obtain a non-uniform feature map and send it to the second sampler;

[0039] The output of the adaptive instance-level mesh generator is divided into several rectangular block sets, which are then fed into an anti-distortion sampling network to extract features. The anti-distortion sampling network performs multi-scale feature anti-distortion on each block set through the inverse transformation of bilinear mapping, aligning the features to the original image coordinate system.

[0040] S5. The second sampler obtains the recovered feature map and inputs it into the Transformer-based SAS DETR model to obtain the image processing result.

[0041] CAMixerSR includes a content-aware mixer and a residual connection module. The content-aware mixer uses convolution operations on simple regions of the image area and a self-attention mechanism to process regions with rich details, depending on the complexity of the image area. The residual connection module fuses multi-scale features and outputs a super-resolution image.

[0042] The adaptive instance-level mesh generator models the target bounding box distribution of the super-resolution image using a Gaussian kernel function, adaptively calculates the saliency coefficient, and dynamically generates a saliency weight map.

[0043] Adaptive calculation of significance coefficients includes:

[0044] ;

[0045] In the formula, For the first The significance coefficient of each bounding box. This is a scale control factor. For the first The width of the bounding box, For the first The height of the bounding box.

[0046] S3 includes constructing separable one-dimensional sampling grids along the x-axis and y-axis of the super-resolution image, adjusting the grid density through a saliency weight map, oversampling high saliency regions, undersampling low saliency regions, and calculating pixel values ​​for non-integer coordinates through bilinear interpolation.

[0047] The SAS DETR model based on Transformer, such as Figure 2 The process includes inputting the recovered feature map into the backbone layer to obtain multi-level features, and inputting the multi-level features sequentially into the top-down score modulation layer, the encoder, the query refinement layer, and the decoder. The encoder has multiple encoder layers connected in series.

[0048] The input to the encoder layer is divided into three branches: one branch directly inputs to the position update layer (Update CorrespondingPositions), one branch inputs to the global score predictor (GSP), and one branch inputs to the global threshold predictor (GTP).

[0049] The results from GSP are input into the Foreground Confidence layer to obtain the foreground confidence score. The foreground confidence score and the filtering threshold of each feature map calculated by GTP are then input into the Dynamic Token Filtering layer. The Dynamic Token Filtering layer is followed by the Layer Filtering layer. The output of the Layer Filtering layer includes background features and salient features. The background features are filtered through the Self Attention mechanism. The obtained salient features are then input into the Position Update layer. The Position Update layer outputs the processing result of the encoder layer.

[0050] Scale-Adaptive Salience Supervision is added to the dynamic token filtering layer and the layer filtering layer.

[0051] The global threshold predictor processes multi-scale features such as... Figure 3 The global threshold predictor consists of four parallel paths. The first path adds the input features to itself and then inputs them into the convolutional layer CONV to obtain the output features of the first path.

[0052] The other three paths have the same structure: the output features of the previous path are upsampled, then multiplied with the input features of this path, then added with the input features of this path, and then input into the convolutional layer to obtain the output features of this path.

[0053] The output features of each path are fused with the output features of the previous path. Finally, the fourth path outputs the total output features, which are then sequentially input into the layer normalization module LN, the activation function layer GELU, the linear layer, and the global average pooling layer GAP to obtain the filtering threshold.

[0054] We introduce the scale-sensitive intersection-union ratio (SIoU) as the loss function for bounding box regression and use the Hungarian algorithm to construct the loss function for the SAS DETR model.

[0055] The loss function of the SAS DETR model includes:

[0056] ;

[0057] In the formula, For loss function, Match the loss for Hungary. For noise reduction loss, To provide auxiliary loss for the encoder, For the focus loss of significance monitoring, , , , They are , , , The corresponding weights.

[0058] Tables 1 to 3 provide detailed comparison results between the present invention and current mainstream object processing methods on multiple datasets. The present invention uses ResNet50 as the backbone network and has been comprehensively evaluated on MS COCO2017, AI-TOD-V2, and a self-built oilfield dataset to verify the effectiveness and generalization ability of the method. The evaluation results of the object detection framework's improvement on the object detection model are shown in Table 1.

[0059] Table 1 Evaluation results of the improvement effect of the object detection framework on the object detection model

[0060] ;

[0061] In Table 1, * indicates training using the present invention. The first column represents various methods, and the first row represents various accuracy metrics. From the MS COCO2017 dataset results shown in Table 1, compared to the original models, training using the target processing framework proposed in this invention significantly improves the metrics of the three classic models, with the largest improvement in FCOS and an AP improvement of 4.9%. The small target metric AP... s The improvement was 6.2%. This demonstrates that the target processing framework has universal enhancement capabilities across different architectural models, and it pays stronger attention to smaller targets.

[0062] The evaluation results of each method on the MS COCO2017 dataset are shown in Table 2.

[0063] Table 2 Evaluation results of each method on the MS COCO2017 dataset

[0064] ;

[0065] As shown in Table 2, SAS DETR achieved an AP of 50.4% on the MS COCO2017 dataset, significantly outperforming other state-of-the-art models. Compared to the baseline model Salience DETR, SAS DETR achieved a 1.2% improvement in AP. Specifically, under the same training conditions, the proposed SAS DETR achieved +4.4% AP compared to Deformable-DETR, while significantly reducing the number of training iterations; and under the same number of training iterations, it achieved +1.4% AP compared to DINO. Notably, compared to other models, SAS DETR significantly improved AP... s The improvements in metrics were generally greater than those in AP metrics, which further demonstrates the model's superiority in handling small target tasks.

[0066] The evaluation results of each method on the AI-TOD-V2 dataset are shown in Table 3.

[0067] Table 3 Evaluation results of each method on the AI-TOD-V2 dataset

[0068] ;

[0069] In the AI-TOD-V2 dataset evaluation shown in Table 3, SAS DETR outperformed other processors on most evaluation metrics. Specifically, this invention achieved an AP of 25.2% on the AI-TOD-V2 test set, a 1.8% improvement over Salience DETR. Furthermore, thanks to scale-adaptive saliency supervision, small targets can more effectively utilize feature information within the processing bounding box. This invention observed a significant improvement in small target processing performance, with AP... vt and AP t The results were improved by 2.4% and 1.9% respectively, further validating the advantages of the method of the present invention in small target processing tasks.

[0070] The evaluation results of each method on the self-built oilfield dataset are shown in Table 4.

[0071] Table 4 Evaluation results of each method on the self-built oilfield dataset

[0072] ;

[0073] In the evaluation of the oilfield self-built dataset shown in Table 4, the mAP of SAS DETR combined with the non-uniform sampling framework was improved by 6.78%, with the AP of the "smoking" category improving by 7.7% and the AP of the "mobile phone" category improving by 8.7%. This shows that the method of the present invention also has good adaptability and practicality in real industrial scenarios.

[0074] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image processing method based on non-uniform sampling and scale-adaptive supervision, characterized in that, include: S1. Obtain the image to be processed, and feed it into the super-resolution reconstruction network CAMixerSR to restore the high-frequency details of the image and generate a super-resolution image; S2. Input the super-resolution image sequentially into the adaptive instance-level mesh generator, the warped sampling network, and the first sampler; The output of the adaptive instance-level mesh generator is divided into several rectangular block sets, which are then fed into an anti-distortion sampling network to extract features. The anti-distortion sampling network performs multi-scale feature anti-distortion on each block set through the inverse transformation of bilinear mapping, aligning the features to the original image coordinate system. S3. The first sampler performs non-uniform distortion sampling on the super-resolution image based on the saliency weight map to obtain a non-uniform image; S4. Input the non-uniform image into the backbone layer to obtain a non-uniform feature map and send it to the second sampler; S5. The second sampler obtains the recovered feature map and inputs it into the Transformer-based SAS DETR model to obtain the image processing result; The SAS DETR model based on Transformer includes inputting the recovered feature map into the backbone layer to obtain multi-level features, and inputting the multi-level features sequentially into the top-down score modulation layer, encoder, query refinement layer, and decoder, wherein the encoder has multiple cascaded encoder layers. The encoder's input is divided into three branches: one branch directly inputs to the position update layer (Update CorrespondingPositions), one branch inputs to the global score predictor (GSP), and one branch inputs to the global threshold predictor (GTP). The GSP result is input into the Foreground Confidence layer to obtain the foreground confidence score. The foreground confidence score and the filtering threshold of each feature map calculated by GTP are input into the Dynamic Token Filtering layer. The Dynamic Token Filtering layer is then connected to the Layer Filtering layer. The output of the Layer Filtering layer includes background features and salient features. The background features are filtered through the Self Attention mechanism. The obtained salient features are input into the Position Update layer. The Position Update layer outputs the processing result of the encoder layer.

2. The image processing method based on non-uniform sampling and scale adaptive supervision according to claim 1, characterized in that, CAMixerSR includes a content-aware mixer and a residual connection module. The content-aware mixer uses convolution operations on simple regions of the image area and a self-attention mechanism to process regions with rich details, depending on the complexity of the image area. The residual connection module fuses multi-scale features and outputs a super-resolution image.

3. The image processing method based on non-uniform sampling and scale adaptive supervision according to claim 2, characterized in that, The adaptive instance-level mesh generator models the target bounding box distribution of the super-resolution image using a Gaussian kernel function, adaptively calculates the saliency coefficient, and dynamically generates a saliency weight map. Adaptive calculation of significance coefficients includes: ； In the formula, For the first The significance coefficient of each bounding box. This is a scale control factor. For the first The width of the bounding box, For the first The height of the bounding box.

4. The image processing method based on non-uniform sampling and scale adaptive supervision according to claim 3, characterized in that, S3 includes constructing separable one-dimensional sampling grids along the x-axis and y-axis of the super-resolution image, adjusting the grid density through a saliency weight map, oversampling high saliency regions, undersampling low saliency regions, and calculating pixel values ​​for non-integer coordinates through bilinear interpolation.

5. The image processing method based on non-uniform sampling and scale adaptive supervision according to claim 1, characterized in that, Add scale-adaptive highlighting supervision to the dynamic token filtering layer and the layer filtering layer.

6. The image processing method based on non-uniform sampling and scale adaptive supervision according to claim 5, characterized in that, The global threshold predictor processes multi-scale features. The global threshold predictor includes four parallel paths. The first path adds the input features to itself and then inputs them into the convolutional layer CONV to obtain the output features of the first path. The other three paths have the same structure: the output features of the previous path are upsampled, then multiplied with the input features of this path, then added with the input features of this path, and then input into the convolutional layer to obtain the output features of this path. The output features of each path are fused with the output features of the previous path. Finally, the fourth path outputs the total output features, which are then sequentially input into the layer normalization module LN, the activation function layer GELU, the linear layer, and the global average pooling layer GAP to obtain the filtering threshold.

7. The image processing method based on non-uniform sampling and scale adaptive supervision according to claim 6, characterized in that, We introduce the scale-sensitive intersection-union ratio (SIoU) as the loss function for bounding box regression and use the Hungarian algorithm to construct the loss function for the SAS DETR model.

8. The image processing method based on non-uniform sampling and scale adaptive supervision according to claim 7, characterized in that, The loss function of the SAS DETR model includes: ； In the formula, For loss function, Match the loss for Hungary. For noise reduction loss, To provide auxiliary loss for the encoder, For the focus loss of significance monitoring, , , , They are , , , The corresponding weights.

Images