An image target detection method and system based on an improved dual-channel Transformer

Abstract

The application provides an image target detection method and system based on an improved double-channel Transformer, and relates to the technical field of data processing. The method realizes accurate separation of target and background features through dynamic division of feature complexity, injects multidimensional position information through four-variable composite position coding, combines fine-grained multimodal alignment to enhance the visual-text semantic association, and outputs the detection result through differential computing power allocation and cross-attention fusion of the double channels. The method improves the target detection accuracy, reduces the missing detection rate of small targets and edge regions, saves computing power consumption, shortens the model training period, and effectively guarantees the robustness and application practicability of the detection result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to an image target detection method and system based on an improved dual-channel Transformer. BACKGROUND

[0002] In actual image recognition applications, the traditional Transformer model still faces three key challenges: first, it is difficult to achieve precise allocation of computing resources, and the indiscriminate calculation of background and target leads to low efficiency; second, it cannot fully learn the spatial position information of the image, especially in small target and edge area positioning; third, it lacks effective mining and semantic association of fine-grained features, making it difficult to distinguish between sub-classes under the same class.

[0003] The feature complexity of the target region and the background region of the image is significantly different. The detailed features such as texture and morphology of the target region (such as lesions in medical images and defects in industrial parts) are the core of recognition, while the background region features are relatively simple. The "equalization" processing strategy of the traditional Transformer will lead to a large amount of computing resources being occupied by invalid background, especially in the early stage of training, the model is difficult to quickly focus on the key features of the target, significantly prolonging the training period. At the same time, the relative position relationship of pixels in the image (such as up / down, center / edge) is crucial for target positioning, but the traditional Transformer only uses two-dimensional absolute position encoding, which cannot effectively capture these relative position information, resulting in high small target missing rate and edge area false detection rate. In addition, in high-precision scenarios such as industrial quality inspection and medical diagnosis, it is necessary to distinguish between "scratches and depressions of the same model parts" and "micro nodules of different subtypes", etc. Fine-grained targets. The traditional model relies on global feature matching and is difficult to extract fine-grained recognition information such as "local texture differences" and "edge shape changes", and there is a lack of alignment mechanism between visual features and class semantics, making it difficult to meet the actual demand for recognition accuracy.

[0004] Although there have been studies to address the above problems, for example, some methods introduce CNN multi-scale convolution to assist in extracting local features, but the feature fusion of CNN and Transformer lacks dynamic adaptability and cannot adjust the computing strategy according to the feature complexity; some methods use relative position encoding to optimize position learning, but only capture the relative relationship in a single direction, which cannot cover complex orientation associations; some methods mine fine-grained features by manually labeling key points, but have poor generalization and high labeling cost, and do not combine with the dual-channel architecture to achieve precise focusing on the target region. These limitations make it difficult for existing models to meet the comprehensive needs of efficiency, accuracy, and fine-grained recognition. SUMMARY

[0005] In view of the problems of low processing efficiency of blurred targets and poor processing effect in complex background of existing Transformer models in image target detection, an image target detection method and system based on an improved dual-channel Transformer are provided.

[0006] In a first aspect, the image target detection method based on the improved dual-channel Transformer adopts the following technical scheme:

[0007] An image target detection method based on an improved dual-channel Transformer includes:

[0008] Feature extraction is performed on an input image, the extracted features are divided into target features and background features by a feature complexity dynamic division method, and are output to different computing power channels respectively;

[0009] A four-variable composite position encoding method is used to inject multi-dimensional position information into the target features, and then deep re-attention calculation is performed;

[0010] An ordinary position encoding method is used to perform lightweight light attention calculation on the background features;

[0011] Fine-grained multi-modal alignment enhancement processing is performed based on the target features, corresponding text descriptions are generated, and alignment between visual features and text semantics is realized;

[0012] Cross-attention learning is performed on the target features and the background features after differential processing, and a target image is output.

[0013] Optionally, the feature complexity dynamic division method includes:

[0014] The input image is converted into a feature map through a convolution embedding layer, and is flattened into a feature matrix, local self-attention calculation is performed to generate attention weights;

[0015] The attention weights are activated by ReLU, LayerNorm and a feedforward neural network to obtain a preliminary feature map, and the preliminary feature map is divided into pixel blocks;

[0016] Information entropy is calculated for each pixel block, and feature complexity is evaluated based on a dynamic threshold;

[0017] The pixel blocks are assigned to a detail processing channel or a global suppression channel according to the feature entropy, and on-demand allocation of computing resources is realized.

[0018] Optionally, the feature entropy is calculated using the information entropy formula ;

[0019] wherein is the pixel value in the pixel block The probability of each batch data is calculated, and a dynamic threshold adjustment mechanism is adopted, and the threshold is updated in real time according to the characteristic distribution of the batch data.

[0020] Optionally, the detail processing channel adopts multi-attention head calculation to strengthen the target feature, and the global suppression channel adopts simplified attention calculation to improve efficiency.

[0021] Optionally, the four-variable composite position encoding includes:

[0022] Original four-variable and orientation factor extraction: calculate the left distance, right distance, upper distance and lower distance of each pixel, and generate a normalized orientation factor;

[0023] Absolute coordinate encoding and composite encoding fusion: encode the pixel absolute coordinates using a sine function, and fuse with the orientation factor to generate a four-dimensional composite position encoding, which is embedded into the Transformer query matrix.

[0024] Optionally, the calculation of the orientation factor includes:

[0025]

[0026] wherein, for each pixel of the input image , the original image resolution , the calculation of the original four-variable is:

[0027] Left distance : the horizontal distance of the pixel to the left boundary of the image, that is ;

[0028] Right distance : the horizontal distance of the pixel to the right boundary of the image, that is ;

[0029] Upper distance : the vertical distance of the pixel to the upper boundary of the image, that is ;

[0030] Lower distance : the vertical distance of the pixel to the lower boundary of the image, that is .

[0031] Optionally, the fine-grained multi-modal alignment enhancement includes:

[0032] Fine-grained attribute mining and text description generation: extract local attributes from target features, and generate structured text descriptions through a lightweight language model;

[0033] Cross-modal contrast learning: align visual features and text representations, and optimize using a triplet loss function;

[0034] Dynamic semantic mapping: in the reasoning phase, the classifier output weight is adjusted by calculating the similarity of visual representation and text library.

[0035] Optionally, the cross-modal contrast learning adopts a triplet loss function ;

[0036] wherein, is the text representation of the difficult negative sample, is the fine-grained visual representation, is the fine-grained text representation, is the marginal parameter, and is weighted and fused with the classification loss.

[0037] Optionally, the dynamic semantic mapping adjusts the class probability by cosine similarity calculation, and the formula is:

[0038]

[0039] wherein, is the class probability output by the original classifier, denotes the adjusted class probability is the adjustment coefficient, is the fine-grained visual representation, is each standard text representation in the pre-trained fine-grained class text library.

[0040] In a second aspect, the present application provides an image target detection system based on an improved double-channel Transformer, comprising:

[0041] A feature extraction module is configured to extract features from an input image, divide the extracted features into target features and background features by a feature complexity dynamic division method, and output the target features and the background features to different computing power channels, respectively.

[0042] A four-variable composite position encoding module is configured to inject multi-dimensional position information into the target features by a four-variable composite position encoding method, and then perform deep re-attention calculation.

[0043] A normal position encoding module is configured to perform normal position encoding on the background features, and perform lightweight light attention calculation.

[0044] An enhancement processing module is configured to perform fine-grained multi-modal alignment enhancement processing based on the target features, generate corresponding text descriptions, and realize alignment of visual features and text semantics.

[0045] An output module is configured to perform cross-attention learning on the target features and the background features that have been processed differently, and output a target image.

[0046] In summary, the application realizes accurate separation of target and background features through dynamic feature complexity division, injects multi-dimensional position information through four-variable composite position coding, enhances visual-textual semantic association through fine-grained multi-modal alignment, and outputs detection results through differential computing power allocation and cross-attention fusion. This makes the target detection accuracy higher, the small target and edge area missing detection rate lower, while saving computing power consumption and shortening the model training cycle, effectively ensuring the robustness and practicality of the detection results. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 is a flowchart of the first embodiment of the image target detection method based on the improved dual-channel Transformer of the application.

[0048] Figure 2 is a model training loss change curve diagram of the improved dual-channel Transformer of the application.

[0049] Figure 3 is a model validation loss change curve diagram of the improved dual-channel Transformer of the application.

[0050] Figure 4 is a model training precision (mAP@0.5) change curve diagram of the improved dual-channel Transformer of the application.

[0051] Figure 5 is a model validation precision (mAP@0.5) change curve diagram of the improved dual-channel Transformer of the application.

[0052] Figure 6 is a structure block diagram of the first embodiment of the image target detection system based on the improved dual-channel Transformer of the application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical scheme and advantages of the application clearer, the application will be further described in detail below with reference to the drawings and examples. It should be understood that the specific examples described herein are only used to explain the application and do not limit the application.

[0054] The embodiment of the application provides an image target detection method based on an improved dual-channel Transformer. Referring to Figure 1 , Figure 1 is a flowchart of the first embodiment of the image target detection method based on the improved dual-channel Transformer of the application.

[0055] In this embodiment, the image target detection method based on the improved dual-channel Transformer includes the following steps:

[0056] Step S10: feature extraction is performed on the input image, and the extracted features are divided into target features and background features by a feature complexity dynamic division method, and are output to different computing power channels respectively.

[0057] The embodiment designs a feature complexity dynamic division method for dividing the input image into target features and background features, and then outputting them to different computing power channels to realize on-demand allocation of computing resources. A four-variable composite position encoding method is designed to generate an encoding containing four-dimensional position information, which is embedded in the target image features to improve the positioning accuracy of the model on the target image features. A fine-grained multi-modal alignment enhancement method is designed to generate corresponding text descriptions combined with target feature information, and multi-modal learning is introduced to realize the alignment of target visual features and text semantics to optimize the final target detection result.

[0058] It should be noted that the feature complexity dynamic division method includes: converting the input image into a feature map through a convolution embedding layer, and flattening it into a feature matrix, performing local self-attention calculation to generate attention weights; performing ReLU activation, LayerNorm and feedforward neural network processing on the attention weights to obtain a preliminary feature map, and dividing it by pixel block; calculating the information entropy of each pixel block and evaluating the feature complexity based on a dynamic threshold; assigning the pixel block to a detail processing channel or a global suppression channel according to the feature entropy to realize on-demand allocation of computing resources.

[0059] It can be understood that the feature entropy calculation uses the information entropy formula ; wherein is the probability of the pixel value being in the pixel block, and a dynamic threshold adjustment mechanism is used, with the threshold being updated in real time according to the feature distribution of the batch data.

[0060] It should be noted that the detail processing channel uses multi-attention head calculation to strengthen the target features, and the global suppression channel uses simplified attention calculation to improve efficiency.

[0061] In specific implementation, the implementation of step S10 specifically includes:

[0062] Input processing and local feature extraction: the input image I is converted into a feature map through a convolution embedding layer (convolution kernel size 3x3, step size 2, output channel number d=256), and the feature map is flattened into an input feature matrix with a sequence length of N . 3x3 local self-attention calculation is performed on the feature matrix to generate attention weights. The calculation formula of the attention weights is shown in equation (1):

[0063] (1)

[0064] is the attention weight for the 3x3 neighborhood of the th pixel, is the query vector and key vector of the pixel, respectively is a trainable weight matrix, is the dimension of the (query / key) vector, is the offset of the neighborhood pixel on the feature map, is a scaling factor to avoid numerical overflow due to attention weights being too large.

[0065] Feature processing and pixel block division: the calculated attention weights are processed by a ReLU nonlinear activation function, and then passed through a LayerNorm layer and a feedforward neural network (FFN) to obtain a preliminary processed feature map . The feature map is divided into 8x8 pixel blocks, and a total of pixel blocks are obtained.

[0066] Feature entropy calculation: for each pixel block, the pixel value distribution (pixel value range 0-255) of all pixel feature vectors in the pixel block is counted, and the information entropy of the pixel block is calculated (as formula (2)):

[0067] (2)

[0068] is the probability of the pixel value being in the pixel block. The pixel value refers to the brightness intensity value of the pixel point in the image on the corresponding color channel, which takes a value in the range of 0 to 255. For a grayscale image, directly represents the gray level of the pixel; for a color image, the image can be converted into a grayscale image before entropy calculation, or the pixel value distribution can be calculated for each of the red, green, and blue color channels. represents the frequency of pixels with a gray value equal to in an 8x8 pixel block, which is obtained by counting the gray distribution of all pixels in the pixel block. A threshold value is set, which can be dynamically adjusted according to the feature distribution of the current batch of data during training, as shown in formula (3):

[0069] (3)

[0070] is the threshold value of the previous batch, is the average feature entropy of all pixel blocks in the current batch, ​is a symbol function (when is 1, the threshold is up-regulated; otherwise, it is -1, the threshold is down-regulated), the step size is set to 0.1 to avoid excessive fluctuation of the threshold. The adaptive threshold setting can make the model flexibly adjust the feature division standard according to the characteristics of different batches of images, and improve the accuracy of feature division.

[0071] into the "detail processing channel", which adopts more complex attention calculation (such as increasing the number of attention heads to 8); the low-entropy pixel block (background feature area) is classified into the "global suppression channel", which adopts simplified attention calculation (such as setting the number of attention heads to 4).

[0072] Step S20: Injecting multi-dimensional position information into the target feature by using a four-variable compound position encoding method, and then performing deep re-attention calculation.

[0073] It should be noted that the four-variable compound position encoding includes:

[0074] Original four variables and direction factor extraction: calculating the left distance, right distance, upper distance and lower distance of each pixel, and generating normalized direction factors;

[0075] Absolute coordinate encoding and compound encoding fusion: using a sine function to encode the absolute coordinates of the pixels, and fusing with the direction factors to generate four-dimensional compound position encoding, which is embedded into the Transformer query matrix.

[0076] It should be noted that the original four variables are normalized to eliminate the influence of image resolution difference on position information, and the horizontal direction factor and the vertical direction factor are generated (such as formula (4)), the value range of the direction factor is [-1, 1], which can directly reflect the relative direction of the pixel (such as >0 indicates that the pixel is left, <0 indicates that it is right) The calculation of the direction factor includes:

[0077] (4)

[0078] wherein, for each pixel of the input image, the original image resolution , the calculation of the original four variables is:

[0079] left distance : the horizontal distance from the pixel to the left boundary of the image, that is ;

[0080] right distance : the horizontal distance from the pixel​ The horizontal distance to the right boundary of the image, i.e. ;

[0081] Top distance : pixel The vertical distance to the top boundary of the image, i.e. ;

[0082] Lower distance : pixel The vertical distance to the bottom boundary of the image, i.e. .

[0083] It should be noted that: when =0 or =0 (i.e., image resolution is 0) or (Extreme cases), default This avoids calculation errors where the denominator is zero. Normalization ensures that the orientation factor is unaffected by image size, accurately reflecting the relative orientation of pixels regardless of image resolution, thus enhancing the model's adaptability to images of different sizes.

[0084] In specific implementation, the method of this embodiment further includes:

[0085] Fusion of absolute coordinate encoding and composite encoding: using a sine function for the absolute coordinates of pixels. Perform initial encoding (as in equation (5)):

[0086]

[0087]

[0088] in, It is the feature dimension. It's a dimensional index. Then, the orientation factor... with absolute coordinate encoding value Perform element-wise multiplication to generate a four-dimensional composite positional code (as shown in Equation (6)), and embed it into the query matrix of the Transformer. In this process, attention calculations consider both absolute position and relative orientation information simultaneously:

[0089] (6)

[0090] in and It queries the next state and the current state of the matrix. These are two parameters of the absolute coordinate encoding value.

[0091] Step S30: Lightweight attention calculation is performed on the background features using general position encoding.

[0092] Step S40: Fine-grained multi-modal alignment enhancement is performed based on the target features to generate corresponding text descriptions and achieve alignment between visual features and text semantics.

[0093] It should be noted that fine-grained multi-modal alignment enhancement includes: fine-grained attribute mining and text description generation: extracting local attributes from target features and generating structured text descriptions through a lightweight language model; cross-modal contrast learning: aligning visual features and text representations, and optimizing using a triplet loss function; dynamic semantic mapping: in the inference stage, the similarity between visual representations and the text library is calculated to adjust the classifier output weight.

[0094] It should be noted that cross-modal contrast learning uses a triplet loss function ;

[0095] wherein, is the text representation of the difficult negative sample, is the fine-grained visual representation, is the fine-grained text representation, is the marginal parameter, and is weighted and fused with the classification loss.

[0096] It can be understood that the dynamic semantic mapping adjusts the class probability through cosine similarity calculation, and the formula is:

[0097]

[0098] wherein, is the class probability output by the original classifier, denotes the adjusted class probability is the adjustment coefficient, is the fine-grained visual representation, is each standard text representation in the pre-trained fine-grained class text library.

[0099] In specific implementation, the specific implementation of step S40 includes:

[0100] Fine-grained attribute mining and text description generation: from the high-entropy pixel blocks output by the "detail processing channel", combined with the position information provided by the four-variable compound position encoding, the attribute candidate region is located. Through 1x1 convolution to compress the channel dimension, and then through 3x3 depth separable convolution to extract local attribute features. These attribute features are input into an attribute classifier (including 3 linear layers, with GELU activation function in the middle), to obtain the class probability of each attribute dimension. Subsequently, the attribute classification result is input into a lightweight language model (based on DistilBERT fine-tuning) to generate structured fine-grained attribute text description .

[0101] Cross-modal contrastive learning: the output features of the “detail processing channel” are mapped to 256-dimensional fine-grained visual representations through a modal connection layer (containing a linear layer and LayerNorm) . At the same time, the attribute text description is input into the DistilBERT model, and the features of the [CLS] token are extracted and mapped to 256-dimensional fine-grained text representations through a linear layer . The and are L2 normalized. Triplet loss is used to realize visual-text alignment, and the loss function is shown in equation (7):

[0102] (7)

[0103] is the text representation of the difficult negative sample, =0.2 is the margin parameter. The contrastive loss is combined with the classification loss (cross-entropy loss) of the dual-channel model, and the total loss is shown in equation (8):

[0104] (8)

[0105] Dynamic semantic mapping:

[0106] During inference, the attribute text description of the input image is generated in real time, and is mapped to the text representation . The cosine similarity between the fine-grained visual representation of the input image and the standard text representation in the pre-trained fine-grained category text library is calculated, as shown in equation (9):

[0107] (9)

[0108] is the standard text representation of the category . The construction of the pre-trained text library and the generation logic of the attribute text description in the training stage are consistent, ensuring semantic consistency.

[0109] The classifier output weight is adjusted according to the similarity result, as shown in equation (10):

[0110] (10)

[0111] a class probability output by an original classifier, as an adjustment coefficient, and output an adjusted class probability .

[0112] Step S50: Cross attention learning is performed on the differentiated target feature and the background feature to output a target image.

[0113] In specific implementation, the embodiment discloses an image target detection method based on an improved dual-channel Transformer, and belongs to the field of image processing and computer vision. In view of the problems of low processing efficiency of fuzzy targets and poor processing effect in complex backgrounds of existing Transformer models in image target detection, an image target detection method based on an improved dual-channel Transformer is proposed. First, an image containing target information is input, and features of the image are extracted. The features are then processed through a 3x3 convolution layer, and target features and background features are divided according to different feature information complexity, so as to avoid semantic interference caused by mixed processing of the two types of features. Then, four-variable compound position encoding is performed on the target features, so that various features can better guarantee their relative position relationship and absolute position relationship. Deep re-attention calculation is then performed to fully mine and strengthen the details of the target features. After obtaining the details of the target features, cross-modal learning is performed by combining the target feature information with text information to avoid the limitations of single modal information. At the same time, ordinary position encoding and light attention calculation are performed on the background features to save computing resources. Finally, cross attention learning is performed on the features of the two channels, and a target image is output. The method effectively improves the accuracy and robustness of image target detection and has practical application value. The proposal of the embodiment is a targeted breakthrough of the existing technical defects in the background of increasing demand for image recognition accuracy and efficiency in the fields of intelligent monitoring, autonomous driving, medical imaging, and industrial quality inspection. These fields not only require the model to quickly process massive image data, but also require high robustness in complex backgrounds, small targets, and fine classification. The three-level architecture of the embodiment meets these core needs.

[0114] In specific implementation, the data set is COCO 2017 target detection data set (containing 80 types of targets, and small targets accounting for 32%); the hardware environment is NVIDIA RTX 3090 graphics card, Intel i9-12900K CPU, and 32 GB memory; all models use the same training strategy (AdamW optimizer, cosine annealing learning rate, Batch Size=16, and training for 120 epochs) to ensure fairness of comparison. The specific experimental results are as follows Figure 2 , Figure 3 , Figure 4 and Figure 5As shown; wherein, Figure 2 is the model training loss change curve, which shows the trend of the training loss value of the neural network model in the 120 rounds of training process. The horizontal coordinate is the training round, and the vertical coordinate is the loss value. From the curve, it can be seen that the model training loss presents a smooth downward trend, the loss value decreases rapidly in the initial stage, and gradually tends to be stable and converges to a lower level in the later stage, and the final training loss is about 0.05~0.1 interval, without obvious shock or rebound phenomenon, indicating that the model training process is stable, and can effectively learn the data features; Figure 3 is the model validation loss change curve, which shows the trend of the validation loss value of the neural network model in the 120 rounds of training process. The horizontal coordinate is the training round, and the vertical coordinate is the loss value. The overall trend of the validation loss curve is consistent with the training loss, showing a smooth downward feature, and the validation loss value is slightly higher than the training loss, which conforms to the normal model training law, and there is no obvious upward rebound throughout the process, indicating that the model does not appear overfitting phenomenon, and has good generalization ability, and the final validation loss converges to a stable level; Figure 4 is the model training precision (mAP@0.5) change curve, which shows the trend of the training precision of the neural network model in the 120 rounds of training process, taking mAP@0.5 as the evaluation index. The horizontal coordinate is the training round, and the vertical coordinate is the average precision mean@0.5. The training precision curve presents a smooth upward trend, the precision increases rapidly in the initial stage, and gradually approaches to the saturation value in the later stage, and the final training precision is about 0.93, without obvious fluctuation or decline, indicating that the model can effectively learn on the training set and has high prediction accuracy; Figure 5 is the model validation precision (mAP@0.5) change curve, which shows the trend of the validation precision of the neural network model in the 120 rounds of training process, taking mAP@0.5 as the evaluation index. The horizontal coordinate is the training round, and the vertical coordinate is the average precision mean@0.5. The validation precision curve has the same trend as the training precision curve, showing a steady upward state, and the validation precision value is close to the training precision, and the final validation precision is about 0.91, with an accuracy improvement of about 44.44%, indicating that the model still has excellent prediction ability on unseen validation data, and has good generalization performance without overfitting problem.

[0115] The specific data comparison is shown in the following table:

[0116]

[0117] Accurate separation of target and background features is achieved through dynamic feature complexity partitioning. Multi-dimensional positional information is injected via four-variable composite positional encoding, and fine-grained multimodal alignment enhances the visual-text semantic association. Detection results are output through dual-channel differentiated computing power allocation and cross-attention fusion. This improves target detection accuracy, reduces the false negative rate of small targets and edge regions, saves computing power, shortens the model training cycle, and effectively ensures the robustness and practicality of the detection results.

[0118] Reference Figure 6 , Figure 6 This is a structural block diagram of the first embodiment of the image target detection system based on the improved dual-channel Transformer of this application.

[0119] like Figure 6 As shown, the image target detection system based on the improved dual-channel Transformer proposed in this application includes:

[0120] Feature extraction module 10 is used to extract features from the input image. It divides the extracted features into target features and background features using a dynamic feature complexity partitioning method, and outputs them to different computing power channels respectively.

[0121] The four-variable composite position encoding module 20 is used to inject multi-dimensional position information into the target features using the four-variable composite position encoding method, and then perform deep attention calculation.

[0122] The ordinary position encoding module 30 is used to perform lightweight, attention-free computation by using ordinary position encoding on background features;

[0123] The enhancement processing module 40 is used to perform fine-grained multimodal alignment enhancement processing based on target features, generate corresponding text descriptions, and achieve alignment between visual features and text semantics.

[0124] Output module 50 is used to perform cross-attention learning on the differentiated target features and background features to output the target image.

[0125] It should be understood that the above are merely illustrative examples and do not constitute any limitation on the technical solution of this application. In specific applications, those skilled in the art can make settings as needed, and this application does not impose any restrictions on this.

[0126] The embodiment realizes accurate separation of target and background features by dynamic division of feature complexity, injects multi-dimensional position information through four-variable composite position coding, enhances visual-textual semantic association by combining fine-grained multi-modal alignment, and outputs detection results through differential allocation of computing power in double channels and cross-attention fusion. The target detection accuracy is improved, the missing detection rate of small targets and edge regions is reduced, the computing power consumption is saved, the model training period is shortened, and the robustness and application practicality of the detection results are effectively guaranteed.

[0127] It should be noted that the above-described workflow is only illustrative and does not limit the scope of protection of the present application. In actual applications, those skilled in the art can select part or all of them according to actual needs to achieve the purpose of the embodiment scheme, and this place does not limit.

[0128] In addition, technical details not described in detail in the embodiment can be referred to the method for image target detection based on improved double-channel Transformer provided by any embodiment of the present application, which will not be repeated here.

[0129] In addition, it should be noted that in this paper, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or includes elements inherent to such process, method, article or system. Without more limitations, the element defined by the statement "includes a" does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0130] The above-mentioned serial numbers of the embodiments of the present application are only for description, not representing the advantages and disadvantages of the embodiments.

[0131] Those skilled in the art can clearly understand the above-mentioned embodiment method can be realized by means of software and the necessary general hardware platform, of course, also can be through hardware, but in many cases the former is the better embodiment. Based on such understanding, the technical solutions of the present application essentially or say the part of the contribution to the prior art can be embodied in the form of software products, the computer software product is stored in a storage medium (such as read-only memory (ReadOnly Memory, ROM) / RAM, disk, optical disc), including a number of instructions to make a terminal device (may be a mobile phone, computer, server, or network equipment, etc.) executes the method of each embodiment of the present application. The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, any equivalent structure or equivalent process transformation made by using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An image object detection method based on improved dual-channel Transformer, characterized in that, The method comprises the following steps: performing feature extraction on an input image, dividing the extracted features into target features and background features through a feature complexity dynamic division method, and outputting the target features and the background features to different computing power channels respectively; injecting multi-dimensional position information into the target features using a four-variable compound position encoding method, and then performing deep heavy attention calculation; injecting multi-dimensional position information into the target features using a four-variable compound position encoding method, and then performing deep heavy attention calculation; performing fine-grained multi-modal alignment enhancement based on the target features, generating corresponding text descriptions, and realizing alignment between visual features and text semantics; performing cross-attention learning on the target features and the background features after differential processing, and outputting a target image; The feature complexity dynamic division method comprises the following steps: converting the input image into a feature map through a convolution embedding layer, flattening the feature map into a feature matrix, performing local self-attention calculation to generate attention weights, and performing ReLU activation, LayerNorm and feedforward neural network processing on the attention weights to obtain a preliminary feature map, and then dividing the preliminary feature map into pixel blocks; calculating the information entropy of each pixel block, and performing feature complexity evaluation based on a dynamic threshold; allocating the pixel blocks to a detail processing channel or a global suppression channel according to the feature entropy, and realizing on-demand allocation of computing resources; The detail processing channel adopts multi-attention head calculation to strengthen the target features, and the global suppression channel adopts simplified attention calculation to improve efficiency. The four-variable compound position encoding comprises the following steps: extracting original four variables and direction factors: calculating the left distance, right distance, upper distance and lower distance of each pixel, and generating normalized direction factors; fusing absolute coordinate encoding and compound encoding: using a sine function to encode the absolute coordinates of the pixels, and fusing the absolute coordinates with the direction factors to generate four-dimensional compound position encoding, which is embedded into a Transformer query matrix; The fine-grained multi-modal alignment enhancement comprises the following steps: fine-grained attribute mining and text description generation: extracting local attributes from the target features, and generating structured text descriptions through a lightweight language model; cross-modal contrast learning: aligning visual features and text representations, and optimizing using a triplet loss function; dynamic semantic mapping: in the inference stage, adjusting the classifier output weight by calculating the similarity between the visual representation and the text library. The probability that the pixel value in the pixel block is x is denoted as p(x), and a dynamic threshold adjustment mechanism is used to update the threshold in real time according to the feature distribution of the batch data.

2. The method of claim 1, wherein, The feature entropy calculation uses the information entropy formula ; The calculation of the direction factor comprises the following steps:

3. The method of claim 1, wherein, The dynamic semantic mapping adjusts the class probability by calculating the cosine similarity, and the formula is as follows: where, for each pixel of the input image , the original image resolution , the original four variables are calculated as: left distance : pixel horizontal distance to the left border of the image, i.e. ; Right distance : pixel horizontal distance to the right image border, i.e. ; Upper distance : Pixel vertical distance to the image upper border, i.e. ; Lower distance : Pixel Vertical distance to the lower image border, i.e. .

4. The method of claim 1, wherein, Cross-modal contrastive learning employs a triplet loss function ; wherein, is a text representation of the difficult negative sample, is a fine-grained visual representation, is a fine-grained text representation, and a is a margin parameter and is fused with the classification loss with a weighting.

5. The method of claim 1, wherein, The method comprises the following steps: wherein, is a class probability output by the original classifier, denotes an adjusted class probability β is an adjustment coefficient, is a fine-grained visual representation, is a standard text representation in the pre-trained fine-grained class text library.

6. An image object detection system based on improved dual-channel Transformer, characterized in that, a feature extraction module for performing feature extraction on an input image, dividing the extracted features into target features and background features through a feature complexity dynamic division method, and outputting the target features and the background features to different computing power channels respectively; a four-variable compound position encoding module for injecting multi-dimensional position information into the target features using a four-variable compound position encoding method, and then performing deep heavy attention calculation; an ordinary position encoding module for injecting multi-dimensional position information into the target features using a four-variable compound position encoding method, and then performing deep heavy attention calculation; ​ The enhancement processing module is configured to perform fine-grained multi-modal alignment enhancement processing based on the target feature, generate corresponding text description, and realize alignment between visual features and text semantics. The output module is configured to perform cross-attention learning on the target feature and the background feature after the differential processing, and output the target image.

Images