Image data processing method based on broken line path mask attention network

Abstract

The embodiment of the invention discloses an image data processing method based on a broken line path mask attention network, and the method comprises the steps: obtaining initial image data, carrying out the preprocessing of the initial image data, and obtaining training data and to-be-processed data; the method comprises the following steps: firstly, establishing an attention expression and a two-dimensional broken line path scanning strategy based on a Mangbar network based on training data, and expanding to generate a broken line path mask; fusing the broken line path mask with a standard attention mechanism, constructing a broken line path mask attention module containing spatial position information, integrating the broken line path mask attention module to a visual converter architecture to form an initial network, and training to obtain a target network; and inputting the to-be-processed data into the target network, and outputting image classification, segmentation and target identification results. According to the method, through fusion of the Mangban sequence modeling capability and the two-dimensional space path mask, image space structure perception is enhanced, model efficiency and precision are considered, unified processing of multi-vision tasks is supported, and an efficient solution is provided for image analysis in a complex scene.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image data processing method based on a broken line path mask attention network. Background Art

[0002] With the widespread application of computer vision technology in tasks such as image classification, segmentation, and object detection, existing technologies have typically used visual transformers and Mamba networks to handle these tasks. However, traditional visual transformers rely on global attention to capture long-range correlations, poorly modeling local spatial relationships and prone to feature confusion in tasks such as semantic segmentation and small object detection. Directly applying Mamba networks destroys the spatial structure of the image due to the one-dimensional scanning path and is also unable to effectively encode two-dimensional dependencies between pixels, ultimately reducing the accuracy and efficiency of multi-vision task processing.

[0003] Therefore, how to integrate the efficient sequence modeling capabilities of the Mamba network with the two-dimensional spatial structure of images to build a visual transformer with both spatial perception and computational efficiency to improve the processing performance of multiple visual tasks is a key issue that needs to be solved urgently. Summary of the Invention

[0004] In view of this, the embodiment of the present application provides an image data processing method based on a zigzag path mask attention network, which can combine the efficient sequence modeling capabilities of the Mamba network with the two-dimensional spatial structure of the image to construct a visual transformer with both spatial perception and computational efficiency, thereby improving the processing performance of multiple visual tasks. The embodiment of the present application provides an image data processing method based on a zigzag path mask attention network as follows:

[0005] An embodiment of the present application provides an image data processing method based on a polyline path mask attention network, comprising:

[0006] Acquiring initial image data, and preprocessing the initial image data to obtain training data and data to be processed;

[0007] Establishing an attention expression based on the Mamba network and a two-dimensional broken line path scanning strategy according to the training data;

[0008] Expanding the two-dimensional zigzag path scanning strategy according to the attention expression to obtain a zigzag path mask;

[0009] Obtain a standard attention mechanism module, and fuse the broken line path mask with the standard attention mechanism module to obtain a broken line path mask attention module, wherein the broken line path mask attention module includes the spatial position of the data to be processed;

[0010] Integrating the polyline path mask attention module into the visual transformer architecture to obtain an initial visual transformer network;

[0011] Inputting the training data into the initial visual transformer network for training to obtain a target visual transformer network;

[0012] The data to be processed is input into a target visual transformer network to obtain an image processing result, wherein the image processing result includes an image classification result, an image segmentation result and a target recognition result.

[0013] In some embodiments, establishing an attention expression based on a Mamba network and a two-dimensional broken line path scanning strategy according to the training data includes:

[0014] Establishing a cyclic recursive mechanism expression of the Mamba network according to the training data and formula (1);

[0015]

[0016] Among them, x i is the pixel of the current training data, h i is the current latent variable, a i is the attenuation factor, B i is the input mapping matrix, y i is the recognition result of the current training data, C i is the output mapping matrix, h i-1 The previous hidden variable, i is the index;

[0017] Convert the loop recursive mechanism expression into an attention expression;

[0018] A two-dimensional broken line path scanning strategy based on the Mamba network is established.

[0019] In some embodiments, converting the cyclic recursive mechanism expression into an attention expression includes:

[0020] According to formula (2), the loop recursive mechanism expression is converted into an attention expression;

[0021]

[0022] Among them, X is the one-dimensional sequence obtained by scanning the current training data along the broken line path, B is the input projection matrix, and C is the output projection matrix. The joint transformation matrix of the input-output mapping, is the transposed matrix of B, L 1D is a one-dimensional causal attenuation mask, ⊙ is the Hadamard product, and Y is the output sequence.

[0023] In some embodiments, the acquiring of image data to be processed, preprocessing of the image to be processed, and obtaining training data and data to be processed include:

[0024] Obtain image data to be processed;

[0025] The image data to be processed is resized, data augmented, and normalized to obtain training data and data to be processed.

[0026] In some embodiments, the expanding the two-dimensional zigzag path scanning strategy according to the attention expression to obtain a zigzag path mask includes:

[0027] Determining a horizontal attenuation factor and a vertical attenuation factor according to the attention expression;

[0028] Acquiring a two-dimensional folded line path type according to the two-dimensional folded line path scanning strategy, wherein the two-dimensional folded line path type includes a horizontal scanning type and a vertical scanning type;

[0029] Calculating the horizontal attenuation factor and the horizontal scanning type to obtain a horizontal scanning weight;

[0030] Calculating the vertical attenuation factor and the vertical scanning type to obtain a vertical scanning weight;

[0031] The horizontal scanning weight and the vertical scanning weight are summed and flattened to obtain a broken line path mask.

[0032] In some embodiments, the target visual transformer network includes a front feature extraction layer, a polyline path mask attention module layer, a downsampling layer, and an output layer.

[0033] In some embodiments, the amount of the training data is 1.35 million, and the amount of the data to be processed is 50,000.

[0034] An embodiment of the present application provides an image data processing device based on a polyline path mask attention network, comprising:

[0035] An acquisition module is used to acquire initial image data, pre-process the initial image data, and obtain training data and data to be processed;

[0036] An establishment module is used to establish an attention expression based on the Mamba network and a two-dimensional broken line path scanning strategy according to the training data;

[0037] a processing module, configured to perform an expansion process on the two-dimensional zigzag path scanning strategy according to the attention expression to obtain a zigzag path mask;

[0038] The processing module is further configured to obtain a standard attention mechanism module, fuse the broken line path mask with the standard attention mechanism module to obtain a broken line path mask attention module, wherein the broken line path mask attention module includes the spatial position of the data to be processed;

[0039] The building module is further configured to integrate the polyline path mask attention module into the visual transformer architecture to obtain an initial visual transformer network;

[0040] The establishment module is further configured to input the training data into the initial visual transformer network for training to obtain a target visual transformer network;

[0041] The processing module is further used to input the data to be processed into the target visual transformer network to obtain image processing results, which include image classification results, image segmentation results and target recognition results.

[0042] The computer device provided in an embodiment of the present application includes a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the method described in the embodiment of the present application is implemented.

[0043] The computer-readable storage medium provided in the embodiment of the present application stores a computer program thereon, and when the computer program is executed by a processor, the method provided in the embodiment of the present application is implemented.

[0044] The embodiment of the present application provides an image data processing method based on a zigzag path mask attention network. The method obtains initial image data, pre-processes the initial image data, and obtains training data and data to be processed. Based on the training data, an attention expression based on a Mamba network and a two-dimensional zigzag path scanning strategy are first established to expand and generate a zigzag path mask. The zigzag path mask is integrated with the standard attention mechanism to construct a zigzag path mask attention module containing spatial position information, and is integrated into the visual transformer architecture to form an initial network. After training, a target network is obtained. The data to be processed is input into the target network to output image classification, segmentation, and target recognition results. In this way, the image spatial structure perception can be enhanced by integrating the Mamba sequence modeling capability with the two-dimensional space path mask, taking into account both model efficiency and accuracy, supporting unified processing of multiple visual tasks, providing an efficient solution for image analysis in complex scenes, and solving the technical problems raised in the background technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments of the present application or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0046] Figure 1 A schematic diagram of an implementation flow of an image data processing method based on a polyline path mask attention network provided in an embodiment of the present application;

[0047] Figure 2 A schematic diagram of the implementation flow of a method for obtaining a broken line path mask in an image data processing method based on a broken line path mask attention network provided in an embodiment of the present application;

[0048] Figure 3 An image data processing device based on a broken line path mask attention network is provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0050] The following description of some of the technologies involved in the embodiments of this application is provided to facilitate understanding and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for the sake of clarity and conciseness, some descriptions of well-known functions and structures are omitted from the following description.

[0051] Figure 1 This is a schematic diagram of the implementation flow of an image data processing method based on a polyline path mask attention network provided by an embodiment of the present application, including steps 101 to 107. Figure 1 This is only an execution order shown in the embodiment of the present application, and does not represent the only execution order of an image data processing method based on a broken line path mask attention network. In the case that the final result can be achieved, Figure 1 The steps shown may be performed in parallel or reversed.

[0052] Step 101: Acquire initial image data, pre-process the initial image data, and obtain training data and data to be processed.

[0053] In an embodiment of the present application, after obtaining the initial image data, the image is uniformly scaled to a fixed size required by the model input (such as 224×224 pixels) to ensure the consistency of the input dimension. The training data is expanded by random horizontal flipping, brightness / contrast adjustment, local area occlusion and other operations to improve the model's anti-interference ability. The pixel values ​​are standardized and adjusted to a range with a mean of 0 and a standard deviation of 1 to accelerate model convergence. After processing, the data is divided into a training data set (for model parameter learning) and a data set to be processed (for inference testing), where the number of training data is 1.35 million and the number of data to be processed is 50,000.

[0054] Step 102: Establish an attention expression based on the Mamba network and a two-dimensional broken line path scanning strategy according to the training data.

[0055] In this embodiment, a Mamba network modeling expression for sequence data is established, including controlling the degree of historical information retention through a decay factor. The current latent variable is updated by a linear combination of the previous latent variable and the current input features. The recursive relationship is equivalently converted into a matrix operation form, and the dependencies between each position in the sequence are represented by a structured mask matrix, where the element values ​​of the mask matrix are the products of the decay factors.

[0056] For the two-dimensional images in the training data, two types of broken line paths are defined to connect any two pixels. 1. Vertical-then-horizontal path: Scans vertically from the starting point to the target row, then horizontally to the target column; 2. Horizontal-then-vertical path: Scans horizontally from the starting point to the target column, then vertically to the target row. A multilayer perceptron is used to learn horizontal and vertical attenuation factors for each pixel, representing the strength of information transfer between adjacent pixels.

[0057] Step 103: Expand the two-dimensional broken line path scanning strategy according to the attention expression to obtain a broken line path mask.

[0058] In an embodiment of the present application, the two-dimensional zigzag path scanning strategy is expanded based on the attention expression logic of the Mamba network. Among them, for any two pixels in the two-dimensional image, the path weights are calculated along the two zigzag paths respectively. The weight of the vertical path first and then the horizontal path is the product of the product of the horizontal attenuation factor and the product of the vertical attenuation factor. The weight of the horizontal path first and then the vertical path is the same. This application will not go into details here. The two types of path weights are combined to generate a two-dimensional zigzag path mask in the form of a four-dimensional tensor, which is then flattened into a two-dimensional matrix. The mask explicitly encodes the spatial dependency relationship between pixels.

[0059] Step 104, obtain the standard attention mechanism module, fuse the broken line path mask with the standard attention mechanism module to obtain the broken line path mask attention module, and the line path mask attention module includes the spatial position of the data to be processed.

[0060] In this embodiment, the generated two-dimensional polyline path mask is integrated with a standard attention mechanism module. Through the Hadamard product operation, the polyline path mask is combined with the attention map used in the attention calculation to construct a polyline path mask attention module. This module constrains the attention calculation by masking, allowing the model to explicitly perceive the spatial position information of the image and enhance the ability to model the spatial correlation of pixels.

[0061] Step 105: Integrate the polyline path mask attention module into the visual transformer architecture to obtain the initial visual transformer network.

[0062] In an embodiment of the present application, the broken line path mask attention module is embedded in the visual transformer infrastructure to form an initial visual transformer network, which includes a front feature extraction layer (such as a multi-layer convolution layer), a broken line path mask attention module layer, a downsampling layer and an output layer.

[0063] Step 106: input the training data into the initial visual transformer network for training to obtain the target visual transformer network.

[0064] In an embodiment of the present application, the training data is input into the initial network, and the Adam optimizer is used for iterative training. The initial learning rate, weight decay coefficient and other parameters are set, and the learning rate is adjusted by the cosine annealing strategy until the model converges to obtain the target visual transformer network.

[0065] Step 107: input the data to be processed into the target visual transformer network to obtain image processing results, which include image classification results, image segmentation results, and target recognition results.

[0066] In this embodiment, the data to be processed is fed into a target visual transformer network for forward inference. For image classification tasks, the network outputs a probability distribution for each category, with the category with the highest probability being the predicted result. For image segmentation tasks, a segmentation mask with the same size as the input image is output, with each pixel corresponding to a predicted category. For object recognition tasks, the detection box coordinates and category label are output, and the final result is obtained after non-maximum suppression.

[0067] The embodiments of the present application solve the problem of insufficient modeling of image spatial structure by traditional visual models through the sequence modeling capability of the Mamba network and the spatial constraints of the two-dimensional broken line path mask, enhance the perception of image spatial structure, balance model efficiency and accuracy, support unified processing of multiple visual tasks, and provide an efficient solution for image analysis in complex scenes.

[0068] In some embodiments, an attention expression based on the Mamba network and a two-dimensional broken line path scanning strategy are established according to the training data, including: establishing a cyclic recursive mechanism expression of the Mamba network according to the training data and formula (1).

[0069]

[0070] Among them, x i is the pixel of the current training data, h i is the current latent variable, a i is the attenuation factor, B i is the input mapping matrix, y i is the recognition result of the current training data, C i is the output mapping matrix, h i-1 The previous hidden variable, i is the index.

[0071] Specifically, after obtaining the training data, a recursive sequence modeling mechanism of the Mamba network is established based on formula (1). The current latent variable is generated by a linear combination of the latent variable at the previous moment and the current input feature. The latent variable at the previous moment is weighted by the attenuation factor to control the degree of retention of historical information (the attenuation factor ranges from 0 to 1, and a larger value indicates more historical information is retained); the current input feature is converted to the latent variable space through the input mapping matrix to achieve dimensional matching. The current latent variable is converted into a recognition result through the output mapping matrix. This result is used to represent the semantic information of the current input feature (such as the category to which the pixel belongs, the feature vector, etc.).

[0072] Furthermore, the loop recursive mechanism expression is converted into an attention expression.

[0073] Specifically, the recursive relation is converted into a matrix operation. Based on the cumulative nature of the attenuation factors during the recursive process, a one-dimensional causal mask matrix is ​​generated. The matrix elements are the products of the attenuation factors at the corresponding positions, representing the dependency strength of each position in the sequence (only historical positions are allowed to affect the current position; future positions have zero influence).

[0074] The latent variable update and output mapping processes are integrated into the form of matrix multiplication. Through the joint operation of the mask matrix and the input-output mapping matrix, the conversion of the recursive mechanism to the attention expression is realized, so that the sequence dependency can be expressed in the form of an explicit matrix.

[0075] Furthermore, a two-dimensional broken line path scanning strategy based on the Mamba network is established.

[0076] Specifically, for any two pixels in a two-dimensional image, 1. Scan the row dimension vertically, moving from the starting row to the target row, then scan the column dimension horizontally, moving from the starting column to the target column. 2. Scan the column dimension horizontally, then scan the row dimension vertically, forming a zigzag scanning path. A multi-layer perceptron learns horizontal and vertical attenuation factors for each pixel, controlling the strength of information transfer between adjacent column pixels in the same row and adjacent row pixels in the same column, respectively (the attenuation factor for nearby pixels is close to 1, while the attenuation factor for distant pixels is close to 0).

[0077] Through the above path design and attenuation factor, the relative spatial position relationship of pixels (such as Manhattan distance) during the scanning process is explicitly encoded, avoiding the destruction of spatial structure caused by traditional one-dimensional scanning.

[0078] This embodiment of the application establishes a recursive mechanism expression for the Mamba network, leveraging the synergy between the attenuation factor and the input mapping matrix to enable the model to effectively capture long-term dependencies between pixel sequences in the training data. Compared to traditional recursive models, the Mamba network's recursive mechanism avoids the vanishing gradient problem by dynamically adjusting the transfer process of latent variables.

[0079] In some embodiments, converting the loop recursive mechanism expression into an attention expression includes: converting the loop recursive mechanism expression into an attention expression according to formula (2).

[0080]

[0081] Where X is the one-dimensional sequence obtained by scanning the current training data along the broken line path, B is the input projection matrix, and C is the output projection matrix. The joint transformation matrix of the input-output mapping, is the transposed matrix of B, L 1D is a one-dimensional causal attenuation mask, ⊙ is the Hadamard product, and Y is the output sequence.

[0082] Specifically, for the 2D images in the training data, the image pixels are converted into a one-dimensional sequence using a 2D zigzag path scanning strategy. Specifically, the image is scanned pixel by pixel along a zigzag path that either moves vertically then horizontally, or horizontally then vertically, to form a one-dimensional input sequence. This sequence preserves the relative positional relationships of pixels in 2D space (e.g., Manhattan distance), avoiding the spatial structure destruction caused by traditional row-first scanning.

[0083] Through a learnable input projection matrix, the feature dimension of each pixel in the one-dimensional input sequence is mapped to the latent variable space of the Mamba network, achieving dimensional matching between the input features and the latent variables. Through a learnable output projection matrix, the features in the latent variable space are mapped back to the output dimension for subsequent task prediction. The output projection matrix is ​​multiplied by the transpose of the input projection matrix to obtain the joint transformation matrix of the input-output mapping. This matrix encodes the linear transformation relationship from input features to output results and is used to characterize the semantic mapping rules of pixel features. For any two positions in the one-dimensional sequence, only the historical position is allowed to influence the current position (the influence weight of the future position is zero); the influence weight of the historical position is determined by the continuous product of the attenuation factors. The continuous product of the attenuation factors of the neighboring positions is close to 1, while the continuous product of the attenuation factors of the distant positions decays with increasing distance, reflecting the causal and attenuation characteristics of the sequence dependency.

[0084] Perform a Hadamard product (element-wise multiplication) on the joint transformation matrix and the one-dimensional causal attenuation mask, so that each element of the joint transformation matrix is ​​weighted by the attenuation mask to form an attention weight matrix with spatial attenuation characteristics;

[0085] The attention weight matrix is ​​multiplied by the input sequence to obtain the output sequence. This process is equivalent to the recursive state update of the Mamba network. The matrix operation explicitly expresses the dependencies between each pixel in the sequence, enabling the model to focus on key feature areas based on the attention weights.

[0086] The embodiment of the present application preserves the spatial structure of the image by scanning the input sequence through a broken line path, and explicitly encodes the attenuation law of pixel dependence through a causal attenuation mask to adapt to the long-range memory characteristics of the Mamba network; the Hadamard product operation of the joint transformation matrix and the mask enhances the model's ability to focus on target features while maintaining computational efficiency.

[0087] In some embodiments, obtaining image data to be processed, preprocessing the image to be processed, and obtaining training data and data to be processed include: obtaining image data to be processed.

[0088] Specifically, original image data containing the target scene is acquired through an image acquisition device.

[0089] Furthermore, the image data to be processed is resized, augmented, and normalized to obtain training data and data to be processed.

[0090] Specifically, the original image obtained is uniformly adjusted to a fixed size. Specifically, according to the model input requirements, a uniform image size is set (such as 224×224 pixels, 512×512 pixels, etc.); bilinear interpolation, nearest neighbor interpolation, or bicubic interpolation methods are used to proportionally scale the length and width of the original image to the target size to ensure that the image content is complete and not distorted. The image is randomly flipped horizontally, flipped vertically, rotated (such as ±15 degrees), and translated to increase the diversity of the image; the brightness, contrast, saturation, and hue of the image are adjusted to simulate image changes under different lighting conditions; partial areas of the image are randomly occluded to force the model to learn local features of the image;

[0091] Normalize the image pixel values ​​so that their distribution conforms to specific statistical characteristics. Divide all pixel values ​​by 255 and scale the pixel value range from [0, 255] to [0, 1]. Calculate the mean and standard deviation based on the statistical characteristics of the training data set, and normalize each pixel value so that its mean is 0 and its standard deviation is 1.

[0092] The processed image data is divided into a training dataset and a to-be-processed dataset. The training dataset is used for model parameter learning and contains image samples that have been augmented and normalized. The to-be-processed dataset is used for model inference testing and contains original image samples that have not been augmented but have been resized and normalized.

[0093] The embodiments of the present application ensure that all input images meet the model requirements by resizing, avoiding processing anomalies caused by size differences; data augmentation simulates various changing scenarios in actual applications, enhancing the model's adaptability to images under different conditions; normalization processing makes the input data distribution more reasonable, accelerates model convergence and improves training stability.

[0094] In the above Figure 1 Based on the above, the embodiment of the present application also provides a method for obtaining a broken line path mask in an image data processing method based on a broken line path mask attention network, such as Figure 2 As shown, Figure 2 A schematic diagram of a method for obtaining a broken line path mask in an image data processing method based on a broken line path mask attention network provided in an embodiment of the present application, including steps 201 to 205:

[0095] Step 201: Determine a horizontal attenuation factor and a vertical attenuation factor according to an attention expression.

[0096] In an embodiment of the present application, two trainable parameters are learned for each pixel in the image through a multi-layer perceptron according to the attention expression. Horizontal attenuation factor: controls the information transmission strength of adjacent column pixels in the same row, with a value range of [0, 1]. The attenuation factor of neighboring pixels is close to 1, and the attenuation factor of distant pixels decreases with increasing distance. Vertical attenuation factor: controls the information transmission strength of adjacent row pixels in the same column, and the rules are consistent with the horizontal attenuation factor. The learning process of this attenuation factor follows the causal attenuation logic in the attention expression, that is, the dependency strength between pixels is characterized by the continuous product of the attenuation factor.

[0097] Step 202 : obtaining a two-dimensional folded line path type according to a two-dimensional folded line path scanning strategy. The two-dimensional folded line path type includes a horizontal scanning type and a vertical scanning type.

[0098] In the embodiment of the present application, two basic broken line path types are defined for the pixel coordinates of a two-dimensional image. 1. Horizontal scanning type (horizontal-then-vertical path): Starting from the starting pixel, first scan along the horizontal direction to the target column, and then scan along the vertical direction to the target row; 2. Vertical scanning type (vertical-then-horizontal path): Starting from the starting pixel, first scan along the vertical direction to the target row, and then scan along the horizontal direction to the target column. The two types of paths cover the orthogonal spatial associations between pixels, ensuring that the relative positional relationship of the pixels is preserved during the scanning process.

[0099] Step 203: Calculate the horizontal attenuation factor and the horizontal scanning type to obtain the horizontal scanning weight.

[0100] In this embodiment, for any two pixels, weights are calculated along a horizontal scan path. 1. Starting from the column coordinates of the starting pixel, scan horizontally toward the target column, multiplying the horizontal attenuation factors of all pixels along the path to obtain the horizontal segment attenuation weight. 2. After reaching the target column, scan vertically toward the target row, multiplying the vertical attenuation factors of all pixels along the path to obtain the vertical segment attenuation weight. The horizontal scan weight is the product of the horizontal segment attenuation weight and the vertical segment attenuation weight.

[0101] Step 204 : Calculate the vertical attenuation factor and the vertical scanning type to obtain a vertical scanning weight.

[0102] In this embodiment, starting from the row coordinate of the starting pixel, the pixel is scanned vertically toward the target row, multiplied by the vertical attenuation factor along the path to obtain the vertical segment attenuation weight. After reaching the target row, the pixel is scanned horizontally toward the target column, multiplied by the horizontal attenuation factor along the path to obtain the horizontal segment attenuation weight. The vertical scan weight is the product of the vertical segment attenuation weight and the horizontal segment attenuation weight.

[0103] Step 205 : summing and flattening the horizontal scanning weight and the vertical scanning weight to obtain a polyline path mask.

[0104] In this embodiment of the present application, the horizontal and vertical scan weights for the same pair of pixels are summed to obtain a fused two-dimensional spatial dependency weight, forming a four-dimensional tensor. The four-dimensional tensor is flattened into a two-dimensional matrix in pixel sequence order to obtain a polyline path mask. This mask is fused with the path weights by multiplying the attenuation factor, explicitly encoding the spatial position dependency between pixels, which can be directly used to constrain the computational process of the attention mechanism.

[0105] The embodiment of the present application uses an attenuation factor multiplication mechanism to make the weight of neighboring pixels significantly higher than that of distant pixels, which is in line with the principle of local correlation of images; by generating masks through attenuation factor multiplication rather than global matrix operations, the computational complexity is controlled at a linear level, which is suitable for large-scale image data processing.

[0106] Although the present application provides method operation steps as described in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-creative work. The order of steps listed in this embodiment is only one way of executing the order of many steps and does not represent the only execution order. When the actual device or client product is executed, it can be executed in sequence or in parallel according to the method shown in this embodiment or the accompanying drawings (for example, in a parallel processor or multi-threaded processing environment).

[0107] like Figure 3 As shown, the embodiment of the present application also provides an image data processing device 300 based on a polyline path mask attention network. The device includes:

[0108] An acquisition module 301 is used to acquire initial image data, pre-process the initial image data, and obtain training data and data to be processed;

[0109] Establishing module 302, for establishing an attention expression based on the Mamba network and a two-dimensional broken line path scanning strategy according to the training data;

[0110] A processing module 303 is configured to perform an expansion process on the two-dimensional polyline path scanning strategy according to the attention expression to obtain a polyline path mask;

[0111] The processing module 303 is further used to obtain a standard attention mechanism module, fuse the broken line path mask with the standard attention mechanism module to obtain a broken line path mask attention module, where the line path mask attention module includes the spatial position of the data to be processed;

[0112] Establishing module 302, further for integrating the polyline path mask attention module into the visual transformer architecture to obtain an initial visual transformer network;

[0113] The establishment module 302 is further used to input the training data into the initial visual transformer network for training to obtain the target visual transformer network;

[0114] The processing module 303 is further used to input the data to be processed into the target visual transformer network to obtain image processing results. The image processing results include image classification results, image segmentation results and target recognition results.

[0115] In some embodiments, the establishment module 302 is further configured to establish a cyclic recursive mechanism expression of the Mamba network based on the training data and formula (1);

[0116]

[0117] Among them, x i is the pixel of the current training data, h i is the current latent variable, a i is the attenuation factor, B i is the input mapping matrix, y i is the recognition result of the current training data, C i is the output mapping matrix, H i-1 The previous hidden variable, i is the index;

[0118] Establishing module 302, further for converting the cyclic recursive mechanism expression into an attention expression;

[0119] The establishing module 302 is further used to establish a two-dimensional broken line path scanning strategy based on the Mamba network.

[0120] In some embodiments, the establishment module 302 is further configured to convert the loop recursive mechanism expression into an attention expression according to formula (2);

[0121]

[0122] Where X is the one-dimensional sequence obtained by scanning the current training data along the broken line path, B is the input projection matrix, and C is the output projection matrix. The joint transformation matrix of the input-output mapping, is the transposed matrix of B, L 1D is a one-dimensional causal attenuation mask, ⊙ is the Hadamard product, and Y is the output sequence.

[0123] In some embodiments, the acquisition module 301 is further configured to acquire image data to be processed;

[0124] The processing module 303 is further configured to resize, augment, and normalize the image data to be processed to obtain training data and data to be processed.

[0125] The processing module 303 is further configured to determine a horizontal attenuation factor and a vertical attenuation factor according to the attention expression;

[0126] The acquisition module 301 is further configured to acquire a two-dimensional folded line path type according to the two-dimensional folded line path scanning strategy, where the two-dimensional folded line path type includes a horizontal scanning type and a vertical scanning type;

[0127] The processing module 303 is further configured to calculate the horizontal attenuation factor and the transverse scanning type to obtain a transverse scanning weight;

[0128] The processing module 303 is further configured to calculate the vertical attenuation factor and the vertical scanning type to obtain a vertical scanning weight;

[0129] The processing module 303 is further configured to sum and flatten the horizontal scanning weights and the vertical scanning weights to obtain a broken line path mask.

[0130] Some modules in the apparatus described herein may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0131] The devices or modules described in the above application embodiments can be implemented by computer chips or physical devices, or by products with certain functions. For ease of description, the above devices are described separately by function in various modules. When implementing the embodiments of this application, the functions of each module can be implemented in the same or multiple software and / or hardware. Of course, a module that implements a certain function can also be implemented by combining multiple sub-modules or sub-units.

[0132] The methods, devices, or modules described in this application can be implemented in the form of computer-readable program code. The controller can be implemented in any appropriate manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to implementing the controller in the form of pure computer-readable program code, it is entirely possible to implement the same function of the controller in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the means for implementing various functions may be considered to be both a software module for implementing the method and a structure within a hardware component.

[0133] An embodiment of the present application further provides a device comprising: a processor; a memory for storing processor-executable instructions; and when the processor executes the executable instructions, the method described in the embodiment of the present application is implemented.

[0134] The embodiments of the present application also provide a non-volatile computer-readable storage medium having a computer program or instruction stored thereon. When the computer program or instruction is executed, the method described in the embodiments of the present application is implemented.

[0135] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist independently, or two or more modules may be integrated into one module.

[0136] The above-mentioned storage medium includes, but is not limited to, random access memory (RAM), read-only memory (ROM), cache, hard disk drive (HDD), or memory card. The memory can be used to store computer program instructions.

[0137] It can be seen from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary hardware. Based on this understanding, the technical solution of the present application can essentially or the part that contributes to the prior art can be embodied in the form of a software product, or it can be embodied through the implementation process of data migration. The computer software product can be stored in a storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0138] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referenced to each other. Each embodiment focuses on the differences from other embodiments. All or part of this application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, mobile communication terminals, multi-processor systems, microprocessor-based systems, programmable electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0139] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit the present application. Although the present application has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some or all of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the present application.

Claims

1. An image data processing method based on a polyline path mask attention network, characterized in that: include: Acquiring initial image data, and preprocessing the initial image data to obtain training data and data to be processed; Establishing an attention expression based on the Mamba network and a two-dimensional broken line path scanning strategy according to the training data; Expanding the two-dimensional zigzag path scanning strategy according to the attention expression to obtain a zigzag path mask; Obtain a standard attention mechanism module, and fuse the broken line path mask with the standard attention mechanism module to obtain a broken line path mask attention module, wherein the broken line path mask attention module includes the spatial position of the data to be processed; Integrating the polyline path mask attention module into the visual transformer architecture to obtain an initial visual transformer network; Inputting the training data into the initial visual transformer network for training to obtain a target visual transformer network; The data to be processed is input into a target visual transformer network to obtain an image processing result, wherein the image processing result includes an image classification result, an image segmentation result and a target recognition result.

2. The method according to claim 1, characterized in that The step of establishing an attention expression based on a Mamba network and a two-dimensional broken line path scanning strategy according to the training data includes: Establishing a cyclic recursive mechanism expression of the Mamba network according to the training data and formula (1); Among them, x i is the pixel of the current training data, h i is the current latent variable, a i is the attenuation factor, B i is the input mapping matrix, y i is the recognition result of the current training data, C i is the output mapping matrix, h i-1 The previous hidden variable, i is the index; Convert the loop recursive mechanism expression into an attention expression; A two-dimensional broken line path scanning strategy based on the Mamba network is established.

3. The method according to claim 2, characterized in that Converting the cyclic recursive mechanism expression into an attention expression includes: According to formula (2), the loop recursive mechanism expression is converted into an attention expression; Among them, X is the one-dimensional sequence obtained by scanning the current training data along the broken line path, B is the input projection matrix, and C is the output projection matrix. The joint transformation matrix of the input-output mapping, is the transposed matrix of B, L 1D is a one-dimensional causal attenuation mask, ⊙ is the Hadamard product, and Y is the output sequence.

4. The method according to claim 1, wherein The step of obtaining image data to be processed and preprocessing the image to be processed to obtain training data and data to be processed includes: Obtain image data to be processed; The image data to be processed is resized, data augmented, and normalized to obtain training data and data to be processed.

5. The method according to claim 1, wherein The step of extending the two-dimensional zigzag path scanning strategy according to the attention expression to obtain a zigzag path mask includes: Determining a horizontal attenuation factor and a vertical attenuation factor according to the attention expression; Acquiring a two-dimensional folded line path type according to the two-dimensional folded line path scanning strategy, wherein the two-dimensional folded line path type includes a horizontal scanning type and a vertical scanning type; Calculating the horizontal attenuation factor and the horizontal scanning type to obtain a horizontal scanning weight; Calculating the vertical attenuation factor and the vertical scanning type to obtain a vertical scanning weight; The horizontal scanning weight and the vertical scanning weight are summed and flattened to obtain a broken line path mask.

6. The method according to claim 1, characterized in that The target visual transformer network includes a front feature extraction layer, a broken line path mask attention module layer, a downsampling layer and an output layer.

7. The method according to claim 1, characterized in that The amount of the training data is 1.35 million, and the amount of the data to be processed is 50,000.

8. An image data processing device based on a polyline path mask attention network, characterized in that: include: An acquisition module is used to acquire initial image data, pre-process the initial image data, and obtain training data and data to be processed; An establishment module is used to establish an attention expression based on the Mamba network and a two-dimensional broken line path scanning strategy according to the training data; a processing module, configured to perform an expansion process on the two-dimensional zigzag path scanning strategy according to the attention expression to obtain a zigzag path mask; The processing module is further configured to obtain a standard attention mechanism module, fuse the broken line path mask with the standard attention mechanism module to obtain a broken line path mask attention module, wherein the broken line path mask attention module includes the spatial position of the data to be processed; The building module is further configured to integrate the polyline path mask attention module into the visual transformer architecture to obtain an initial visual transformer network; The establishment module is further configured to input the training data into the initial visual transformer network for training to obtain a target visual transformer network; The processing module is further used to input the data to be processed into the target visual transformer network to obtain image processing results, which include image classification results, image segmentation results and target recognition results.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images