Scene text detection method based on VMamba framework and channel-spatial attention mechanism

Abstract

The present application relates to a scene text detection method, device and computer equipment based on VMamba framework and channel-spatial attention mechanism. The method comprises: obtaining a scene image reference sample data set and an enhanced sample data set containing text; constructing a deformable VMamba feature encoder based on the VMamba framework using deformable convolution to obtain multi-scale text instance feature maps corresponding to each scene image; obtaining a balanced feature map using interpolation and maximum pooling methods; constructing a GASM module based on the channel-spatial attention mechanism to obtain pixel-by-pixel enhanced text instance features; constructing a Fourier feature decoder to obtain a reconstructed text instance contour; and using a loss function to train a scene text detection model with an enhanced sample data set as input and a text instance contour as output. The present method can realize scene text detection with a global receptive field, fast convergence and precise positioning.

Description

Technical Field

[0001] The present application relates to the technical field of scene text detection and target detection, and in particular to a scene text detection method, device and computer equipment based on VMamba framework and channel-space attention mechanism. Background Art

[0002] Text detection can be regarded as a special form of object detection and is widely used in fields such as text information extraction and advanced driver assistance systems. However, the diversity of text shapes, complex backgrounds, and changing lighting conditions in real scenes often lead to loss of image details, posing a major challenge to text detection. Traditional methods rely on salient text features and manually design features through techniques such as connected component analysis or sliding windows, but it is difficult to capture contextual information in natural scenes and has limited processing capabilities for multi-directional or arbitrarily shaped text.

[0003] In recent years, researchers have proposed a variety of text detection methods. Although there have been some improvements in the scene text detection problem, there are still certain problems with the detection of scene text of arbitrary shapes, including the scarcity of scene text of arbitrary shapes, which leads to insufficient training data for ordinary scene text detection models; the visual infrastructure used by traditional scene text detection models lacks the ability to model long-range dependencies, or has quadratic computational complexity, cannot be expanded, and has a slow convergence speed; the cross activation mechanism in the VMamba framework has encountered limitations in capturing local information of complex text; and focusing only on single-dimensional features will hinder the representation ability of scene text detection models of arbitrary shapes. Summary of the invention

[0004] Based on this, it is necessary to provide a scene text detection method, device and computer equipment based on the VMamba framework and channel-space attention mechanism that can have a global receptive field, fast convergence and precise positioning to address the above technical problems.

[0005] In a first aspect, the present application provides a scene text detection method based on a VMamba framework and a channel-space attention mechanism, the method being used to detect text in a scene, comprising:

[0006] Step S1, obtaining a scene image reference sample dataset containing text, applying a data enhancement strategy to the reference sample dataset, and generating an enhanced sample dataset;

[0007] Step S2, constructing a deformable VMamba feature encoder based on the VMamba framework using deformable convolution, applying the deformable VMamba feature encoder to process each scene image in the enhanced sample data set, and obtaining a multi-scale text instance feature map corresponding to each scene image;

[0008] Step S3, using interpolation and maximum pooling methods to obtain a balanced feature map according to the multi-scale text instance feature map; constructing a GASM module based on the channel-space attention mechanism to perform channel-space attention calculation and channel shuffling on the balanced feature map to obtain the text instance features after pixel-by-pixel enhancement;

[0009] Step S4, constructing a Fourier feature decoder, which is used to obtain a pixel-by-pixel text area mask image and a Fourier feature vector according to the text instance features, and using an inverse discrete Fourier transform to obtain a reconstructed text instance contour; from the input to the output direction, the deformable VMamba feature encoder, the GASM module and the Fourier feature decoder constitute a network structure to be trained, and the loss function is used to train the network structure to be trained to generate a scene text detection model with an enhanced sample data set as input and a text instance contour as output.

[0010] In one embodiment, applying a data enhancement strategy to a reference sample dataset in step S1 to generate an enhanced sample dataset includes:

[0011] The reference sample dataset is randomly cropped, randomly horizontally flipped, and randomly rotated, and the brightness, saturation, and contrast are adjusted to generate an enhanced sample dataset.

[0012] In one embodiment, step S2 comprises:

[0013] Step S201, using the Stem module to divide each scene image in the enhanced sample data set input into the deformable VMamba feature encoder into image patches of a specified size;

[0014] Step S202, sequentially input the image patch into the first stage, the second stage, the third stage, and the fourth stage of the deformable VMamba feature encoder, wherein the first stage extracts and outputs a first-size feature map, the second stage extracts and outputs a second-size feature map, the third stage extracts and outputs a third-size feature map, and the fourth stage extracts and outputs a fourth-size feature map;

[0015] Step S203: input the second size feature map, the third size feature map, and the fourth size feature map into the target deformable convolution, and output a multi-scale text instance feature map.

[0016] In one embodiment, step S203 includes:

[0017] The second-size feature map, the third-size feature map, and the fourth-size feature map are input into the deformable convolution with a convolution kernel size of 3×3, and the text instance feature maps of the corresponding scales output by the deformable VMamba feature encoder are obtained respectively, where the deformable convolution is shown in the following formula:

[0018] ;

[0019] in, Represents the features of the input feature map at position p, Represents the feature of the output feature map corresponding to the input feature map at position p, Indicates k The weights of the sampling positions, Indicates k The preset offset of the sampling position, Indicates k The learnable offset of the sampling positions, Indicates k A learnable modulation scalar at each sampling position.

[0020] In one embodiment, step S3 comprises:

[0021] Balanced semantic features are obtained based on multi-scale text instance feature maps through interpolation and maximum pooling methods;

[0022] The embedded Gaussian non-local attention mechanism is used to refine the balanced semantic features and obtain the balanced feature map;

[0023] The GASM module is built based on the channel-spatial attention mechanism, and the channel dimension in the balanced feature map is arranged to the last dimension; the channel attention map is generated through the fully connected layer and the ReLU activation function, and the balanced feature map and the channel attention map are multiplied pixel by pixel to obtain the channel-enhanced feature map;

[0024] Divide the channel-enhanced feature map into a target number of groups, where the number of channels in each group is determined by dividing the total number of channels by the number of groups, and divide the channels in each group into at least one subgroup, and combine different subgroups in each group by disrupting the order of channels in each group, and restore the disrupted feature map to its original shape to obtain a shuffled feature map;

[0025] The shuffled feature map is convolved, and batch normalization and ReLU activation function are used to generate a spatial attention map. The shuffled feature map is multiplied pixel by pixel with the spatial attention map to output the pixel-by-pixel enhanced text instance features.

[0026] In one embodiment, a channel attention map is generated by a fully connected layer and a ReLU activation function, and the balanced feature map and the channel attention map are multiplied pixel by pixel to obtain a channel-enhanced feature map including:

[0027] The number of input channels of the balanced feature map is reduced through the fully connected layer, and the complex relationship between channels is modeled using the ReLU activation function. The number of input channels is increased again to generate a channel attention map.

[0028] Multiply the balanced feature map and the channel attention map pixel by pixel to obtain the channel enhanced feature map. The formula is as follows:

[0029] ;

[0030] ;

[0031] in, ω c is the channel attention map, is the feature map after channel enhancement, for function, represents pixel-by-pixel multiplication, is the original input feature map, is the fully connected layer, is the transpose operation.

[0032] In one embodiment, a convolution operation is performed on the shuffled feature map, and a batch normalization and ReLU activation function are used to generate a spatial attention map. The shuffled feature map is multiplied pixel by pixel with the spatial attention map, and the output pixel-by-pixel enhanced text instance features include:

[0033] The shuffled feature map is input into the first 7×7 convolutional layer to reduce the number of channels of the shuffled feature map;

[0034] Batch normalization and ReLU activation function are used to stabilize the training process and perform nonlinear transformations to model complex relationships in space. The number of channels of the feature map is restored to the shuffled size through the second 7×7 convolutional layer to generate a spatial attention map.

[0035] Multiply the shuffled feature map and the spatial attention map pixel by pixel to obtain the pixel-by-pixel enhanced text instance features output by the GASM module. The formula is as follows:

[0036] ;

[0037] ;

[0038] in, ω s is the spatial attention map, It is the feature map output by the GASM module. for function, represents pixel-by-pixel multiplication, is the shuffled feature map, For batch normalization, is the convolutional layer.

[0039] In one embodiment, step S4 comprises:

[0040] Convolution is performed on the pixel-by-pixel enhanced text instance features to obtain the classification result for predicting whether each pixel belongs to the text area and the regression result for predicting the Fourier series coefficient of each pixel;

[0041] The classification results are normalized to obtain the probability distribution of the identified text area and the probability distribution of the identified text center area; the text area score and the text center area score are weighted multiplied to obtain a pixel-by-pixel classification score map, and a binarization operation is performed to obtain a text area mask map;

[0042] The Fourier feature vector corresponding to the text area in the text area mask image is extracted through thresholding operation, and the reconstructed text instance contour is obtained through inverse discrete Fourier transform;

[0043] The loss function based on text instance category and Fourier feature vector is used to train the network structure composed of deformable VMamba feature encoder, GASM module and Fourier feature decoder from input to output to obtain the scene text detection model and output the predicted text instance contour.

[0044] In the second aspect, the present application also provides a scene text detection device based on the VMamba framework and the channel-space attention mechanism, including:

[0045] The sample augmentation module is used to obtain a reference sample dataset of scene images containing text, apply a data augmentation strategy to the reference sample dataset, and generate an enhanced sample dataset;

[0046] A feature encoder construction module is used to construct a deformable VMamba feature encoder based on the VMamba framework using deformable convolution, and the deformable VMamba feature encoder is used to process each scene image in the enhanced sample data set to obtain a multi-scale text instance feature map corresponding to each scene image;

[0047] The GASM module constructs a module for obtaining a balanced feature map based on the multi-scale text instance feature map by using interpolation and maximum pooling methods; the GASM module is constructed based on the channel-spatial attention mechanism to perform channel-spatial attention calculation and channel shuffling on the balanced feature map to obtain the text instance features after pixel-by-pixel enhancement;

[0048] The scene text detection model generation module is used to construct a Fourier feature decoder, which is used to obtain a pixel-by-pixel text area mask and Fourier feature vector based on the text instance features, and use the discrete Fourier inverse transform to obtain the reconstructed text instance contour; from the input to the output direction, the deformable VMamba feature encoder, the GASM module and the Fourier feature decoder constitute the network structure to be trained, and the loss function is used to train the network structure to generate a scene text detection model with the enhanced sample data set as input and the text instance contour as output.

[0049] In a third aspect, the present application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of a scene text detection method based on a VMamba framework and a channel-space attention mechanism when executing the computer program.

[0050] The scene text detection method, device and computer equipment based on the VMamba framework and the channel-space attention mechanism obtain a reference sample data set of scene images containing text, apply a data enhancement strategy to the reference sample data set, and generate an enhanced sample data set; in terms of target sample augmentation, by introducing a data enhancement strategy, the model is helped to better learn the inherent laws of the data, and the problems of overfitting and poor robustness of the model are effectively solved; based on the VMamba framework, a deformable VMamba feature encoder is constructed by using deformable convolution, and the deformable VMamba feature encoder is applied to process each scene image in the enhanced sample data set to obtain a multi-scale text instance feature map corresponding to each scene image; in terms of feature encoding, by introducing the visual basic framework VMamba based on the state-space model, the problem of lack of global perspective or secondary computational complexity of the traditional visual infrastructure is effectively solved, and at the same time, by combining deformable convolution, VMamba is adaptively modified in the field of scene text detection, so that the model can appropriately focus on the local foreground receptive field while maintaining the cross-shaped activation; interpolation and maximum pooling methods are used to implement the multi-scale text instance feature map according to the multi-scale text instance. A balanced feature map is obtained by converting the feature map into a GASM module based on the channel-space attention mechanism, which is used to perform channel-space attention calculation and channel shuffling on the balanced feature map to obtain the text instance features after pixel-by-pixel enhancement. In terms of feature enhancement, the irregular distribution of arbitrary-shaped text instances in natural scenes is utilized to construct a GASM module to perform channel-space enhancement and channel shuffling on the feature map, thereby realizing adaptive feature refinement in the channel and spatial dimensions. A Fourier feature decoder is constructed to obtain a pixel-by-pixel text region mask map and Fourier feature vector according to the text instance features, and the inverse discrete Fourier transform is used to obtain the reconstructed text instance contour. From the input to the output direction, the deformable VMamba feature encoder, the GASM module and the Fourier feature decoder constitute the network structure to be trained, and the loss function is used to train the network structure to be trained to generate a scene text detection model with the enhanced sample data set as input and the text instance contour as output. In terms of feature decoding, the conversion of the text instance contour from the spatial domain to the Fourier domain is realized through the construction of the Fourier feature vector, which can better distinguish the contour boundary of the text instance of arbitrary shape. At the same time, the features obtained from the deformable VMamba feature encoder have both global and local information, and the features enhanced by the GASM module can better adapt to the characteristics of complex backgrounds, thereby improving the detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a flowchart of a scene text detection method based on VMamba framework and channel-space attention mechanism in one embodiment;

[0052] Figure 2 is an overall framework diagram of a scene text detection model in one embodiment;

[0053] Figure 3 is a schematic diagram of a deformable VMamba feature encoder in one embodiment;

[0054] Figure 4 is a schematic diagram of a GASM module in one embodiment;

[0055] Figure 5 Schematic diagram of a Fourier feature decoder in one embodiment. DETAILED DESCRIPTION

[0056] Text detection can be regarded as a special form of target detection and is widely used in fields such as text information extraction and advanced driver assistance systems. However, the diversity of text shapes (such as curved, tilted or diverse fonts), complex backgrounds and changing lighting conditions in real scenes often lead to loss of image details, posing a major challenge to text detection. Traditional methods rely on salient text features and manually design features through techniques such as connected component analysis (CCA) or sliding windows (SW), but it is difficult to capture contextual information in natural scenes and has limited processing capabilities for multi-directional or arbitrarily shaped text.

[0057] Since 2016, with the development of deep learning in object detection and semantic segmentation, research has gradually applied it to text detection, deriving two major methods: regression-based and segmentation-based. Regression-based methods regard text detection as a task of predicting text region bounding boxes or quadrilaterals, but have limited efficiency when modeling complex geometric layouts or highly curved text. Researchers proposed SLPR (A deep learning based plate recognition framework, a text detection algorithm) to improve irregular text detection and QuadBox to introduce quadrilateral prediction to deal with tilted text. It should be pointed out that neither SLPR nor QuadBox solves the problem of uneven detection caused by different text sizes. To this end, the MSR (multi-scale shape regression) proposed by researchers improves the detection ability of small texts through multi-scale features. Segmentation-based methods detect text regions pixel by pixel, but may lack global coherence. TextMountain and MOST (A Multi-Oriented Scene Text Detector with Localization Refinement, a text detection algorithm) combine segmentation and regression methods to improve detection performance, but perform poorly in dense scenes. To this end, PixelLink achieves text instance separation through pixel-level links. Although these methods have made breakthroughs in horizontal and multi-directional text detection, there are still significant challenges in dealing with the global homogeneity and highly diverse local features of arbitrarily shaped text.

[0058] In recent years, with the increasing complexity of the geometric layout of text in natural scenes, the research focus has shifted to the detection of arbitrary-shaped text, mainly involving text shape modeling and visual infrastructure optimization. In text shape modeling, researchers have proposed adaptive fitting of arbitrary-direction and curved text through parameterized Bezier curves, and some have performed geometric modeling between shape space and parameter space through a bidirectional mapping mechanism to uniformly represent the geometric layout of text. However, these methods rely on spatial domain modeling, and there are problems such as complex post-processing or poor adaptability to highly curved text. To this end, some people model text contours in the Fourier domain through Fourier transform, which improves modeling accuracy and computational efficiency. In terms of visual infrastructure, convolutional neural networks (CNNs) perform well in capturing local features, but lack the ability to model long-range dependencies. Vision transformers (ViTs) provide a global perspective through a self-attention mechanism, but have high computational complexity. Researchers have also proposed a state space model (SSM) that shows great potential in modeling long-term dependencies and is superior to ViTs in scalability. In order to overcome the limitations of SSM in processing visual data, researchers proposed the VMamba model, which successfully extended SSM to the visual field containing spatial information by introducing the selective scanning 2D (SS2D, 2D-Selective-Scan) module. This method not only retains the ability to model long-term dependencies, but also significantly reduces the computational cost.

[0059] Although the above text detection methods have improved the scene text detection problem, there are still certain problems with the detection of scene text of arbitrary shapes: the scarcity of scene text of arbitrary shapes will lead to insufficient training data for ordinary scene text detection models; the visual infrastructure used by existing scene text detection models lacks the ability to model long-range dependencies, or has quadratic computational complexity, cannot be expanded, and has a slow convergence speed; the cross activation mechanism in the VMamba framework has encountered limitations in capturing local information of complex text; focusing only on single-dimensional features will hinder the representation ability of scene text detection models of arbitrary shapes.

[0060] The purpose of this application is to overcome the problems in the field of scene text detection, such as the scarcity of scene texts of arbitrary shapes, resulting in insufficient training data; the visual infrastructure used by traditional scene text detection models lacks the ability to model long-range dependencies, or has quadratic computational complexity, cannot be expanded, and has a slow convergence speed; the cross activation mechanism in the VMamba framework has encountered limitations in capturing local information of complex texts; and only focusing on single-dimensional features will hinder the representation ability of scene text detection models of arbitrary shapes. A scene text detection method based on the VMamba framework and a channel-space attention mechanism is provided.

[0061] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0062] In an exemplary embodiment, Figure 1 As shown, a scene text detection method based on the VMamba framework and the channel-space attention mechanism is provided, including:

[0063] Step S1, obtaining a scene image reference sample dataset containing text, applying a data enhancement strategy to the reference sample dataset, and generating an enhanced sample dataset;

[0064] Exemplarily, by applying data augmentation strategies, including random cropping, random rotation, random horizontal flipping, color jittering, and contrast jittering, multiple transformed samples, i.e., enhanced sample datasets, are generated; these enhanced samples are used together with the annotated data for model training, and the metadata is used as auxiliary information for subsequent debugging, evaluation, and result mapping to achieve the purpose of sample augmentation;

[0065] Step S2, constructing a deformable VMamba feature encoder based on the VMamba framework using deformable convolution, applying the deformable VMamba feature encoder to process each scene image in the enhanced sample data set, and obtaining a multi-scale text instance feature map corresponding to each scene image;

[0066] Exemplarily, for the data obtained by the data enhancement strategy in step S1, in view of the limitations of insufficient fitting ability or excessive computational overhead of the traditional visual basic framework, based on the VMamba framework, according to the global homogeneity and locality of the text instance, the VMamba framework is optimized by deformable convolution to emphasize the limitations of the cross activation, a deformable VMamba feature encoder is constructed, and an output multi-scale text instance feature map is obtained;

[0067] Step S3, using interpolation and maximum pooling methods to obtain a balanced feature map according to the multi-scale text instance feature map; constructing a GASM module based on the channel-space attention mechanism to perform channel-space attention calculation and channel shuffling on the balanced feature map to obtain the text instance features after pixel-by-pixel enhancement;

[0068] Exemplarily, in order to further improve the detection accuracy of scene text instances and make full use of the perception of global and local information by the deformable VMamba feature encoder, a GASM module based on the channel-space attention mechanism is designed to enhance the features pixel by pixel. After the feature map is obtained in step S2, the information flow is balanced by interpolation and maximum pooling. According to the irregular distribution of arbitrary shape text instances in natural scenes, channel-space attention calculation and channel shuffling are performed on the balanced features, that is, the balanced feature map, to obtain the text instance features after pixel-by-pixel enhancement.

[0069] Step S4, constructing a Fourier feature decoder, which is used to obtain a pixel-by-pixel text area mask image and a Fourier feature vector according to the text instance features, and using an inverse discrete Fourier transform to obtain a reconstructed text instance contour; from the input to the output direction, the deformable VMamba feature encoder, the GASM module and the Fourier feature decoder constitute a network structure to be trained, and the loss function is used to train the network structure to be trained to generate a scene text detection model with an enhanced sample data set as input and a text instance contour as output.

[0070] Exemplarily, based on the result of the GASM module in step S3, the pixel-by-pixel text region mask and Fourier feature vector are obtained in sequence through the Fourier feature decoder, and the reconstructed text instance contour is obtained using the inverse discrete Fourier transform. In addition, by designing a loss function, the classification and regression results of the text instance are trained, and finally a scene text detection model with a global receptive field, fast convergence and precise positioning is obtained.

[0071] Specifically, the overall framework of the scene text detection model in the scene text detection method based on the VMamba framework and the channel-space attention mechanism is as follows: Figure 2 As shown in the figure, firstly, the image is input as the reference sample data set, and the data enhancement strategy is adopted for the reference sample data set. After the deformable VMamba feature encoder, feature enhancement module and Fourier feature decoder, the result is output. The Fourier feature decoder includes obtaining the text area mask map by normalization and obtaining the Fourier feature vector containing the real part and the imaginary part.

[0072] In the above-mentioned scene text detection method based on VMamba framework and channel-space attention mechanism, in terms of target sample augmentation, the data enhancement strategy is introduced to help the scene text detection model better learn the inherent laws of the data, and effectively solve the problems of model overfitting and poor robustness; in terms of feature encoding, the problem of lack of global perspective or quadratic computational complexity of the traditional visual infrastructure is effectively solved by introducing VMamba, a visual basic framework based on the state-space model, and adaptively modifying VMamba in the field of scene text detection by combining deformable convolution, so that the scene text detection model can properly focus on the local foreground receptive field while maintaining the cross-shaped activation; in terms of feature enhancement, the GASM module is constructed by utilizing the irregular distribution characteristics of arbitrary shape text instances in natural scenes, and the feature map is channel-space enhanced and channel shuffled, thereby realizing adaptive feature refinement in the channel and spatial dimensions; in terms of feature decoding, the text instance contour is transformed from the spatial domain to the Fourier domain through the construction of the Fourier feature vector, which can better distinguish the contour boundary of the arbitrary shape text instance. At the same time, the features obtained from the deformable VMamba feature encoder have both global and local information, and the features enhanced by the GASM module can better adapt to the characteristics of complex backgrounds, thereby improving the detection accuracy.

[0073] In an exemplary embodiment, applying a data enhancement strategy to the reference sample dataset in step S1 to generate an enhanced sample dataset includes:

[0074] The reference sample dataset is randomly cropped, randomly horizontally flipped, and randomly rotated, and the brightness, saturation, and contrast are adjusted to generate an enhanced sample dataset.

[0075] Specifically, step S1 includes:

[0076] Step S101, the training data set used in this application is CTW1500, Total-Text, ICDAR2015, which contains 3255 different scene text images, of which 2255 images are annotated with polygons and 1000 images are annotated with quadrilaterals, which are the reference sample data sets;

[0077] Step S102, using the built-in LoadImageFromFile method of the mmdet framework, reads the image data set in RGB format, ignores the direction information, and loads the annotation data of the image, including polygon annotations, polygon borders, and classification labels;

[0078] Step S103, randomly cropping the image read in step S102 to 800×800 pixels, maintaining the aspect ratio, adjusting the ratio range to 0.75 to 2.5 times, then applying a random cropping operation, the cropping ratio is 0.5, and maintaining the minimum area ratio of the cropped area not less than 20% by iterating once; after cropping, randomly flipping the image horizontally, with a flipping probability of 0.5;

[0079] Step S104, then performing a random enhancement operation on the image returned in step S103, including:

[0080] Center cropping is applied randomly with a minimum edge ratio of 0.3 and an execution probability of 0.8;

[0081] Randomly apply rotation operations with a maximum rotation angle of 30°, use canvas expansion to ensure the image is intact after rotation, and execute with a probability of 0.5;

[0082] Randomly select two different size scaling operations, namely 800×800 pixels with aspect ratio preservation and non-aspect ratio preservation, with execution probabilities of 0.6 and 0.4;

[0083] Step S105, applying the ColorJitter method built into the mmdet framework to the image randomly enhanced in step S104 to adjust the brightness, saturation and contrast to increase the diversity of the image data, wherein the brightness variation range is 32 / 255, and the saturation and contrast variation ratio is 0.5, and the data-enhanced image, the corresponding annotation data and its metadata (including image path, original size, adjusted size and scaling factor) are packaged together, wherein the enhanced image is the enhanced sample data set, which is used together with the annotation data for model training, wherein the enhanced sample data set is used as input feature, and the annotation data provides supervision information; and the metadata is used as auxiliary information for subsequent debugging, evaluation and result mapping, thereby achieving the purpose of target sample augmentation.

[0084] In an exemplary embodiment, step S2 includes:

[0085] Step S201, using the Stem module to divide each scene image in the enhanced sample data set input into the deformable VMamba feature encoder into image patches of a specified size;

[0086] Step S202, sequentially input the image patch into the first stage, the second stage, the third stage, and the fourth stage of the deformable VMamba feature encoder, wherein the first stage extracts and outputs a first-size feature map, the second stage extracts and outputs a second-size feature map, the third stage extracts and outputs a third-size feature map, and the fourth stage extracts and outputs a fourth-size feature map;

[0087] Step S203: input the second size feature map, the third size feature map, and the fourth size feature map into the target deformable convolution, and output a multi-scale text instance feature map.

[0088] In an exemplary embodiment, step S203 includes:

[0089] The second-size feature map, the third-size feature map, and the fourth-size feature map are input into the deformable convolution with a convolution kernel size of 3×3, and the text instance feature maps of the corresponding scales output by the deformable VMamba feature encoder are obtained respectively, where the deformable convolution is shown in the following formula:

[0090] ;

[0091] in, Represents the features of the input feature map at position p, Represents the feature of the output feature map corresponding to the input feature map at position p, Indicates k The weights of the sampling positions, Indicates k The preset offset of the sampling position, Indicates k The learnable offset of the sampling positions, Indicates k A learnable modulation scalar at each sampling position.

[0092] Exemplarily, step S2 is specifically:

[0093] Step S201, sending the enhanced sample image generated in step S1 to the deformable VMamba encoder to extract text features, that is, sending each scene image in the enhanced sample data set to the deformable VMamba encoder, where the enhanced sample data set is used to extract text features. Since directly processing two-dimensional image data will result in high computational costs, the encoder uses a Stem module composed of 4×4 convolutions to divide the input image into 4×4 patches, so that the subsequent VSS Block module can extract features;

[0094] Step S202, the image patch obtained in step S201 is sequentially input into Stage 1-4 of VMamba, where Stage 1 is the first stage, Stage 2 is the second stage, Stage 3 is the third stage, and Stage 4 is the fourth stage. Except for the first stage, each stage is composed of a downsampling and several VSSBlock modules. The following is the process in each stage:

[0095] In Stage 1, the input feature map is 96×200×200 and passes through two VSS Block modules. The structure of each VSS Block module is as follows:

[0096] Layer normalization,

[0097] SiLU activation function,

[0098] The convolution kernel size is 3×3 and the depth of the convolution is 1.

[0099] Selective scanning 2D module,

[0100] Layer normalization,

[0101] SiLU activation function,

[0102] And the input skip connection is added,

[0103] The final output size is 96×200×200 feature map, that is, the first size feature map;

[0104] In Stage2, the input feature map of size 96×200×200 is first downsampled to 192×100×100. Except for the difference between the input size and the output size, the rest is the same as Stage1, that is, after two VSS Block modules, the final output feature map of size 192×100×100 is the second size feature map;

[0105] In Stage3, the input feature map of size 192×100×100 is first downsampled to 384×50×50, and then passed through eight VSS Block modules. Except for the input size, output size and number of VSS Block modules, the rest are the same as Stage2. The final output feature map of size 384×50×50 is the feature map of the third size.

[0106] In Stage 4, the input feature map of size 384×50×50 is first downsampled to 768×25×25, and then passes through two VSS Block modules. Except for the input size, output size and number of VSS Block modules, the rest are the same as Stage 3. The final output feature map of size 768×25×25 is the fourth size feature map.

[0107] In step S203, considering the global homogeneity and locality of text instances, and the limitations of the cross activation mechanism in the VMamba framework in capturing complex local text information, the deformable convolution (DConv) method shown in the following formula is used to enable the model to focus on local areas more effectively, thereby having the ability to focus on the local foreground receptive field:

[0108] ;

[0109] in and Represent the features of the input feature map and the output feature map at position p, respectively. and Respectively represent k The weights and preset offsets of the sampling positions, and Respectively represent k Learnable offset and modulation scalar for each sampling position;

[0110] Further, it is observed that Stage 1 usually contains more low-level edge and texture information and has a low correlation with high-level semantic features. Therefore, not using deformable convolution (DConv) in this layer helps improve computational efficiency without significantly affecting detection performance. The multi-scale feature maps extracted from Stage 2-4 in step S202, i.e., the second size feature map, the third size feature map, and the fourth size feature map are input into the DConv with a convolution kernel size of 3×3, thereby obtaining the multi-scale text instance feature map output by the deformable VMamba feature encoder.

[0111] Specifically, the deformable VMamba feature encoder is as follows: Figure 3 As shown, after the Stem module and two VSS Block modules, the first-size feature map is obtained, after downsampling and two VSS Block modules, the second-size feature map is obtained and input into DConv with a convolution kernel size of 3×3, after downsampling and eight VSS Block modules, the third-size feature map is obtained and input into DConv with a convolution kernel size of 3×3, and after downsampling and two VSS Block modules, the fourth-size feature map is obtained and input into DConv with a convolution kernel size of 3×3.

[0112] In an exemplary embodiment, step S3 includes:

[0113] Balanced semantic features are obtained based on multi-scale text instance feature maps through interpolation and maximum pooling methods;

[0114] The embedded Gaussian non-local attention mechanism is used to refine the balanced semantic features and obtain the balanced feature map;

[0115] The GASM module is built based on the channel-spatial attention mechanism, and the channel dimension in the balanced feature map is arranged to the last dimension; the channel attention map is generated through the fully connected layer and the ReLU activation function, and the balanced feature map and the channel attention map are multiplied pixel by pixel to obtain the channel-enhanced feature map;

[0116] Divide the channel-enhanced feature map into a target number of groups, where the number of channels in each group is determined by dividing the total number of channels by the number of groups, and divide the channels in each group into at least one subgroup, and combine different subgroups in each group by disrupting the order of channels in each group, and restore the disrupted feature map to its original shape to obtain a shuffled feature map;

[0117] The shuffled feature map is convolved, and batch normalization and ReLU activation function are used to generate a spatial attention map. The shuffled feature map is multiplied pixel by pixel with the spatial attention map to output the pixel-by-pixel enhanced text instance features.

[0118] In an exemplary embodiment, a channel attention map is generated by a fully connected layer and a ReLU activation function, and the balanced feature map and the channel attention map are multiplied pixel by pixel to obtain a channel-enhanced feature map including:

[0119] The number of input channels of the balanced feature map is reduced through the fully connected layer, and the complex relationship between channels is modeled using the ReLU activation function. The number of input channels is increased again to generate a channel attention map.

[0120] Multiply the balanced feature map and the channel attention map pixel by pixel to obtain the channel enhanced feature map. The formula is as follows:

[0121] ;

[0122] ;

[0123] in, ω c is the channel attention map, is the feature map after channel enhancement, for function, represents pixel-by-pixel multiplication, is the original input feature map, is the fully connected layer, is the transpose operation.

[0124] In an exemplary embodiment, a convolution operation is performed on the shuffled feature map, and a batch normalization and ReLU activation function are used to generate a spatial attention map. The shuffled feature map is multiplied pixel by pixel with the spatial attention map, and the output pixel-by-pixel enhanced text instance features include:

[0125] The shuffled feature map is input into the first 7×7 convolutional layer to reduce the number of channels of the shuffled feature map;

[0126] Batch normalization and ReLU activation function are used to stabilize the training process and perform nonlinear transformations to model complex relationships in space. The number of channels of the feature map is restored to the shuffled size through the second 7×7 convolutional layer to generate a spatial attention map.

[0127] Multiply the shuffled feature map and the spatial attention map pixel by pixel to obtain the pixel-by-pixel enhanced text instance features output by the GASM module. The formula is as follows:

[0128] ;

[0129] ;

[0130] in, ω s is the spatial attention map, It is the feature map output by the GASM module. for function, represents pixel-by-pixel multiplication, is the shuffled feature map, For batch normalization, is the convolutional layer.

[0131] Exemplarily, step S3 is specifically:

[0132] Step S301, after being encoded by the deformable VMamba feature encoder, the low-level and high-level information of the text instance present complementary characteristics. Therefore, in order to make full use of the deformable VMamba feature encoder's perception of global and local information, the best information integration method should ensure that the integrated features have balanced low-level and high-level information at each resolution. The multi-scale text instance feature map obtained in step S2 is scaled to the size of the intermediate layer by interpolation and maximum pooling. Optionally, the intermediate layer is the third-size feature map. The second-size feature map is interpolated to the size of the third-size feature map, and the fourth-size feature map is converted to the size of the third-size feature map by maximum pooling.

[0133] Furthermore, the balanced semantic features are obtained by the following formula:

[0134] ;

[0135] in, C represents balanced semantic features, L represents the number of multi-level features, represents the resolution level of the feature, and 、 are the lowest and highest level feature indices, respectively;

[0136] Step S302, the balanced semantic features obtained in step S301 are further refined using the embedded Gaussian non-local attention mechanism, and rescaled using the same but reverse process, that is, the balanced semantic features of the third-size feature map size are interpolated to the fourth-size feature map size, and the balanced semantic features of the third-size feature map size are converted to the second-size feature map size using the maximum pooling method. Through the above method, the features from the low layer to the high layer are simultaneously fused to obtain the balanced features, that is, the balanced feature map;

[0137] Step S303, in order to further improve the detection accuracy of scene text instances, based on the fact that arbitrary-shaped text instances are irregularly distributed in natural scenes, a GASM module is constructed based on channel-spatial attention, and the dimensions of the balanced feature map output in step S302 are rearranged so that the channel dimension is located at the last dimension. The number of input channels is increased from 1 to 1 through the fully connected layer. Dimensionality reduction to , and then use the ReLU activation function to model the complex relationship between channels, and finally reduce the number of channels from Re-upgrade to , and restore the dimension position, exemplarily, r=4, to generate an adaptive pixel-by-pixel channel weight, i.e., a channel attention map; the input feature map and the channel attention map are multiplied pixel by pixel to obtain an enhanced feature map, as shown in the following formula:

[0138] ;

[0139] ;

[0140] in, ω c is the channel attention map, is the feature map after channel enhancement, for function, represents pixel-by-pixel multiplication, is the original input feature map, is the fully connected layer, is the transpose operation;

[0141] Step S304: After channel attention enhancement, there may still be a problem that the information between channels is not fully mixed. In order to give full play to the effects of channel attention and spatial attention, channel shuffling is introduced to divide the channel enhanced feature map obtained in step S303 into several groups. For example, when divided into 4 groups, each group contains Number of channels, is the total number of channels, and the channels in each group are divided into several subgroups. The order of channels in each group is disrupted to combine different subgroups in each group, and finally the disrupted feature map is restored to its original shape to obtain the shuffled feature map, as shown in the following formula:

[0142] ;

[0143] in, is the shuffled feature map, Shuffle operations for channels;

[0144] Step S305, relying solely on channel attention and channel shuffling may not fully utilize the spatial dimension information. In order to enable the model to capture the important details of the text in the feature map. Further applying spatial attention, the shuffled feature map obtained in step S304 is input into a 7×7 convolution layer, that is, the first 7×7 convolution layer, reducing the number of channels to the original , to reduce the number of parameters and computational burden, for example, r = 4. After the convolution operation, batch normalization and ReLU activation function are used to stabilize the training process and perform nonlinear transformations. Then, another 7×7 convolution layer, i.e., the second 7×7 convolution layer, is used to restore the number of channels of the feature map to the size after shuffling, and obtain the spatial attention map to ensure that the spatial features match the input feature map after reconstruction. The shuffled feature map and the spatial attention map are multiplied pixel by pixel to obtain the pixel-by-pixel enhanced text instance features output by the GASM module, as shown in the following formula:

[0145] ;

[0146] ;

[0147] in, ω s is the spatial attention map, It is the pixel-by-pixel enhanced text instance feature output by the GASM module, i.e., the feature map. for function, represents pixel-by-pixel multiplication, is the shuffled feature map, For batch normalization, is the convolutional layer.

[0148] Specifically, the GASM module is as follows Figure 4 As shown, the size of the input feature map is ,go through MLP After balancing, the feature map size becomes , after the ReLU activation function and MLP After balancing, the size of the feature map changes back ,go through After the function is processed, the channel attention map is obtained, and the channel attention map is multiplied pixel by pixel with the input feature map; after the channel is shuffled, the feature map is input into the 7×7 Conv layer, the size becomes After the feature map is processed by BN, it is input into another 7×7 Conv Layer, size changed back , the feature map is processed by BN and then The spatial attention map is processed by the function, and the spatial attention map is multiplied pixel by pixel with the feature map after channel shuffle to obtain a size of The output feature map of .

[0149] In an exemplary embodiment, step S4 includes:

[0150] Convolution is performed on the pixel-by-pixel enhanced text instance features to obtain the classification result for predicting whether each pixel belongs to the text area and the regression result for predicting the Fourier series coefficient of each pixel;

[0151] The classification results are normalized to obtain the probability distribution of the identified text area and the probability distribution of the identified text center area; the text area score and the text center area score are weighted multiplied to obtain a pixel-by-pixel classification score map, and a binarization operation is performed to obtain a text area mask map;

[0152] The Fourier feature vector corresponding to the text area in the text area mask image is extracted through thresholding operation, and the reconstructed text instance contour is obtained through inverse discrete Fourier transform;

[0153] The loss function based on text instance category and Fourier feature vector is used to train the network structure composed of deformable VMamba feature encoder, GASM module and Fourier feature decoder from input to output to obtain the scene text detection model and output the predicted text instance contour.

[0154] Exemplarily, step S4 is specifically as follows:

[0155] Step S401, for the pixel-by-pixel enhanced text instance features output by the GASM module in step S3, two different convolution layers with a convolution kernel size of 3×3 and a step size of 1 are used to respectively obtain a classification result for predicting whether each pixel belongs to a text area and a regression result for predicting the Fourier series coefficient of each pixel;

[0156] Step S402, after the convolution operation in step S401, normalize the first two channels and the last two channels of the classification result to obtain the probability distribution of the identified text area and the probability distribution of the identified text center area, i.e., the corresponding area score. In order to filter out the low-quality predictions around the text boundary, the text area score and the text center area score are weighted multiplied to obtain a pixel-by-pixel classification score map, and a binarization operation is performed to obtain a text area mask map;

[0157] Step S403, using a complex-valued function To describe any closed text contour, let is a periodic function with a period of 1 and ,and then It can be expressed by Fourier series expansion, and the inverse Fourier transform can be used to reconstruct the text contour, as shown in the following formula:

[0158] ;

[0159] Among them, the Fourier eigenvector It can be expressed as ,in represents the real part, Represents the imaginary part. Since the sampling points are discrete values, it is impossible to accurately obtain the text contour. Analytical form, then use the discrete periodic Fourier transform to transform the continuous function Converted into discrete form, that is ,in ,So It can be obtained by the discrete Fourier series coefficient formula shown in the following formula:

[0160] ;

[0161] After obtaining the text region mask image in step S402, the Fourier feature vectors corresponding to these regions in step S401 are extracted by thresholding operation, including the real part and the imaginary part, and the reconstructed text instance contour is obtained by the discrete Fourier inverse transform formula shown in the following formula:

[0162] ;

[0163] in, Indicates the frequency kThe initial state of the Fourier series is This means the frequency is k The circular motion trajectory of the series, and , ;

[0164] Step S404, after obtaining the reconstructed text instance contour in step S403, the detection model is trained using the loss function shown in the following formula:

[0165] ;

[0166] in, It is a loss function related to the text instance category, which is used to identify the text area and the text center area; It is the loss function of the Fourier feature vector of the text instance, which is used to improve the detection accuracy. The two loss functions are shown in the following formula:

[0167] ;

[0168] ;

[0169] in, and They are the cross entropy losses of the text area and the text center area respectively. Indicates smoothness loss, is the inverse Fourier transform, is a pixel index set within the text area. , and Represents its true Fourier eigenvector and predicted Fourier eigenvector, weight Used to distinguish the contributions of different regions, when pixels When in the center of the text area, ,otherwise ;

[0170] By combining the two loss functions, the category and location of the text instance can be accurately grasped, the detection accuracy of the model can be improved, and the predicted text instance contour is output through NMS (non maximum suppression) to achieve scene text detection.

[0171] Specifically, the Fourier feature decoder is as follows Figure 5As shown in the figure, the feature map output by the GASM module is normalized to obtain the probability distribution of the identified text area and the probability distribution of the identified text center area, and the weighted product and binarization operations are performed to obtain the text area mask map; the Fourier feature vector containing the real part and the imaginary part is extracted and concatenated through the threshold operation, and the discrete Fourier inverse transform formula is used to obtain the reconstructed text instance contour, and the predicted text instance contour is output through NMS.

[0172] At least a portion of the steps in the flowcharts involved in the embodiments described above may include multiple steps or multiple stages. These steps or stages may be executed at different times, or may be executed in turn or alternating with other steps or at least a portion of the steps or stages in other steps.

[0173] Based on the same inventive concept, the embodiment of the present application also provides a scene text detection device based on the VMamba framework and the channel-space attention mechanism for implementing the scene text detection method based on the VMamba framework and the channel-space attention mechanism involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of the scene text detection device based on the VMamba framework and the channel-space attention mechanism provided below can be found in the above limitations on the scene text detection method based on the VMamba framework and the channel-space attention mechanism, which will not be repeated here.

[0174] In an exemplary embodiment, a scene text detection device based on a VMamba framework and a channel-space attention mechanism is provided, wherein:

[0175] The sample augmentation module is used to obtain a reference sample dataset of scene images containing text, apply a data augmentation strategy to the reference sample dataset, and generate an enhanced sample dataset;

[0176] A feature encoder construction module is used to construct a deformable VMamba feature encoder based on the VMamba framework using deformable convolution, and the deformable VMamba feature encoder is used to process each scene image in the enhanced sample data set to obtain a multi-scale text instance feature map corresponding to each scene image;

[0177] The GASM module constructs a module for obtaining a balanced feature map based on the multi-scale text instance feature map by using interpolation and maximum pooling methods; the GASM module is constructed based on the channel-spatial attention mechanism to perform channel-spatial attention calculation and channel shuffling on the balanced feature map to obtain the text instance features after pixel-by-pixel enhancement;

[0178] The scene text detection model generation module is used to construct a Fourier feature decoder, which is used to obtain a pixel-by-pixel text area mask and Fourier feature vector based on the text instance features, and use the discrete Fourier inverse transform to obtain the reconstructed text instance contour; from the input to the output direction, the deformable VMamba feature encoder, the GASM module and the Fourier feature decoder constitute the network structure to be trained, and the loss function is used to train the network structure to generate a scene text detection model with the enhanced sample data set as input and the text instance contour as output.

[0179] Each module in the scene text detection device based on the VMamba framework and the channel-space attention mechanism can be implemented in whole or in part by software, hardware, and a combination thereof. Each of the above modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each of the above modules.

[0180] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0181] For those skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, and these modifications and improvements are all within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A scene text detection method based on VMamba framework and channel-spatial attention mechanism, characterized in that: The method is used to detect text in a scene, and includes: Step S1, obtaining a scene image reference sample dataset containing text, applying a data enhancement strategy to the reference sample dataset, and generating an enhanced sample dataset; Step S2, constructing a deformable VMamba feature encoder based on the VMamba framework using deformable convolution, applying the deformable VMamba feature encoder to process each scene image in the enhanced sample data set, and obtaining a multi-scale text instance feature map corresponding to each scene image; wherein step S2 includes: Step S201, using the Stem module to divide each scene image in the enhanced sample data set input into the deformable VMamba feature encoder into image patches of a specified size; Step S202, sequentially input the image patch into the first stage, the second stage, the third stage, and the fourth stage of the deformable VMamba feature encoder, wherein the first stage extracts and outputs a first-size feature map, the second stage extracts and outputs a second-size feature map, the third stage extracts and outputs a third-size feature map, and the fourth stage extracts and outputs a fourth-size feature map; Step S203, inputting the second size feature map, the third size feature map, and the fourth size feature map into the target deformable convolution, and outputting a multi-scale text instance feature map; Wherein, step S203 includes: The second-size feature map, the third-size feature map, and the fourth-size feature map are input into the deformable convolution with a convolution kernel size of 3×3, and the text instance feature maps of the corresponding scales output by the deformable VMamba feature encoder are obtained respectively, where the deformable convolution is shown in the following formula: Among them, x(p) represents the feature of the input feature map at position p, y(p) represents the feature of the output feature map corresponding to the input feature map at position p, and w k represents the weight of the Kth sampling position, p k represents the preset offset of the Kth sampling position, Δp k represents the learnable offset of the Kth sampling position, Δm k represents the learnable modulation scalar at the Kth sampling position; Step S3, using interpolation and maximum pooling methods to obtain a balanced feature map according to the multi-scale text instance feature map; constructing a GASM module based on the channel-space attention mechanism to perform channel-space attention calculation and channel shuffling on the balanced feature map to obtain the text instance features after pixel-by-pixel enhancement; Step S4, constructing a Fourier feature decoder, which is used to obtain a pixel-by-pixel text area mask image and a Fourier feature vector according to the text instance features, and using a discrete Fourier inverse transform to obtain a reconstructed text instance contour; from the input to the output direction, the deformable VMamba feature encoder, the GASM module and the Fourier feature decoder constitute a network structure to be trained, and the loss function is used to train the network structure to be trained to generate a scene text detection model with an enhanced sample data set as input and a text instance contour as output.

2. The method according to claim 1, characterized in that In step S1, the reference sample data set is applied with a data enhancement strategy to generate an enhanced sample data set, which includes: The reference sample dataset is randomly cropped, randomly horizontally flipped, and randomly rotated, and the brightness, saturation, and contrast are adjusted to generate an enhanced sample dataset.

3. The method according to claim 2, characterized in that The step S3 comprises: Balanced semantic features are obtained based on multi-scale text instance feature maps through interpolation and maximum pooling methods; The embedded Gaussian non-local attention mechanism is used to refine the balanced semantic features and obtain the balanced feature map; The GASM module is built based on the channel-spatial attention mechanism, and the channel dimension in the balanced feature map is arranged to the last dimension; the channel attention map is generated through the fully connected layer and the ReLU activation function, and the balanced feature map and the channel attention map are multiplied pixel by pixel to obtain the channel-enhanced feature map; Divide the channel-enhanced feature map into a target number of groups, where the number of channels in each group is determined by dividing the total number of channels by the number of groups, and divide the channels in each group into at least one subgroup, and combine different subgroups in each group by disrupting the order of channels in each group, and restore the disrupted feature map to its original shape to obtain a shuffled feature map; The shuffled feature map is convolved, and batch normalization and ReLU activation function are used to generate a spatial attention map. The shuffled feature map is multiplied pixel by pixel with the spatial attention map to output the pixel-by-pixel enhanced text instance features.

4. The method according to claim 3, characterized in that The channel attention map is generated by the fully connected layer and the ReLU activation function, and the balanced feature map and the channel attention map are multiplied pixel by pixel to obtain the channel enhanced feature map, which includes: The number of input channels of the balanced feature map is reduced through the fully connected layer, and the complex relationship between channels is modeled using the ReLU activation function. The number of input channels is increased again to generate a channel attention map. Multiply the balanced feature map and the channel attention map pixel by pixel to obtain the channel enhanced feature map. The formula is as follows: ω c =σ(Permute(MLP(Permute(F input )))) Among them, ω c is the channel attention map, F channel is the feature map after channel enhancement, σ is the Sigmoid function, Indicates pixel-by-pixel multiplication, F input is the original input feature map, MLP is the fully connected layer, and Permute is the transposition operation.

5. The method according to claim 4, characterized in that The convolution operation is performed on the shuffled feature map, and a batch normalization and ReLU activation function are used to generate a spatial attention map. The shuffled feature map is multiplied pixel by pixel with the spatial attention map, and the pixel-by-pixel enhanced text instance features are outputted, including: The shuffled feature map is input into the first 7×7 convolutional layer to reduce the number of channels of the shuffled feature map; Batch normalization and ReLU activation function are used to stabilize the training process and perform nonlinear transformations to model complex relationships in space. The number of channels of the feature map is restored to the shuffled size through the second 7×7 convolutional layer to generate a spatial attention map. Multiply the shuffled feature map and the spatial attention map pixel by pixel to obtain the pixel-by-pixel enhanced text instance features output by the GASM module. The formula is as follows: ω s =σ(BN(Conv(BN(Conv(F shuffle ))))) Among them, ω s is the spatial attention map, F output is the feature map output by the GASM module, σ is the Sigmoid function, Indicates pixel-by-pixel multiplication, F shuffle is the shuffled feature map, BN is batch normalization, and Conv is the convolutional layer.

6. The method according to claim 5, characterized in that The step S4 comprises: Convolution is performed on the pixel-by-pixel enhanced text instance features to obtain the classification result for predicting whether each pixel belongs to the text area and the regression result for predicting the Fourier series coefficient of each pixel; The classification results are normalized to obtain the probability distribution of the identified text area and the probability distribution of the identified text center area; the text area score and the text center area score are weighted multiplied to obtain a pixel-by-pixel classification score map, and a binarization operation is performed to obtain a text area mask map; The Fourier feature vector corresponding to the text area in the text area mask image is extracted through thresholding operation, and the reconstructed text instance contour is obtained through discrete Fourier inverse transform; The loss function based on text instance category and Fourier feature vector is used to train the network structure composed of deformable VMamba feature encoder, GASM module and Fourier feature decoder from input to output to obtain the scene text detection model and output the predicted text instance contour.

7. A scene text detection device based on VMamba framework and channel-space attention mechanism, characterized in that: The device comprises: The sample augmentation module is used to obtain a reference sample dataset of scene images containing text, apply a data augmentation strategy to the reference sample dataset, and generate an enhanced sample dataset; The feature encoder construction module is used to construct a deformable VMamba feature encoder based on the VMamba framework using deformable convolution, and use the deformable VMamba feature encoder to process each scene image in the enhanced sample data set to obtain a multi-scale text instance feature map corresponding to each scene image; wherein the feature encoder construction module is also used to execute the Stem module to divide each scene image in the enhanced sample data set input into the deformable VMamba feature encoder into image patches of specified sizes; the image patches are sequentially input into the first stage, the second stage, the third stage, and the fourth stage of the deformable VMamba feature encoder, wherein the first stage extracts and outputs a first-size feature map, and the second stage extracts and outputs a first-size feature map. tage extracts and outputs a second-size feature map, the third Stage extracts and outputs a third-size feature map, and the fourth Stage extracts and outputs a fourth-size feature map; the second-size feature map, the third-size feature map, and the fourth-size feature map are input into the target deformable convolution, and a multi-scale text instance feature map is output; wherein the feature encoder construction module is also used to execute the input of the second-size feature map, the third-size feature map, and the fourth-size feature map into the target deformable convolution, and the output of the multi-scale text instance feature map includes inputting the second-size feature map, the third-size feature map, and the fourth-size feature map into a deformable convolution with a convolution kernel size of 3×3, and respectively obtaining text instance feature maps of corresponding scales output by the deformable VMamba feature encoder, wherein the deformable convolution is shown in the following formula: Among them, x(p) represents the feature of the input feature map at position p, y(p) represents the feature of the output feature map corresponding to the input feature map at position p, and w k represents the weight of the Kth sampling position, p k represents the preset offset of the Kth sampling position, Δp k represents the learnable offset of the Kth sampling position, Δm k represents the learnable modulation scalar at the Kth sampling position; The GASM module constructs a module for obtaining a balanced feature map based on the multi-scale text instance feature map by using interpolation and maximum pooling methods; the GASM module is constructed based on the channel-spatial attention mechanism to perform channel-spatial attention calculation and channel shuffling on the balanced feature map to obtain the text instance features after pixel-by-pixel enhancement; The scene text detection model generation module is used to construct a Fourier feature decoder, which is used to obtain a pixel-by-pixel text area mask and Fourier feature vector based on the text instance features, and use the discrete Fourier inverse transform to obtain the reconstructed text instance contour; from the input to the output direction, the deformable VMamba feature encoder, the GASM module and the Fourier feature decoder constitute the network structure to be trained, and the loss function is used to train the network structure to generate a scene text detection model with the enhanced sample data set as input and the text instance contour as output.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

Images