Industrial image recognition method and system based on Mama and Conv

Abstract

The invention discloses an industrial image recognition method and system based on Mama and Conv, and relates to the technical field of image recognition. The method comprises the following steps: acquiring a to-be-detected industrial image; according to the method, an image recognition network is utilized to recognize a to-be-detected industrial image, firstly, a preliminary feature extraction module composed of Conv is utilized to extract preliminary features of the to-be-detected industrial image, and then a depth feature extraction module composed of Mama and Conv is utilized to further extract local features and global features according to the preliminary features; and the local features and the global features are fused to obtain fused features, and finally, a classifier is utilized to perform image classification according to the fused features. The method can effectively solve the problems that in the prior art, local detail capturing is insufficient, the long-distance dependence modeling capacity is limited, and the computing resource requirement is high.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to an industrial image recognition method and system based on Mamba and Conv. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art. Current deep learning-based image recognition methods typically employ a combination of CNN, Transformer, Mamba, and Transformer modules to extract multi-scale features from the input image, followed by a classifier network to achieve image classification. However, existing technologies still suffer from the following shortcomings in practical applications: First, while traditional convolutional neural networks (CNNs) possess excellent spatial local perception capabilities in extracting local features and effectively capturing local texture information in images, their receptive field is limited and their ability to model long-range dependencies is insufficient. When dealing with complex scenes or tasks with strong global context dependencies, CNNs often need to stack a large number of convolutional layers to expand the receptive field. This not only leads to a significant increase in network structure, number of parameters, and computational overhead, but also easily causes problems such as gradient vanishing and slow training convergence, reducing the model's training efficiency and inference performance.

[0003] Secondly, Transformer-based visual models, through self-attention mechanisms, model long-distance dependencies in the feature space, enhancing their ability to model global information and demonstrating high recognition accuracy in large-scale visual tasks. However, the pure Transformer architecture performs global interrelationship calculations on input features, resulting in computational complexity that increases quadratically with input resolution. This leads to significant computational and memory overhead in high-resolution images or large-scale data scenarios, making efficient deployment on resource-constrained terminal devices difficult.

[0004] To address the aforementioned issues, hybrid architectures combining Mamba and Transformer, Conv and Transformer, and Mamba and Conv have emerged in recent years. Among these, the Mamba and Transformer and Conv and Transformer combinations, by introducing local convolutions or state space units before and after the global modeling module, can reduce the redundancy of pure self-attention computation to some extent and decrease some computational load. However, because they still retain the explicit modeling process for global features, the overall computational complexity remains high, making it difficult to meet the needs of rapid inference in low-computing-power environments. The Mamba and Conv combination methods mostly employ simple module concatenation or stacking, failing to achieve deep fusion of local and long-sequence dependency information. This makes it difficult to fully leverage the efficient long-range information capture advantages of state space models and the local feature extraction capabilities of convolution, resulting in insufficient overall feature extraction and representation capabilities, limiting the application effectiveness and deployment efficiency in complex image scenarios.

[0005] Some existing studies employ a Transformer-Mamba hybrid encoder to fuse shallow and deep semantic features from the feature map, and use stacked decoders to decode the features output by the encoder to obtain a feature sequence, which is then input into the prediction head for prediction. However, the hybrid encoder needs to iteratively process multi-scale feature maps, and the overall complexity increases after stacking decoders, resulting in excessive computational overhead, making it difficult to meet real-time requirements in low-computing-power environments.

[0006] In summary, existing hybrid structures suffer from several drawbacks in industrial image recognition, including global computational redundancy, quadratic expansion of memory and computing power with resolution, failure to deeply integrate local and long-range dependencies, and increased latency due to multi-scale iterative decoding. These shortcomings make it difficult to achieve real-time high-precision recognition on low-computing-power devices. Summary of the Invention

[0007] To address the shortcomings of existing technologies, the present invention aims to provide an industrial image recognition method and system based on Mamba and Conv, which achieves deep fusion of local and global features. This method maintains the accurate capture of local textures by convolutional branches while utilizing Mamba branches for efficient modeling of long sequences or global contexts. Furthermore, it dynamically adjusts the importance of the two types of features through a gating mechanism, thereby effectively solving the problems of insufficient capture of local details, limited ability to model long-distance dependencies, and high computational resource requirements in existing technologies.

[0008] To achieve the above objectives, the present invention is implemented through the following technical solution: The first aspect of this invention provides an industrial image recognition method based on Mamba and Conv, comprising the following steps: Acquire industrial images to be inspected; An image recognition network is used to identify industrial images to be detected. First, a preliminary feature extraction module composed of Conv is used to extract preliminary features of the industrial images to be detected. Then, a deep feature extraction module composed of Mamba and Conv is used to further extract local and global features based on the preliminary features. The local and global features are then fused to obtain fused features. Finally, a classifier is used to classify the image based on the fused features.

[0009] Furthermore, the preliminary feature extraction module includes two Conv2D modules with the same structure connected in sequence, and the deep feature extraction module includes two MambaConv Mixer modules with the same structure connected in sequence.

[0010] Furthermore, the Conv2D module contains a Conv2D layer, a batch normalization layer, and a ReLU activation function layer.

[0011] Furthermore, the MambaConv Mixer module includes a MambaConv Mixer structure for residual connections and a deep feature representation module. The deep feature representation module is used to further adjust and optimize the extracted features. The MambaConv Mixer structure includes four modules: a Mamba-based global feature extraction module, a Conv-based local feature extraction module, a Gate calculation module, and a global and local feature fusion module. Specifically, the Mamba-based global feature extraction module extracts global features, the Conv-based local feature extraction module extracts local features, the Gate calculation module assigns weights to the extracted features, and the global and local feature fusion module fuses the global and local features.

[0012] Furthermore, after the preliminary feature extraction module extracts the preliminary features of the industrial image to be detected, it divides the preliminary features into two parts, which are then input into the Mamba-based global feature extraction module and the Conv-based local feature extraction module, respectively.

[0013] Furthermore, the specific steps for extracting global features using the Mamba-based global feature extraction module are as follows: Extract preliminary image texture features and change image dimensions; By using a state-space model to model the long-distance dependence of features that change the image dimension, preliminary global features are obtained. The dimensions of the initial global features are restored to obtain the final global features.

[0014] Furthermore, the specific steps for assigning weights to the extracted features using the Gate calculation module are as follows: The global and local features are concatenated after being subjected to average pooling operations respectively. The concatenated features are normalized to obtain the calculated Gate value, and the corresponding feature weights are assigned based on the Gate value.

[0015] A second aspect of the present invention provides an industrial image recognition system based on Mamba and Conv, comprising: The data acquisition module is configured to acquire industrial images to be inspected; The data processing module is configured to use an image recognition network to recognize the industrial image to be detected. First, a preliminary feature extraction module composed of Conv is used to extract preliminary features of the industrial image to be detected. Then, a deep feature extraction module composed of Mamba and Conv is used to further extract local and global features based on the preliminary features. The local and global features are then fused to obtain fused features. Finally, a classifier is used to classify the image based on the fused features.

[0016] A third aspect of the present invention provides a computer-readable storage medium storing a computer program adapted to be loaded by a processor and to execute steps in the industrial image recognition method based on Mamba and Conv as described in the first aspect of the present invention.

[0017] A fourth aspect of the present invention provides a computer device comprising: A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the industrial image recognition method based on Mamba and Conv as described in the first aspect of the present invention.

[0018] This invention discloses an industrial image recognition method and system based on Mamba and Conv. The technical solution has the following beneficial effects: 1. This invention combines local convolutional branches with Mamba state space branches and uses a gating mechanism to weight the outputs of both as needed. It can automatically adjust the ratio of local to global information according to the characteristics of the input scene, so that the network has a better ability to take into account local details and global contextual relationships, and improves the completeness of industrial image recognition in expressing multi-scale features.

[0019] 2. Compared to the Transformer architecture, which relies solely on self-attention for global modeling, this invention introduces a state-space module into the Mamba branch to recursively model long sequence dependencies, reducing computational complexity from quadratic to linear. Simultaneously, by using convolutional branches to handle local information capture, it avoids the global branch's simultaneous global scan of all spatial locations, reducing the demand for GPU memory and computing power under high-resolution input. This results in lower inference speed and computational resource consumption under the same conditions, offering better resource friendliness and deployment feasibility.

[0020] 3. The MambaCNN network proposed in this invention adopts a parallel dual-branch design with low coupling, clear structure, and strong adaptability. The feature dimension, number of layers, and gating weight configuration of the convolutional branch and state space branch can be flexibly adjusted according to specific application scenarios, making it easy to integrate into visual backbone networks of different sizes. It also supports rapid migration and use in industrial image classification vision tasks, reducing the network redesign and debugging cycle.

[0021] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of the industrial image recognition method based on Mamba and Conv in Embodiment 1 of the present invention; Figure 2 This is a diagram showing the overall architecture of the MambaConv Mixer in Embodiment 1 of the present invention. Detailed Implementation

[0024] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0025] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof. The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0026] Example 1: This invention provides an industrial image recognition method based on Mamba and Conv. A MambaCNN network is designed to improve the accuracy and computational efficiency of industrial image recognition, reduce the hardware deployment threshold, and offer better versatility and scalability. The MambaCNN network achieves simultaneous and efficient extraction of local detail information and globally dependent features within the same network structure through efficient parallel coupling of local convolutional branches and state space modules, combined with an adaptive gating mechanism. Specifically, it includes two sequentially connected Conv2D modules, two MambaConv Mixer modules, and a classifier. The Conv2D modules are used to change the feature map size and extract preliminary image features. The MambaConv Mixer modules are used to change the feature map size and depth, extract local and global features, and fuse them. The classifier is used to classify the fused features of the image. The acquired industrial image is input into the MambaCNN network to perform image classification, obtaining the image classification result.

[0027] like Figure 1 As shown, the specific steps include: Step 1: Acquire the industrial image to be inspected.

[0028] In this embodiment, the MambaCNN network is preferably applied to image recognition tasks with macroscopic geometric features (such as the number of gear teeth and the distribution of shaft holes), such as for part type identification. Therefore, the acquired industrial image to be detected is the industrial image of the corresponding device in the required task.

[0029] Step 2: Use an image recognition network to identify the industrial image to be detected.

[0030] Step 2.1: First, use the preliminary feature extraction module composed of Conv to extract the preliminary features of the industrial image to be detected.

[0031] In one specific implementation, the preliminary feature extraction module includes two identical Conv2D modules connected in sequence. The Conv2D modules are used to change the feature map size and extract preliminary features of the image. The Conv2D module contains a Conv2D layer, a batch normalization layer (BN layer), and a ReLU activation function layer.

[0032] Step 2.2: Then, the deep feature extraction module composed of Mamba and Conv is used to further extract local and global features based on the preliminary features, and the local and global features are fused to obtain fused features.

[0033] In one specific implementation, the deep feature extraction module includes two MambaConv Mixer modules with identical structures connected sequentially. The MambaConv Mixer module is used to change the feature map size, extract local and global features at depth, and fuse them. The MambaConv Mixer module contains a MambaConv Mixer structure that performs residual connections and a deep feature representation module.

[0034] The deep feature representation module is used to further adjust and optimize the extracted features, enhancing feature extraction capabilities. The deep feature representation module consists of a combination of a Conv2D layer, a ReLU activation function layer, and another Conv2D layer.

[0035] like Figure 2 As shown, the MambaConv Mixer structure includes four modules: a global feature extraction module based on Mamba, a local feature extraction module based on Conv, a gate calculation module, and a global and local feature fusion module. Specifically, the global feature extraction module based on Mamba extracts global features, the local feature extraction module based on Conv extracts local features, the gate calculation module assigns weights to the extracted features, and the global and local feature fusion module fuses the global and local features.

[0036] After the preliminary feature extraction module extracts the preliminary features of the industrial image to be detected, it divides these preliminary features into two parts, which are then input into the Mamba-based global feature extraction module and the Conv-based local feature extraction module, respectively, to extract the global features of the image. With local features The Gate calculation module is based on and Computational Dynamics Dynamic allocation and The weights will eventually be... , and Input the global and local feature fusion modules to obtain the fused features of the image.

[0037] Specifically, the following steps are included: Step 2.2.1: Extract global features using the Mamba-based global feature extraction module.

[0038] The structure of the global feature extraction module based on Mamba is as follows: a combination of Conv2D layer and SiLU activation function layer, Rearrange layer (dimension rearrangement layer), a combination of Conv1D layer and SiLU activation function layer, SSM layer, and Rearrange layer are connected in sequence. Step 2.2.1.1: Extract preliminary image texture features and change image dimensions.

[0039] In one specific implementation, to accommodate the size of the SSM layer, the initial features of the image are first input into a combination of a Conv2D layer and a SiLU activation function layer to extract image texture features, and then the image dimensions are changed through a Rearrange layer. The features with the changed image dimensions are then input into a combination of a Conv1D layer and a SiLU activation function layer for further image feature extraction.

[0040] Specifically, the first Rearrange layer changes the image dimensions, for example, changing the original image dimensions [B,C,L,P] to [B,C,L×P], where B is the batch size, C is the number of channels, L is the height of the feature map, and P is the width of the feature map; the second Rearrange layer restores the image dimensions to [B,C,L,P].

[0041] Step 2.2.1.2: Use the state-space model to model the long-distance dependency of features that change the image dimensions to obtain preliminary global features.

[0042] Specifically, the SSM layer is the State Space Model Layer, a core component in Mamba for efficiently modeling long-range dependencies. It inputs the image features extracted from the combination of the Conv1D layer and the SiLU activation function layer into the SSM layer to model long-range dependencies.

[0043] A standard Mamba SSM layer mainly consists of the following components: 1. Input projection: Projecting the input... The input channels mapped to the state space are in the following form: .

[0044] in, Input at time t The result after mapping to the state space, For mapping functions; For time steps.

[0045] 2. State Update Equation: The state update equation describes the evolution of the sequence state over time. , In the formula, B are trainable parameters; It is a state vector.

[0046] 3. Output Equation: Mapping the state vector to the output. : .

[0047] In the formula, This is for outputting the projection matrix.

[0048] 4. Convolution Implementation: To achieve efficient implementation, Mamba discretizes the response of SSM and represents it as a one-dimensional convolution kernel, which is then convolved with the input sequence to complete the entire process.

[0049] 5. Dynamic Gating: Mamba introduces a lightweight gating mechanism: .

[0050] In the formula, This is the final output result; for The result after mapping to the state space; It is a dynamically calculated gate value; It is a convolutional kernel calculated based on SSM parameters; This represents convolution.

[0051] Step 2.2.1.3: Restore the dimensions of the preliminary global features to obtain the final global features.

[0052] The initial global features are then passed through a Rearrange layer to restore the altered image dimensions, resulting in the final global features of the image. .

[0053] Step 2.2.2: Extract local features using the Conv-based local feature extraction module.

[0054] In one specific implementation, the Conv-based local feature extraction module consists of two Conv2D layers connected sequentially with a SiLU activation function layer. The first Conv2D layer and SiLU activation function layer are used to extract image texture features from the initial features; the second Conv2D layer and SiLU activation function layer are used to further extract local image features based on the image texture features, thus obtaining the local features of the image. .

[0055] Step 2.2.3: Use the Gate calculation module to assign weights to the extracted features.

[0056] In one specific implementation, the Gate calculation module is used to calculate the weight ratio of the Mamba-based global feature extraction module and the Conv-based local feature extraction module.

[0057] Specifically, the following steps are included: Step 2.2.3.1: Perform average pooling operations on the global features and local features respectively, and then concatenate them to obtain the concatenated features.

[0058] Specifically, the Gate calculation module first... and Perform average pooling operation, and then... and Then, the parts are assembled.

[0059] Step 2.2.3.2: Normalize the spliced ​​features to obtain the calculated Gate value, and assign corresponding feature weights based on the Gate value.

[0060] Specifically, the concatenated features are input into a linear layer and normalized using a sigmoid activation function layer to obtain... value.

[0061] Step 2.2.4: Use the global and local feature fusion module to fuse global and local features.

[0062] In one specific implementation, the global and local feature fusion module first obtains global features through a preliminary fusion formula. With local features Preliminary fusion characteristics : .

[0063] Secondly, the initial fusion features will be... Input a Conv2D layer for deep fusion to obtain the fused features of the image.

[0064] Step 2.3: Finally, use a classifier to classify the image based on the fused features.

[0065] In one specific implementation, the classifier is used to classify images based on fused features. The classifier's structure is specifically a combination of a normalization layer, an average pooling layer, a Flatten layer, and a fully connected layer (Linear layer).

[0066] Specifically, the image fusion features are first normalized and averaged through a normalization layer and an average pooling layer. Then, the features are flattened by a Flatten layer, transforming the features into a size of [B,C,M], where M is the product of the width and height of the feature map, laying the groundwork for the subsequent fully connected layer. Finally, a fully connected layer is used to classify the industrial image to obtain the image classification result.

[0067] Example 2: Embodiment 2 of the present invention provides an industrial image recognition system based on Mamba and Conv, comprising: The data acquisition module is configured to acquire industrial images to be inspected; The data processing module is configured to use an image recognition network to recognize the industrial image to be detected. First, a preliminary feature extraction module composed of Conv is used to extract preliminary features of the industrial image to be detected. Then, a deep feature extraction module composed of Mamba and Conv is used to further extract local and global features based on the preliminary features. The local and global features are then fused to obtain fused features. Finally, a classifier is used to classify the image based on the fused features.

[0068] Example 3: Embodiment 3 of the present invention provides a computer-readable storage medium storing a computer program adapted for loading by a processor and executing the steps of the industrial image recognition method based on Mamba and Conv as described in Embodiment 1 of the present invention.

[0069] Example 4: Embodiment 4 of the present invention provides a computer device, the device comprising: A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the steps of the industrial image recognition method based on Mamba and Conv as described in Embodiment 1 of the present invention.

[0070] The steps and methods involved in Examples 2, 3 and 4 above correspond to those in Example 1. For specific implementation details, please refer to the relevant description section of Example 1.

[0071] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application. In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data processing device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, an optical medium, or a semiconductor medium, etc. The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An industrial image recognition method based on Mamba and Conv, characterized in that, Includes the following steps: Acquire industrial images to be inspected; An image recognition network is used to identify industrial images to be detected. First, a preliminary feature extraction module composed of Conv is used to extract preliminary features of the industrial images to be detected. Then, a deep feature extraction module composed of Mamba and Conv is used to further extract local and global features based on the preliminary features. The local and global features are then fused to obtain fused features. Finally, a classifier is used to classify the image based on the fused features.

2. The industrial image recognition method based on Mamba and Conv as described in claim 1, characterized in that, The preliminary feature extraction module consists of two Conv2D modules with the same structure connected in sequence, and the deep feature extraction module consists of two MambaConv Mixer modules with the same structure connected in sequence.

3. The industrial image recognition method based on Mamba and Conv as described in claim 2, characterized in that, The Conv2D module contains a Conv2D layer, a batch normalization layer, and a ReLU activation function layer.

4. The industrial image recognition method based on Mamba and Conv as described in claim 2, characterized in that, The MambaConvMixer module includes a MambaConv Mixer structure for residual connections and a deep feature representation module. The deep feature representation module is used to further adjust and optimize the extracted features. The MambaConv Mixer structure includes four modules: a Mamba-based global feature extraction module, a Conv-based local feature extraction module, a Gate calculation module, and a global and local feature fusion module. Specifically, the Mamba-based global feature extraction module extracts global features, the Conv-based local feature extraction module extracts local features, the Gate calculation module assigns weights to the extracted features, and the global and local feature fusion module fuses the global and local features.

5. The industrial image recognition method based on Mamba and Conv as described in claim 4, characterized in that, initially... After the feature extraction module extracts the preliminary features of the industrial image to be detected, it divides the preliminary features into two parts and inputs them into the global feature extraction module based on Mamba and the local feature extraction module based on Conv, respectively.

6. The industrial image recognition method based on Mamba and Conv as described in claim 4, characterized in that, The specific steps for extracting global features using the Mamba-based global feature extraction module are as follows: Extract preliminary image texture features and change image dimensions; By using a state-space model to model the long-distance dependence of features that change the image dimension, preliminary global features are obtained. The dimensions of the initial global features are restored to obtain the final global features.

7. The industrial image recognition method based on Mamba and Conv as described in claim 4, characterized in that, The specific steps for assigning weights to the extracted features using the Gate calculation module are as follows: The global and local features are concatenated after being subjected to average pooling operations respectively. The concatenated features are normalized to obtain the calculated Gate value, and the corresponding feature weights are assigned based on the Gate value.

8. An industrial image recognition system based on Mamba and Conv, characterized in that, include: The data acquisition module is configured to acquire industrial images to be inspected; The data processing module is configured to use an image recognition network to recognize the industrial image to be detected. First, a preliminary feature extraction module composed of Conv is used to extract preliminary features of the industrial image to be detected. Then, a deep feature extraction module composed of Mamba and Conv is used to further extract local and global features based on the preliminary features. The local and global features are then fused to obtain fused features. Finally, a classifier is used to classify the image based on the fused features.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and executed as described in any one of claims 1-7.

10. A computer device, characterized in that, include: A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program that, when executed by the processor, implements the industrial image recognition method based on Mamba and Conv as described in any one of claims 1-7.

Images