Two-dimensional code image feature extraction method and device based on CNN-MALT hybrid architecture

Abstract

The application is suitable for the technical field of computer application, and provides a two-dimensional code image feature extraction method and device based on a CNN-MALT hybrid architecture, which comprises the following steps: inputting a target two-dimensional code image into a CNN preprocessing module of a preset lightweight image feature extraction network to generate first structure perception features; inputting the first structure perception features into a MALT encoder of the preset lightweight image feature extraction network, performing feature extraction on the first structure perception features based on a proxy attention mechanism to generate deep features; and inputting the deep features into a feature fusion module of the preset lightweight image feature extraction network to generate target features. Thus, the structure features of the two-dimensional code image are perceived by the CNN preprocessing module, the deep features are generated by the MALT encoder based on the proxy attention mechanism, and the deep features are fused by the feature fusion module, so as to ensure that the calculation complexity is reduced while the two-dimensional code image feature extraction accuracy is improved.

Description

Technical Field

[0001] This application belongs to the field of computer application technology, and in particular relates to a method and apparatus for extracting QR code image features based on a CNN-MALT hybrid architecture. Background Technology

[0002] QR codes, as an efficient information carrier, have been widely used in complex industrial environments such as logistics management, mobile payment, and industrial traceability. However, in practical applications, the acquired QR code images are often severely degraded due to factors such as uneven lighting, motion blur, physical wear, partial occlusion, or perspective distortion, resulting in low-quality QR code images. The feature recognizability of such QR code images is significantly reduced, leading to a decrease in the accuracy of decoding algorithms. Therefore, accurately extracting QR code image features has become a key issue for achieving reliable recognition and decoding.

[0003] In related technologies, features of QR code images are usually extracted through deep learning, such as through the Transformer model. However, the Transformer model has high computational complexity, which makes it difficult to meet the deployment requirements of resource-constrained edge devices. Summary of the Invention

[0004] The purpose of this application is to provide a method, apparatus, electronic device, and computer-readable storage medium for QR code image feature extraction based on a CNN-MALT hybrid architecture. This can solve the problem in related technologies where QR code image feature extraction using Transformer models, etc., is difficult to deploy on resource-constrained edge devices due to the high computational complexity of the Transformer model.

[0005] In a first aspect, embodiments of this application provide a method for extracting QR code image features based on a CNN-MALT hybrid architecture, comprising: inputting a target QR code image into a CNN preprocessing module of a preset lightweight image feature extraction network to generate a first structure-aware feature corresponding to the target QR code; inputting the first structure-aware feature into a MALT encoder of the preset lightweight image feature extraction network, and extracting features from the first structure-aware feature through the MALT encoder based on a surrogate attention mechanism to generate a depth feature corresponding to the target QR code image; and inputting the depth feature into a feature fusion module of the preset lightweight image feature extraction network to generate a target feature corresponding to the target QR code image.

[0006] In one possible implementation of the first aspect, the above-mentioned feature extraction of the first structurally perceptual features by the MALT encoder based on the surrogate attention mechanism includes: The first structural perception feature is transformed into a second structural perception feature of a preset dimension by using a MALT encoder. The MALT encoder extracts features from the second structure-aware features based on the surrogate attention mechanism to generate deep features.

[0007] Optionally, in another possible implementation of the first aspect, the aforementioned MALT encoder includes L MALT layers, where L is a positive integer. The MALT encoder extracts features from the second structurally perceptual features based on a surrogate attention mechanism to generate deep features, including: The second structure-aware feature is input into the first MALT layer, and proxy attention is calculated on the second structure-aware feature matrix to generate the first deep feature corresponding to the first MALT layer. Input the (i-1)th deep feature into the i-th MALT layer, and perform proxy attention calculation on the (i-1)th deep feature to generate the i-th deep feature corresponding to the i-th MALT layer, where i is an integer greater than 1 and less than or equal to L.

[0008] Optionally, in another possible implementation of the first aspect, the above-mentioned surrogate attention calculation on the (i-1)th deep feature to generate the i-th deep feature corresponding to the i-th MALT layer includes: Perform proxy attention calculation on the (i-1)th deep feature to generate the ith proxy attention feature; The second structure-aware feature matrix is ​​input into the i-th MALT layer, and the i-th residual connection matrix is ​​generated according to the i-th preset residual projection matrix and the i-th preset fusion weight. The i-th deep feature is generated based on the i-th agent attention feature and the i-th residual connection matrix.

[0009] Optionally, in another possible implementation of the first aspect, the j-th MALT layer includes a first parameter matrix and a second parameter matrix, wherein the first parameter matrix is ​​a shared parameter matrix of all MALT layers, and the second parameter matrix is ​​a unique parameter matrix corresponding to the j-th MALT layer, where j is an integer greater than or equal to 1 and less than or equal to L.

[0010] Optionally, in another possible implementation of the first aspect, the feature fusion module that inputs deep features into a preset lightweight image feature extraction network to generate target features corresponding to the target QR code image includes: L deep features are input into the feature fusion module to select a preset number of deep features from the L deep features; The feature fusion module fuses a preset number of deep features to generate target features.

[0011] Optionally, in another possible implementation of the first aspect, the aforementioned CNN preprocessing module includes depthwise separable convolutional layers and small kernel convolutional layers. The aforementioned CNN preprocessing module, which inputs the target QR code image into a preset lightweight image feature extraction network, generates the first structure-aware features corresponding to the target QR code, including: Initial feature extraction is performed on the target QR code image using depthwise separable convolutional layers to generate the first feature map; Deep feature extraction is performed on the first feature map using a small kernel convolutional layer to generate the first structure-aware feature.

[0012] Secondly, this application also provides a QR code image feature extraction device based on a CNN-MALT hybrid architecture, comprising: a first generation module for inputting a target QR code image into a CNN preprocessing module of a preset lightweight image feature extraction network to generate a first structure-aware feature corresponding to the target QR code; a second generation module for inputting the first structure-aware feature into a MALT encoder of a preset lightweight image feature extraction network, and extracting features from the first structure-aware feature through the MALT encoder based on a surrogate attention mechanism to generate a depth feature corresponding to the target QR code image; and a third generation module for inputting the depth feature into a feature fusion module of a preset lightweight image feature extraction network to generate a target feature corresponding to the target QR code image.

[0013] In one possible implementation of the second aspect, the aforementioned second generation module includes: The first generation unit is used to perform dimensional transformation on the first structure-aware features through the MALT encoder to generate a second structure-aware feature with a preset dimension. The second generation unit is used to extract features from the second structure-aware features using the MALT encoder based on the surrogate attention mechanism, and generate deep features.

[0014] Optionally, in another possible implementation of the second aspect, the MALT encoder described above includes L MALT layers, where L is a positive integer, and the second generation unit described above is specifically used for: The second structure-aware feature is input into the first MALT layer, and proxy attention is calculated on the second structure-aware feature matrix to generate the first deep feature corresponding to the first MALT layer. Input the (i-1)th deep feature into the i-th MALT layer, and perform proxy attention calculation on the (i-1)th deep feature to generate the i-th deep feature corresponding to the i-th MALT layer, where i is an integer greater than 1 and less than or equal to L.

[0015] Optionally, in another possible implementation of the second aspect, the aforementioned second generating unit is further used for: Perform proxy attention calculation on the (i-1)th deep feature to generate the ith proxy attention feature; The second structure-aware feature matrix is ​​input into the i-th MALT layer, and the i-th residual connection matrix is ​​generated according to the i-th preset residual projection matrix and the i-th preset fusion weight. The i-th deep feature is generated based on the i-th agent attention feature and the i-th residual connection matrix.

[0016] Optionally, in another possible implementation of the second aspect, the j-th MALT layer includes a first parameter matrix and a second parameter matrix. The first parameter matrix is ​​a shared parameter matrix of all MALT layers, and the second parameter matrix is ​​a unique parameter matrix corresponding to the j-th MALT layer, where j is an integer greater than or equal to 1 and less than or equal to L.

[0017] Optionally, in another possible implementation of the second aspect, the aforementioned third generation module includes: The selection unit is used to input L deep features into the feature fusion module to select a preset number of deep features from the L deep features; The third generation unit is used to fuse a preset number of deep features through the feature fusion module to generate target features.

[0018] Optionally, in another possible implementation of the second aspect, the aforementioned CNN preprocessing module includes depthwise separable convolutional layers and small kernel convolutional layers, and the aforementioned first generation module includes: The fourth generation unit is used to perform initial feature extraction on the target QR code image through a depthwise separable convolutional layer to generate the first feature map; The fifth generation unit is used to perform deep feature extraction on the first feature map through a small kernel convolutional layer to generate the first structure-aware feature.

[0019] Thirdly, this application also provides an electronic device. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the computer program to implement any of the implementations of the first aspect described above.

[0020] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the method of any of the implementations of the first aspect described above.

[0021] Fifthly, this application also provides a computer program product that, when run on an electronic device, causes the electronic device to execute any of the implementation methods of the first aspect described above.

[0022] Compared with the prior art, the beneficial effects of this application embodiment are as follows: the structural features of the QR code image are perceived by the CNN preprocessing module, ensuring the structural integrity of the QR code image; the depth features of the QR code image are extracted by the MALT encoder based on the surrogate attention mechanism, capturing the global dependencies of the QR code image in an approximately linear complexity manner, significantly reducing computational overhead; and the depth features are fused by the feature fusion module, reducing redundant features while enhancing feature expression, thereby ensuring the extraction accuracy of QR code image features, reducing computational complexity, and improving the deployment success rate on edge devices with limited computing resources. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating a QR code image feature extraction method based on a CNN-MALT hybrid architecture provided in an embodiment of this application; Figure 2 This is a schematic diagram of the process for generating agent attention features in a MALT layer according to an embodiment of this application; Figure 3 This is a schematic diagram of a process for scaling depth features according to an embodiment of this application; Figure 4 This is a schematic diagram of the structure of the QR code image feature extraction device based on the CNN-MALT hybrid architecture provided in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0025] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0026] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0027] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0028] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0029] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0030] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0031] QR codes, as an efficient information carrier, have been widely used in complex industrial environments such as logistics management, mobile payment, and industrial traceability. However, in practical applications, the acquired QR code images are often severely degraded due to factors such as uneven lighting, motion blur, physical wear, partial occlusion, or perspective distortion, resulting in low-quality QR code images. The feature recognizability of such QR code images is significantly reduced, leading to a decrease in the accuracy of decoding algorithms. Therefore, accurately extracting QR code image features has become a key issue for achieving reliable recognition and decoding.

[0032] In related technologies, features of QR code images are usually extracted through deep learning, such as through the Transformer model. However, the Transformer model has high computational complexity, which makes it difficult to meet the deployment requirements of resource-constrained edge devices.

[0033] Existing lightweight methods, such as model compression and knowledge distillation, reduce model parameters, but cannot guarantee model performance, i.e., cannot guarantee the accuracy of QR code image feature extraction.

[0034] Based on this, this application proposes a method, apparatus, electronic device, storage medium, and computer program for extracting QR code image features based on a CNN-MALT hybrid architecture.

[0035] The following description, with reference to the accompanying drawings, details the QR code image feature extraction method, apparatus, electronic device, storage medium, and computer program based on the CNN-MALT hybrid architecture provided in this application.

[0036] Figure 1 The illustration shows a flowchart of a QR code image feature extraction method based on a CNN-MALT hybrid architecture provided in an embodiment of this application.

[0037] Step 101: Input the target QR code image into the CNN preprocessing module of the preset lightweight image feature extraction network to generate the first structural perception feature corresponding to the target QR code.

[0038] It should be noted that the QR code image feature extraction method based on the CNN-MALT hybrid architecture of this application embodiment can be executed by the QR code image feature extraction device based on the CNN-MALT hybrid architecture of this application embodiment. The QR code image feature extraction device based on the CNN-MALT hybrid architecture of this application embodiment can be configured in any electronic device to execute the QR code image feature extraction method based on the CNN-MALT hybrid architecture of this application embodiment.

[0039] As one possible implementation, the target QR code image can be a QR code image captured by a sensor (such as a camera), or a low-quality QR code image that is deformed, blurred, or occluded due to factors such as uneven lighting, motion blur, physical wear, partial occlusion, or perspective distortion, or it can be a normal QR code image.

[0040] In one possible implementation of this application, the target QR code image can be input into a preprocessing module of a pre-defined lightweight image feature extraction network, which is a convolutional neural network (CNN). The CNN preprocessing module may include convolutional layers, pooling layers, etc. The spatial invariance and translation invariance of the CNN can be used to perceive the structural features of the target QR code image and generate the first structural perceptual features of the target QR code image.

[0041] For example, the first structural sensing features may include "line structure features", "corner features", "T-joint features", "L-joint features" of the target QR code image, etc.

[0042] It should be noted that the various network layers included in the CNN preprocessing module listed above, and the various features included in the first structure perception feature, are only exemplary. In actual use, they can be determined according to actual usage needs and application scenarios. This application embodiment does not limit them in this regard.

[0043] Optionally, to reduce computational costs and ensure the accuracy of QR code image feature extraction, features of the target QR code image can be extracted using depthwise separable convolutional layers and small kernel convolutional layers. That is, in one possible implementation of this application embodiment, the aforementioned CNN preprocessing module includes depthwise separable convolutional layers and small kernel convolutional layers. The target QR code image is input into the CNN preprocessing module of a preset lightweight image feature extraction network to generate the first structure-aware features corresponding to the target QR code, which may include: Initial feature extraction is performed on the target QR code image using depthwise separable convolutional layers to generate the first feature map; Deep feature extraction is performed on the first feature map using a small kernel convolutional layer to generate the first structure-aware feature.

[0044] As one possible implementation, the CNN preprocessing module may include depthwise separable convolutional layers and little-kernel convolutional layers. These two layers can work together in series within the CNN preprocessing module to jointly extract features from the target QR code image. It should be noted that depthwise separable convolutional layers and little-kernel convolutional layers can refer to individual operation layers, or they can be sub-modules composed of multiple operation layers of the same type. This application embodiment does not limit this.

[0045] First, initial feature extraction of the target QR code image can be performed using depthwise separable convolutional layers to generate the first feature map. This can be divided into two steps: depthwise convolution and pointwise convolution. In the depthwise convolution step, the operation can be performed only in the spatial dimensions (i.e., height and width) of the input target QR code image, while remaining independent in the channel dimension. Specifically, assuming the target QR code image has C channels, it can be processed by C independent convolution kernels, each responsible for performing a two-dimensional convolution operation on one channel. This process extracts the unique spatial patterns within each channel (such as edges or textures under specific color components), but does not perform cross-channel information fusion. The number of channels in the output feature map of the depthwise convolution step remains consistent with the input, still being C. Next, after the depthwise convolution step, a pointwise convolution step is performed. As an example, a 1×1 convolution kernel can be used to linearly combine and fuse the C feature maps output by the aforementioned depthwise convolution in the channel dimension. Each 1×1 convolution kernel can generate the first feature map by weighting and summing the values ​​of all C channels at the same spatial location with learnable weights. The first feature map can include preliminary, low-level visual structural information of the target QR code image, and its dimension can be determined based on the number of 1×1 convolutional kernels.

[0046] This approach reduces computational complexity and the number of parameters while maintaining the accuracy of feature extraction.

[0047] Next, deep feature extraction can be performed on the first feature map using small kernel convolutional layers to generate first structure-aware features. The small kernel convolutional layer can include convolutional layers with small kernel sizes (e.g., 3×3, 2×2, etc.). In this embodiment, the CNN preprocessing module can include one or more small kernel convolutional layers, but is not limited to this.

[0048] Specifically, each small-kernel convolutional layer receives a first feature map from the output of a depthwise separable convolutional layer as input. Within its local receptive field (e.g., a 3×3 region), it combines and transforms the first feature map. By stacking multiple such layers consecutively, increasingly complex feature representations can be gradually constructed "from local to global," thereby generating the first structure-aware features. In other words, the first feature map can be constructed hierarchically from the small-kernel convolutional layer into a structure-aware representation specific to the QR code image, thus generating the first structure-aware features. Through depthwise separable convolutional layers and small-kernel convolutional layers, it is ensured that the CNN preprocessing module, while maintaining a lightweight design, provides high-precision input features rich in complete geometric constraints for the subsequent Multi-Agent Lightweight Transformer (MALT) encoder.

[0049] For example, the first kernel convolutional layer can combine the basic edges in the first feature map into simple corners or endpoints; the second kernel convolutional layer can further combine the simple corners or endpoints to identify intermediate geometric structures such as "L" shapes, "T" shapes, or short line segments; deeper kernel convolutional layers can combine these intermediate geometric structures into patterns that are highly correlated with the characteristics of QR code symbols, such as local fragments of the positioning graphic, the prototype of the alignment pattern, etc.

[0050] Step 102: Input the first structure-aware features into the MALT encoder of the preset lightweight image feature extraction network, and use the MALT encoder to extract features from the first structure-aware features based on the surrogate attention mechanism to generate the depth features corresponding to the target QR code image.

[0051] As one possible implementation, the pre-defined lightweight image feature extraction network may include a MALT encoder. The MALT encoder can extract features from the first structural perception features based on a surrogate attention mechanism. By utilizing the long-distance dependency modeling capability of the surrogate attention mechanism, the first structural perception features corresponding to the target QR code image under extreme conditions such as severe occlusion, deformation, and blurring can be globally associated. Furthermore, since the surrogate attention mechanism introduces a surrogate matrix with a feature count much smaller than that of the first structural perception features as a connection matrix for information interaction, the dense computation mode of fully connected layers can be transformed into an efficient two-stage sparse computation mode. Therefore, feature extraction based on the surrogate attention mechanism can reduce the computational complexity of feature extraction while ensuring the accuracy of feature extraction.

[0052] Optionally, since the data representation method used in the MALT encoder may differ from that in the CNN preprocessing module, the first structure-aware features can first undergo dimensionality transformation to convert them into dimensions suitable for the MALT encoder. That is, in one possible implementation of this application, the above-mentioned feature extraction of the first structure-aware features using the MALT encoder based on a surrogate attention mechanism may include: The first structural perception feature is transformed into a second structural perception feature of a preset dimension by using a MALT encoder. The MALT encoder extracts features from the second structure-aware features based on the surrogate attention mechanism to generate deep features.

[0053] As one possible implementation, the features of each channel of the first structural sensing feature can be flattened first, and the first structural sensing feature can be dimensionally transformed through a learnable linear projection layer to generate a second structural sensing feature with a preset dimension.

[0054] As an example, suppose the first structure-aware feature It is a three-dimensional tensor with shape [H, W, C], where H and W represent the first structural perceptual features. The spatial height and width, C represents the number of feature channels, and the MALT encoder expects to receive an input format that is a two-dimensional matrix sequence with shape [N, D], where N = H × W, representing the total number of spatial locations (i.e., the first structure-aware features). The number of features (D is a preset dimension). First, the first structural perceptual features can be... Flatten the features of each channel, and perceive the first structural features. The shape is transformed to [N, C]. Then, the first structure's perceptual features can be obtained through a learnable linear projection layer. Perform the following transformation:

[0055] in, Let be a learnable projection matrix with shape [C, D]. Let be a learnable bias vector with shape [N, D]. The second structural sensing feature has a shape of [N, D].

[0056] As one possible implementation, feature extraction can be performed on the second structure-aware features based on a surrogate attention mechanism. First, a learnable surrogate matrix A can be pre-defined, with a shape of [M, D], where M is much smaller than N, and M represents the number of surrogates. Second, the second structure-aware features can be mapped to the query space using a learnable query projection matrix to generate a query matrix Q. Similarly, the second structure-aware features can be mapped to the key space using a learnable key projection matrix to generate a key matrix K. Finally, the second structure-aware features can be mapped to the value space using a learnable value projection matrix to generate a value matrix V. Next, deep features can be generated based on Q, K, V, and A. The specific method for calculating surrogate attention is as follows: The first step is to calculate the dot product between the query matrix Q and the transpose of the surrogate matrix A, and then use the softmax function. Normalization is performed to obtain the correlation strength between each feature in the second structure-aware feature set and each agent. The second step involves calculating the dot product between the transpose of the agent matrix A and the key matrix K, also processed using the softmax function. Normalization is performed, and then the normalized result is weighted and summed in the value matrix V. Finally, the result obtained in the first step is multiplied by the result obtained in the second step, thus generating the surrogate attention feature. At this point, each original feature in the second-structure perceptual feature has obtained a new representation enhanced based on global contextual information, and the surrogate attention feature can be used as a deep feature. The formula for calculating the surrogate attention feature can be expressed as:

[0057] in, This represents the agent's attention characteristics.

[0058] It should be noted that H, W, C, N, D, and M are all positive integers, and their specific values ​​can be determined according to actual usage requirements and application scenarios. This application does not limit these values.

[0059] Optionally, the MALT encoder may include L stacked MALT layers, where L is a positive integer. Each MALT layer can perform surrogate attention calculation on the output of the previous MALT layer to generate multiple depth features, thereby improving the accuracy of QR code image feature extraction. That is, in one possible implementation of this application embodiment, the above-mentioned MALT encoder includes L MALT layers, and the above-mentioned feature extraction of the second structure-aware features based on the surrogate attention mechanism using the MALT encoder to generate depth features may include: The second structure-aware feature is input into the first MALT layer, and proxy attention is calculated on the second structure-aware feature matrix to generate the first deep feature corresponding to the first MALT layer. Input the (i-1)th deep feature into the i-th MALT layer, and perform proxy attention calculation on the (i-1)th deep feature to generate the i-th deep feature corresponding to the i-th MALT layer, where i is an integer greater than 1 and less than or equal to L.

[0060] In one possible implementation, the second structure-aware features can be input into the first MALT layer. The first MALT layer can then map the second structure-aware features to the query space using a learnable query projection matrix, generating a query matrix. The second structure-aware features can be mapped to the key space using a learnable key projection matrix to generate a key matrix. The second structure-aware features can be mapped to the value space using a learnable value projection matrix to generate a value matrix. And then according to , , And A, generate the first agent attention feature corresponding to the first MALT layer. And can As the output of the first MALT layer, i.e. the first deep feature, the specific generation method can be referred to the description in the above embodiments, and will not be repeated here.

[0061] Next, the (i-1)th deep feature can be input into the i-th MALT layer, where i is an integer greater than 1 and less than or equal to L. For example, the first deep feature can be input into the second MALT layer, the second deep feature can be input into the third MALT layer, and so on. This will not be elaborated further here.

[0062] In the i-th MALT layer, the (i-1)-th deep feature can be mapped to the query space using a learnable query projection matrix, thus generating the query matrix. The (i-1)th deep feature can be mapped to the key space using a learnable key projection matrix to generate the key matrix. The (i-1)th deep feature can be mapped to the value space using a learnable value projection matrix to generate a value matrix. And then according to , , And A, generate the i-th agent attention feature corresponding to the i-th MALT layer. And can As the output of the i-th MALT layer, i.e. the i-th depth feature, the specific generation method can be referred to the description in the above embodiments, and will not be repeated here.

[0063] It should be noted that the query projection matrix, key projection matrix and value projection matrix corresponding to each of the above MALT layers may be the same or different. The specifics can be determined according to actual usage requirements and application scenarios. This application embodiment does not limit this.

[0064] Optionally, residual connections can be added to each MALT layer to ensure that even if the MALT encoder performs layer-by-layer depth feature extraction, its output depth features are still constrained by the original geometric structure (i.e., the second structure-aware features). That is, in one possible implementation of this application, the above-mentioned surrogate attention calculation for the (i-1)th depth feature to generate the i-th depth feature corresponding to the i-th MALT layer can include: Perform proxy attention calculation on the (i-1)th deep feature to generate the ith proxy attention feature; The second structure-aware feature matrix is ​​input into the i-th MALT layer, and the i-th residual connection matrix is ​​generated according to the i-th preset residual projection matrix and the i-th preset fusion weight. The i-th deep feature is generated based on the i-th agent attention feature and the i-th residual connection matrix.

[0065] In one possible implementation of this application, a proxy attention feature can be generated by performing proxy attention calculation on the second structural perception feature through a first MALT layer. The specific calculation method can be referred to the detailed description in the above embodiments, and will not be repeated here. The second structure-aware feature matrix can be input into the first MALT layer, and the first residual connection matrix can be generated according to the first preset residual projection matrix and the first preset fusion weights. Then, the first proxy attention feature... The sum of the sum and the first residual connection matrix is ​​determined as the output of the first MALT layer, i.e., the first deep feature.

[0066] Next, surrogate attention can be calculated on the first deep feature to generate the second surrogate attention feature. The second structure-aware feature matrix can be input into the second MALT layer. Based on the second preset residual projection matrix and the second preset fusion weight, the second residual connection matrix is ​​generated. Then, the sum of the second agent attention feature and the second residual connection matrix can be determined as the output of the second MALT layer, i.e., the second depth feature.

[0067] And so on, such as Figure 2 As shown, the (i-1)th depth feature output from the (i-1)th MALT layer can be input into the ith MALT layer, and surrogate attention can be calculated on the (i-1)th depth feature to generate the ith surrogate attention feature. The second structurally perceptual feature matrix can be input into the i-th MALT layer. Based on the i-th preset residual projection matrix and the i-th preset fusion weights, the i-th residual connection matrix is ​​generated. The sum of the i-th proxy attention feature and the i-th residual connection matrix can be determined as the i-th deep feature. The calculation formula can be expressed as follows: +

[0068] in, This represents the i-th preset fusion weight. This represents the i-th preset residual projection matrix. and All are learnable parameters, and each MALT layer and They can be the same or different, and the embodiments in this application do not limit this.

[0069] This implementation method reduces the computational complexity to O(NM).

[0070] Optionally, to further reduce the weight of QR code image feature extraction while maintaining its accuracy, the parameter matrices within each MALT layer (such as query mapping matrix, key mapping matrix, value mapping matrix, and preset residual projection matrix) can be designed using hierarchical parameter decomposition. The parameter matrix of the j-th MALT layer (where j is an integer greater than or equal to 1 and less than or equal to L) is... It can include the first parameter matrix and the second parameter matrix ,Right now It can be represented as:

[0071] Wherein, the first parameter matrix The second parameter matrix is ​​a shared parameter matrix across all MALT layers, which can be shared by all L MALT layers. It learns the general, fundamental linear transformation pattern common to all MALT layers. This is the unique parameter matrix corresponding to the j-th MALT layer, which can be specifically adjusted or fine-tuned for the j-th MALT layer.

[0072] Optional, second parameter matrix It can be learned from two low-rank matrices, i.e. Assume the dimension of the first structural perception matrix is ​​D. , ,rank .

[0073] By using hierarchical parameter decomposition design, a large number of parameters can be reused, avoiding redundancy caused by completely independent parameters in each layer. Without significantly sacrificing performance, the size of the preset lightweight image feature extraction network is greatly reduced.

[0074] Step 103: Input the depth features into the feature fusion module of the preset lightweight image feature extraction network to generate the target features corresponding to the target QR code image.

[0075] In one possible implementation, the pre-defined lightweight image feature extraction network may include a feature fusion module. This module inputs various depth features, performs size unification and channel alignment on each feature, and then stitches and fuses the processed features to generate the target features corresponding to the target QR code image.

[0076] Optionally, representative deep features can be selected for fusion to reduce computational resource consumption, achieving model lightweighting while maintaining the accuracy of QR code image feature extraction. That is, in one possible implementation of this application embodiment, the feature fusion module that inputs deep features into a preset lightweight image feature extraction network to generate target features corresponding to the target QR code image may include: L deep features are input into the feature fusion module to select a preset number of deep features from the L deep features; The feature fusion module fuses a preset number of deep features to generate target features.

[0077] In one possible implementation of this application, the number of selected depth features can be preset, and a preset number of target MALT layers can be determined based on the preset number. The depth features output by each target MALT layer are determined as a preset number of depth features selected from L depth features.

[0078] As an example, such as Figure 3 As shown, the MALT encoder includes L MALT layers. In order to make the selected deep features representative, the shallow MALT layer with rich details, the medium-depth MALT layer with balanced details and semantics, the deeper MALT layer with rich semantics, and the deepest MALT layer with global semantics can be identified as the target MALT layer. For example, the L / 4, L / 2, L3 / 4 and L MALT layers can be identified as the target MALT layers, and the deep features output by the L / 4, L / 2, L3 / 4 and L MALT layers can be selected.

[0079] As an example, with L=12 and the preset quantity being 4, the depth features output from the 3rd, 6th, 9th, and 12th MALT layers can be selected.

[0080] Next, since the depth features output by different MALT layers may have different sizes and number of channels, scale unification and channel alignment can be performed on the selected depth features. As an example, scale unification can be achieved by upsampling operations (such as bilinear interpolation) or downsampling operations to adjust the spatial size of the selected depth features to a preset size. Channel alignment can be achieved by using 1x1 convolutions to adjust the number of channels for each depth feature to a uniform target value.

[0081] For example, such as Figure 3As shown, the size of the depth feature output by the L3 / 4th MALT layer can be used as the preset size, and the depth features output by the L / 4th MALT layer and the L / 2th MALT layer can be upsampled, while the depth feature output by the Lth MALT layer can be downsampled, thereby performing scale unification processing on each depth feature, and the processed depth features at each scale can be fused.

[0082] Finally, the processed deep features are concatenated along the channel dimension to generate highly information-dense and complementary target features.

[0083] It should be noted that the above-described deep feature fusion method is merely exemplary and can be determined according to actual usage requirements and application scenarios. For example, feature fusion can also be performed through weighted summation or other methods, which is not limited in this application embodiment.

[0084] This approach incorporates both spatial details such as precise edges and corners from shallow features (ensuring accuracy in localization and module segmentation) and high-level semantic information from deep features, including information about the QR code version, overall structure, and reasonable inferences about damaged areas based on context. This approach achieves both the accuracy and robustness of feature extraction and the lightweighting of the pre-defined lightweight image feature extraction network.

[0085] The QR code image feature extraction method based on the CNN-MALT hybrid architecture provided in this application uses a CNN preprocessing module to perceive the structural features of the QR code image, ensuring the structural integrity of the QR code image; it uses a MALT encoder to extract the depth features of the QR code image based on a proxy attention mechanism, capturing the global dependencies of the QR code image in an approximately linear complexity manner, significantly reducing computational overhead; and it uses a feature fusion module to fuse the depth features, reducing redundant features while enhancing feature representation, thereby ensuring the accuracy of QR code image feature extraction, reducing computational complexity, and improving the deployment success rate on edge devices with limited computing resources.

[0086] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0087] Corresponding to the QR code image feature extraction method based on the CNN-MALT hybrid architecture in the above embodiment, Figure 4 The diagram shows a structural block diagram of a QR code image feature extraction device based on a CNN-MALT hybrid architecture provided in this application embodiment. For ease of explanation, only the parts related to this application embodiment are shown.

[0088] Reference Figure 4 The device 40 includes: The first generation module 41 is used to input the target QR code image into the CNN preprocessing module of the preset lightweight image feature extraction network to generate the first structural perception feature corresponding to the target QR code. The second generation module 42 is used to input the first structure-aware features into the MALT encoder of the preset lightweight image feature extraction network, and extract features from the first structure-aware features through the MALT encoder based on the surrogate attention mechanism to generate the depth features corresponding to the target QR code image. The third generation module 43 is used to input the depth features into the feature fusion module of the preset lightweight image feature extraction network to generate the target features corresponding to the target QR code image.

[0089] In practical use, the QR code image feature extraction device based on the CNN-MALT hybrid architecture provided in this application embodiment can be configured in any electronic device to execute the aforementioned QR code image feature extraction method based on the CNN-MALT hybrid architecture.

[0090] The QR code image feature extraction device based on the CNN-MALT hybrid architecture provided in this application uses a CNN preprocessing module to perceive the structural features of the QR code image, ensuring the structural integrity of the QR code image; it uses a MALT encoder to extract the depth features of the QR code image based on a surrogate attention mechanism, capturing the global dependencies of the QR code image in an approximately linear complexity manner, significantly reducing computational overhead; and it uses a feature fusion module to fuse the depth features, reducing redundant features while enhancing feature representation, thereby ensuring the accuracy of QR code image feature extraction, reducing computational complexity, and improving the deployment success rate on edge devices with limited computing resources.

[0091] In one possible implementation of this application, the second generation module 42 includes: The first generation unit is used to perform dimensional transformation on the first structure-aware features through the MALT encoder to generate a second structure-aware feature with a preset dimension. The second generation unit is used to extract features from the second structure-aware features using the MALT encoder based on the surrogate attention mechanism, and generate deep features.

[0092] Optionally, in another possible implementation of this application, the MALT encoder includes L MALT layers, where L is a positive integer, and the second generation unit is specifically used for: The second structure-aware feature is input into the first MALT layer, and proxy attention is calculated on the second structure-aware feature matrix to generate the first deep feature corresponding to the first MALT layer. Input the (i-1)th deep feature into the i-th MALT layer, and perform proxy attention calculation on the (i-1)th deep feature to generate the i-th deep feature corresponding to the i-th MALT layer, where i is an integer greater than 1 and less than or equal to L.

[0093] Optionally, in another possible implementation of this application, the second generating unit is further used for: Perform proxy attention calculation on the (i-1)th deep feature to generate the ith proxy attention feature; The second structure-aware feature matrix is ​​input into the i-th MALT layer, and the i-th residual connection matrix is ​​generated according to the i-th preset residual projection matrix and the i-th preset fusion weight. The i-th deep feature is generated based on the i-th agent attention feature and the i-th residual connection matrix.

[0094] Optionally, in another possible implementation of this application, the j-th MALT layer includes a first parameter matrix and a second parameter matrix. The first parameter matrix is ​​a shared parameter matrix of all MALT layers, and the second parameter matrix is ​​a unique parameter matrix corresponding to the j-th MALT layer, where j is an integer greater than or equal to 1 and less than or equal to L.

[0095] Optionally, in another possible implementation of this application, the third generation module 43 mentioned above includes: The selection unit is used to input L deep features into the feature fusion module to select a preset number of deep features from the L deep features; The third generation unit is used to fuse a preset number of deep features through the feature fusion module to generate target features.

[0096] Optionally, in another possible implementation of this application, the CNN preprocessing module includes depthwise separable convolutional layers and small kernel convolutional layers, and the first generation module 41 includes: The fourth generation unit is used to perform initial feature extraction on the target QR code image through a depthwise separable convolutional layer to generate the first feature map; The fifth generation unit is used to perform deep feature extraction on the first feature map through a small kernel convolutional layer to generate the first structure-aware feature.

[0097] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0098] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0099] To implement the above embodiments, this application also proposes an electronic device.

[0100] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application.

[0101] like Figure 5 As shown, the above-mentioned electronic device 200 includes: The system includes a memory 210 and at least one processor 220, and a bus 230 connecting different components (including the memory 210 and the processor 220). The memory 210 stores a computer program, which, when executed by the processor 220, implements the QR code image feature extraction method based on the CNN-MALT hybrid architecture described in this application embodiment.

[0102] Bus 230 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0103] Electronic device 200 typically includes a variety of electronic device readable media. These media can be any available media that can be accessed by electronic device 200, including volatile and non-volatile media, removable and non-removable media.

[0104] Memory 210 may also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 240 and / or cache memory 250. Electronic device 200 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 260 may be used to read and write non-removable, non-volatile magnetic media (… Figure 5 Not shown; usually referred to as a "hard drive"). Although Figure 5 As not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 230 via one or more data media interfaces. Memory 210 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this application.

[0105] A program / utility 280 having a set (at least one) of program modules 270 may be stored in, for example, memory 210. Such program modules 270 include—but are not limited to—an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 270 typically perform the functions and / or methods described in the embodiments of this application.

[0106] Electronic device 200 can also communicate with one or more external devices 290 (e.g., keyboard, pointing device, display 291, etc.), and with one or more devices that enable a user to interact with electronic device 200, and / or with any device that enables electronic device 200 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 292. Furthermore, electronic device 200 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 293. As shown, network adapter 293 communicates with other modules of electronic device 200 via bus 230. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0107] The processor 220 performs various functional applications and data processing by running programs stored in the memory 210.

[0108] It should be noted that the implementation process and technical principles of the electronic device in this embodiment are explained in the foregoing description of the QR code image feature extraction method based on the CNN-MALT hybrid architecture of this application embodiment, and will not be repeated here.

[0109] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0110] This application provides a computer program product that, when run on an electronic device, enables the electronic device to perform the steps described in the various method embodiments above.

[0111] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographic device / electronic device, a recording medium, a computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0112] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0113] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0114] In the embodiments provided in this application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings or direct couplings or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0115] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0116] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for extracting features from QR code images based on a CNN-MALT hybrid architecture, characterized in that, include: The target QR code image is input into the CNN preprocessing module of a preset lightweight image feature extraction network to generate the first structural perception feature corresponding to the target QR code. The first structure-aware feature is input into the MALT encoder of the preset lightweight image feature extraction network. The MALT encoder extracts features from the first structure-aware feature based on the surrogate attention mechanism to generate the depth features corresponding to the target QR code image. The depth features are input into the feature fusion module of the preset lightweight image feature extraction network to generate the target features corresponding to the target QR code image.

2. The QR code image feature extraction method based on a CNN-MALT hybrid architecture as described in claim 1, characterized in that, The step of extracting features from the first structure-aware features using the MALT encoder based on a surrogate attention mechanism to generate depth features corresponding to the target QR code image includes: The first structure-aware feature is dimensionally transformed by the MALT encoder to generate a second structure-aware feature with a preset dimension. The deep features are generated by extracting features from the second structure-aware features using the MALT encoder based on the proxy attention mechanism.

3. The QR code image feature extraction method based on a CNN-MALT hybrid architecture as described in claim 2, characterized in that, The MALT encoder comprises L MALT layers, where L is a positive integer. The process of extracting features from the second structure-aware features using the MALT encoder based on the surrogate attention mechanism to generate the deep features includes: The second structure-aware feature is input into the first MALT layer, and proxy attention is calculated on the second structure-aware feature matrix to generate the first depth feature corresponding to the first MALT layer; The (i-1)th depth feature is input into the i-th MALT layer, and proxy attention is calculated on the (i-1)th depth feature to generate the i-th depth feature corresponding to the i-th MALT layer, where i is an integer greater than 1 and less than or equal to L.

4. The QR code image feature extraction method based on the CNN-MALT hybrid architecture as described in claim 3, characterized in that, The step of performing proxy attention calculation on the (i-1)th depth feature to generate the i-th depth feature corresponding to the i-th MALT layer includes: Perform proxy attention calculation on the (i-1)th depth feature to generate the ith proxy attention feature; The second structure-aware feature matrix is ​​input into the i-th MALT layer, and the i-th residual connection matrix is ​​generated according to the i-th preset residual projection matrix and the i-th preset fusion weight. The i-th deep feature is generated based on the i-th agent attention feature and the i-th residual connection matrix.

5. The QR code image feature extraction method based on a CNN-MALT hybrid architecture as described in any one of claims 3-4, characterized in that, The j-th MALT layer includes a first parameter matrix and a second parameter matrix. The first parameter matrix is ​​a shared parameter matrix for all MALT layers, and the second parameter matrix is ​​a unique parameter matrix corresponding to the j-th MALT layer, where j is an integer greater than or equal to 1 and less than or equal to L.

6. The QR code image feature extraction method based on a CNN-MALT hybrid architecture as described in any one of claims 3-4, characterized in that, The step of inputting the depth features into the feature fusion module of the preset lightweight image feature extraction network to generate the target features corresponding to the target QR code image includes: The L deep features are input into the feature fusion module to select a preset number of deep features from the L deep features; The feature fusion module fuses a preset number of deep features to generate the target feature.

7. The QR code image feature extraction method based on a CNN-MALT hybrid architecture as described in any one of claims 1-4, characterized in that, The CNN preprocessing module includes depthwise separable convolutional layers and small kernel convolutional layers. The step of inputting the target QR code image into the CNN preprocessing module of a preset lightweight image feature extraction network to generate the first structure-aware features corresponding to the target QR code includes: The target QR code image is initially extracted using the depthwise separable convolutional layer to generate a first feature map. The first feature map is subjected to deep feature extraction by a small kernel convolutional layer to generate the first structure-aware feature.

8. A QR code image feature extraction device based on a CNN-MALT hybrid architecture, characterized in that, include: The first generation module is used to input the target QR code image into the CNN preprocessing module of the preset lightweight image feature extraction network to generate the first structure-aware feature corresponding to the target QR code. The second generation module is used to input the first structure-aware features into the MALT encoder of the preset lightweight image feature extraction network, and to extract features from the first structure-aware features through the MALT encoder based on the surrogate attention mechanism to generate the depth features corresponding to the target QR code image. The third generation module is used to input the depth features into the feature fusion module of the preset lightweight image feature extraction network to generate the target features corresponding to the target QR code image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it causes the electronic device to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by an electronic device, it implements the method as described in any one of claims 1 to 7.

Images