Code type classification method and device based on sampling enhancement, equipment, storage medium and program product

Abstract

The application is suitable for the technical field of image processing, and provides a code type classification method and device based on sampling enhancement, equipment, a storage medium and a program product. The method comprises the following steps: inputting a to-be-classified barcode image into a feature extraction module of a preset classification network to generate a sampling feature map; inputting the sampling feature map into a sampling enhancement module of the preset classification network to generate a saliency score map; inputting the sampling feature map and the saliency score map into a reactivation module of the preset classification network to generate a reactivation feature map; and inputting the reactivation feature map into a classification prediction module of the preset classification network to determine a code type category corresponding to the barcode image. Thus, the sampling enhancement module performs spatial saliency detection on the sampling feature map, and the sampling feature map is subjected to feature reactivation based on the saliency score map, so that the network can effectively distinguish the code region and the background region with a small overhead and suppress background interference, thereby improving the accuracy of code type classification while ensuring real-time performance.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, and in particular relates to a code classification method, apparatus, electronic device, computer-readable storage medium and computer program product based on sampling enhancement. Background Technology

[0002] Barcode technology (including 1D and 2D barcodes), as a highly efficient data information carrier, has been widely used in logistics management, mobile payment, industrial manufacturing, medical traceability, and other fields. In the actual decoding process, the decoder needs to call the corresponding decoding algorithm based on the barcode type of the image to be decoded. If the barcode type cannot be correctly determined in advance, the decoder must try multiple decoding algorithms sequentially until successful decoding, resulting in a significant reduction in decoding efficiency. Therefore, classifying the acquired barcode image by barcode type before decoding is a crucial step in achieving efficient decoding.

[0003] In related technologies, traditional algorithms based on manually designed features are typically used to classify barcode images. However, this method is limited by fixed feature extraction rules and struggles to maintain stable classification performance in complex industrial environments (such as changes in lighting, motion blur, and background interference), leading to issues with classification accuracy. Summary of the Invention

[0004] The purpose of this application is to provide a code classification method, apparatus, electronic device and computer-readable storage medium based on sampling enhancement, which can solve the problem in related technologies that code classification is limited by fixed feature extraction rules, resulting in low accuracy of code classification.

[0005] In a first aspect, embodiments of this application provide a barcode classification method based on sampling enhancement, comprising: inputting a barcode image to be classified into a feature extraction module of a preset classification network to extract features from the barcode image and generate a sampling feature map corresponding to the barcode image, wherein the barcode image includes a code region; inputting the sampling feature map into a sampling enhancement module of the preset classification network to perform spatial saliency detection on the sampling feature map and generate a saliency score map, wherein each pixel value in the saliency score map represents the probability that the pixel corresponding to the pixel value belongs to the code region; inputting the sampling feature map and the saliency score map into a reactivation module of the preset classification network to perform feature reactivation on the sampling feature map according to the saliency score map and generate a reactivated feature map; and inputting the reactivated feature map into a classification prediction module of the preset classification network to predict the code category of the reactivated feature map and determine the code category corresponding to the barcode image.

[0006] In one possible implementation of the first aspect, the sampling enhancement module includes a multi-branch convolutional unit and an activation unit. The sampling enhancement module, which inputs the sampled feature map into a preset classification network, performs spatial saliency detection on the sampled feature map to generate a saliency score map, including: The sampled feature map is input into at least one convolutional branch of the multi-branch convolutional unit to generate the first feature map corresponding to each convolutional branch, wherein each convolutional branch has a different functional type. Each first feature map is input into a feedforward neural network of a multi-branch convolutional unit, and a nonlinear transformation is performed on each first feature map to generate a second feature map corresponding to each first feature map. Each second feature map is input into the fusion unit of the multi-branch convolutional unit to fuse the second feature maps and generate a fused feature map. The fused feature map is input into the activation unit, and the fused features are activated pixel by pixel to generate a saliency score map.

[0007] Optionally, in another possible implementation of the first aspect, the feature extraction module includes an initial convolutional layer, a first inverse residual structure, a residual connection unit, and a second inverse residual structure. The feature extraction module, which inputs the barcode image to be classified into a preset classification network, performs feature extraction on the barcode image to generate a sampled feature map corresponding to the barcode image, including: The barcode image is input into the initial convolutional layer to generate the first sampled feature map; The first sampled feature map is input into the first inverse residual structure, and the inverse residual transformation is performed on the first sampled feature map to generate the second sampled feature map; The third sampled feature map is generated by adding the first sampled feature map and the second sampled feature map element by element through the residual connection unit. The third sampled feature map is input into the second inverse residual structure, and a downsampled inverse residual transformation is performed on the third sampled feature map to generate a sampled feature map.

[0008] Optionally, in another possible implementation of the first aspect, the above-mentioned inputting the sampled feature map and saliency score map into the reactivation module of the preset classification network, and performing feature reactivation on the sampled feature map according to the saliency score map to generate a reactivated feature map, includes: The reactivation module multiplies the sampled feature map element-wise with the saliency score map to generate a reactivation feature map.

[0009] Optionally, in another possible implementation of the first aspect, the classification prediction module includes a classification prediction head and N inverted residual structures, where N is an integer greater than 1. The classification prediction module that inputs the reactivated feature map into the preset classification network performs code type prediction on the reactivated feature map to determine the code type corresponding to the barcode image, including: The reactivated feature map is input into the first inverse residual structure to perform downsampling inverse residual transformation on the reactivated feature map, generating the first high-order feature map corresponding to the reactivated feature map; The i-th higher-order feature map is input into the (i+1)-th inverse residual structure to perform downsampling inverse residual transformation on the i-th higher-order feature map, generating the (i+1)-th higher-order feature map corresponding to the reactivation feature map, where i is an integer greater than or equal to 1 and less than N; The Nth high-order feature map is input into the classification prediction head, and feature compression and category mapping are performed on the Nth high-order feature map to determine the code system category.

[0010] Alternatively, in another possible implementation of the first aspect, the aforementioned pre-trained classification network is pre-trained in the following manner: Obtain the training dataset, which includes multiple training samples. Each training sample contains a barcode image sample, the barcode category label corresponding to the barcode image sample, and the corresponding binarized segmentation map. Input the barcode image samples into the initial classification network to determine the predicted saliency score map and the predicted code type category; Based on the predicted saliency score map and the binarized segmentation map, determine the first loss value of the initial classification network; Based on the code system prediction category and the code system category label, determine the second loss value of the initial classification network; Based on the first loss value and the second loss value, the initial classification network is iteratively trained to generate the preset classification network.

[0011] Secondly, this application also provides a sampling-enhanced barcode classification device, comprising: a first generation module, used to input a barcode image to be classified into a feature extraction module of a preset classification network, extract features from the barcode image, and generate a sampling feature map corresponding to the barcode image, wherein the barcode image includes a code region; a second generation module, used to input the sampling feature map into a sampling enhancement module of the preset classification network, perform spatial saliency detection on the sampling feature map, and generate a saliency score map, wherein each pixel value in the saliency score map represents the probability that the pixel corresponding to the pixel value belongs to the code region; a third generation module, used to input the sampling feature map and the saliency score map into a reactivation module of the preset classification network, perform feature reactivation on the sampling feature map according to the saliency score map, and generate a reactivated feature map; and a determination module, used to input the reactivated feature map into a classification prediction module of the preset classification network, perform code category prediction on the reactivated feature map, and determine the code category corresponding to the barcode image.

[0012] In one possible implementation of the second aspect, the sampling enhancement module includes a multi-branch convolutional unit and an activation unit; correspondingly, the second generation module includes: The first generation unit is used to input the sampled feature map into at least one convolutional branch of the multi-branch convolutional unit and generate the first feature map corresponding to each convolutional branch respectively, wherein the functional types of each convolutional branch are different. The second generation unit is used to input each first feature map into the feedforward neural network of the multi-branch convolutional unit, and perform nonlinear transformations on each first feature map to generate the second feature map corresponding to each first feature map. The fusion unit is used to input each second feature map into the fusion unit of the multi-branch convolutional unit, and fuse the second feature maps to generate a fused feature map. The third generation unit is used to input the fused feature map into the activation unit, activate the fused features pixel by pixel, and generate a saliency score map.

[0013] Optionally, in another possible implementation of the second aspect, the feature extraction module includes an initial convolutional layer, a first inverse residual structure, a residual connection unit, and a second inverse residual structure; correspondingly, the first generation module includes: The fourth generation unit is used to input the barcode image into the initial convolutional layer and generate the first sampled feature map; The fifth generation unit is used to input the first sampled feature map into the first inverse residual structure, perform inverse residual transformation on the first sampled feature map, and generate the second sampled feature map. The sixth generation unit is used to add the first sampled feature map and the second sampled feature map element by element through the residual connection unit to generate the third sampled feature map; The seventh generation unit is used to input the third sampled feature map into the second inverse residual structure, perform downsampling inverse residual transformation on the third sampled feature map, and generate a sampled feature map.

[0014] Alternatively, in another possible implementation of the second aspect, the aforementioned third generation module includes: The eighth generation unit is used to generate a reactivation feature map by multiplying the sampled feature map and the saliency score map element by element through the reactivation module.

[0015] Optionally, in another possible implementation of the second aspect, the classification prediction module includes a classification prediction head and N inverted residual structures, where N is an integer greater than 1; correspondingly, the determination module includes: The ninth generation unit is used to input the reactivated feature map into the first inverse residual structure to perform downsampling inverse residual transformation on the reactivated feature map and generate the first high-order feature map corresponding to the reactivated feature map. The tenth generation unit is used to input the i-th higher-order feature map into the (i+1)-th inverse residual structure to perform downsampling inverse residual transformation on the i-th higher-order feature map and generate the (i+1)-th higher-order feature map corresponding to the reactivated feature map, where i is an integer greater than or equal to 1 and less than N. The determination unit is used to input the Nth high-order feature map into the classification prediction head, perform feature compression and category mapping on the Nth high-order feature map, and determine the code system category.

[0016] Alternatively, in another possible implementation of the second aspect, the aforementioned pre-trained classification network is pre-trained in the following manner: Obtain the training dataset, which includes multiple training samples. Each training sample contains a barcode image sample, the barcode category label corresponding to the barcode image sample, and the corresponding binarized segmentation map. Input the barcode image samples into the initial classification network to determine the predicted saliency score map and the predicted code type category; Based on the predicted saliency score map and the binarized segmentation map, determine the first loss value of the initial classification network; Based on the code system prediction category and the code system category label, determine the second loss value of the initial classification network; Based on the first loss value and the second loss value, the initial classification network is iteratively trained to generate the preset classification network.

[0017] Thirdly, this application also provides an electronic device. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the computer program to implement any of the implementations of the first aspect described above.

[0018] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the method of any of the implementations of the first aspect described above.

[0019] Fifthly, this application also provides a computer program product that, when run on an electronic device, causes the electronic device to execute any of the implementation methods of the first aspect described above.

[0020] The beneficial effects of this application embodiment compared with the prior art are as follows: A sampled feature map containing code structure information is generated through a preset classification network feature extraction module, providing a feature basis for subsequent code region localization. Then, a sampling enhancement module performs lightweight spatial saliency detection on the sampled feature map to generate a saliency score map. Next, a reactivation module reactivates the sampled feature map based on the saliency score map, suppressing the feature response of the background region and enhancing the feature response of the code region. Finally, a classification prediction module predicts the category of the reactivated feature map and outputs the code type category. Because the sampling enhancement module and the reactivation module distinguish between the code region and the background region of the barcode image, the classification prediction module can perform category prediction without interference from the background region, thus improving the accuracy of code type classification. Furthermore, since no complex calculations are required in the sampling enhancement module and the reactivation module, the real-time performance of code type classification is guaranteed. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic flowchart of a code classification method based on sampling enhancement provided in an embodiment of this application; Figure 2 This is a schematic diagram of the first inverted residual structure provided in an embodiment of this application; Figure 3 This is a schematic diagram of the second inverted residual structure provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a multi-branch convolutional unit provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of a classification prediction head provided in an embodiment of this application; Figure 6 This is a flowchart illustrating a code classification method based on sampling enhancement provided in another embodiment of this application; Figure 7 This is a schematic diagram of the process for training a preset classification network provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of a code classification device based on sampling enhancement provided in an embodiment of this application; Figure 9 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0023] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0024] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0025] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0026] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0027] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0028] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0029] Barcode technology (including 1D and 2D barcodes), as a highly efficient data information carrier, has been widely used in logistics management, mobile payment, industrial manufacturing, and medical traceability. In the actual decoding process, the decoder needs to call the corresponding decoding algorithm based on the barcode type of the image to be decoded. The barcode type refers to the encoding standard type of the barcode. For example, 1D barcodes include EAN, UPC, ISBN, and Codba codes; 2D barcodes include Fast Response Matrix (FRM), Data Matrix (DMM), and Aztec codes. If the barcode type cannot be correctly determined beforehand, the decoder must try multiple decoding algorithms sequentially until successful decoding, resulting in a significant reduction in decoding efficiency. Therefore, classifying the acquired barcode image by barcode type before decoding is a crucial step in achieving efficient decoding.

[0030] In related technologies, traditional algorithms based on manually designed features or deep learning-based methods are commonly used to classify barcode images. However, the former is limited by fixed feature extraction rules and struggles to maintain stable classification performance in complex industrial environments (such as changes in lighting, motion blur, and background interference), leading to issues with classification accuracy. While the latter improves classification accuracy to some extent, it often comes at the cost of increased model complexity and computational overhead, making it difficult to meet the high real-time requirements of industrial scenarios.

[0031] Based on this, this application proposes a sampling-enhanced code classification method, apparatus, electronic device, storage medium, and computer program, which are described in detail below.

[0032] The sampling-enhanced code classification method, apparatus, electronic device, storage medium, and computer program provided in this application will be described in detail below with reference to the accompanying drawings.

[0033] Figure 1 The diagram shows a schematic flowchart of a code classification method based on sampling enhancement provided in an embodiment of this application.

[0034] Step 101: Input the barcode image to be classified into the feature extraction module of the preset classification network to extract features from the barcode image and generate a sampled feature map corresponding to the barcode image, wherein the barcode image includes the code region.

[0035] As an example, a barcode image includes a code area and a background area, and its dimensions can be 3×H×W, where H and W are the height and width of the image, respectively, and 3 represents the RGB three color channels, but it is not limited to this.

[0036] The pre-trained classification network is a pre-trained convolutional neural network (the pre-training method is detailed below), which may include a feature extraction module, a sampling enhancement module, a reactivation module, and a classification prediction module.

[0037] In one possible implementation of this application, a feature extraction module can perform multi-level feature extraction and spatial downsampling on the barcode image to obtain a sampled feature map. The spatial size of the sampled feature map can be smaller than the original input image to improve the efficiency of subsequent code classification.

[0038] In one example, the feature extraction module may include an initial convolutional layer, a first inverse residual structure, a residual connection unit, and a second inverse residual structure. Step 101 above may include: Step 1011: Input the barcode image into the initial convolutional layer to generate the first sampled feature map; Step 1012: Input the first sampled feature map into the first inverse residual structure, perform inverse residual transformation on the first sampled feature map, and generate the second sampled feature map; Step 1013: The first sampled feature map and the second sampled feature map are added element by element through the residual connection unit to generate the third sampled feature map; Step 1014: Input the third sampled feature map into the second inverse residual structure, perform downsampling inverse residual transformation on the third sampled feature map, and generate a sampled feature map.

[0039] Steps 1011-1014 are described in detail below.

[0040] As an example, in step 1011, the stride of the initial convolutional layer is 2, including ( (a positive integer) number of convolutional kernels. Through... Each convolutional kernel performs a convolution operation on the barcode image. Each kernel slides along the spatial dimension of the barcode image with a stride of 2, performing a weighted summation of the pixel values ​​within the covered area. This involves calculating the linear weighted sum of each pixel and its neighboring pixels, resulting in one channel of the first sampled feature map. Because the stride is 2, the spatial size of the first sampled feature map is reduced to half the size of the barcode image. Since the number of convolutional kernels determines the number of output channels, the number of channels in the first sampled feature map is equal to the number of convolutional kernels. The essence of this process is to map the barcode image from the RGB pixel space to a shallow feature space through convolution operations, thereby achieving both spatial downsampling and channel-level mapping.

[0041] For example, suppose the input barcode image has dimensions of 3×224×224. The initial convolutional layer has a stride of 2 and contains 32 convolutional kernels. After the initial convolutional layer, the output first sampled feature map has dimensions of 32×112×112, that is, a feature map with a length and width of 112 and 32 channels.

[0042] By using the initial convolutional layer, texture information of the barcode image can be extracted to obtain smoother features, while reducing the resolution of the feature map and improving the network's operating efficiency.

[0043] In step 1012, as Figure 2 As shown, the first inverse residual structure consists of a first 1×1 convolutional layer, a depthwise separable convolutional layer with a stride of 1, and a second 1×1 convolutional layer connected in sequence. The first 1×1 convolutional layer and the depthwise separable convolutional layer are each followed by a Rectified Linear Unit (ReLU) activation function with an upper limit of y. The second 1×1 convolutional layer is connected to a linear layer.

[0044] Among them, the ReLU activation function with an upper limit of y (hereinafter referred to as ReLUy) can limit the range of activation values ​​to 0 to y (y is a positive number), reduce the accuracy loss during subsequent quantization calculations, and improve the classification accuracy of the preset classification network.

[0045] For example, y can be 6, and the mathematical expression for ReLU6 is: ReLU6(x) = min(max(x,0),6) As one possible implementation, the inverse residual transformation can include dimensionality upscaling, depthwise separable convolution, and dimensionality reduction of the first sampled feature map.

[0046] Specifically, firstly, the first sampled feature map is updimensionalized using a first 1×1 convolutional layer, that is, the number of channels in the first sampled feature map is expanded by a factor of the original expansion (e.g., 6 times). Then, the data in the updimensionalized first sampled feature map is corrected using the ReLUy activation function (e.g., y = 6). This maps the first sampled feature map from a low-dimensional space to a high-dimensional space and corrects the feature data. In this way, the large amount of feature data in the high-dimensional space is beneficial for subsequent convolutional layers to extract richer texture and structural information, and the corrected feature data can reduce the accuracy loss during subsequent quantization calculations.

[0047] Next, spatial features are extracted from the first sampled feature map after dimensionality upscaling using depthwise separable convolutional layers, and the extracted feature data is then corrected using the ReLUy activation function (e.g., y = 6). Depthwise separable convolution decomposes standard convolution into two steps: depthwise convolution and pointwise convolution. Depthwise convolution uses a 3×3 kernel to independently perform spatial convolution on each channel of the first sampled feature map after dimensionality upscaling, without changing the number of channels. Pointwise convolution uses a 1×1 kernel to fuse information between channels in the output of the depthwise convolution. Compared to standard convolution, depthwise separable convolution reduces the number of parameters and computational cost.

[0048] Finally, the output of the depthwise separable convolutional layer is dimensionality-reduced by a second 1×1 convolutional layer and a linear layer, compressing the high-dimensional features back into a low-dimensional space to generate a second sampled feature map. The size of the second sampled feature map is the same as that of the first sampled feature map. The use of a linear layer here preserves complete feature information and avoids information collapse at the low-dimensional bottleneck.

[0049] For example, the first sampled feature map has a dimension of 32×112×112. First, the number of channels is expanded from 32 to 192 (expansion factor of 6) through a first 1×1 convolutional layer and activated by ReLU6. The output feature map of the first 1×1 convolutional layer has a dimension of 192×112×112. A depthwise separable convolutional layer (with a stride of 1) extracts spatial features from the output feature map of the first 1×1 convolutional layer and activates it by ReLU6. The output feature map of the depthwise separable convolutional layer has a dimension of 192×112×112. Then, the number of channels is compressed back from 192 to 32 through a second 1×1 convolutional layer and a linear layer to generate the second sampled feature map, which has a dimension of 32×112×112.

[0050] In step 1013, the first sampled feature map and the second sampled feature map have the same spatial size H / 2×W / 2 and number of channels. The residual connection unit adds the first sampled feature map and the second sampled feature map element by element. Assume the first sampled feature map is... The second sampling feature map is For each spatial location (i,j) and each channel c, calculate (c,i,j)= (c,i,j)+ (c,i,j) yields the third sampling feature map. .

[0051] By using residual connections, the input feature map (first sampled feature map) is superimposed on the output feature map (second sampled feature map) after inverse residual transformation. This allows the original low-level feature information to be passed across layers, preventing the loss of important low-level structural information in deep networks during feature transformation. Specifically, while inverse residual transformation can extract rich texture and structural features, it may weaken the original low-level information such as contours and edges during dimensionality upscaling and downscaling. By directly superimposing the original first sampled feature map onto the transformed second sampled feature map through residual connections, subsequent network layers can still reference the original low-level structural information while extracting high-level semantic features, thereby enhancing the perception of code region boundaries.

[0052] Next, proceed to step 1014. For example... Figure 3As shown, the second inverse residual structure consists of a first 1×1 convolutional layer, a depthwise separable convolutional layer with a stride of 2, and a second 1×1 convolutional layer connected in sequence. The first 1×1 convolutional layer and the depthwise separable convolutional layer are each followed by a ReLUy activation function (e.g., y = 6), and the second 1×1 convolutional layer is connected to a linear layer.

[0053] Downsampling inverse residual transformation can be based on inverse residual transformation, by setting the stride of depthwise separable convolution to 2, so that the spatial size of the third sampled feature map is reduced to 1 / 2 of the original.

[0054] Specifically, the third sampled feature map first undergoes dimensionality upscaling through a first 1×1 convolutional layer, expanding the number of channels by a factor of 6 (e.g., 6 times). Then, it passes through a depthwise separable convolutional layer with a stride of 2. This convolutional layer extracts spatial features while simultaneously reducing the spatial size of the output feature map to half that of the input feature map due to the stride of 2. Therefore, after the second inverse residual structure downsampling inverse residual transformation, the height of the output feature map is H / 4 and the width is W / 4. Finally, the number of channels is compressed to the target dimension through a second 1×1 convolutional layer and a linear layer, resulting in the sampled feature map. The target dimension can be larger than the original dimension of the third sampled feature map to compensate for the loss of feature representation ability caused by the reduction in spatial information after the depthwise separable convolutional layer.

[0055] By using depthwise separable convolutions with a stride of 2, spatial downsampling is achieved while feature extraction is completed, expanding the receptive field of subsequent network layers and enabling the network to capture code structure information over a wider range. At the same time, the reduction in spatial size reduces the computational load of subsequent modules and improves inference speed.

[0056] For example, the third sampled feature map First, the number of channels is expanded from 32 to 192 through the first 1×1 convolutional layer, to the feature map. Then, spatial features are extracted and downsampled simultaneously using a depthwise separable convolution with a stride of 2 to obtain the feature map. Finally, the number of channels is compressed from 192 to 64 (target dimension) through a second 1×1 convolutional layer and a linear layer to obtain the sampled feature map. .

[0057] Through the above steps, the initial convolutional layer performs preliminary downsampling and channel mapping, compressing the original barcode image to a suitable processing size. A first inverse residual structure (stride of 1) refines features while maintaining resolution, and residual connections preserve shallow structural information. A second inverse residual structure (stride of 2) performs spatial downsampling while completing feature extraction, expanding the receptive field and reducing subsequent computational load. This provides a feature base rich in texture and edge information for subsequent barcode region localization, while also creating the prerequisites for real-time processing.

[0058] It should be noted that the stride, number of convolutional kernels, and upper limit y listed above are merely examples. In actual use, the stride, number of convolutional kernels, and upper limit y can be determined according to actual usage requirements and application scenarios. This application embodiment does not limit these parameters.

[0059] Step 102: Input the sampled feature map into the sampling enhancement module of the preset classification network, perform spatial saliency detection on the sampled feature map, and generate a saliency score map. In the saliency score map, each pixel value represents the probability that the pixel corresponding to the pixel value belongs to the code region.

[0060] Spatial saliency detection refers to calculating the saliency response of each spatial location in the sampled feature map as belonging to the target region (i.e., the code region), thereby generating a saliency score map with the same spatial size as the sampled feature map. The pixel value of each pixel in the saliency score map represents the probability that the corresponding pixel belongs to the code region, with a value ranging from 0 to 1.

[0061] In one embodiment, the sampling enhancement module includes a multi-branch convolutional unit and an activation unit, and step 102 may include: Step 1021: Input the sampled feature map into at least one convolutional branch of the multi-branch convolutional unit to generate the first feature map corresponding to each convolutional branch, wherein the functional types of each convolutional branch are different. Step 1022: Input each first feature map into the feedforward neural network of the multi-branch convolutional unit, and perform nonlinear transformation on each first feature map to generate the second feature map corresponding to each first feature map. Step 1023: Input each second feature map into the fusion unit of the multi-branch convolutional unit to fuse the second feature maps and generate a fused feature map; Step 1024: Input the fused feature map into the activation unit, activate the fused features pixel by pixel, and generate a saliency score map.

[0062] Steps 1021-1024 are explained in detail below.

[0063] A multi-branch convolutional unit can include one or more convolutional branches, a corresponding feedforward network (FFN) for each convolutional branch, and a fusion unit. Each convolutional branch has a different functional type, meaning that each convolutional branch has a different focus in terms of feature extraction objectives and computation methods.

[0064] For example, such as Figure 4 As shown, the branch convolutional unit includes five convolutional branches: Vanilla Convolution (VC), Central Difference Convolution (CDC), Angular Difference Convolution (ADC), Horizontal Difference Convolution (HDC), and Vertical Difference Convolution (VDC). Each of the five convolutional branches corresponds to one FNN, and each FNN transmits its output to the fusion unit. The fusion unit includes stacked sub-units and a 1×1 convolutional layer.

[0065] Among them, the VC branch refers to the standard convolution operation, that is, using a learnable convolution kernel to perform weighted summation on the input sampled feature map, which is used to extract low-frequency feature information (such as illumination and color distribution); the CDC branch calculates gradients along multiple directions, calculates the difference between the center pixel and the neighboring pixels and sums them with weights, which is used to calculate the difference between the center pixel and the neighboring pixels, highlighting edge information and enhancing robustness to illumination changes; the ADC branch calculates differences along the horizontal direction, which is used to capture gradient changes at different angles, so that the network can adapt to features in different directions; the HDC branch is used to extract features in the horizontal direction; and the VDC branch calculates differences along the vertical direction, which is used to extract features in the vertical direction.

[0066] An FNN consists of two 1×1 convolutional layers and an intermediate Gaussian Error Linear Unit (GELU) activation function concatenated together. Its mathematical expression is: FFN(x) = GELU(xW1 + b1)W2 + b2 Where W1 and W2 are the weights of the convolutional kernels of the pre-trained classification network, and b1 and b2 are the biases of the convolutional kernels of the pre-trained classification network.

[0067] As an example, the input of the sampled feature map can be fed in parallel to one or more convolutional branches of the branch convolutional unit, with each convolutional branch extracting features from the sampled feature map and generating a first feature map.

[0068] Then, each convolutional branch inputs the generated first feature map into its corresponding FFN, which performs non-linear reorganization of the first feature map along the channel dimension. The first 1×1 convolution mixes and increases the dimension of the first feature map across the channels, and the GELU activation function introduces a non-linear transformation. The second 1×1 convolution compresses the increased and non-linearly transformed first feature map back to its original dimension, generating the second feature map.

[0069] Next, each second feature map is input into the stacking sub-unit of the fusion unit. The stacking sub-unit splices the second feature maps in the channel dimension to form a stacked feature map with the number of channels equal to the sum of the number of channels of all the second feature maps.

[0070] For example, the second feature maps of the five convolutional branches are all The stacked feature map is obtained by concatenating the features along the channel dimension. .

[0071] Next, the stacked feature map output from the stacked sub-units is input into a 1×1 convolutional layer. The 1×1 convolutional layer has 1 kernel. Through the 1×1 convolutional layer, the stacked feature map is reduced to 1 channel dimensionality, generating a single-channel fused feature map.

[0072] For example, stacked feature maps After inputting a 1×1 convolutional layer with 1 kernel, the output is a fused feature map. .

[0073] Finally, the fused feature map is input into the activation unit, which performs pixel-by-pixel activation on the fused feature map, mapping the pixel value of each pixel in the fused feature map from the real number domain to the (0,1) interval to generate a saliency score map. The pixel value of each pixel in the saliency score map can be used as the saliency score of that pixel.

[0074] As an example, the activation unit may include the Sigmoid activation function, whose mathematical expression is: σ(x) = 1 / (1 + e^(-x)) The Sigmoid activation function converts the real values ​​of each pixel in the fused feature map into probability values ​​between 0 and 1 to generate a saliency score map. In the saliency score map, the closer the pixel value (i.e., the saliency score) is to 1, the higher the probability that the pixel belongs to the code region, and the closer the pixel value is to 0, the higher the probability that the pixel belongs to the background region.

[0075] In a specific example, the fused feature map has a dimension of 1×56×56. The Sigmoid layer performs pixel-by-pixel activation on this fused feature map to generate a saliency score map with a dimension of 1×56×56. Each pixel value in this saliency score map represents the probability that the pixel belongs to a code region.

[0076] By utilizing multi-branch convolutions with different functional types, the network can better learn features from different dimensions of the sampled feature maps. Simultaneously, the non-linear activation function in FFN allows the network to fit complex functions, performing non-linear refinement on the first feature map output from each convolutional branch. Finally, fusion and activation are used to generate a saliency score map with clear probabilistic meaning. This approach avoids high-complexity computations or iterative optimizations, distinguishing code regions from the background with relatively low computational overhead, thus balancing detection accuracy and computational efficiency.

[0077] Step 103: Input the sampled feature map and saliency score map into the reactivation module of the preset classification network, and perform feature reactivation on the sampled feature map according to the saliency score map to generate a reactivated feature map.

[0078] As one possible implementation, feature reactivation can refer to recalibrating each spatial location of the sampled feature map using the saliency score map as spatial weights. The feature responses of locations with high saliency scores (code regions) are preserved or enhanced, while the feature responses of locations with low saliency scores (background regions) are suppressed.

[0079] In one example, step 103 above may include: Step 1031: The sampled feature map is multiplied element-wise with the saliency score map by the reactivation module to generate a reactivation feature map.

[0080] As an example, the sampled feature map Significance score plot The input is processed by the reactivation module, which then converts the sampled feature map into a single image. Significance score plot Element-wise multiplication generates reactivation feature maps. .

[0081] For example, for the spatial location (i,j) (i,j are integers greater than or equal to 0 and less than 56), (i,j)= (i,j)× (i,j).

[0082] Step 104: Input the reactivated feature map into the classification prediction module of the preset classification network, perform code type prediction on the reactivated feature map, and determine the code type corresponding to the barcode image.

[0083] One possible implementation is to use a classification prediction module to perform high-order semantic abstraction and category mapping on the reactivation feature map, outputting the code type category. After the reactivation feature map is processed by the sampling enhancement module, the features of the code region are enhanced and the features of the background region are suppressed. However, the features at this point are still in the mid-level feature space and have not yet formed a high-order semantic representation that can be used for classification. The classification prediction module gradually expands the receptive field of the network through multiple consecutive downsampling, enabling the preset classification network to capture the global layout structure of the code (such as the three positioning back-character patterns of the fast response matrix code and the L-shaped edge-finding pattern of the data matrix code). At the same time, it gradually abstracts the features from the texture level to the semantic level, providing high-order semantic features for the final category judgment. Finally, the classification prediction head compresses the high-order semantic features into a one-dimensional vector and maps it to the category space, outputting the code type category.

[0084] In one example, the classification prediction module includes a classification prediction head and N inverted residual structures, where N is an integer greater than 1. Step 104 above may include: Step 1041: Input the reactivated feature map into the first inverse residual structure to perform downsampling inverse residual transformation on the reactivated feature map and generate the first high-order feature map corresponding to the reactivated feature map. Step 1042: Input the i-th higher-order feature map into the (i+1)-th inverse residual structure to perform downsampling inverse residual transformation on the i-th higher-order feature map and generate the (i+1)-th higher-order feature map corresponding to the reactivated feature map, where i is an integer greater than or equal to 1 and less than N. Step 1043: Input the Nth high-order feature map into the classification prediction head, perform feature compression and category mapping on the Nth high-order feature map, and determine the code system category.

[0085] As one possible implementation, the inverted residual structure can be as follows: Figure 3 As shown, its specific description can be found in the description of the second inverse residual structure in the above embodiments, and will not be repeated here. The specific description of the downsampling inverse residual transformation can be found in the relevant description in the above embodiments, and will not be repeated here.

[0086] Among them, such as Figure 5 As shown, the classification prediction head can consist of a Global Average Pooling (GAP) layer, a linear layer, and a normalized exponential function layer.

[0087] As an example, with N=3, the reactivated feature map is input into the first inverse residual structure to perform downsampling inverse residual transformation on the reactivated feature map, generating the first high-order feature map corresponding to the reactivated feature map.

[0088] Next, the first high-order feature map is input into the second inverse residual structure to perform downsampling inverse residual transformation on the first high-order feature map, generating the second high-order feature map corresponding to the reactivated feature map.

[0089] Then, the second higher-order feature map is input into the third inverse residual structure to perform downsampling inverse residual transformation on the second higher-order feature map, generating the third higher-order feature map corresponding to the reactivated feature map.

[0090] For example, reactivation feature maps After the first inverse residual structure (step size 2) performs downsampling inverse residual transformation, the first high-order feature map has a dimension of 128×28×28; after the second inverse residual structure (step size 2) performs downsampling inverse residual transformation, the second high-order feature map has a dimension of 256×14×14; after the third inverse residual structure (step size 2) performs downsampling inverse residual transformation, the third high-order feature map has a dimension of 512×7×7.

[0091] Next, the third high-order feature map is input into the GAP layer of the classification prediction head, and feature compression is performed on the third high-order feature map through the GAP layer. Specifically, in each channel of the third high-order feature map, the arithmetic mean of the pixel values ​​at all spatial locations in that channel is calculated, and each channel is compressed into a single value, thereby compressing the multi-dimensional high-order feature map into a one-dimensional feature vector.

[0092] By using the GAP layer, the response of each channel can cover the entire code region, eliminating the influence of spatial location on classification, improving the robustness of the network, and reducing the number of input parameters for the linear layer to prevent overfitting.

[0093] Then, the one-dimensional feature vector is input into a linear layer, which performs class mapping. This involves multiplying the one-dimensional feature vector by a weight matrix (pre-determined by a pre-trained classification network) and adding a bias (pre-determined by a pre-trained classification network). The output of the linear layer is a score vector, with the dimension being the number of code system categories. Each score in the score vector represents the confidence level of the one-dimensional feature vector corresponding to a certain code system category.

[0094] Finally, the score vector is input into a normalized exponential function layer. This layer normalizes the score vector by mapping each element (confidence level) to a positive number using an exponential function and normalizing it to a probability distribution, ensuring that the sum of probabilities for each code type is 1. The code type corresponding to the highest probability is then taken as the code type of the barcode image.

[0095] For example, if there are 9 code type categories, and the third higher-order feature map has a dimension of 512×7×7, the GAP layer compresses this higher-order feature map to generate a one-dimensional feature vector with a dimension of 1×512. A linear layer maps this one-dimensional feature vector to a 9-dimensional code type category space, generating a score vector with a dimension of 1×9. A normalized exponential function layer normalizes this score vector, generating probability distributions for the 9 code type categories. For example, the probability distributions for code type category 1, code type category 2, ..., code type category 9 are: [0.124, 0.045, 0.019, 0.010, 0.554, 0.005, 0.037, 0.204, 0.002]. The maximum probability is 0.554, corresponding to code type category 5. Therefore, the code type category of the barcode image is code type category 5.

[0096] The following is combined Figure 6 The specific process of one embodiment of this application is described.

[0097] In one possible implementation of this application, firstly, the barcode image I to be classified is input into the feature extraction module of a preset classification network. The first sampled feature map is generated by an initial convolutional layer with a stride of 2 in the feature extraction module. The first sampled feature map is then input into a first inverse residual structure with a stride of 1 in the feature extraction module. After inverse residual transformation of the first sampled feature map by the first inverse residual structure, a second sampled feature map is generated. The first and second sampled feature maps are then input into a residual connection unit to generate a third sampled feature map. The third sampled feature map is then input into a second inverse residual structure with a stride of 2 for downsampling inverse residual transformation to generate a sampled feature map. Subsequently, the sampled feature map The input to the sampling enhancement module generates a fused feature map through its multi-branch convolutional units. This fused feature map is then input to the sigmoid activation unit of the sampling enhancement module to generate a saliency score map. Next, the sampled feature map... The saliency score map is input into the reactivation module, and reactivation feature maps are generated through element-wise multiplication. Finally, the reactivated feature map The input to the classification prediction module undergoes continuous downsampling inverse residual transformation through three downsampling inverse residual structures, and is then input to the classification prediction head of the classification prediction module to determine the code type of barcode image I.

[0098] In one possible implementation of this application, the aforementioned pre-trained classification network is pre-trained in the following manner: Step 1501: Obtain the training dataset. The training dataset includes multiple training samples. Each training sample contains a barcode image sample, the barcode category label corresponding to the barcode image sample, and the corresponding binarized segmentation map. Step 1502: Input the barcode image sample into the initial classification network to determine the predicted saliency score map and the predicted code type category; Step 1503: Determine the first loss value of the initial classification network based on the predicted saliency score map and the binarized segmentation map; Step 1504: Determine the second loss value of the initial classification network based on the code system prediction category and the code system category label; Step 1505: Based on the first loss value and the second loss value, iteratively train the initial classification network to generate the preset classification network.

[0099] Among them, the barcode image sample can be an image containing a one-dimensional barcode or a two-dimensional barcode.

[0100] As an example, to improve the robustness of the preset classification network, low-quality barcode images can be collected under conditions such as uneven lighting, motion blur, physical wear, partial occlusion, or perspective distortion. Furthermore, corresponding barcode image samples are collected for all barcode types.

[0101] The code category label can refer to the code category to which the barcode image sample belongs, such as EAN code, UPC code, ISBN code, Codba code, Fast Response Matrix code, Data Matrix code, Aztec code, etc.

[0102] A binarized segmentation image is a binary mask image corresponding to a barcode image sample, where a pixel value of 1 indicates that the pixel belongs to the barcode region, and a pixel value of 0 indicates that the pixel belongs to the background region.

[0103] In one possible implementation, after collecting barcode image samples, the code type category label and binarized segmentation map corresponding to each barcode image sample can be labeled. Each barcode image sample and its corresponding code type category label and binarized segmentation map form a training sample, and multiple training samples form a training dataset.

[0104] In a specific example, the training dataset contains 100,000 training samples. Each training sample contains a barcode image sample, the corresponding code category label, and the corresponding binarized segmentation map.

[0105] After obtaining the training dataset, input the barcode image sample of each training sample in the training dataset into the initial classification network, and then process it as follows: Figure 7 The process shown is as follows: First, the barcode image sample enters the initial convolutional layer, which performs preliminary downsampling and channel mapping on the barcode image sample to generate a first sampled feature map. Then, the first sampled feature map undergoes inverse residual transformation through a first inverse residual structure (with a stride of 1) to generate a second sampled feature map. The first and second sampled feature maps are then added element-wise through a residual connection unit to generate a third sampled feature map. The third sampled feature map undergoes downsampling inverse residual transformation through a second inverse residual structure (with a stride of 2) to generate a sampled feature map.

[0106] Next, the sampled feature map is input into the sampling enhancement module of the initial classification network. Parallel feature extraction is performed on the sampled feature map using multi-branch differential convolutional units (including ordinary convolutional branches, central differential convolutional branches, angular differential convolutional branches, horizontal differential convolutional branches, and vertical differential convolutional branches). The outputs of each convolutional branch are nonlinearly transformed by a feedforward neural network and stacked along the channel dimension. Then, they are compressed to a single channel by a 1×1 convolution. Finally, the image is normalized pixel-by-pixel by a Sigmoid activation function to generate the barcode image sample prediction saliency score map. The specific meaning of the prediction saliency score map is the same as that of the saliency score map, where each pixel value represents the predicted probability that the pixel belongs to the code region. The specific implementation of this step is the same as the related embodiments described above and will not be repeated here.

[0107] Then, the sampled feature map and the predicted saliency score map are input into the reactivation module of the initial classification network, and the reactivation feature map is generated by multiplying them element by element.

[0108] Then, the reactivated feature map is input into the classification prediction module of the initial classification network. After N downsampling inverse residual structures (e.g., N=3), downsampling and semantic abstraction are performed continuously to gradually expand the receptive field to capture the global structural features of the code. The features are abstracted from the texture level to the semantic level and then compressed into a one-dimensional feature vector through the GAP layer in the classification prediction head. After being mapped to the category space through a linear layer, the prediction scores of each code type are obtained. Finally, the prediction scores are converted into a prediction probability distribution through a normalized exponential function, and the category corresponding to the highest probability is taken as the predicted code type of the barcode image sample.

[0109] Next, the predicted saliency score map is compared with the manually labeled binarized segmentation map, and the first loss value of the initial classification network is calculated using the Binary Cross Entropy Loss (BCE Loss) function. The mathematical expression is: / N Where N is the number of training samples in a training batch. This is the true label (i.e., the binarized segmentation map) of the i-th sample in this batch. This is a graph showing the prediction significance score of the i-th sample predicted by the initial classification network. and They have the same dimensions.

[0110] The first loss value is used to measure the difference between the predicted saliency score map and the true binarized segmentation map. When the probability distribution of code regions in the predicted saliency score map differs from the true binarized segmentation map... Figure 1 When the two values ​​are equal, the first loss value is smaller; when the difference between the two values ​​is large, the first loss value is larger.

[0111] Then, the predicted code type category of each barcode image sample is compared with the true code type category label, and a second loss value is calculated using the multi-class cross-entropy loss function. (Multi-class cross-entropy loss function) The mathematical expression is:

[0112] Where N is the number of samples in a batch, and C is the total number of code system categories. For the i-th barcode image sample, the true label is in the c-th barcode category. Let be the predicted probability of the c-th code category for the i-th sample by the predefined classification network. This represents the predicted probability of the j-th code category for the i-th sample by the predefined classification network.

[0113] The second loss value measures the difference between the predicted code category and the true code category label. The second loss value is smaller when the predicted category matches the true category; and larger when the predicted category does not match the true category.

[0114] Next, the first loss value and the second loss value are added together to generate the total loss value. :

[0115] Then, gradient backpropagation is performed on the initial classification network based on the total loss value. The gradient of each trainable parameter (including convolutional layer weights, biases, normalized layer parameters, etc.) in the initial classification network is calculated, and the network parameters of the initial classification network are updated using a gradient descent optimization algorithm (such as adaptive moment estimation or stochastic gradient descent). This completes one iteration of training.

[0116] After each training iteration, the network parameters are updated in the direction that reduces the total loss. This iterative process is repeated until the preset number of iterations is reached (e.g., 100 training epochs, with each epoch iterating through the training dataset once). At this point, training is stopped, and the current network is designated as the preset classification network.

[0117] The barcode classification method based on sampling enhancement provided in this application generates a sampled feature map containing barcode structure information through a preset classification network feature extraction module, providing a feature basis for subsequent barcode region localization. Then, a sampling enhancement module performs lightweight spatial saliency detection on the sampled feature map to generate a saliency score map. Next, a reactivation module reactivates the sampled feature map based on the saliency score map, suppressing the feature response of the background region and enhancing the feature response of the barcode region. Finally, a classification prediction module predicts the category of the reactivated feature map and outputs the barcode category. Because the sampling enhancement and reactivation modules distinguish between the barcode region and the background region of the barcode image, the classification prediction module can predict the category without interference from the background region, thus improving the accuracy of barcode classification. Furthermore, since no complex calculations are required in the sampling enhancement and reactivation modules, the real-time performance of barcode classification is ensured.

[0118] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0119] Corresponding to the code classification method based on sampling enhancement described in the above embodiments, Figure 8 The diagram shows a structural block diagram of a code classification device based on sampling enhancement provided in an embodiment of this application. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0120] Reference Figure 8 The device 80 includes: The first generation module 81 is used to input the barcode image to be classified into the feature extraction module of the preset classification network, extract features from the barcode image, and generate a sampled feature map corresponding to the barcode image, wherein the barcode image includes the code region; The second generation module 82 is used to input the sampled feature map into the sampling enhancement module of the preset classification network, perform spatial saliency detection on the sampled feature map, and generate a saliency score map, wherein each pixel value in the saliency score map represents the probability that the pixel corresponding to the pixel value belongs to the code region. The third generation module 83 is used to input the sampled feature map and the saliency score map into the reactivation module of the preset classification network, and to reactivate the sampled feature map according to the saliency score map to generate a reactivated feature map. The determination module 84 is used to input the reactivated feature map into the classification prediction module of the preset classification network, perform code type prediction on the reactivated feature map, and determine the code type corresponding to the barcode image.

[0121] In practical use, the code classification device based on sampling enhancement provided in this application embodiment can be configured in any electronic device to execute the aforementioned code classification method based on sampling enhancement.

[0122] The code classification device based on sampling enhancement provided in this application generates a sampled feature map containing code structure information through a preset classification network feature extraction module, providing a feature basis for subsequent code region localization. Then, a sampling enhancement module performs lightweight spatial saliency detection on the sampled feature map to generate a saliency score map. Next, a reactivation module reactivates the sampled feature map based on the saliency score map, suppressing the feature response of the background region and enhancing the feature response of the code region. Finally, a classification prediction module predicts the category of the reactivated feature map and outputs the code category. Because the sampling enhancement module and the reactivation module distinguish between the code region and the background region of the barcode image, the classification prediction module can predict the category without interference from the background region, thus improving the accuracy of code classification. Furthermore, since no complex calculations are required in the sampling enhancement module and the reactivation module, the real-time performance of code classification is guaranteed.

[0123] In one possible implementation of this application, the sampling enhancement module includes a multi-branch convolutional unit and an activation unit; correspondingly, the second generation module 82 includes: The first generation unit is used to input the sampled feature map into at least one convolutional branch of the multi-branch convolutional unit and generate the first feature map corresponding to each convolutional branch respectively, wherein the functional types of each convolutional branch are different. The second generation unit is used to input each first feature map into the feedforward neural network of the multi-branch convolutional unit, and perform nonlinear transformations on each first feature map to generate the second feature map corresponding to each first feature map. The fusion unit is used to input each second feature map into the fusion unit of the multi-branch convolutional unit, and fuse the second feature maps to generate a fused feature map. The third generation unit is used to input the fused feature map into the activation unit, activate the fused features pixel by pixel, and generate a saliency score map.

[0124] Optionally, in another possible implementation of this application, the feature extraction module includes an initial convolutional layer, a first inverse residual structure, a residual connection unit, and a second inverse residual structure; correspondingly, the first generation module 81 includes: The fourth generation unit is used to input the barcode image into the initial convolutional layer and generate the first sampled feature map; The fifth generation unit is used to input the first sampled feature map into the first inverse residual structure, perform inverse residual transformation on the first sampled feature map, and generate the second sampled feature map. The sixth generation unit is used to add the first sampled feature map and the second sampled feature map element by element through the residual connection unit to generate the third sampled feature map; The seventh generation unit is used to input the third sampled feature map into the second inverse residual structure, perform downsampling inverse residual transformation on the third sampled feature map, and generate a sampled feature map.

[0125] Optionally, in another possible implementation of this application, the third generation module 83 includes: The eighth generation unit is used to generate a reactivation feature map by multiplying the sampled feature map and the saliency score map element by element through the reactivation module.

[0126] Optionally, in another possible implementation of this application, the classification prediction module includes a classification prediction head and N inverted residual structures, where N is an integer greater than 1; correspondingly, the determination module 84 includes: The ninth generation unit is used to input the reactivated feature map into the first inverse residual structure to perform downsampling inverse residual transformation on the reactivated feature map and generate the first high-order feature map corresponding to the reactivated feature map. The tenth generation unit is used to input the i-th higher-order feature map into the (i+1)-th inverse residual structure to perform downsampling inverse residual transformation on the i-th higher-order feature map and generate the (i+1)-th higher-order feature map corresponding to the reactivated feature map, where i is an integer greater than or equal to 1 and less than N. The determination unit is used to input the Nth high-order feature map into the classification prediction head, perform feature compression and category mapping on the Nth high-order feature map, and determine the code system category.

[0127] Alternatively, in another possible implementation of this application, the aforementioned pre-trained classification network is pre-trained in the following manner: Obtain the training dataset, which includes multiple training samples. Each training sample contains a barcode image sample, the barcode category label corresponding to the barcode image sample, and the corresponding binarized segmentation map. Input the barcode image samples into the initial classification network to determine the predicted saliency score map and the predicted code type category; Based on the predicted saliency score map and the binarized segmentation map, determine the first loss value of the initial classification network; Based on the code system prediction category and the code system category label, determine the second loss value of the initial classification network; Based on the first loss value and the second loss value, the initial classification network is iteratively trained to generate the preset classification network.

[0128] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0129] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0130] To implement the above embodiments, this application also proposes an electronic device.

[0131] Figure 9 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application.

[0132] like Figure 9 As shown, the above-mentioned electronic device 200 includes: The system includes a memory 210 and at least one processor 220, and a bus 230 connecting different components (including the memory 210 and the processor 220). The memory 210 stores a computer program, which, when executed by the processor 220, implements the code classification method based on sampling enhancement as described in the embodiments of this application.

[0133] Bus 230 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0134] Electronic device 200 typically includes a variety of electronic device readable media. These media can be any available media that can be accessed by electronic device 200, including volatile and non-volatile media, removable and non-removable media.

[0135] Memory 210 may also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 240 and / or cache memory 250. Electronic device 200 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 260 may be used to read and write non-removable, non-volatile magnetic media (… Figure 9 Not shown; usually referred to as a "hard drive"). Although Figure 9 As not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 230 via one or more data media interfaces. Memory 210 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this application.

[0136] A program / utility 280 having a set (at least one) of program modules 270 may be stored in, for example, memory 210. Such program modules 270 include—but are not limited to—an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 270 typically perform the functions and / or methods described in the embodiments of this application.

[0137] Electronic device 200 can also communicate with one or more external devices 290 (e.g., keyboard, pointing device, display 291, etc.), and with one or more devices that enable a user to interact with electronic device 200, and / or with any device that enables electronic device 200 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 292. Furthermore, electronic device 200 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 293. As shown, network adapter 293 communicates with other modules of electronic device 200 via bus 230. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0138] The processor 220 performs various functional applications and data processing by running programs stored in the memory 210.

[0139] It should be noted that the implementation process and technical principles of the electronic device in this embodiment are explained in the foregoing description of the code classification method based on sampling enhancement in the embodiments of this application, and will not be repeated here.

[0140] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0141] This application provides a computer program product that, when run on an electronic device, enables the electronic device to perform the steps described in the various method embodiments above.

[0142] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographic device / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0143] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0144] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0145] In the embodiments provided in this application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings or direct couplings or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0146] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0147] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A code classification method based on sampling enhancement, characterized in that, include: The barcode image to be classified is input into the feature extraction module of the preset classification network to extract features from the barcode image and generate a sampled feature map corresponding to the barcode image, wherein the barcode image includes the code region; The sampled feature map is input into the sampling enhancement module of the preset classification network to perform spatial saliency detection on the sampled feature map and generate a saliency score map, wherein each pixel value in the saliency score map represents the probability that the pixel corresponding to the pixel value belongs to the code region; The sampled feature map and the saliency score map are input into the reactivation module of the preset classification network, and the sampled feature map is reactivated according to the saliency score map to generate a reactivated feature map; The reactivated feature map is input into the classification prediction module of the preset classification network to predict the code type of the reactivated feature map and determine the code type corresponding to the barcode image.

2. The method as described in claim 1, characterized in that, The sampling enhancement module includes multi-branch convolutional units and activation units. The sampling enhancement module, which inputs the sampled feature map into the preset classification network, performs spatial saliency detection on the sampled feature map to generate a saliency score map, including: The sampled feature map is input into at least one convolutional branch of the multi-branch convolutional unit to generate a first feature map corresponding to each convolutional branch, wherein each convolutional branch has a different functional type. Each of the first feature maps is input into the feedforward neural network of the multi-branch convolutional unit, and a nonlinear transformation is performed on each of the first feature maps to generate a second feature map corresponding to each of the first feature maps. Each of the second feature maps is input into the fusion unit of the multi-branch convolutional unit to fuse the second feature maps and generate a fused feature map. The fused feature map is input into the activation unit, and the fused features are activated pixel by pixel to generate the saliency score map.

3. The method as described in claim 1, characterized in that, The feature extraction module includes an initial convolutional layer, a first inverse residual structure, residual connection units, and a second inverse residual structure. The step of inputting the barcode image to be classified into the feature extraction module of a preset classification network, extracting features from the barcode image, and generating a sampled feature map corresponding to the barcode image includes: The barcode image is input into the initial convolutional layer to generate a first sampled feature map; The first sampled feature map is input into the first inverse residual structure, and the first sampled feature map is transformed by inverse residual to generate the second sampled feature map. The first sampled feature map and the second sampled feature map are added element by element through the residual connection unit to generate a third sampled feature map; The third sampled feature map is input into the second inverse residual structure, and a downsampled inverse residual transformation is performed on the third sampled feature map to generate the sampled feature map.

4. The method as described in claim 1, characterized in that, The step of inputting the sampled feature map and the saliency score map into the reactivation module of the preset classification network, and reactivating the sampled feature map according to the saliency score map to generate a reactivated feature map includes: The reactivation module generates the reactivation feature map by multiplying the sampled feature map element by element with the saliency score map.

5. The method as described in claim 1, characterized in that, The classification prediction module includes a classification prediction head and N inverted residual structures, where N is an integer greater than 1. The step of inputting the reactivated feature map into the classification prediction module of the preset classification network, performing code type prediction on the reactivated feature map, and determining the code type corresponding to the barcode image includes: The reactivated feature map is input into the first inverse residual structure to perform downsampling inverse residual transformation on the reactivated feature map, thereby generating the first high-order feature map corresponding to the reactivated feature map; The i-th higher-order feature map is input into the (i+1)-th inverse residual structure to perform downsampling inverse residual transformation on the i-th higher-order feature map, generating the (i+1)-th higher-order feature map corresponding to the reactivated feature map, where i is an integer greater than or equal to 1 and less than N; The Nth higher-order feature map is input into the classification prediction head, and feature compression and category mapping are performed on the Nth higher-order feature map to determine the code system category.

6. The method according to any one of claims 1-5, characterized in that, The pre-trained classification network is pre-trained in the following manner: Obtain a training dataset, which includes multiple training samples. Each training sample contains a barcode image sample, a barcode category label corresponding to the barcode image sample, and a corresponding binarized segmentation map. The barcode image samples are input into the initial classification network to determine the predicted saliency score map and the predicted code type category; Based on the predicted saliency score map and the binarized segmentation map, determine the first loss value of the initial classification network; Based on the code system prediction category and the code system category label, determine the second loss value of the initial classification network; The initial classification network is iteratively trained based on the first loss value and the second loss value to generate the preset classification network.

7. A code classification device based on sampling enhancement, characterized in that, include: The first generation module is used to input the barcode image to be classified into the feature extraction module of the preset classification network, extract features from the barcode image, and generate a sampled feature map corresponding to the barcode image, wherein the barcode image includes a code region; The second generation module is used to input the sampled feature map into the sampling enhancement module of the preset classification network, perform spatial saliency detection on the sampled feature map, and generate a saliency score map, wherein each pixel value in the saliency score map represents the probability that the pixel corresponding to the pixel value belongs to the code region; The third generation module is used to input the sampled feature map and the saliency score map into the reactivation module of the preset classification network, and perform feature reactivation on the sampled feature map according to the saliency score map to generate a reactivated feature map; The determination module is used to input the reactivated feature map into the classification prediction module of the preset classification network, perform code type prediction on the reactivated feature map, and determine the code type corresponding to the barcode image.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it causes the electronic device to implement the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by an electronic device, it implements the method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, Includes a computer program that, when run on an electronic device, causes the electronic device to perform the method as described in any one of claims 1 to 6.

Images