One-dimensional code system determination method and device based on multiple modes and storage medium

Abstract

The invention relates to the technical field of graphic codes, and discloses a one-dimensional code system determination method and device based on multiple modes and a storage medium, and the method comprises the steps: inputting a one-dimensional code image into an image encoder, and obtaining a first image feature map; inputting the code system list into a text encoder to obtain a plurality of first text feature maps; performing feature enhancement processing on the first image feature map and the plurality of first text feature maps through a feature enhancement module to obtain a second image feature map and a plurality of second text feature maps; performing normalization processing on the second image feature map and the plurality of second text feature maps to obtain a third image feature map and a plurality of third text feature maps; respectively determining the similarity between the third image feature map and each third text feature map in the plurality of third text feature maps to obtain a plurality of similarities; and determining the code system corresponding to the first text feature map corresponding to the maximum similarity in the plurality of similarities as the code system of the one-dimensional code image. According to the invention, the one-dimensional code system can be accurately determined.

Description

Technical Field

[0001] This application relates to the field of graphic code technology, specifically to a method, device, and storage medium for determining a one-dimensional code system based on multimodality. Background Technology

[0002] With the development of technology, in order to improve the efficiency of obtaining information and ensure the security of information, a technology of storing information through graphic codes has emerged. Common graphic codes include one-dimensional codes and two-dimensional codes.

[0003] One-dimensional barcodes are widely used in the retail, logistics, medical, and manufacturing industries. In practical applications, an image of the barcode is captured by a camera and then decoded to obtain the information stored within it. However, due to the numerous types of barcodes, different barcode types require different decoding methods. Determining the barcode's barcode type before decoding and then using the corresponding decoding method can improve decoding efficiency. Currently, a method for accurately determining the barcode's barcode type is lacking. Summary of the Invention

[0004] In view of the above problems, this application provides a method, device and storage medium for determining the code system of a one-dimensional code based on multimodality, which is used to solve the problem that the code system of a one-dimensional code cannot be accurately determined in the prior art.

[0005] According to one aspect of the embodiments of this application, a method for determining the code scheme of a one-dimensional code based on multimodality is provided. The method includes: inputting a one-dimensional code image with a code scheme to be determined into a trained image encoder to obtain a first image feature map output by the trained image encoder; inputting a code scheme list into a trained text encoder to obtain a plurality of first text feature maps output by the trained text encoder, wherein the code scheme list includes name texts of various code schemes, the plurality of first text feature maps correspond one-to-one with the various code schemes, and the code scheme of the one-dimensional code image belongs to one of the various code schemes; performing feature enhancement processing on the first image feature map and the plurality of first text feature maps respectively through a feature enhancement module to obtain a second image feature map and a plurality of second text feature maps, wherein the plurality of second text feature maps correspond to the plurality of first text feature maps. Figure 1 A one-to-one correspondence is achieved; the second image feature map and the plurality of second text feature maps are normalized respectively to obtain a third image feature map and a plurality of third text feature maps, wherein the plurality of third text feature maps correspond to the plurality of second text feature maps. Figure 1One-to-one correspondence; determine the similarity between each third text feature map in the third image feature map and the plurality of third text feature maps respectively, to obtain multiple similarities, wherein the plurality of similarities are correlated with the plurality of third text feature maps. Figure 1 One-to-one correspondence; the code system corresponding to the first text feature map with the highest similarity among the multiple similarities is determined as the code system of the one-dimensional code image.

[0006] In one alternative approach, the feature enhancement module includes multiple convolutional normalized Gaussian activation branches, each of which includes cascaded convolutional layers, normalization layers, and Gaussian activation function layers, and the kernel sizes of the multiple convolutional layers included in the multiple convolutional normalized Gaussian activation branches are different.

[0007] In one optional approach, the step of performing feature enhancement processing on the first image feature map and the plurality of first text feature maps respectively through the feature enhancement module to obtain a second image feature map and a plurality of second text feature maps includes: inputting the first image feature map into each of the convolutional normalized Gaussian activation branches in the multi-path convolutional normalized Gaussian activation branch to obtain a plurality of first feature maps output by the multi-path convolutional normalized Gaussian activation branch; stacking the plurality of first feature maps in the channel dimension to obtain a second feature map; performing dimensionality reduction processing on the second feature map through a one-dimensional convolutional layer to obtain a second image feature map; for each of the plurality of first text feature maps: inputting the first text feature map into each of the convolutional normalized Gaussian activation branches in the multi-path convolutional normalized Gaussian activation branch to obtain a plurality of third feature maps output by the multi-path convolutional normalized Gaussian activation branch; stacking the plurality of third feature maps in the channel dimension to obtain a fourth feature map; and performing dimensionality reduction processing on the fourth feature map through the one-dimensional convolutional layer to obtain a second text feature map.

[0008] In one alternative approach, the trained image encoder includes a cascaded graph representation embedding module, a graph representation evolution module, and a global graph structure compression module; the trained text editor includes a cascaded word embedding module, multiple Transformer encoders, and a word feature compression module.

[0009] In one optional embodiment, the graph representation embedding module includes an overlapping convolution module and a TopK operator; the overlapping convolution module includes multiple cascaded units, each unit including a cascaded convolutional layer, a normalization layer, and a Gaussian activation function layer; the overlapping convolution module is used to receive the one-dimensional barcode image, process the one-dimensional barcode image to obtain graph node features, and output the graph node features to the TopK operator and the graph representation evolution module respectively; the TopK operator is used to calculate the graph node features to obtain an adjacency matrix, and output the adjacency matrix to the graph representation evolution module; the graph representation evolution module is used to receive the graph node features and the adjacency matrix, update the graph node features and the adjacency matrix to obtain updated graph node features, and output the updated graph node features to the global graph compression module; the global graph structure compression module is used to receive the updated graph node features, compress the updated graph node features to obtain the first image feature map.

[0010] In one optional embodiment, the word embedding module is used to receive the code list, convert the name text of the multiple code types into multiple one-dimensional vectors, and output the multiple one-dimensional vectors to the multiple Transformer encoders, wherein the multiple one-dimensional vectors correspond one-to-one with the name text of the multiple code types; the multiple Transformer encoders are used to receive the multiple one-dimensional vectors, learn the multiple one-dimensional vectors to obtain multiple learned features, and output the multiple learned features to the word feature compression module, wherein the multiple learned features correspond one-to-one with the multiple one-dimensional vectors; the word feature compression module is used to receive the multiple learned features and compress the multiple learned features to obtain the multiple first text feature maps.

[0011] In one optional approach, the trained image encoder and the trained text encoder are trained through the following steps: inputting a one-dimensional barcode training image into the image encoder to be trained, obtaining a training image feature map output by the image encoder; determining a first probability that the one-dimensional barcode training image belongs to each of the multiple barcode systems based on the training image feature map; determining a first loss value based on the first probability using a cross-entropy loss function; inputting training name text into the text encoder to be trained, obtaining a training text feature map output by the text encoder, wherein the training name text is the name text of the barcode system of the one-dimensional barcode training image; determining a second probability that the training name text belongs to the name text of each barcode system in the barcode system list based on the training text feature map; determining a second loss value based on the second probability using a cross-entropy loss function; determining a total loss value based on the first loss value and the second loss value; adjusting the weight parameters and bias parameters of the image encoder and the text encoder to be trained based on the total loss value, respectively, to obtain the trained image encoder and the trained text encoder.

[0012] In one alternative approach, determining the total loss value based on the first loss value and the second loss value includes: determining the first loss value (loss). I The first weight w1, and the second loss value loss T The second weight w2, where w2 > w1 > 0, and w1 + w2 = 1; through the formula loss all =w1×loss I +w2×loss T Determine the total loss value. all .

[0013] According to another aspect of the embodiments of this application, a one-dimensional code system determination device based on multimodality is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the one-dimensional code system determination method based on multimodality as described above.

[0014] According to another aspect of the embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the multimodal one-dimensional code system determination method as described above.

[0015] In this embodiment, when determining the code scheme of a one-dimensional barcode, comprehensive information is obtained from different perspectives by comprehensively considering both the one-dimensional barcode image and the code scheme's name text. This improves the accuracy and robustness of classifying different one-dimensional barcode code schemes, thus enhancing the accuracy of the final determined code scheme. Furthermore, determining the code scheme by comparing the similarity between image features and code scheme name text features reduces the probability of misjudgment and improves the reliability of the classification results. Moreover, since the code scheme's name text information can enhance the explanation and clarification of the one-dimensional barcode classification results, the multimodal one-dimensional barcode code scheme determination method provided in this embodiment also possesses good interpretability.

[0016] The above description is merely an overview of the technical solutions of the embodiments of this application. In order to better understand the technical means of the embodiments of this application and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of this application more obvious and understandable, specific implementation methods of this application are described below. Attached Figure Description

[0017] The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0018] Figure 1 A flowchart illustrating the training method of the model provided in an embodiment of this application is shown;

[0019] Figure 2 This paper shows a schematic diagram of the structure of the image encoder and text encoder provided in the embodiments of this application;

[0020] Figure 3 A schematic diagram of the structure of the overlapping convolution module provided in an embodiment of this application is shown;

[0021] Figure 4 The diagram illustrates the structure of the evolution module provided in an embodiment of this application.

[0022] Figure 5 A flowchart of a one-dimensional code system determination method based on multimodality provided in an embodiment of this application is shown;

[0023] Figure 6 A schematic diagram of the feature enhancement module provided in an embodiment of this application is shown;

[0024] Figure 7 A schematic diagram of the structure of a one-dimensional code system determination device based on multimodality provided in an embodiment of this application is shown. Detailed Implementation

[0025] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein.

[0026] One-dimensional barcodes are composed of black and white stripes, using stripes and spaces of varying widths arranged according to specific rules to represent data. The width and spacing of these stripes and spaces determine the encoded content. There are many types of one-dimensional barcodes, including UPC-E, EAN-13, EAN-8, UPC-A, Code39, Code93, Code128, ITF25, and CodaBar. Different barcode systems result in different stripe and space widths and spacings. Different one-dimensional barcode systems require different decoding methods.

[0027] Currently, decoders used for decoding one-dimensional codes cannot accurately distinguish all code types. This forces the decoder to try multiple decoding methods until successful decoding. This method is inefficient and unsuitable for scenarios requiring fast decoding.

[0028] Before decoding a one-dimensional code, accurately identifying its code type and then directly using the decoding method for that code type would significantly improve decoding efficiency, eliminating the need to try multiple decoding methods one by one. However, since the appearance of one-dimensional codes with different code types is not significantly different, it is difficult to determine the code type of a one-dimensional code directly from its appearance.

[0029] Although the appearance of one-dimensional barcodes with different code systems is not significantly different, the name texts of the code systems differ considerably. For example, the name texts of one-dimensional barcodes with code system UPC-E (whose code system name text is UPC-E) and one-dimensional barcodes with code system EAN-13 (whose code system name text is EAN-13) are quite different. Therefore, if the name text of each code system can be associated with the one-dimensional barcode of that code system, then when determining the code system of a one-dimensional barcode, the code system of the name text with the most similar name text to the one-dimensional barcode can be determined.

[0030] Based on this, this application proposes a model training method. By training the model, the model learns the relationship between one-dimensional codes of various codes and the name text of the code system, and then the code system of the one-dimensional code can be determined using the trained model. The model includes an image encoder and a text encoder.

[0031] Figure 1This diagram illustrates a flowchart of a training method for a model provided in an embodiment of this application. The method is executed by a terminal device, which may include one or more processors, such as a server, touchscreen phone, smartphone, tablet computer, portable electronic device, or other electronic device. The processor may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application; no limitation is made herein. The one or more processors included in the terminal device may be of the same type, such as one or more CPUs; or they may be of different types, such as one or more CPUs and one or more ASICs; no limitation is made herein. Figure 1 As shown, the method includes the following steps:

[0032] Step 110: Input the one-dimensional code training image into the image encoder to be trained to obtain the training image feature map output by the image encoder to be trained.

[0033] In this embodiment, if it is necessary to use the trained model to determine the one-dimensional code belonging to certain code systems, then the one-dimensional codes of these code systems are used as one-dimensional code training images. Furthermore, in this step, the one-dimensional code training images are one-dimensional code images with known code systems.

[0034] Figure 2 A schematic diagram of the structure of the image encoder and text encoder provided in the embodiments of this application is shown. Figure 2 As shown, the image encoder includes a graph representation embedding module, a graph representation evolution module, and a global graph structure compression module. The image encoder is a novel visual graph neural network backbone used for visual tasks such as image classification, object detection, and semantic segmentation. The overlapping convolution module extracts local features from the one-dimensional code training image and converts them into a graph structure, outputting graph node features and an adjacency matrix. The graph representation evolution module receives these graph node features and the adjacency matrix, updates them, and outputs a graph node feature matrix. The global graph structure compression module is responsible for compressing this graph node feature matrix to obtain the training image feature map.

[0035] The diagram illustrates that the embedding module includes an overlapping convolution module and a TopK operator. The overlapping convolution module comprises multiple cascaded units, each of which includes cascaded convolutional layers, normalization layers, and Gaussian activation function layers.

[0036] This section uses an overlapping convolutional module consisting of three cascaded units as an example for explanation. Figure 3 A schematic diagram of the overlapping convolution module provided in an embodiment of this application is shown. Figure 3As shown, the overlapping convolutional module consists of three cascaded units: the first unit, the second unit, and the third unit. Each unit includes a cascaded convolutional layer, a normalization layer, and a Gaussian activation function layer. It is worth noting that this description only uses an overlapping convolutional module with three cascaded units as an example, and does not limit the number of units included in the overlapping convolutional module. The overlapping convolutional module can also include two units or four or more units.

[0037] In this embodiment, the convolutional layers in the overlapping convolutional module have a window size of 3, a stride of 2, and padding of 1. The convolutional layers in the overlapping convolutional module provided in this application can overlap their convolutional windows, thereby promoting information interaction among multiple local blocks of the feature map input to the overlapping convolutional model. The normalization layer in the overlapping convolutional module can prevent gradient explosion. In this embodiment, by setting a Gaussian activation function (GELU function) in the overlapping convolutional module, compared to the ReLU function layer, a perturbation is added near the zero value, making the model smoother and reducing the gradient vanishing problem.

[0038] After the one-dimensional code training image is input into the overlapping convolution module, it is processed by the overlapping convolution module to obtain graph node features X. The overlapping convolution module then transmits the graph node features X to the TopK operator and the graph representation evolution module, respectively. Where X∈R D×H / 8×W / 8 D is the number of channels, and H and W are the height and width of the one-dimensional code training image.

[0039] The TopK operator calculates the K nearest neighbors of each graph node based on the graph node features X output by the overlapping convolution module. Specifically, the TopK operator finds the K nearest neighbors of each node based on the Euclidean distance of each node's features, obtaining an adjacency matrix A, and then transmits the adjacency matrix A to the graph representation evolution module. Where A ∈ RT HW / 64×HW / 64 In this embodiment of the application, by utilizing Euclidean distance as a metric, it is possible to better distinguish between one-dimensional code regions (i.e., target regions) and non-one-dimensional code regions (i.e., background regions) in one-dimensional code training images.

[0040] Figure 4 A schematic diagram illustrating the structure of the evolution module provided in an embodiment of this application is shown. For example... Figure 4 As shown, the graph representation evolution module consists of four graph filtering modules and three graph update modules alternately connected in series. After receiving the graph node features X transmitted by the overlapping convolution module and the adjacency matrix A transmitted by the TopK operator, the graph representation evolution module outputs a graph node feature Z.

[0041] Specifically, the first graph filtering module updates the graph node features based on the graph node features X and the adjacency matrix A, obtaining an updated graph node feature matrix Y, and then transmits the updated graph node feature matrix Y to the first graph update module and the second graph filtering model. The mathematical model of the first graph filtering module is shown in equation (1).

[0042] Y = AXW (1)

[0043] Where Y is the feature matrix output by the first graph filtering module, and W is a matrix with learnable parameters.

[0044] The first graph update module receives the updated graph node feature matrix Y from the graph filtering module and outputs an updated adjacency matrix A' to the second graph filtering module. Specifically, the graph update module consists of a TopK operator and a pooling layer. The structure of the TopK operator is the same as that of the TopK operator in the graph representation embedding module. The pooling layer performs double average pooling on the updated graph node feature matrix Y.

[0045] Figure 4 The second to fourth graph filtering modules are the same as the first graph filtering module; therefore, their functions are similar to those of the first graph filtering module and will not be repeated here. Similarly, the second and third graph update modules are the same as the first graph update module; therefore, their functions are similar to those of the first graph update module and will not be repeated here.

[0046] The graph shows that the evolution module outputs updated graph node features to the global graph structure compression module, which then compresses these updated features. Specifically, the global graph structure compression module compresses the received features through a global pooling layer, resulting in a training image feature map I representing the one-dimensional code training image. C1 can be set as needed, for example, to 784.

[0047] Step 120: Determine the first probability that the one-dimensional code training image belongs to each of the multiple code systems based on the feature map of the training image.

[0048] In this step, "multiple code schemes" refers to the code schemes that the one-dimensional code training image may belong to. Specifically, in this step, the first probability of the one-dimensional code training image belonging to each of these multiple code schemes is determined based on the feature map of the training image using the softmax function. For example, since one-dimensional code systems include UPC-E, EAN-13, EAN-8, UPC-A, Code39, Code93, Code128, ITF25, and CodaBar, the code system of a one-dimensional code training image belongs to one of these systems. Therefore, in this step, the softmax function is used to determine the first probability that the code system of the one-dimensional code training image belongs to UPC-E, EAN-13, EAN-8, UPC-A, Code39, Code93, Code128, ITF25, and CodaBar based on the feature map I of the training image.

[0049] Step 130: Determine the first loss value based on the first probability using the cross-entropy loss function.

[0050] Since the code scheme of the one-dimensional code training image is known, in this step, the first loss value (loss) is determined based on the code scheme of the one-dimensional code training image and the first probability determined in step 120 using the cross-entropy loss function. I .

[0051] Step 140: Input the training name text into the text encoder to be trained to obtain the training text feature map output by the text encoder to be trained.

[0052] The training name text refers to the name text of the code format of the one-dimensional barcode training image. For example, if the code format of the one-dimensional barcode training image is Code128, then the training name text in this step is Code128.

[0053] The role of a text encoder is to extract feature information from text; it is composed of BERT. BERT (Bidirectional Encoder Representations from Transformers) is a bidirectional encoding representation model based on Transformers. It has achieved remarkable results in the field of Natural Language Processing (NLP), breaking records on multiple NLP tasks.

[0054] like Figure 2As shown, the text editor includes a cascaded word embedding module, multiple Transformer encoders, and a word feature compression module. The number of Transformer encoders can be determined as needed; this embodiment does not limit the number of Transformer encoders. Here, we will use a text editor with 12 Transformer encoders as an example.

[0055] The training text is processed by a text encoder to obtain text features T, T∈R. 1×CT C T For the feature dimension, C in the embodiments of this application T The value is 784.

[0056] After the training name text is input into the word embedding module, the word embedding module receives the training name text, converts it into a one-dimensional vector, and outputs this one-dimensional vector to the Transformer encoder. For example, if the training name text is Code128, since it contains only one word, the word embedding module will convert this word into a word vector O. we O we ∈R 1×784 After the training name text is converted into word vectors, it enters 12 cascaded Transformer encoders so that the text encoder can learn from them.

[0057] Each Transformer encoder consists of a multi-head self-attention (MHSA) mechanism, layer normalization (LN), and a feed-forward neural network (FFNN). Word vectors O we After inputting the first Transformer encoder, the new word vector O is output after MHSA processing. we ', where O we '∈R 1×768 The overall process of MHSA is shown in equation (2).

[0058] O we = softmax(Q) we ×K we T / d k V we (2)

[0059] Word vectors are first processed by three matrices W with learnable parameters. Q W K and W V Perform matrix multiplication to obtain Q. weK we and V we Among them, W Q ∈R 784×1 W K ∈R 784×1 W V ∈R 784×1 Q we ∈R 1×1 K we ∈R 1×1 V we ∈R 1×1 Then Q we and K we T Perform matrix multiplication and divide by a coefficient for scaling, then perform softmax to convert it into similarity scores between word vectors, and finally multiply by V. we Obtain the learned features O we In this embodiment of the application, d k It is 28.

[0060] Layer normalization (LN) normalizes the features output by MHSA. In this embodiment, LN is used to normalize the features instead of batch normalization for all samples of each feature. Since layer normalization normalizes the features of each sample, each feature is subtracted from its mean and divided by its standard deviation, resulting in each sample having a feature with zero mean and unit variance.

[0061] A feedforward neural network consists of two linear layers and a ReLU activation function. The first linear layer increases the dimensionality of the features, and the second linear layer reduces the dimensionality of the features back to their original dimensions. The feedforward neural network outputs the learned features O. we Feedforward neural networks, through the dimensionality expansion of hidden layers and the ReLU activation function, enable the Transformer encoder to learn more complex feature relationships and richer feature combinations.

[0062] O we =Linear(ReLU(Linear(O we '))) (3)

[0063] The word feature compression module receives the learned text feature map O from the output of the 12th Transformer encoder. we The feature is compressed. Specifically, the word feature compression module compresses the feature through a global pooling layer and outputs a training text feature map T representing the word. In this embodiment, C2 is 784.

[0064] Step 150: Determine the second probability that the training name text belongs to each code system in the code system list based on the training text feature map.

[0065] In this step, the code system list includes the name text of each code system from the various code systems discussed in step 120. For example, if step 120 determines the first probability that a one-dimensional code training image belongs to each of the code systems UPC-E, EAN-13, EAN-8, UPC-A, Code39, Code93, Code128, ITF25, and CodaBar, then the code system list in this step includes UPC-E, EAN-13, EAN-8, UPC-A, Code39, Code93, Code128, ITF25, and CodaBar. Accordingly, in this step, the second probability that the training text feature map T belongs to UPC-E, EAN-13, EAN-8, UPC-A, Code39, Code93, Code128, ITF25, and CodaBar is determined using the softmax function.

[0066] Step 160: Determine the second loss value based on the second probability using the cross-entropy loss function.

[0067] In this step, the second loss value is determined using the cross-entropy loss function based on the training name text and the second probability determined in step 150. T .

[0068] Step 170: Determine the total loss value based on the first loss value and the second loss value.

[0069] In this embodiment of the application, preferably, through loss all =w1×loss I +w2×loss T Determine the total loss value all Where w1 > 0, w2 > 0, and w1 + w2 = 1. w1 and w2 can be determined as needed.

[0070] As mentioned earlier, the appearance of one-dimensional codes of different codes is not significantly different, while the names of the text of different codes are quite different. Therefore, preferably, by setting w2 > w1, increasing the loss of the training text feature map helps the model learn the discriminativeness of one-dimensional codes of different codes, so that the finally trained image encoder and trained text encoder can more accurately distinguish one-dimensional codes of various codes. For example, w2 is 0.75 and w1 is 0.25.

[0071] Step 180: Adjust the weight parameters and bias parameters of the image encoder and the text encoder to be trained according to the total loss value to obtain the trained image encoder and the trained text encoder.

[0072] In this embodiment of the application, by inputting the one-dimensional code training image and the code name text of the one-dimensional code (i.e., the training name text) into the model to be trained, the model can learn the encoding rules of the one-dimensional code corresponding to the code name text, so that the code system of the one-dimensional code to be decoded can be determined using the trained model.

[0073] Figure 5 A flowchart of a multimodal one-dimensional code system determination method provided in an embodiment of this application is shown. This method is executed by a terminal device. The terminal device may be a terminal device including one or more processors, such as a barcode scanner, touchscreen phone, smartphone, tablet computer, portable electronic device, or other electronic device. The processor may be a central processing unit (CPU), an ASIC, or one or more integrated circuits configured to implement the embodiments of this application; no limitation is made herein. The one or more processors included in the terminal device may be processors of the same type, such as one or more CPUs; or they may be processors of different types, such as one or more CPUs and one or more ASICs; no limitation is made herein. Figure 5 As shown, the method includes the following steps:

[0074] Step 210: Input the one-dimensional code image of the code system to be determined into the trained image encoder to obtain the first image feature map output by the trained image encoder.

[0075] The structure and function of the trained image encoder can be found in [reference needed]. Figure 1 The provided embodiments will not be described in detail here. In the embodiments of this application, the first image feature map is I1.

[0076] Step 220: Input the code list into the trained text encoder to obtain multiple first text feature maps output by the trained text encoder.

[0077] The code list includes the names of various code systems, and multiple first text feature maps correspond one-to-one with each code system. The code system of the one-dimensional code image in step 210 belongs to one of these multiple code systems. For example, if the code list includes the names of N code systems, then the text features obtained in this step... It is an N×C2 matrix, and each row of the matrix corresponds to a first text feature map in this embodiment. Since there are 9 code systems for one-dimensional codes, N is 9 in this embodiment.

[0078] The structure and function of a trained text encoder can be found in [reference]. Figure 1 The provided embodiments will not be described in detail here.

[0079] Step 230: The feature enhancement module performs feature enhancement processing on the first image feature map and multiple first text feature maps respectively, resulting in a second image feature map and multiple second text feature maps. The multiple second text feature maps are integrated with the multiple first text feature maps. Figure 1 One-to-one correspondence.

[0080] Figure 6 A schematic diagram of the feature enhancement module provided in an embodiment of this application is shown. Figure 6 As shown, the feature enhancement module includes three convolutional normalized Gaussian activation (GNMA) branches. Each GNMA branch comprises a cascaded convolutional layer, a normalization layer, and a Gaussian activation function layer, and the kernel sizes of the multiple convolutional layers in the three GNMA branches are different. Specifically, the first convolutional layer has a kernel size of 1, a stride of 1, and padding of 0; the second convolutional layer has a kernel size of 3, a stride of 1, and padding of 1; and the third convolutional layer has a kernel size of 5, a stride of 1, and padding of 2.

[0081] In this embodiment, by setting the feature enhancement module to include three different convolutional layers, the three different convolutional layers can extract different features after the feature map is input into the feature enhancement module, thereby increasing the information included in the final output feature map of the feature enhancement module. Furthermore, by setting three-way convolutional normalized Gaussian activation branches instead of four or more, both the processing efficiency and the enhancement effect of the feature enhancement module on the feature map can be balanced.

[0082] It is worth noting that the number of convolutional normalized Gaussian activation branches included in the feature enhancement module is not limited in the embodiments of this application. The feature enhancement module may also include two convolutional normalized Gaussian activation branches or four or more convolutional normalized Gaussian activation branches.

[0083] In this step, specifically, the first image feature map I1 is input into the first, second, and third convolutional normalized Gaussian activation branches, respectively. Finally, the features output from the three branches are stacked (concat) along the channel dimension, and the dimensionality of the stacked features is reduced using a one-dimensional convolutional layer to obtain the second image feature map I2, where I2∈R. 1×C1 In this embodiment, the robustness of the model is enhanced by applying nonlinear transformations to the feature map through normalization layers and Gaussian activation function layers. By integrating the output features of convolutional layers at different scales through stacking operations, the contextual features of the one-dimensional code can be better obtained, thereby enhancing the model's understanding of the image.

[0084] Similarly, text features T1∈R N×C2 The first, second, and third convolutional normalized Gaussian activation branches are input respectively, and the feature map T2, T2∈R, is finally obtained from the feature enhancement module. N×C2 Each row of this matrix corresponds to a second text feature map in an embodiment of this application.

[0085] Step 240: Normalize the second image feature map and the multiple second text feature maps respectively to obtain a third image feature map and multiple third text feature maps. The multiple third text feature maps are consistent with the multiple second text feature maps. Figure 1 One-to-one correspondence.

[0086] The second image feature map I2 is normalized by L2 using the following formula (4) to obtain the third image feature map I3.

[0087] I3=I2 / ||I2||2 (4)

[0088] The feature map T2 is obtained by L2 normalization of the feature map T2 using the following equation (5).

[0089] T3=T2 / ||T2||2 (5)

[0090] Step 250: Determine the similarity between each third text feature map in the third image feature map and the multiple third text feature maps respectively, obtaining multiple similarity scores. These multiple similarity scores are related to the multiple third text feature maps. Figure 1 One-to-one correspondence.

[0091] The similarity S between the third image feature map and the third text feature map is determined by the following equation (6), where S∈R 1×N .

[0092] S=I3T3 / ||I3||2||T3||2 (6)

[0093] In this application, the similarity between the third image feature map and the third text feature map is determined to achieve the fusion of image information and text information, thereby achieving the purpose of determining the one-dimensional code system based on multimodal information and improving the accuracy of the determined one-dimensional code system.

[0094] Step 260: Determine the code system of the first text feature map corresponding to the maximum similarity among multiple similarities as the code system of the one-dimensional code image.

[0095] In this step, the code category corresponding to the maximum value in S is taken as the code category of the one-dimensional code to be determined, and the classification result of the image is obtained by the following equation (7).

[0096] Result=argmax(S).index() (7)

[0097] In this embodiment, when determining the code scheme of a one-dimensional barcode, comprehensive information is obtained from different perspectives by comprehensively considering the one-dimensional barcode image and the code scheme's name text. This improves the accuracy and robustness of classifying different one-dimensional barcode code schemes, thus enhancing the accuracy of the final determined code scheme. Furthermore, determining the code scheme by comparing the similarity between image features and the code scheme's name text features reduces the probability of misjudgment and improves the reliability of the classification results. Moreover, since the code scheme's name text information is part of the one-dimensional barcode's name information, it can enhance the explanation and clarification of the one-dimensional barcode classification results. In other words, the multimodal one-dimensional barcode code scheme determination method provided in this embodiment also possesses good interpretability. Figure 7 The diagram shows a schematic of the structure of a one-dimensional barcode system determination device based on multimodality provided in an embodiment of this application. The specific embodiments of this application do not limit the specific implementation of the one-dimensional barcode system determination device based on multimodality.

[0098] like Figure 7 As shown, the one-dimensional code system determination device 300 based on multimodal mode may include a processor 302 and a memory 304.

[0099] The memory 304 is used to store the computer program 306. The memory 304 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk drive. The computer program 306 may include computer-executable instructions.

[0100] The processor 302 is used to execute the computer program 306 to implement the above-described embodiments of the one-dimensional code system determination method based on multimodality and / or the model training method.

[0101] Processor 302 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The one or more processors included in the multimodal one-dimensional code determination device 300 may be processors of the same type, such as one or more CPUs; or they may be processors of different types, such as one or more CPUs and one or more ASICs.

[0102] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described embodiments of the one-dimensional code system determination method based on multimodality and / or the model training method.

[0103] This application provides a computer program that can be executed by a processor to implement the above-described embodiments of the one-dimensional code system determination method and / or model training method based on multimodality.

[0104] This application provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described embodiments of the one-dimensional code system determination method based on multimodality and / or the model training method.

[0105] In the several embodiments provided in this application, any function, if implemented as a software functional module / unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the technical solution of this application can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or other electronic device) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing computer program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0106] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, the embodiments of this application are not directed to any particular programming language. It should be understood that the content of this application described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of this application.

[0107] It should be noted that the above embodiments are illustrative of this application and not restrictive, and those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In claims enumerating several means, several units or modules of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.

[0108] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for determining a one-dimensional code system based on multimodal characteristics, characterized in that, The method includes: The one-dimensional code image of the code system to be determined is input into the trained image encoder to obtain the first image feature map output by the trained image encoder; The code list is input into the trained text encoder to obtain multiple first text feature maps output by the trained text encoder. The code list includes the name text of multiple code systems, and the multiple first text feature maps correspond one-to-one with the multiple code systems. The code system of the one-dimensional code image belongs to one of the multiple code systems. The feature enhancement module performs feature enhancement processing on the first image feature map and the plurality of first text feature maps respectively to obtain a second image feature map and a plurality of second text feature maps, wherein the plurality of second text feature maps correspond one-to-one with the plurality of first text feature maps; The second image feature map and the plurality of second text feature maps are normalized respectively to obtain a third image feature map and a plurality of third text feature maps, wherein the plurality of third text feature maps correspond one-to-one with the plurality of second text feature maps; The similarity between each third text feature map in the third image feature map and the plurality of third text feature maps is determined respectively to obtain a plurality of similarity values, wherein the plurality of similarity values ​​correspond one-to-one with the plurality of third text feature maps; The code system corresponding to the first text feature map with the highest similarity among the multiple similarities is determined as the code system of the one-dimensional code image.

2. The method according to claim 1, characterized in that, The feature enhancement module includes multiple convolutional normalized Gaussian activation branches. Each of the multiple convolutional normalized Gaussian activation branches includes cascaded convolutional layers, normalization layers, and Gaussian activation function layers, and the kernel sizes of the multiple convolutional layers included in the multiple convolutional normalized Gaussian activation branches are different.

3. The method according to claim 2, characterized in that, The step of performing feature enhancement processing on the first image feature map and the plurality of first text feature maps respectively through the feature enhancement module to obtain a second image feature map and a plurality of second text feature maps includes: The first image feature map is input into each of the convolutional normalized Gaussian activation branches in the multi-path convolutional normalized Gaussian activation branch to obtain multiple first feature maps output by the multi-path convolutional normalized Gaussian activation branch. The multiple first feature maps are stacked along the channel dimension to obtain a second feature map; The second feature map is obtained by dimensionality reduction of the second feature map through a one-dimensional convolutional layer; For each of the plurality of first text feature maps: The first text feature map is input into each of the convolutional normalized Gaussian activation branches in the multi-path convolutional normalized Gaussian activation branch to obtain multiple third feature maps output by the multi-path convolutional normalized Gaussian activation branch. The multiple third feature maps are stacked along the channel dimension to obtain a fourth feature map; The fourth feature map is reduced in dimensionality by the one-dimensional convolutional layer to obtain the second text feature map.

4. The method according to claim 1, characterized in that, The trained image encoder includes a cascaded graph representation embedding module, a graph representation evolution module, and a global graph structure compression module; The trained text editor includes a cascaded word embedding module, multiple Transformer encoders, and a word feature compression module.

5. The method according to claim 4, characterized in that, The graph representation embedding module includes an overlapping convolution module and a TopK operator; The overlapping convolution module includes multiple cascaded units, each of which includes a cascaded convolutional layer, a normalization layer, and a Gaussian activation function layer. The overlapping convolution module is used to receive the one-dimensional code image, process the one-dimensional code image to obtain graph node features, and output the graph node features to the TopK operator and the graph representation evolution module, respectively. The TopK operator is used to calculate and process the graph node features to obtain the adjacency matrix, and outputs the adjacency matrix to the graph representation evolution module; The graph representation evolution module is used to receive the graph node features and the adjacency matrix, update the graph node features and the adjacency matrix to obtain the updated graph node features, and output the updated graph node features to the global graph compression module. The global graph structure compression module is used to receive the updated graph node features and compress the updated graph node features to obtain the first image feature map.

6. The method according to claim 4, characterized in that, The word embedding module is used to receive the code list, convert the name text of the multiple code types into multiple one-dimensional vectors, and output the multiple one-dimensional vectors to the multiple Transformer encoders, wherein the multiple one-dimensional vectors correspond one-to-one with the name text of the multiple code types; The plurality of Transformer encoders are used to receive the plurality of one-dimensional vectors, and the plurality of Transformer encoders are used to learn the plurality of one-dimensional vectors to obtain a plurality of learned features, and output the plurality of learned features to the word feature compression module, wherein the plurality of learned features correspond one-to-one with the plurality of one-dimensional vectors; The word feature compression module is used to receive the multiple learned features and compress the multiple learned features to obtain the multiple first text feature maps.

7. The method according to claim 1, characterized in that, The trained image encoder and the trained text encoder are obtained by training the image encoder and the text encoder to be trained through the following steps: The one-dimensional barcode training image is input into the image encoder to be trained to obtain the training image feature map output by the image encoder to be trained. Based on the feature map of the training image, determine the first probability that the one-dimensional code training image belongs to each of the multiple code systems; The first loss value is determined based on the first probability using the cross-entropy loss function; The training name text is input into the text encoder to be trained to obtain the training text feature map output by the text encoder to be trained, wherein the training name text is the code name text of the one-dimensional code training image. Based on the training text feature map, determine the second probability that the training name text belongs to each code system in the code system list; The second loss value is determined based on the second probability using the cross-entropy loss function; The total loss value is determined based on the first loss value and the second loss value; The weight parameters and bias parameters of the image encoder and the text encoder to be trained are adjusted according to the total loss value to obtain the trained image encoder and the trained text encoder.

8. The method according to claim 7, characterized in that, The step of determining the total loss value based on the first loss value and the second loss value includes: Determine the first loss value. I The first weight w1, and the second loss value loss T The second weight w2, where w2 > w1 > 0, and w1 + w2 = 1; Through the formula loss all =w1×loss I +w2×loss T Determine the total loss value. all .

9. A one-dimensional code system determination device based on multimodality, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the one-dimensional code system determination method based on multimodality as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the one-dimensional code system determination method based on multimodality as described in any one of claims 1 to 8.

Images