End-to-end container number identification method and system

Abstract

The present invention discloses an end-to-end container number recognition method and system. The method comprises: inputting a container image into a pre-trained recognition model; performing feature extraction and segmentation on the input image using a saliency foreground perception network; performing character enhancement on a text saliency feature map using an LSTM network; encoding time steps using a position encoding layer and outputting a query vector; outputting a feature vector using a self-attention mechanism model, using the text saliency feature map as a value input, the character enhancement features as a key input, and the query vector as a query input; and outputting the feature vector using a linear layer to obtain a recognition result. The system comprises an input module, a saliency foreground perception module, a character feature enhancement module, a position encoding module, a self-attention mechanism module, and a linear output module. The present invention can accurately locate and recognize container numbers and dimensions, and is widely applicable in the field of data recognition.

Description

Technical Field

[0001] The present invention relates to the field of data identification, and in particular to an end-to-end container number identification method and system. Background Art

[0002] Container number recognition requires locating the position of key text in the image and performing corresponding recognition. For real-life scenarios, if background technology methods are used to process images containing text, we need to divide the process into two stages: the first stage is text detection, which locates the text position in the image; the second stage is text recognition, which recognizes the text area obtained in the first stage. This requires training two models separately, which is relatively cumbersome. In addition, incorrect text detection results will further affect the text recognition results, causing error accumulation. Currently, the detection and recognition methods for horizontal text are relatively mature, but the recognition effect for vertical text, text with large tilt angles, and text captured under various adverse conditions (such as fog, darkness, rain, etc.) is poor. In the text detection stage, in addition to the container number information we need, the container image also contains a lot of irrelevant information, which will interfere with the final result. In the text recognition stage, most existing text recognition methods use contextual information. Heavy reliance on contextual information can lead to attention drift problems, resulting in poor recognition results. Summary of the Invention

[0003] In order to solve the above technical problems, the purpose of the present invention is to provide an end-to-end container number identification method and system, which can accurately locate and identify the container number and container size.

[0004] The first technical solution adopted by the present invention is: an end-to-end container number identification method, comprising the following steps:

[0005] Obtain container images and input them into the pre-trained recognition model;

[0006] Based on the saliency foreground perception network, the input image is feature extracted and segmented to obtain the text saliency feature map;

[0007] Based on the LSTM network, the text salient feature map is enhanced to obtain character enhancement features;

[0008] Encode the time steps based on the position encoding layer and output the query vector;

[0009] Based on the self-attention mechanism model, it takes the text saliency feature map as the value input, the character enhancement feature as the key input, and the query vector as the query input, and outputs the feature vector;

[0010] The feature vector is output based on the linear layer to obtain the recognition result.

[0011] Furthermore, it also includes:

[0012] Obtain the video to be tested and extract frames from it to obtain a container image.

[0013] Furthermore, the step of extracting and segmenting the input image based on the saliency foreground perception network to obtain a text saliency feature map specifically includes:

[0014] Get container images;

[0015] Perform preliminary feature extraction on the container image to obtain the original image feature map;

[0016] Introducing the attention similarity of different points, the original image feature map is reshaped by latitude to obtain a global feature map;

[0017] Add the global feature map to the original image feature map to obtain the global salient feature map;

[0018] The global salient feature map is fused to obtain the text proposal box;

[0019] The text proposal boxes are mapped back to the global salient feature map to obtain the text salient feature map.

[0020] Furthermore, the step of introducing the attention similarity of different points and reshaping the original image feature map according to latitude to obtain the global feature map specifically includes:

[0021] Introducing attention similarity at different points;

[0022] Convolution and reshaping are performed on the three dimensions of the original image feature map to obtain the reshaped first dimension, the reshaped second dimension, and the reshaped third dimension;

[0023] The reshaped first dimension and the reshaped second dimension are compared and processed to obtain a similarity matrix;

[0024] Multiply the reshaped third dimension with the similarity matrix to obtain the global feature map.

[0025] Furthermore, the formula for introducing attention similarity at different points is expressed as follows:

[0026]

[0027] In the above formula, subscript i is the index of the output position of the global feature response, subscript j lists the index of all possible global positions, X is the input feature map, W θ and W Φ is the weight matrix in dimensions θ and Φ that the network needs to learn.

[0028] Furthermore, the step of performing character enhancement on the text salient feature map based on the LSTM network to obtain character enhancement features specifically includes:

[0029] A two-layer LSTM is used to process the text salient feature map and capture global context information.

[0030] The hidden variables of the second layer LSTM are convolved to obtain character enhancement features.

[0031] Furthermore, during the training process of the recognition model, text detection loss and text recognition loss are used for model optimization.

[0032] Furthermore, the expression of the text detection loss is as follows:

[0033]

[0034] In the above formula, P tex(i) and G tex(i) are the predicted value of the i-th pixel in the text segmentation result and the true value of the text area, respectively.

[0035] Furthermore, the expression of the text detection loss is as follows:

[0036]

[0037] In the above formula, I is a graph from the set Q of all text proposal boxes, S is the label containing all text character categories, L is the length of the text sequence of a text proposal box, log(P(S l |l, I)) means that the prediction result of the lth character in the predicted sequence is S l The conditional probability of .

[0038] The second technical solution adopted by the present invention is: an end-to-end container number identification system, comprising:

[0039] An input module, used to obtain container images and input the container images into the pre-trained recognition model;

[0040] The salient foreground perception module extracts and segments the input image based on the salient foreground perception network to obtain a text salient feature map;

[0041] The character feature enhancement module performs character enhancement on the text salient feature map based on the LSTM network to obtain character enhancement features;

[0042] The position encoding module encodes the time steps based on the position encoding layer and outputs the query vector;

[0043] The self-attention mechanism module is based on the self-attention mechanism model, takes the text saliency feature map as the value input, the character enhancement feature as the key input, and the query vector as the query input, and outputs a feature vector;

[0044] The linear output module outputs the feature vector based on the linear layer to obtain the recognition result.

[0045] The beneficial effects of the method and system of the present invention are as follows: the present invention adopts an end-to-end approach to simultaneously detect and recognize container images in the same model. The final output is obtained through our saliency foreground perception network, independent character feature enhancement module, position encoding layer and self-attention model. The present invention is more robust and practical for the recognition of container numbers and sizes. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a flowchart of the steps of an end-to-end container number identification method of the present invention;

[0047] Figure 2 This is a schematic diagram of an application of a specific embodiment of the present invention;

[0048] Figure 3 This is a schematic diagram of data processing for a saliency foreground perception network specifically implemented in the present invention;

[0049] Figure 4 The present invention is a structural block diagram of an end-to-end container number identification system. DETAILED DESCRIPTION

[0050] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are provided for ease of description only and do not limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted based on the understanding of those skilled in the art.

[0051] Reference Figure 1 and Figure 2 The present invention provides an end-to-end container number identification method, which includes the following steps:

[0052] S1. Obtain container images and input them into the pre-trained recognition model.

[0053] In addition, the video can be directly processed and frame extracted to obtain the image for subsequent recognition. The results of all images can be classified into container number recognition results and container size recognition results.

[0054] S2. Extract and segment the input image based on the saliency foreground perception network to obtain a text saliency feature map.

[0055] S2.1. First, use a commonly used deep neural network (such as ResNet50, also known as a residual neural network) to perform preliminary feature extraction on the input container image to obtain the original feature map X [bs, c, h, w] (bs is the batch size, c is the number of channels, h and w are the height and width of the feature map, and matrix operations mainly focus on the latter three dimensions).

[0056] S2.2. After initial feature extraction, the salient foreground-aware network introduces a self-attention mechanism to guide better segmentation. For the original feature map X, in order to make the pixels between each point in the corresponding area of ​​the key text have more similarity, and the pixels corresponding to other irrelevant information have less similarity, the attention similarity of different points is introduced:

[0057]

[0058] The subscript i in the above formula is the index of the output position of the global feature response, the subscript j lists the index of all possible global positions, X is the input feature map, W θ and W Φ is the weight matrix in dimensions θ and Φ that the network needs to learn.

[0059] S2.3. Perform 1x1x1 convolution on the three different dimensions θ (first dimension), Φ (second dimension) and g (third dimension) of the feature map of size [bs, c, h, w] (bs is the batch size, c is the number of channels, h and w are the height and width of the feature map, and matrix operations are mainly performed on the last three different dimensions).

[0060] S2.4. Reshape the θ and g dimensions into [bs, (h*w), c], and reshape the Φ dimension into [bs, c, (h*w)]. Multiply the reshaped θ and Φ dimensions and process them through sofamax to form the similarity matrix S1 of [bs, (h*w), (h*w)].

[0061] S2.5. Multiply the reshaped g dimension by the similarity matrix S1 to obtain Y i :

[0062]

[0063] Where W g is the weight matrix that the network needs to learn in the g dimension. Our method can increase the receptive field, rather than being limited to a small local receptive field. The global feature map Y is obtained by calculating the similarity between each point in the feature map on a global scale.

[0064] S2.6. Use the residual structure to add the resulting global feature map Y of S2.5 to the input original image feature map X to obtain the global salient feature map F.saliency :

[0065] F saliency =X+Y,

[0066] The obtained global salient feature map F saliency This can make the feature values ​​belonging to the container number text area more concentrated to better distinguish irrelevant information on the image. The overall process diagram of S2.1-S2.6 refers to Figure 3 .

[0067] S2.7, extract the global salient feature map F saliency Fusion can aggregate network features at different levels into the final features, perform segmentation, and obtain text proposal boxes.

[0068] In addition, using text detection L det The loss supervises the obtained text proposal boxes (during the training process of the model):

[0069]

[0070] Among them, P tex(i) and G tex(i) They are the predicted value of the i-th pixel in the text segmentation result and the true value of the text area (binary, 1 indicates that there is valid text information, and 0 indicates that it does not exist)

[0071] S2.8. Rotate the obtained text proposal boxes uniformly so that they are wider than they are high.

[0072] S2.9. Perform a perspective transformation on the rotated text proposal box to correct any unreasonable container number or letter representations in the text area image. The independent character feature enhancement module enhances the salient features output by the saliency foreground perception network, enabling it to encode rich semantically rich character information and positional information.

[0073] S2.10. Map the obtained text proposal box back to the global saliency feature map F saliency , get the text saliency feature map F text .

[0074] S3. Obtain the key through the independent character feature enhancement module.

[0075] S3.1. Obtained text saliency feature map F text For each row of , a two-layer long short-term memory network (LSTM) with a hidden state size of 128 is used to capture global context information. For all rows, LSTM shares parameters to avoid overfitting and reduce the number of parameters:

[0076]

[0077]

[0078] and are the hidden variables of the first and second LSTM layers respectively.

[0079] S3.2, Through the 3x3 convolution operation, the character enhancement feature F is obtained enhanced :

[0080] F enhanced =conv(F·F),

[0081] F is included Tensor, conv represents 3x3 convolution.

[0082] S4. Get the query through the position encoding layer.

[0083] S4.1. The position encoding layer encodes the time steps, inputs a key vector for each time step and outputs a specific query vector q t , the embedding vector at each time step remains unchanged across different decoding sequences.

[0084] S5. Transform the text saliency feature map F text As the value input of the self-attention model, the character enhanced feature F enhanced As the key input of the self-attention mechanism model, the query vector q t As the query input of the self-attention model, the feature vector g is obtained through the self-attention model t .

[0085] In addition, the model training process is optimized using the following loss:

[0086]

[0087] Where I is a graph from the set Q of all text proposal boxes, S is a label containing all text character categories, L is the length of the text sequence of a text proposal box, 10g(P(S l |l, I)) means that the prediction result of the lth character in the predicted sequence is S l The conditional probability of .

[0088] Merge text detection L det and text recognition loss L rec , optimize in the same model:

[0089] Ltotal =αL det +βL rec ,

[0090] α and β are adjustable parameters during model training. Here, α is set to 0.8 and β is set to 1.

[0091] S6, eigenvector g t After the linear layer, we get the end-to-end text detection map and text recognition results.

[0092] S7. Filter out incorrect parts according to ISO6346 (1995) standard to obtain the final output.

[0093] Specifically, since there are a large number of recognition results in video recognition, the characters that appear the most times in the recognition results of container number and container size are taken as the final output. If there are character strings that appear the same number of times in a result, the character string that is recognized first is taken.

[0094] This paper proposes a novel end-to-end container number recognition method based on images and videos. By innovating on traditional text detection and recognition methods, this joint model utilizes a saliency foreground-aware network, an independent character feature enhancement module, a position encoding layer, and a self-attention model to simultaneously address both text detection and recognition. Container number and number information recognition differs from traditional text recognition. Container images contain a significant amount of irrelevant information that needs to be filtered out, and the container number and size have no contextual semantic connection. Traditional encoding-decoding-based text recognition methods weaken positional cues and strengthen contextual connections as the decoding time step increases, potentially leading to alignment drift and misrecognition. Our method, however, is more robust and practical for container number and size recognition. Furthermore, while most existing container number methods rely on image-based detection and recognition, our method incorporates detection and recognition of container numbers and sizes in container videos. Our method effectively recognizes vertically non-semantic text and mixed text with letters and numbers at steep angles, achieving high accuracy. Our method also achieves good results even in adverse shooting conditions such as fog and darkness. Our method has achieved ideal recognition accuracy in a variety of environments, different perspectives, and multiple scales, and has basically met the conditions for trial operation in a production environment.

[0095] Our approach was compared with two commercial algorithms and tested on three large datasets. Algorithms were considered to have failed if they misidentified or missed container numbers. The test data is as follows:

[0096] Table 1: Yangpu checkpoint on-site pictures as verification library 1

[0097] algorithm Total number of pictures success fail Recognition rate This patented method 3000 2731 269 91.03％ Business Algorithms 1 3000 2512 488 83.73％ Business Algorithms 2 3000 2483 517 82.77％

[0098] Table 2: Zengcheng checkpoint site photos as verification library 2

[0099] algorithm Total number of pictures success fail Recognition rate This patented method 2000 1935 65 96.75％ Business Algorithms 1 2000 1338 662 66.9％ Business Algorithms 2 2000 1957 43 97.85％

[0100] Table 2: Datian checkpoint site pictures as verification library 3

[0101] algorithm Total number of pictures success fail Recognition rate This patented method 613 459 154 74.88％ Business Algorithms 1 613 336 277 54.81％ Business Algorithms 2 613 321 292 52.37％

[0102] After testing against three validation libraries, our static image recognition method performed better than the two existing methods. Recognition errors primarily occurred in low-light conditions and when the container number printing was damaged or obscured. Our image and video stream-based algorithms have met the requirements of production environments and have been officially put into production.

[0103] like Figure 4 As shown, an end-to-end container number identification system includes:

[0104] An input module, used to obtain container images and input the container images into the pre-trained recognition model;

[0105] The salient foreground perception module extracts and segments the input image based on the salient foreground perception network to obtain a text salient feature map;

[0106] The character feature enhancement module performs character enhancement on the text saliency feature map based on the LSTM network to obtain character enhancement features;

[0107] The position encoding module encodes the time steps based on the position encoding layer and outputs the query vector;

[0108] The self-attention mechanism module is based on the self-attention mechanism model, takes the text saliency feature map as the value input, the character enhancement feature as the key input, and the query vector as the query input, and outputs a feature vector;

[0109] The linear output module outputs the feature vector based on the linear layer to obtain the recognition result.

[0110] The contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0111] An end-to-end container number identification device:

[0112] at least one processor;

[0113] at least one memory for storing at least one program;

[0114] When the at least one program is executed by the at least one processor, the at least one processor implements the end-to-end container number identification method as described above.

[0115] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0116] A storage medium storing processor-executable instructions, wherein the processor-executable instructions are used to implement the above-mentioned end-to-end container number identification method when executed by the processor.

[0117] The contents of the above method embodiments are all applicable to the present storage medium embodiment. The functions specifically implemented by the present storage medium embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0118] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. An end-to-end container number identification method, characterized in that: The following steps are involved: Obtain container images and input them into the pre-trained recognition model; Based on the saliency foreground perception network, the input image is feature extracted and segmented to obtain the text saliency feature map; Based on the LSTM network, the text salient feature map is enhanced to obtain character enhancement features; Encode the time steps based on the position encoding layer and output the query vector; Based on the self-attention mechanism model, it takes the text saliency feature map as the value input, the character enhancement feature as the key input, and the query vector as the query input, and outputs the feature vector; Output the feature vector based on the linear layer to obtain the recognition result; The step of extracting and segmenting the input image based on the saliency foreground perception network to obtain a text saliency feature map specifically includes: Get container images; Perform preliminary feature extraction on the container image to obtain the original image feature map; Introducing the attention similarity of different points, the original image feature map is reshaped by latitude to obtain a global feature map; Add the global feature map to the original image feature map to obtain the global salient feature map; The global salient feature map is fused to obtain the text proposal box; Map the text proposal box back to the global salient feature map to obtain the text salient feature map; The step of introducing attention similarity at different points and reshaping the original image feature map by latitude to obtain a global feature map specifically includes: Introducing attention similarity at different points; Convolution and reshaping are performed on the three dimensions of the original image feature map to obtain the reshaped first dimension, the reshaped second dimension, and the reshaped third dimension; Multiply the reshaped first dimension and the reshaped second dimension and perform sofamax processing to obtain the similarity matrix; Multiply the reshaped third dimension with the similarity matrix to obtain the global feature map.

2. The end-to-end container number identification method according to claim 1, characterized in that: Also includes: Obtain the video to be tested and extract frames from it to obtain a container image.

3. The end-to-end container number identification method according to claim 2, characterized in that: The formula for introducing attention similarity at different points is expressed as follows: In the above formula, subscript i is the index of the output position of the global feature response, subscript j lists the index of all possible global positions, X is the input feature map, W θ and W Φ is the weight matrix in dimensions θ and Φ that the network needs to learn.

4. The end-to-end container number identification method according to claim 3, characterized in that: The step of performing character enhancement on the text salient feature map based on the LSTM network to obtain character enhancement features specifically includes: A two-layer LSTM is used to process the text salient feature map and capture global context information. The hidden variables of the second layer LSTM are convolved to obtain character enhancement features.

5. The end-to-end container number identification method according to claim 1, characterized in that: During the training process of the recognition model, text detection loss and text recognition loss are used for model optimization.

6. The end-to-end container number identification method according to claim 5, characterized in that: The expression of the text detection loss is as follows: In the above formula, P tex(i) and G tex(i) are the predicted value of the i-th pixel in the text segmentation result and the true value of the text area, respectively.

7. The end-to-end container number identification method according to claim 5, characterized in that: The expression of the text detection loss is as follows: In the above formula, I is a graph from the set Q of all text proposal boxes, S is the label containing all text character categories, L is the length of the text sequence of a text proposal box, log(P(S l |l,I)) means that the prediction result of the lth character in the predicted sequence is S l The conditional probability of .

8. An end-to-end container number identification system, characterized in that: The method for performing an end-to-end container number identification method according to claim 1 comprises: An input module, used to obtain container images and input the container images into the pre-trained recognition model; The salient foreground perception module extracts and segments the input image based on the salient foreground perception network to obtain a text salient feature map; The character feature enhancement module performs character enhancement on the text salient feature map based on the LSTM network to obtain character enhancement features; The position encoding module encodes the time steps based on the position encoding layer and outputs the query vector; The self-attention mechanism module is based on the self-attention mechanism model, takes the text saliency feature map as the value input, the character enhancement feature as the key input, and the query vector as the query input, and outputs a feature vector; The linear output module outputs the feature vector based on the linear layer to obtain the recognition result.

Images