Express delivery sheet recognition and self-correction method and recognition system

Abstract

The application discloses a kind of express sheet identification and self-correction method, identification system.Therein express sheet identification and self-correction method includes: obtaining the image of express carries out text block detection, and the character content of text block is pre-identified;According to the character content of pre-identified text block and the position of text block, relationship matching analysis is carried out, to obtain the interrelation of text block and the category of text block;According to the interrelation of text block and the character of text block pre-identified, the character of text block pre-identified is respectively inferred to realize self-correction;Self-corrected character information and text block category are used as the identification result of express sheet.The application can improve the identification accuracy of express sheet, and the scope of application is wider.

Description

Technical Field

[0001] This invention relates to the technical field of OCR (Optical Character Recognition), and more particularly to a method for recognizing and self-correcting express delivery waybills. Background Technology

[0002] Because it is currently difficult to obtain and share recipient and sender information for packages from different courier companies, township-level courier forwarding stations cannot directly sort incoming packages. To reduce the workload of manual sorting, OCR technology is needed to recognize courier waybills. However, the layout structure of courier waybills varies greatly from company to company, making layout analysis based solely on vision difficult. Furthermore, many courier waybills have issues such as random scribbling and missing parts, resulting in low accuracy of conventional OCR algorithms.

[0003] Currently, the recognition of express delivery waybill information typically treats the waybill as a special complex table, and uses table layout analysis and character OCR algorithms for parsing and recognition. There are three main methods:

[0004] 1. Directly detect the text box position. This method uses visual algorithms such as object detection or semantic segmentation to detect the specific location of the text boxes. Simultaneously, it performs OCR recognition on the text boxes and then derives row, column, and cell information from the spatial arrangement of the text boxes. This information is then combined with the text content of each text box to generate a spreadsheet. However, this method is highly dependent on OCR detection results and manually designed rules. Different styles of waybills require separate development, resulting in very poor adaptability and scalability for new waybill formats.

[0005] 2. Layout analysis using table line detection. This method involves detecting corner points using object detection algorithms, or extracting table lines using image morphological transformations, texture extraction, and edge detection. Then, information about rows, columns, and merged cells is derived from the corner points and table lines. OCR recognition is then performed on the text boxes within the cells, and finally, the layout analysis results are combined to generate a spreadsheet. However, this method requires relatively clear and complete table lines and is not suitable for the layout analysis of complex tables without lines, such as express delivery waybills.

[0006] 3. End-to-end neural network learning. Using specially designed neural network models, such as TableBank, image-to-text technology converts table images into a structured description language (e.g., HTML tags defining table structure), containing table layout and text content, directly generating spreadsheets. However, this solution has poor robustness. For different types of express delivery waybills, some types may exhibit poor recognition performance, and there's no way to intervene and correct this at intermediate stages. Model repair and adjustment are difficult, making it unsuitable for engineering deployment.

[0007] Therefore, how to provide a highly accurate method for recognizing and self-correcting express waybills is a technical problem that needs to be solved. Summary of the Invention

[0008] To address the technical problems of low accuracy and narrow application range of existing recognition algorithms, this invention proposes a method and system for recognizing and self-correcting express waybills.

[0009] The express waybill recognition and self-correction method proposed in this invention includes:

[0010] Step 1: Obtain an image of the express delivery waybill, perform text block detection, and pre-recognize the character content of the text blocks;

[0011] Step 2: Perform relationship matching analysis based on the character content and position of the pre-identified text blocks to obtain the interrelationships between text blocks and the category of each text block;

[0012] Step 3: Based on the relationships between text blocks and the character content of the pre-identified text blocks, perform semantic analysis and reason about the character content of the text blocks to achieve self-correction.

[0013] Step 4: Use the character information obtained from self-correction and the category of the text block as the recognition result of the express waybill.

[0014] Furthermore, data frames are extracted from the video stream and input into a preset express delivery waybill detection model to obtain the image of the express delivery waybill.

[0015] Furthermore, step 2 includes:

[0016] Step 2.1: Embedding encoding is performed on each text block as a unit. Each text block is encoded as an embedding vector. The embedding vectors of all text blocks are concatenated and transformed to obtain the encoding matrix of all text blocks in the form of express delivery slips.

[0017] Step 2.2: Analyze the relative positional relationships between the text blocks to obtain the positional encoding matrix of all text blocks on a waybill;

[0018] Step 2.3: Perform multi-head self-attention reasoning on the encoding matrix and position encoding matrix of all text blocks of a courier waybill to extract the semantic relationship between each text block and other text blocks, and form a relationship matching matrix;

[0019] Step 2.4: Obtain the category of the text block based on the relationship matching matrix.

[0020] Furthermore, step 2.1 includes:

[0021] The text content of each text block is segmented using a word segmentation algorithm to obtain S([c1,c2,ci,……,cn]), where ci is the smallest semantic unit of the text. The text is then mapped to a one-dimensional text word vector S0 with a length of L0 by querying the word list.

[0022] Based on the text block numbering order, the word vectors of each text are concatenated into a text mapping matrix. Where M0 is the maximum number of text blocks;

[0023] Set two trainable matrix variables. and N0 is the length of the model's latent variables, expressed as E t =Z t ×K t +b t Calculate the encoding matrix of the text blocks of the express delivery waybill.

[0024] Furthermore, step 2.2 includes:

[0025] The center point coordinates of each text block are sorted and numbered in a certain order, and the resulting text block number is denoted as d. The corresponding positional encoding vector has a length of N0 and a value of P. d , where P d The calculation formula is:

[0026]

[0027] The text block encoding vectors are concatenated into a positional encoding matrix according to their numbering order.

[0028] Furthermore, the relationship matching matrix is ​​used for inference via an FFN network to obtain the category of the text block. The formula for calculating the loss function during inference is as follows: Where M0 is the maximum number of text blocks, types is the number of text block categories, and y di y represents the label value for a certain category. di_pre_prob It is the probability value of that category that is output during inference.

[0029] Furthermore, step 3 includes:

[0030] Step 3.1: Embed the characters within each text block to obtain the character encoding matrix for each text block;

[0031] Step 3.2: Concatenate the relation matching matrix of a waybill with all its character encoding matrices to obtain the text block input encoding matrix. The concatenation formula is X. i =RE i ||E i RE i It is the relation matching vector corresponding to the i-th text block, with a length of N0, E i It is the character embeddings encoding matrix corresponding to the i-th text block, with shape [L0, N0], X i This represents the input encoding matrix of the i-th text block, with shape [L0+1, N0].

[0032] Step 3.3: Input the text block into the encoding matrix and perform multi-head self-attention inference to obtain the character feature matrix;

[0033] Step 3.4: Obtain the encoding matrix of each corrected text block based on the character feature matrix.

[0034] Furthermore, step 3.1 includes:

[0035] The pre-identified characters in the text block are segmented using a word segmentation algorithm to obtain S([c1,c2,ci,……,cn]), where ci is the smallest semantic unit of the character;

[0036] By querying the vocabulary, the text is mapped to a one-dimensional text word vector S0 with a length of L0.

[0037] Set up a trainable dictionary matrix and

[0038] The character encoding matrix corresponding to the text block is calculated based on two dictionary matrices.

[0039] The express waybill recognition system proposed in this invention includes:

[0040] A camera is used to capture images;

[0041] An industrial control computer is used to acquire formatted text images from images, perform text block detection, and pre-identify the text content of the text blocks. Based on the pre-identified text content and position of the text blocks, a relationship matching analysis is performed to obtain the relationship between the text blocks and their positions, as well as the category of the text blocks. Based on the category of the text blocks and semantic analysis, the text content of the text blocks is inferred to achieve self-correction. The self-corrected text information is used as the recognition result of the formatted text.

[0042] Furthermore, the camera is provided in at least two sets.

[0043] Compared with the prior art, the present invention has the following beneficial effects.

[0044] This invention designs a relation matching module based on an attention mechanism, comprehensively considering text semantics and text block position, to achieve reliable classification of text blocks. Different express delivery waybills have significantly different layouts; some show the recipient first and the sender second, while others show the sender first (such as Aneng Logistics), and the text identifiers also vary. Conventional table parsing algorithms cannot accurately parse the layouts of various express delivery waybills, and simple semantic analysis cannot accurately distinguish between recipient and sender information. This system, however, comprehensively considers contextual semantics and text block position, achieving reliable classification of text blocks.

[0045] This invention designs a text self-correction module based on an attention mechanism. Based on semantic analysis of the text, it combines contextual relationship matching information to detect and correct errors in the original text. During express delivery, it is common for parts of the waybill to be written on, covered up, or damaged. Conventional OCR algorithms cannot effectively identify damaged content, while this system can perform in-depth semantic analysis of the original recognition results and context based on OCR pre-recognition, effectively detecting and correcting damaged content. Furthermore, the system adopts a text-generative network structure, which can support error correction for missing or extra characters, significantly improving the accuracy of text self-correction.

[0046] This invention designs a multi-stage, multi-branch training method based on the dependencies between algorithm modules, and designs specific data augmentation strategies based on the characteristics of each model dataset, which effectively reduces the workload of dataset collection and processing, lowers the difficulty of model training, and enhances the generalization ability of the model. Attached Figure Description

[0047] The present invention will now be described in detail with reference to the embodiments and accompanying drawings, wherein:

[0048] Figure 1 This is a system architecture diagram of an embodiment of the present invention.

[0049] Figure 2This is a flowchart of an identification and self-correction algorithm according to an embodiment of the present invention.

[0050] Figure 3 This is a structural diagram of an identification and self-correction algorithm according to an embodiment of the present invention.

[0051] Figure 4 This is a network structure diagram of the Transformer Encoder module of a relation matching algorithm according to an embodiment of the present invention.

[0052] Figure 5 This is a network structure diagram of the Transformer Encoder module of a self-correction algorithm according to an embodiment of the present invention. Detailed Implementation

[0053] To make the technical problems to be solved, the technical solutions, and the beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0054] Therefore, a feature pointed out in this specification is used to describe one feature of one embodiment of the invention, and does not imply that every embodiment of the invention must have the described feature. Furthermore, it should be noted that this specification describes many features. Although certain features may be combined to illustrate possible system designs, these features may also be used in other combinations not explicitly stated. Therefore, unless otherwise stated, the described combinations are not intended to be limiting.

[0055] The express waybill recognition and self-correction method proposed in this invention mainly includes the following steps.

[0056] Step 1: Obtain an image of the formatted text, perform text block detection, and pre-identify the character content of the text blocks. In other words, the first step is to obtain the text blocks and their approximate content.

[0057] Step 2 involves performing relationship matching analysis based on the character content of the pre-identified text blocks and the positions between them to determine the category of each text block. Different companies' express delivery waybills have specific formatting rules. For example, on Shentong Express, the recipient's name and phone number precede the address, while on Tehuisong Express, they follow. Similarly, on Aneng Logistics waybills, the sender's information precedes the recipient's information, while on SF Express, the recipient's information precedes the sender's information. Many logistics waybills only contain recipient information without sender information. Furthermore, for a multi-line address, different lines correspond to different text blocks, and there are close semantic relationships between the addresses on each line. Therefore, semantic analysis of the relationships between text blocks can effectively improve the accuracy of text block classification.

[0058] Step 3: Based on the relationships between text blocks and the character content of the pre-identified text blocks, perform semantic analysis and reason about the character content of the text blocks to achieve self-correction.

[0059] Step 4: Finally, the self-corrected character information and text block category are used as the recognition result of the express waybill.

[0060] Although there are many courier companies, each company's waybills follow certain patterns. For example, Yunda Logistics only offers a few fixed delivery methods such as "door-to-door delivery," "no upstairs delivery," and "self-pickup." Zhongtong Express uses "delivery" to identify the recipient's address, while Aneng Express uses "consignor" to identify the recipient's name. Debon Logistics uses "****" to protect privacy, while Tehuisong uses "^_^" for privacy protection. Furthermore, Yuantong's tracking numbers begin with "YT," and JD.com's tracking numbers begin with "JD." Therefore, fully utilizing the relationships between the text blocks in the courier waybill can help check and correct the semantic meaning within each text block, thereby improving the accuracy of text correction.

[0061] Through the above steps, this invention extracts all text blocks using a text block detection algorithm, and then performs relationship matching analysis based on the pre-identified characters and position coordinates of the text blocks, thereby realizing layout analysis and semantic self-correction of express delivery waybills.

[0062] In one embodiment, the express delivery label image of the present invention is obtained by extracting data frames from a video stream and inputting the data frames into a preset express delivery label detection model. When an express package passes by, a video stream of the express package is captured by taking a picture. Data frames are extracted from these video streams, and the image corresponding to the data frame is input into a preset format text type detection model to obtain the express delivery label image on the express package.

[0063] Express delivery waybill detection requires an algorithm. Considering that this algorithm runs on the camera side, this invention uses a lightweight YOLOV4-tiny network for express delivery waybill detection. A single frame of high-definition express delivery waybill image is read from the camera, and the high-definition image is resized to a preset resolution [416, 416, 3]. The YOLOV4-tiny model is then used to process the input image x... img ∈R 3×416×416 Through reasoning and prediction, the category of the formatted text is determined to be a courier waybill.

[0064] In one embodiment, the present invention employs the DBNet algorithm for text block detection. The backbone network uses ResNet, with an additional FPN structure to obtain a feature map F. Then, a probability map, a threshold map, and a binary map are obtained through inference from the feature map F, thereby achieving text box segmentation. The loss function is: L = L S +α×Lb +β×L t L S It is the probabilistic graphical loss function, L b It is the binary graph loss function, L t It is a threshold plot loss function, with α taking the value 1.0, β taking the value 10, and L... S and L b The solution is obtained using binary cross-entropy (BCE), L t L1 loss is used for calculation.

[0065] In one embodiment, the present invention employs a single-line Chinese character recognition algorithm to pre-recognize the character content of text blocks. This single-line Chinese character recognition algorithm, also known as an object detection algorithm, is mainly responsible for pre-recognizing the detected text blocks and consists of three parts: backbone, detection neck, and detection head. The backbone uses the lightweight network GhostNet to maximize detection capability while significantly reducing computational load. The detection neck uses the LSTM algorithm to perform semantic association between the recognized Chinese characters, achieving efficient use of feature information. The detection head uses the CTC algorithm, which supports end-to-end recognition of variable-length text. The loss function is CTC Loss: L(S) = -∑ (x,z)∈S ln p(zx), where S represents the training set space, x is the inferred input sequence, z represents the labeled sequence, and p(zx) represents the cumulative value of all output sequences in x that can be mapped to z.

[0066] In one embodiment, step 2 of the present invention performs relationship matching analysis based on the character content and position of the pre-identified text block to obtain the mutual relationship between text blocks and the category of each text block, specifically including the following steps.

[0067] Step 2.1: Embedding encoding is performed on each text block as a unit. Each text block is encoded as an embedding vector. The embedding vectors of all text blocks are concatenated and transformed to obtain the encoding matrix of all text blocks in the form of express delivery slips.

[0068] Step 2.1 can be further divided into the following steps.

[0069] The text content of each text block is segmented using a word segmentation algorithm to obtain S([c1,c2,ci,……,cn]), where ci is the smallest semantic unit of the text. The text is then mapped to a one-dimensional text word vector S0 with a length of L0 by querying the word list.

[0070] Based on the text block numbering order, the word vectors of each text are concatenated into a text mapping matrix. Where M0 is the maximum number of text blocks;

[0071] Set two trainable matrix variables. and N0 is the length of the model's latent variables, expressed as E t =Z t ×K t +b t Calculate the encoding matrix of the text blocks of the express delivery waybill.

[0072] Step 2.2: Analyze the relative positional relationships between the text blocks to obtain the positional encoding matrix of all text blocks on a waybill;

[0073] In step 2.2, the center point coordinates of each text block are sorted and numbered in a certain order, and the resulting text block number is denoted as d. The corresponding position encoding vector has a length of N0 and a value of P. d , where P d The calculation formula is:

[0074]

[0075] The text block encoding vectors are concatenated into a positional encoding matrix according to their numbering order.

[0076] Step 2.3: Perform multi-head self-attention reasoning on the encoding matrix and position encoding matrix of all text blocks of a courier waybill to extract the semantic relationship between each text block and other text blocks, and form a relationship matching matrix;

[0077] Step 2.4: Obtain the category of the text block based on the relationship matching matrix.

[0078] In one embodiment, the relation matching matrix is ​​used to obtain the category of the text block through FFN network inference, and the formula for calculating the loss function during inference is as follows: Where M0 is the maximum number of text blocks, types is the number of text block categories, and y di y represents the label value for a certain category. di_pre_prob It is the probability value of that category that is output during inference.

[0079] Step 2 above can also be called a relation matching algorithm. The relation matching algorithm uses text blocks as the basic unit to perform semantic relation reasoning on the text blocks contained in the entire express delivery waybill. The main structure is as follows: Figure 3As shown, the algorithm inputs are the location information of text blocks and the pre-identified character content, and the outputs are the relationship matching matrix and the category of each text block. First, based on the pre-identified character information of the text blocks, each text block is embedded into an embedding vector. The embedding vectors of all text blocks are concatenated and transformed to obtain the overall text block encoding matrix of the express delivery waybill. Since the number of text blocks in each express delivery waybill is fixed, the maximum number of text blocks can be set to 64. Then, the overall text block encoding matrix and the location encoding of each text block are input into a multi-level self-attention module for inference, obtaining the relationship matching matrix between the text blocks. This relationship matching matrix has two main functions: first, it obtains the specific category of each text block (such as recipient address, sender name, etc.) through a forward network inference; second, it is input into the subsequent text self-correction algorithm to improve the text self-correction effect by utilizing the relationships between text blocks.

[0080] This relation matching algorithm can also be divided into text block embedding calculation method, text block position embedding calculation method, and text block attention module calculation method.

[0081] The text block embedding calculation method mainly includes the following steps.

[0082] First, a word segmentation algorithm is used to segment the character content of the text block using the tokenizer method, resulting in S([c1,c2,ci,……,cn]), where ci is the smallest semantic unit of the character; Chinese characters are represented by individual characters, and English words by individual words. Then, through a word lookup, the characters are mapped to a one-dimensional word vector S. i , where i is the character number, the vector length is L0 (L0 is the longest number of characters in the text block, which is 128 in one embodiment), and any insufficient part is padded with 0.

[0083] The word vectors of each text block are concatenated into a text mapping matrix according to the corresponding text block numbering order (the text blocks are numbered using a text block position embedding calculation method). (Where M0 is the maximum number of text blocks, which is 64 in one embodiment of the present invention), and any insufficient part is padded with 0.

[0084] Set two trainable matrix variables. and (where N0 is the length of the model's latent variables, which is 1024 in one embodiment of this invention), through E t =Z t ×K t +b t Calculate the text block embeddings matrix of the express delivery waybill.

[0085] The text block position embedding calculation method mainly includes the following steps.

[0086] For the text embeddings matrix The corresponding position encoding matrix is First, sort and number each text block according to its center point coordinates, from left to right and from top to bottom. The text block number is denoted as d, and the corresponding position encoding vector has a length of N0 and a value of P. d , where P d The calculation formula is:

[0087]

[0088] Then, the text block position encoding vectors are concatenated into a position encoding matrix according to their numbering order. The purpose of 'i' is to distinguish whether 'd' is odd or even, and to make the numbers in adjacent positions farther apart.

[0089] The text block attention module calculation method includes the following steps.

[0090] Text block embeddings matrix E t and position encoding matrix E p After inference by the text block attention module, the relation matching matrix RE for the express delivery waybill is obtained. The text block attention module is a multi-level self-attention network, the specific structure of which is as follows: Figure 4 As shown, the text block embeddings matrix E t and position encoding matrix E p The summation is used as the input variable for the K and Q channels, while the V channel directly inputs the text block embeddings matrix E. t Then, multi-head self-attention inference is performed. The feature matrix output by the text block self-attention module is used as the input text block feature matrix for the next-level text block self-attention module.

[0091] The number N of cascaded text block self-attention modules can be flexibly selected according to the type and quantity of text blocks. In one embodiment of the present invention, N=6. The text feature matrix finally output by the multi-level self-attention modules is the relation matching matrix RE of the express delivery waybill, which is input into the text self-correction module for the next step of inference.

[0092] Loss function calculation is also required when training the relationship matching model.

[0093] The relation matching matrix RE output by the text block attention module is used for inference through a feedforward neural network (FFN) to directly derive the category of each text block. The loss function used is categorical cross-entropy, calculated as follows: Where M0 is the maximum number of text blocks, types is the number of text block categories (taking a courier waybill as an example, text block categories can include: courier company name, recipient name, recipient phone number, etc.), y di y represents the label value for a certain category. di_pre_prob It is the probability value for this category output by the model.

[0094] This invention employs a self-correction model to automatically correct characters within a single text block based on the semantic information within the block and the semantic relationships between text blocks. The model's input consists of pre-identified text characters and the relation matching matrix RE output by the relation matching module; the output is the self-corrected text information. The main process involves embedding the characters within the text block, adding this to the corresponding character position encoding, concatenating it with the relation matching matrix RE, and then using self-attention reasoning to obtain the corrected semantic text.

[0095] Self-correction mainly includes the calculation methods of the character embeddings module, the calculation methods of the attention encoding module, and the calculation of the loss function of the text self-correction algorithm.

[0096] The character embeddings module calculation method includes the following steps.

[0097] Step 1: Use the tokenizer method to segment the text to obtain S([c1,c2,ci,……,cn]), where ci is the smallest semantic unit of the text, Chinese characters are extracted and English words are extracted; through word lookup, the text is mapped to a one-dimensional word vector S0 with a length of L0 (L0 is the longest number of characters in the text block, which is 128 in this system), and any insufficient part is padded with 0.

[0098] Step 2: Generate word segmentation position vectors according to the word segmentation order in the input encoding vector. The position vector corresponding to S0 is set as P according to the character order. i ([1,2,…,L0]), the total length of the word segmentation position vector is the same as the total length of the input encoding vector, both being L0.

[0099] Step 3: Set up a trainable dictionary matrix and Where W0 is the total number of character codes, L0 is the longest character count in the text block, and N0 is the length of the model's latent variables. The Embeddings code and position Embeddings code of the corresponding character are calculated using a lookup table. Adding these together yields the character Embeddings code matrix for the text block.

[0100] Step 4: Concatenate the relation matching matrix RE and the character encoding matrix, using the formula: X i =RE i ||E i , where RE i It is the relation matching vector corresponding to the i-th text block, with a length of N0, E i It is the character embeddings encoding matrix corresponding to the i-th text block, with shape [L0, N0], X i This represents the input encoding matrix for the i-th text block, with shape [formula missing].

[0101] [L0+1,N0].

[0102] The character attention encoding module uses N cascaded Transformer Encoder modules (specific structure as follows) Figure 5 The input value is composed of the text block input encoding matrix X (as shown). i The output value is the current character feature matrix, which is used as the input value for the next level Transformer Encoder module for the next step of model inference.

[0103] The text self-correction algorithm calculates its loss function by passing the text feature matrix output by the character attention module through a feedforward neural network (FFN) to directly derive the corrected encoding matrix for each text block. The loss function uses categorical cross-entropy, and the calculation formula is as follows: Where L0 is the maximum number of characters in each text block, num is the total number of Chinese characters, and y di The label value for classifying a character, y di_pre_prob This is the probability value for classifying this character in the model output.

[0104] Each of the above algorithms and models needs to be trained in advance. There are strong data dependencies between the various algorithm models in this invention, and training needs to be carried out step by step.

[0105] During the training of the express waybill detection and pre-recognition algorithms, the express waybill detection algorithm is trained directly using labeled express waybill image samples, and data augmentation methods such as random cropping, rotation, brightness, color, and noise reduction can be used. The text block detection and single-line Chinese recognition modules can directly use publicly available OCR datasets, while also incorporating express waybill-related datasets for training.

[0106] Training of the relation matching algorithm. The dataset file for the relation matching algorithm is labeled in XML format. Each express waybill sample is a master node, and the text blocks contained therein are corresponding child nodes. Each text block child node contains attributes such as text block number, pre-identified text, and text block category number.

[0107] The first step involves using a trained text block detection algorithm to infer the coordinates and dimensions of all text blocks in the collected express delivery waybill samples. These text blocks are then extracted and saved as text block images, numbered from left to right and top to bottom. A single-line Chinese character recognition algorithm is then used to infer the pre-recognized text corresponding to each text block image. The sample name, text block number, and pre-recognized text are then entered into the dataset file according to XML specifications. Finally, a manual process determines the text block category number based on the text block information and adds it to the corresponding node attributes in the annotation file, resulting in the original annotation file.

[0108] The second step is to create a dictionary of similar-looking characters for commonly used characters in express delivery, and then to obfuscate the pre-identified text in the original annotation file according to a fixed ratio, including randomly replacing similar-looking characters in 40%, randomly deleting words in 20%, randomly adding words in 20%, and randomly reversing the word order in 20%, thereby obtaining a data augmentation annotation file.

[0109] The third step involves mixing the original annotation files and the data augmentation annotation files, randomly reordering them, dividing them into training and validation sets, and completing the training of the relation matching algorithm.

[0110] The following section introduces the training of the text self-correction algorithm, that is, the training of the self-correction model.

[0111] Since the text self-correction algorithm requires the relation matching matrix output by the relation matching algorithm as input during training, the data annotation format of the two algorithms must be consistent. In this invention, the text self-correction algorithm dataset is based on the relation matching algorithm dataset, with an added "text label" attribute on the child nodes of the text block to represent the actual text characters of that text block.

[0112] The first step is to construct the text self-correction algorithm dataset by manually verifying the actual text characters in the "text label" attribute of each text block child node, based on the relation matching algorithm dataset.

[0113] The second step is to divide the prepared dataset into training and validation sets proportionally, and then begin algorithm training. The text self-correction algorithm must be trained using the main node of the express delivery waybill as the basic unit. First, all child node data within the main node are input into the relation matching algorithm for inference to obtain the corresponding relation matching matrix. Then, along with the corresponding attribute values ​​from the label file, this matrix is ​​input into the text self-correction algorithm for model training.

[0114] This invention also protects an express delivery waybill recognition system, which includes a camera and an industrial control computer. The camera is used to acquire raw images.

[0115] The industrial control computer is used to acquire express delivery waybill images from the original image, perform text block detection, and pre-recognize the text blocks. Based on the pre-recognized text content and the position of the text blocks, relationship matching analysis is performed to obtain the interrelationships and categories of the text blocks. Based on the interrelationships and semantics of the characters within the text blocks, the pre-recognized text of the text blocks is self-corrected. The self-corrected text information and text block category are used as the final recognition result.

[0116] In one embodiment, the camera is provided with at least two sets, which respectively capture express waybills at different height ranges to prevent image blurring due to insufficient depth of field.

[0117] Figure 1 This paper illustrates a specific embodiment of the express delivery waybill recognition system of the present invention. The system includes cameras 1 and 4, embedded inference modules 2 and 5, an industrial control computer 3, and a backend server 6.

[0118] Each module is connected and communicates via wired or wireless means. Specifically, embedded inference module 2 is connected to camera 1 via MIPI bus; embedded inference module 5 is connected to camera 4 via MIPI bus; embedded inference module 2 is connected to industrial computer 3 via Ethernet; embedded inference module 2 is connected to industrial computer 3 via Ethernet; embedded inference module 5 is connected to industrial computer 3 via Ethernet; and industrial computer 3 is connected to the backend server 6 via wired or 4G channel.

[0119] As shown in the system framework diagram, each module performs its own specific function. Cameras 1 and 4 are primarily responsible for acquiring video information within the shooting area and transmitting the acquired digital signals to embedded inference modules 2 and 5. Embedded inference modules 2 and 5 are responsible for compressing the images and performing algorithmic calculations to detect and extract the express delivery waybill image. The industrial control computer 3 is responsible for OCR recognition and layout analysis of the express delivery waybill image, and for performing character self-correction based on semantics. It then displays the express delivery image and recognition results on the monitor in real time, while simultaneously uploading the image data, recognition results, and configuration information to the backend server 6. The backend server 6 is responsible for the storage and management of the image data, recognition results, and configuration information, and sends the recognition results to other systems via API interfaces.

[0120] The main workflow of this system can be roughly divided into three stages. The first stage is the waybill detection stage. The device extracts a frame of image data from the video stream, preprocesses it, and then uses the waybill detection model to calculate the position and size of the waybill, extracting it as a separate waybill image. The second stage is the character pre-recognition stage. First, text block detection is performed on the corrected waybill. Then, an OCR algorithm is used cyclically to perform OCR pre-recognition on each text block. If the number of text blocks, the number of characters, and the confidence level of the pre-recognition results reach a set threshold, subsequent character self-correction is performed. The third stage is the character self-correction stage. The relationship matching module, based on the position of the text blocks and the content of the pre-recognized characters, uses an attention mechanism to infer and determine the relationships between the text blocks, thereby predicting the category of the text block (such as recipient address content, sender phone number content, etc.). Simultaneously, the output relationship vector is sent to the self-correction module to complete the error correction of the pre-recognized text characters.

[0121] Figure 3 The diagram illustrates the collaboration between the various algorithm modules. After the camera captures video, the corresponding data frames are extracted and used as input images to the waybill detection module. The waybill detection module then runs the waybill detection algorithm to filter out the waybill images from the data frames.

[0122] The image of the express delivery waybill is used as input data to the waybill pre-recognition module. The waybill pre-recognition module consists of two modules: a text block detection module and a single-line Chinese character recognition module. The text block detection module uses a text block detection algorithm to detect all text blocks on each express delivery waybill. The single-line Chinese character recognition algorithm pre-recognizes the character content of each text block.

[0123] Next, the text blocks are input into the relation matching module, which includes a text block position encoding module (named Text Block Position Embeddings in the diagram), a text block encoding module (named Text Block Embeddings in the diagram), a text block attention module, and an FFN module. The text block position encoding module generates a text block position encoding matrix based on the coordinates of the text blocks. The text block encoding module generates a text block encoding matrix based on the character content of the pre-identified text blocks, and both are then input into the text block attention module to obtain the relation matching matrix. The relation matching matrix is ​​then used as input to the FFN module to obtain the corresponding text block category, i.e., classification.

[0124] In addition, the relation matching matrix is ​​input into the text self-correction module. The text self-correction module includes a character encoding module (referred to as the character Embeddings module in the diagram), a character attention module, and an FFN module.

[0125] The character encoding module encodes the characters in each text block to generate a character encoding matrix. The character encoding matrix and the relation matching matrix are input into the character attention module to obtain the character feature matrix. The character feature matrix is ​​then processed by the FFN module to obtain self-corrected character information, i.e., self-corrected text information.

[0126] It should be noted that the FFN modules of the relation matching module and the text self-correction module are independent FFN modules.

[0127] This invention captures images of express delivery waybills using a camera, extracts all text blocks using a text block detection algorithm, and then performs relationship matching analysis based on the position coordinates of the text blocks and the pre-recognized text to achieve waybill layout analysis and OCR semantic self-correction.

[0128] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for recognizing and self-correcting express delivery waybills, characterized in that, include: Step 1: Obtain an image of the express delivery waybill, perform text block detection, and pre-recognize the character content of the text blocks; Step 2: Perform relationship matching analysis based on the character content and position of the pre-identified text blocks to obtain the interrelationships between text blocks and the category of each text block; Step 3: Based on the relationships between text blocks and the character content of the pre-identified text blocks, perform semantic analysis and reason about the character content of the text blocks to achieve self-correction. Step 4: Use the character information and text block category obtained from the self-correction as the recognition result of the express delivery waybill; Step 2 includes: Step 2.1: Embedding encoding is performed on each text block as a unit. Each text block is encoded as an embedding vector. The embedding vectors of all text blocks are concatenated and transformed to obtain the encoding matrix of all text blocks in the form of express delivery slips. Step 2.2: Analyze the relative positional relationships between the text blocks to obtain the positional encoding matrix of all text blocks on a waybill; Step 2.3: Perform multi-head self-attention reasoning on the encoding matrix and position encoding matrix of all text blocks of a courier waybill to extract the semantic relationship between each text block and other text blocks, and form a relationship matching matrix; Step 2.4: Obtain the category of the text block based on the relationship matching matrix; Step 3 includes: Step 3.1: Embed the characters within each text block to obtain the character encoding matrix for each text block; Step 3.2: Concatenate the relation matching matrix of a waybill with all its character encoding matrices to obtain the text block input encoding matrix. The concatenation formula is as follows: , It is the relation matching vector corresponding to the i-th text block, with a length of , It is the character embeddings encoding matrix corresponding to the i-th text block, with shape [formula missing]. , This represents the input encoding matrix for the i-th text block, with shape [formula missing]. ; Step 3.3: Input the text block into the encoding matrix and perform multi-head self-attention inference to obtain the character feature matrix; Step 3.4: Obtain the encoding matrix of each corrected text block based on the character feature matrix.

2. The express waybill recognition and self-correction method as described in claim 1, characterized in that, Data frames are extracted from the video stream and input into a preset express delivery waybill detection model to obtain the image of the express delivery waybill.

3. The express waybill recognition and self-correction method as described in claim 1, characterized in that, Step 2.1 includes: Each text block is segmented using a word segmentation algorithm to obtain S([c1,c2,ci,……,cn]), where ci is the smallest semantic unit of the text. Then, through a word lookup, the text is mapped to a one-dimensional text word vector. The vector length is ; Based on the text block numbering order, the word vectors of each text are concatenated into a text mapping matrix. ,in Maximum number of text blocks; Set two trainable matrix variables. and , The length of the model's latent variables is determined by... Calculate the encoding matrix of the text blocks of the express delivery waybill. .

4. The express waybill recognition and self-correction method as described in claim 3, characterized in that... Step 2.2 includes: The center point coordinates of each text block are sorted and numbered in a certain order, and the resulting text block numbers are denoted as follows: The corresponding positional encoding vector length is All values ​​are ,in The calculation formula is: ； The text block encoding vectors are concatenated into a positional encoding matrix according to their numbering order. .

5. The express waybill recognition and self-correction method as described in claim 1, characterized in that, The relation matching matrix is ​​used for inference via an FFN network to obtain the category of the text block. The formula for calculating the loss function during inference is as follows: ,in, This represents the maximum number of text blocks. The number of categories for text blocks. For the label value of a certain category, It is the probability value of that category that is output during inference.

6. The express waybill recognition and self-correction method as described in claim 1, characterized in that, Step 3.1 includes: The pre-identified characters in the text block are segmented using a word segmentation algorithm to obtain S([c1,c2,ci,……,cn]), where ci is the smallest semantic unit of the character; By querying the vocabulary, text is mapped to one-dimensional text word vectors. The vector length is ; Set up a trainable dictionary matrix and ; The character encoding matrix corresponding to the text block is calculated based on two dictionary matrices.

7. A waybill recognition system for express delivery, characterized in that, include: A camera is used to capture images; An industrial control computer is used to acquire formatted text images from images, perform text block detection, and pre-identify the text content of the text blocks. Based on the pre-identified text content and position of the text blocks, a relationship matching analysis is performed to obtain the relationship between the text blocks and their positions, as well as the category of the text blocks. Based on the category of the text blocks and semantic analysis, the text content of the text blocks is inferred to achieve self-correction. The self-corrected text information is used as the recognition result of the formatted text. The process involves performing relational matching analysis based on the text content and position of pre-identified text blocks to obtain the relationships between text blocks and their positions, as well as the category of each text block. This is done by embedding each text block into an embedding vector, concatenating and transforming the embedding vectors of all text blocks to obtain the encoding matrix of all text blocks, with the waybill as the unit. The relative positional relationships between the text blocks are analyzed to obtain the positional encoding matrix of all text blocks on a waybill. Multi-head self-attention reasoning is then applied to the encoding matrix and positional encoding matrix of all text blocks on a waybill to extract the semantic relationships between each text block and other text blocks, forming a relational matching matrix. The category of each text block is then determined based on the relational matching matrix. When performing semantic analysis and reasoning on the character content of text blocks to achieve self-correction, the characters within each text block are embedded and encoded to obtain the character encoding matrix for each text block. The relation matching matrix of a waybill and all its character encoding matrices are concatenated to obtain the text block input encoding matrix. The concatenation formula is as follows: , It is the relation matching vector corresponding to the i-th text block, with a length of , It is the character embeddings encoding matrix corresponding to the i-th text block, with shape [formula missing]. , This represents the input encoding matrix for the i-th text block, with shape [formula missing]. The text blocks are input into the encoding matrix for multi-head self-attention inference to obtain the character feature matrix; the corrected encoding matrix of each text block is obtained based on the character feature matrix.

8. The express delivery waybill recognition system as described in claim 7, characterized in that, The camera is provided in at least two sets.

Images