Driver ticket automatic recognition and filling method based on multi-mode neural network

Abstract

The application provides a driver's bill automatic identification and filling method based on a multi-mode neural network, comprising the following steps: S1: extracting multi-mode features; S2: training a bill category model; S3: predicting the category of a newly uploaded bill; and S4: extracting target filling information from a bill original picture; the method matches a label text block with a neighboring text box, does not need to rely on a fixed template, can adapt to various non-standardized bills, can effectively identify and extract information from disordered format bills, and greatly improves the adaptation capability for various irregular bill in a freight scene; a multi-mode neural network is used to classify bills from a picture visual feature and a text semantic feature double mode, solves the problem that only text recognition is easy to confuse, reduces bill category identification errors; meanwhile, target label positioning, position matching and content rule checking are used to accurately extract label values, avoid information matching errors and filling confusion, and realize accurate data entry.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and particularly to an automatic recognition and filling method for driver's bills based on a multi-modal neural network. Background Art

[0002] In the freight logistics link of the transportation industry, truck drivers need to rely on the logistics management system software to complete the filling of various expenses and business data, and at the same time need to upload the corresponding bill photos as vouchers; the types of bills that drivers need to fill are complex, including both expense invoices such as highway tolls, fuel costs, urea costs, repair costs, loading and unloading costs, telephone bills, parking fees, etc., and non-expense documents such as weighing lists, loading and unloading lists, and waybills. Moreover, the information dimensions to be filled in for various bills are diverse; drivers' daily operations mostly rely on mobile phones. Due to the objective conditions of small font sizes of bills and inconvenient mobile phone button operations, it is extremely easy for drivers to make mistakes such as selecting the wrong form category and filling in incorrect values, and they need to fill in again repeatedly, consuming a lot of time and energy;

[0003] To solve the problems of low bill filling efficiency and easy errors, the main solution in the industry at present is the picture OCR information extraction method, that is, to identify the text information in the bill photo through OCR technology, and then automatically fill the recognition result into the corresponding form of the logistics management system; by extracting the text information at the specified position in the bill picture and using this information as the label value for matching and filling. The technical logic of this method is applicable to application scenarios with unified target styles and stable content structures, such as information extraction of standardized vouchers such as bank cards, identity cards, and national unified standard shopping invoices;

[0004] However, the existing picture OCR information extraction method has the following drawbacks in the bill processing scenario of the freight industry:

[0005] 1. Poor adaptability to non-standard bills: Most bills in the freight scenario have no unified format standard, and the formats of highway tolls and parking fee invoices in different provinces vary significantly; documents such as loading and unloading fees and repair fees are mostly non-standard receipts issued by freight yards and repair shops, without a fixed layout; weighing lists, loading and unloading lists, waybills, etc. have completely different styles due to different freight yards, and the arrangement methods of label names and label values are diverse, including horizontal arrangement, vertical arrangement, and even mixed arrangement of the two. If the bill photo is taken obliquely, there will also be a situation where the information is arranged obliquely. The existing picture OCR information extraction method cannot effectively identify such bills with messy formats;

[0006] 2. High error rate in document recognition, prone to double recognition bias: Existing image OCR information extraction methods rely solely on text information for document processing, which easily leads to errors in tag value extraction and makes it impossible to accurately match the information in the document with the system entry items; at the same time, errors in document category recognition are prone to occur, resulting in chaotic system numerical entry and failure to achieve effective data entry.

[0007] 3. Poor scalability: With the development of the freight industry, the types of invoices continue to increase and the formats become more diversified. Existing image OCR information extraction methods based on extracting information from fixed locations cannot adapt to the dynamic changes in invoice formats. It is necessary to continuously adjust the extraction rules for new invoice types, which not only increases the technical maintenance cost, but also causes the invoice recognition error rate to continue to increase with the increase of invoice types, and cannot fundamentally solve the problem of automated data entry. Summary of the Invention

[0008] The purpose of this invention is to address the shortcomings of existing technologies by proposing an automatic driver ticket recognition and filling method based on a multimodal neural network.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] The method for automatic driver ticket recognition and filling based on multimodal neural networks includes the following steps:

[0011] S1: Extract multimodal features;

[0012] Includes the following sub-steps:

[0013] S11: Perform OCR text recognition on the original image of the invoice to generate text information;

[0014] The original image of the invoice is input into a general OCR recognition tool. The general OCR recognition tool scans the text area in the original image of the invoice and recognizes the corresponding text information. Each original image of the invoice corresponds to unique text information, which is in string format.

[0015] S12: Perform word segmentation on the text information to obtain a word vector matrix;

[0016] The text information in string form corresponding to the original image of each ticket is split into multiple independent and ordered words according to the Chinese word structure using a word segmentation tool, forming a word sequence corresponding to the original image of the ticket;

[0017] The tf-idf or word2Vec component is used to convert each word in the word sequence into a set of numerical word vectors. All word vectors are combined in the order of the words to obtain the corresponding word vector matrix.

[0018] S13: Obtain the text feature representation of the original image of each ticket;

[0019] The internal state of the RNN network in the multimodal neural network is initialized beforehand, and the initial internal state values ​​are set. , Within the interval [0, 1]; let the total number of word vectors in each word vector matrix be n, where n is a positive integer;

[0020] The multimodal neural network includes an RNN network, a CNN network, and a feature fusion module;

[0021] For each original ticket image, the word vectors in the word vector matrix are sequentially input into the RNN network. The RNN network outputs its internal state value at the current time step using the following formula:

[0022] ;

[0023] in, Let be the internal state value of the (t-1)th word vector; W is the preset weight matrix; b is the preset offset term; tanh is the hyperbolic tangent activation function; Let t be the t-th word vector in the word vector matrix;

[0024] Concatenate all the internal state values ​​obtained from each word vector matrix in word vector order to obtain the corresponding one-dimensional vector. ,Will This serves as a textual feature representation of the original image of the ticket;

[0025] S14: Input the original image of the ticket into a CNN network to form a feature vector;

[0026] The original image of the ticket is preprocessed, including size standardization and color standardization.

[0027] The preprocessed original image of the ticket is input into a CNN network. The CNN network performs 2D convolutional feature extraction to obtain a feature vector representing the visual features of the original ticket image, denoted as . ;

[0028] S15: Obtain the multimodal features of the original image of the invoice;

[0029] The text feature representation of the original ticket image output in step S13 is... The feature vector of the visual features of the original image of the ticket obtained in step S14 The input is fed into the feature fusion module, which performs a weighted fusion of the text feature representation weight coefficients and the visual feature vector weight coefficients based on pre-set weights to obtain the multimodal features of the original image of the ticket. ;

[0030] S2: Train the bill category model;

[0031] Includes the following sub-steps:

[0032] S21: Multimodal feature classification and combination;

[0033] Collect various known categories of documents in freight scenarios, such as weighbridge slips, highway toll invoices, and waybills; obtain multimodal features of known categories of documents based on the method in step S1; classify and combine the multimodal features to obtain a multimodal feature set for each category of documents, denoted as . ;

[0034] ;

[0035] Where i represents the type of bill and M represents the number of bills in this type. This is the i-th multimodal feature of bill category i;

[0036] S22: Calculate the correlation of multimodal features;

[0037] Element-wise, a combined inner product operation is performed on all multimodal features in the multimodal feature set of each type of ticket, and the result is used as the exponent. Substitute into the formula In this process, the characteristic correlation of this type of bill is obtained. ;

[0038] The element-level comprehensive inner product operation refers to multiplying all multimodal features in the multimodal feature set of each type of ticket according to their positions and then summing them up.

[0039] The feature relevance of each category of collected tickets By summing the results, we obtain the comprehensive multimodal feature relevance of all types of tickets. ;

[0040] ;

[0041] in, The result of the multimodal feature element-level comprehensive inner product operation for bill category i is given, where n is the total number of bill categories.

[0042] S23: Train the RNN network;

[0043] The training set is composed of all the ticket categories and their corresponding multimodal feature sets obtained in step S21.

[0044] Will As the training objective, the RNN network is iteratively trained using the training set; during training, the weight matrix W and the bias term b are continuously adjusted; when When the RNN network converges to its maximum value within a preset number of training rounds, the training is complete. At this point, the weight matrix W and the offset term b are the optimal parameters of the RNN network.

[0045] S3: Predict the type of newly uploaded invoice;

[0046] For newly uploaded ticket images by drivers, their multimodal features T are extracted using step S1. The multimodal features T are then combined with the M multimodal features of each category of tickets in the training set using element-wise inner product operations. The results are then substituted into... In the process, the predicted Probability of Return (POR) value for each category corresponding to the ticket image is obtained, and the ticket category with the largest predicted POR value is selected as the ticket category to which the ticket image belongs.

[0047] S4: Extract the target information to be filled in from the original image of the invoice;

[0048] Includes the following sub-steps:

[0049] S41: Obtain the corresponding information for text blocks and text boxes;

[0050] For invoice images that have completed invoice category prediction, text blocks and text boxes are extracted using an OCR recognition tool;

[0051] Specifically, text block extraction: The OCR recognition tool performs optical character recognition processing on the ticket image. By scanning the text area of ​​the ticket image, continuous text strings are recognized as a text block. Using blank areas as the segmentation basis, non-continuous text is split into multiple independent text blocks, thus completing the text block extraction.

[0052] Text box extraction: While recognizing text blocks, the OCR recognition tool matches a unique text box for each text block. The text box is a quadrilateral that is approximately rectangular and encloses the text block. The coordinates of the four vertices of the text box indicate the position of the text block in the ticket image.

[0053] Each text block and its corresponding text box form a set of TEXT-BOX key-value pairs. All TEXT-BOX key-value pairs of each ticket image that has completed category prediction are integrated to obtain a list of TEXT-BOX key-value pairs containing the ticket image.

[0054] S42: Find the tag text block;

[0055] Based on the business needs of freight industry invoice filling, the specific data items to be extracted and entered into the system for each invoice category are set in advance and denoted as the target values ​​to be extracted; each target value to be extracted corresponds to a unique target label.

[0056] Based on the ticket category to which the ticket image belongs, the system retrieves the set of tags and keywords corresponding to the target value to be extracted; the system contains the set of tags and keywords corresponding to each ticket category; the set of tags and keywords contains all possible expressions of the target tag and configures the matching priority for each expression;

[0057] Traverse all text blocks in the TEXT-BOX key-value pairs obtained in step S41. After symbol filtering for each text block, match the text block with all possible expressions of the target tag in turn based on the matching priority order in the tag keyword set. When the corresponding content of the text block completely matches a certain expression in the tag keyword set, stop traversing and matching, determine the text block as the tag text block corresponding to the current target value to be extracted, and proceed to step S43. If no corresponding expression is matched after traversing all text blocks, it is determined that there is no valid target tag in the ticket image, and the current target value extraction process is terminated.

[0058] S43: Get the coordinates of the label text box;

[0059] Based on the one-to-one correspondence between text blocks and text boxes in the TEXT-BOX key-value pair list, the text box corresponding to the tag text block is retrieved, and the coordinates of the four vertices of the text box are obtained. The text box is denoted as the tag text box S.

[0060] S={A( , ), B( , ), C( , ), D( , )};

[0061] Among them, A, B, C, and D are the top left, top right, bottom right, and bottom left vertices of the label text box S, respectively;

[0062] S44: Find other nearby text boxes and retrieve the target text box;

[0063] Based on the general layout rules of freight industry documents, other nearby text boxes are searched in the order of right side first and bottom side second; the text box corresponding to the feature value to be extracted is recorded as the target text box, and the target text box is obtained.

[0064] Includes the following sub-steps:

[0065] S441: Search for other text boxes to the right of label text box S, and determine whether the target text box exists;

[0066] Let the other text boxes be labeled T;

[0067] T={E( , ), F( , ), G( , ), H( , )};

[0068] Among them, E, F, G, and H are the top left, top right, bottom right, and bottom left vertices of the other text box T, respectively;

[0069] Match the label text box S against the other text boxes on its right in turn, and determine whether the following two conditions are met simultaneously:

[0070] (1) > and > ;

[0071] (2) > and < ;

[0072] The intersection point of the extension of line segment FE of other text boxes T and the extension of line segment DB of label text box S is M1. , The intersection point M2 is the extension of line segment GH of other text box T and the extension of line segment BD of label text box S. , The midpoint of line segment BD is M. , );

[0073] When both conditions are met, other corresponding text boxes are identified as target text boxes, and step S443 is initiated.

[0074] If the above two conditions cannot be met at the same time, then there is no target text box in the other text boxes to the right of the label text box S, and proceed to step S442;

[0075] S442: Search for other text boxes below the label text box S and determine if the target text box exists;

[0076] Match the label text box S sequentially with the other text boxes below it, and determine whether the following two conditions are met simultaneously:

[0077] (1) > and > ;

[0078] (2) < and > ;

[0079] The intersection point of the extension of line segment HE of other text box T and the extension of line segment CD of label text box S is... The intersection point of the extension of line segment GF of other text box T and the extension of line segment DC of label text box S is... The midpoint of line segment DC is ;

[0080] When both conditions are met, other corresponding text boxes are identified as target text boxes, and step S443 is initiated.

[0081] If both of the above conditions cannot be met simultaneously, then the target text box will not exist in any of the other text boxes below the label text box S; the process will end.

[0082] S443: Get the text block corresponding to the target text box;

[0083] Based on the TEXT-BOX key-value pair list of the ticket image, obtain the text block corresponding to each of the identified target text boxes, and retrieve the content of the text block.

[0084] S444: Verify the content of the text block;

[0085] Based on the business requirements for invoice filling in the freight industry, corresponding content verification rules are formulated for each target value to be extracted in advance; the content verification rules include the usage requirements of numbers, letters and symbols.

[0086] The text block content extracted in step S443 is matched and verified with the content verification rules of the corresponding target value to be extracted.

[0087] If the content of the text block conforms to the content validation rules, then the content of the text block is determined to be a valid target input value, and the extraction is complete;

[0088] If the content of a text block does not conform to the content validation rules, the content of the text block is determined to be an invalid target input value. Steps S441-S444 are repeated until a valid target input value is obtained, at which point the extraction is complete. The valid target input value is then filled into the corresponding specific data item in the system. Alternatively, if no valid target input value is obtained after traversing all nearby text boxes, an "Extraction Failed" message is displayed.

[0089] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0090] This method matches the label text block with the adjacent text box without relying on a fixed template. It can adapt to various non-standard documents with horizontal, vertical, mixed arrangement and tilted shooting. It can effectively identify and extract information from messy documents such as highway tolls, parking fees, non-standard receipts, weighbridge slips, waybills, loading and unloading lists, and greatly improve the adaptability to various irregular documents in freight scenarios.

[0091] A multimodal neural network is used to classify invoices from both visual features of images and semantic features of text, which solves the problem of confusion caused by relying solely on text recognition and reduces errors in invoice category recognition. At the same time, target label localization, location matching, and content rule verification are used to accurately extract label values, avoiding information matching errors and data entry confusion, and achieving accurate data entry.

[0092] This method does not rely on fixed-location extraction rules. When adding new invoice types, it only requires simple configuration of tag keywords and verification rules to support them. There is no need to redevelop the extraction template. It can adapt to the dynamic changes in invoice types and formats, reduce system maintenance costs, and fundamentally achieve sustainable automated filling. The recognition accuracy will not decrease due to the increase in the number of invoices. Attached Figure Description

[0093] Figure 1 This is a flowchart illustrating the steps of the driver ticket automatic recognition and filling method based on multimodal neural networks of the present invention. Detailed Implementation

[0094] To provide a further understanding of the purpose, structure, features, and functions of the present invention, detailed descriptions are provided below with reference to specific embodiments.

[0095] like Figure 1 As shown, the automatic driver ticket recognition and filling method based on multimodal neural networks includes the following steps:

[0096] S1: Extract multimodal features;

[0097] Includes the following sub-steps:

[0098] S11: Perform OCR text recognition on the original image of the invoice to generate text information;

[0099] The original image of the invoice is input into a general OCR recognition tool (such as PaddleOCR, EasyOCR, etc.). The general OCR recognition tool scans the text area in the original image of the invoice and recognizes the corresponding text information; each original image of the invoice corresponds to unique text information; the text information is in string form.

[0100] S12: Perform word segmentation on the text information to obtain a word vector matrix;

[0101] Using word segmentation tools such as jieba, the text information in string form corresponding to the original image of each ticket is split into multiple independent and ordered words according to the Chinese word structure (Chinese word formation habits and semantic integrity), forming a word sequence corresponding to the original image of the ticket;

[0102] The tf-idf or word2Vec component is used to convert each word in the word sequence into a set of numerical word vectors. All word vectors are combined in the order of the words to obtain the corresponding word vector matrix.

[0103] S13: Obtain the text feature representation of the original image of each ticket;

[0104] The internal state of the RNN network in the multimodal neural network is initialized beforehand, and the initial internal state values ​​are set. , Within the interval [0, 1]; let the total number of word vectors in each word vector matrix be n, where n is a positive integer;

[0105] The multimodal neural network includes an RNN network, a CNN network, and a feature fusion module;

[0106] For each original ticket image, the word vectors in the word vector matrix are sequentially input into the RNN network. The RNN network outputs its internal state value at the current time step using the following formula:

[0107] ;

[0108] in, Let be the internal state value of the (t-1)th word vector; W is the preset weight matrix; b is the preset offset term; tanh is the hyperbolic tangent activation function; Let t be the t-th word vector in the word vector matrix;

[0109] Concatenate all the internal state values ​​obtained from each word vector matrix in word vector order to obtain the corresponding one-dimensional vector. ,Will This serves as a textual feature representation of the original image of the ticket;

[0110] S14: Input the original image of the ticket into a CNN network to form a feature vector;

[0111] The original image of the ticket is preprocessed, including size unification (scaling original images of tickets of different sizes and resolutions to a fixed size) and color standardization (converting the original image of the ticket to a standard RGB three-channel format, etc.).

[0112] The preprocessed original image of the invoice is input into a CNN network. The CNN network uses pre-defined multi-size convolutional kernels as a sliding window to scan the original image region by region. It extracts visual features such as lines, contours, and color blocks through multi-layer two-dimensional convolution operations. The extracted visual features are then processed by multi-layer convolution and pooling, and finally integrated and encoded to obtain a feature vector representing the visual features of the original image of the invoice, denoted as [feature vector]. ;

[0113] S15: Obtain the multimodal features of the original image of the invoice;

[0114] The text feature representation of the original ticket image output in step S13 is... The feature vector of the visual features of the original image of the ticket obtained in step S14 The input is fed into the feature fusion module, which performs a weighted fusion of the text feature representation weight coefficients and the visual feature vector weight coefficients based on pre-set weights to obtain the multimodal features of the original image of the ticket. ;

[0115] In this embodiment, the weight of the text feature representation of the original ticket image is set to 0.7, and the weight of the feature vector of the visual features of the original ticket image is set to 0.3. Therefore, the multimodal features... .

[0116] S2: Train the bill category model;

[0117] Includes the following sub-steps:

[0118] S21: Multimodal feature classification and combination;

[0119] Collect various known categories of documents in freight scenarios, such as weighbridge slips, highway toll invoices, and waybills; obtain multimodal features of known categories of documents based on the method in step S1; classify and combine the multimodal features to obtain a multimodal feature set for each category of documents, denoted as . ;

[0120] ;

[0121] Where i represents the type of bill and M represents the number of bills in this type. This is the i-th multimodal feature of bill category i;

[0122] For example, the multimodal feature set of weighbridge slips. ;

[0123] Multimodal feature set of highway toll invoices ;

[0124] Multimodal feature set of waybills ;

[0125] S22: Calculate the correlation of multimodal features;

[0126] Element-wise, a combined inner product operation is performed on all multimodal features in the multimodal feature set of each type of ticket, and the result is used as the exponent. Substitute into the formula In this process, the characteristic correlation of this type of bill is obtained. ;

[0127] The feature relevance of each category of collected tickets By summing the results, we obtain the comprehensive multimodal feature relevance of all types of tickets. ;

[0128] ;

[0129] in, The result of the multimodal feature element-level comprehensive inner product operation for bill category i is given, where n is the total number of bill categories.

[0130] For example, = + +...+ ;

[0131] S23: Train the RNN network;

[0132] The training set is composed of all the ticket categories and their corresponding multimodal feature sets obtained in step S21.

[0133] Will As the training objective, the RNN network is iteratively trained using the training set; during training, the weight matrix W and the bias term b are continuously adjusted; when When the RNN network converges to its maximum value within a preset number of training rounds, the training is complete. At this point, the weight matrix W and the offset term b are the optimal parameters of the RNN network.

[0134] S3: Predict the type of newly uploaded invoice;

[0135] For newly uploaded ticket images by drivers, their multimodal features T are extracted using step S1. The multimodal features T are then combined with the M multimodal features of each category of tickets in the training set using element-wise inner product operations. The results are then substituted into... In the process, the predicted Probability of Return (POR) value for each category corresponding to the ticket image is obtained, and the ticket category with the largest predicted POR value is selected as the ticket category to which the ticket image belongs.

[0136] S4: Extract the target information to be filled in from the original image of the invoice;

[0137] Includes the following sub-steps:

[0138] S41: Obtain the corresponding information for text blocks and text boxes;

[0139] For invoice images that have completed invoice category prediction, text blocks and text boxes are extracted using an OCR recognition tool;

[0140] Specifically, text block extraction: The OCR recognition tool performs optical character recognition processing on the ticket image. By scanning the text area of ​​the ticket image, continuous text strings are recognized as a text block. Using blank areas as the segmentation basis, non-continuous text is split into multiple independent text blocks, thus completing the text block extraction.

[0141] Text box extraction: While recognizing text blocks, the OCR recognition tool matches a unique text box for each text block. The text box is a quadrilateral that is approximately rectangular and encloses the text block. The coordinates of the four vertices of the text box indicate the position of the text block in the ticket image.

[0142] For example, if the four vertices are arranged in a clockwise order of top left, top right, bottom right, and bottom left, and denoted as vertices A, B, C, and D respectively, then A = ( , B = () , C = () , D = () , );

[0143] Let TEXT denote the text block and BOX denote the text box, where BOX = {A( , ), B( , ), C( , ), D( , )};

[0144] Each text block and its corresponding text box form a set of TEXT-BOX key-value pairs. All TEXT-BOX key-value pairs of each ticket image that has completed category prediction are integrated to obtain a list of TEXT-BOX key-value pairs containing the ticket image.

[0145] S42: Find the tag text block;

[0146] Based on the business needs of freight industry invoice filling, the specific data items to be extracted and entered into the system for each invoice category are set in advance and denoted as the target values ​​to be extracted; each target value to be extracted corresponds to a unique target label.

[0147] Based on the ticket category to which the ticket image belongs, the system retrieves the set of tags and keywords corresponding to the target value to be extracted; the system contains the set of tags and keywords corresponding to each ticket category; the set of tags and keywords contains all possible expressions of the target tag and configures the matching priority for each expression;

[0148] Traverse all text blocks in the TEXT-BOX key-value pairs obtained in step S41. After symbol filtering for each text block, match the text block with all possible expressions of the target tag in turn based on the matching priority order in the tag keyword set. When the corresponding content of the text block completely matches a certain expression in the tag keyword set, stop traversing and matching, determine the text block as the tag text block corresponding to the current target value to be extracted, and proceed to step S43. If no corresponding expression is matched after traversing all text blocks, it is determined that there is no valid target tag in the ticket image, and the current target value extraction process is terminated.

[0149] S43: Get the coordinates of the label text box;

[0150] Based on the one-to-one correspondence between text blocks and text boxes in the TEXT-BOX key-value pair list, the text box corresponding to the tag text block is retrieved, and the coordinates of the four vertices of the text box are obtained. The text box is denoted as the tag text box S.

[0151] S={A( , ), B( , ), C( , ), D( , )};

[0152] Among them, A, B, C, and D are the top left, top right, bottom right, and bottom left vertices of the label text box S, respectively;

[0153] S44: Find other nearby text boxes and retrieve the target text box;

[0154] Based on the general layout rules of freight industry documents, other nearby text boxes are searched in the order of right side first and bottom side second; the text box corresponding to the feature value to be extracted is recorded as the target text box, and the target text box is obtained.

[0155] Specifically, the general formatting rules for freight industry invoices are as follows:

[0156] The “Label” text box and the “Value” text box have two positional relationships: left-right and top-bottom.

[0157] Left-right relationship: The "Value" text box is to the right of the "Label" text box, specifically, either slightly above or slightly below.

[0158] Top-bottom relationship: The "Value" text box is below the "Label" text box, specifically to the left or right.

[0159] Then, search for nearby text boxes in the order of first the right side and then the bottom side;

[0160] Includes the following sub-steps:

[0161] S441: Search for other text boxes to the right of label text box S, and determine whether the target text box exists;

[0162] Let the other text boxes be labeled T;

[0163] T={E( , ), F( , ), G( , ), H( , )};

[0164] Among them, E, F, G, and H are the top left, top right, bottom right, and bottom left vertices of the other text box T, respectively;

[0165] Match the label text box S against the other text boxes on its right in turn, and determine whether the following two conditions are met simultaneously:

[0166] (1) > and > ;

[0167] (2) > and < ;

[0168] The intersection point of the extension of line segment FE of other text boxes T and the extension of line segment DB of label text box S is M1. , The intersection point M2 is the extension of line segment GH of other text box T and the extension of line segment BD of label text box S. , The midpoint of line segment BD is M. , );

[0169] When both conditions are met, other corresponding text boxes are identified as target text boxes, and step S443 is initiated.

[0170] If the above two conditions cannot be met at the same time, then there is no target text box in the other text boxes to the right of the label text box S, and proceed to step S442;

[0171] S442: Search for other text boxes below the label text box S and determine if the target text box exists;

[0172] Match the label text box S sequentially with the other text boxes below it, and determine whether the following two conditions are met simultaneously:

[0173] (1) > and > ;

[0174] (2) < and > ;

[0175] The intersection point of the extension of line segment HE of other text box T and the extension of line segment CD of label text box S is... The intersection point of the extension of line segment GF of other text box T and the extension of line segment DC of label text box S is... The midpoint of line segment DC is ;

[0176] When both conditions are met, other corresponding text boxes are identified as target text boxes, and step S443 is initiated.

[0177] If both of the above conditions cannot be met simultaneously, then the target text box will not exist in any of the other text boxes below the label text box S; the process will end.

[0178] S443: Get the text block corresponding to the target text box;

[0179] Based on the TEXT-BOX key-value pair list of the ticket image, obtain the text block corresponding to each of the identified target text boxes, and retrieve the content of the text block.

[0180] S444: Verify the content of the text block;

[0181] Based on the business requirements for invoice filling in the freight industry, corresponding content verification rules are formulated for each target value to be extracted in advance; the content verification rules include the usage requirements of numbers, letters and symbols.

[0182] For example: The gross weight / tare weight / net weight on the weighing slip must be a number (including decimal points), and units (tons, kilograms, t, kg, etc.) can be attached.

[0183] The amount on the highway toll invoice must be a number (including two decimal places), and may include currency units such as "yuan".

[0184] Document numbers for various types of documents: can be a combination of numbers and letters, and some may contain separators such as horizontal lines / underscores;

[0185] The text block content extracted in step S443 is matched and verified with the content verification rules of the corresponding target value to be extracted.

[0186] If the content of the text block conforms to the content validation rules, then the content of the text block is determined to be a valid target input value, and the extraction is complete;

[0187] If the content of a text block does not conform to the content validation rules, the content of the text block is determined to be an invalid target input value. Steps S441-S444 are repeated until a valid target input value is obtained, at which point the extraction is complete. The valid target input value is then filled into the corresponding specific data item in the system. Alternatively, if no valid target input value is obtained after traversing all nearby text boxes, an "Extraction Failed" message is displayed.

[0188] This invention proposes an automatic driver invoice recognition and filling method based on a multimodal neural network. First, it constructs a neural network using both text and image modalities to accurately identify invoice categories, solving the problem of inaccurate category differentiation based solely on text keywords, as freight drivers' invoices often contain many repetitive keywords. Simultaneously, a text block search algorithm is designed to address the issue of inaccurate value retrieval due to different invoice styles for the same type. This method has no special requirements for target information and does not require separate templates for each type of document; simple configuration achieves excellent extraction results.

[0189] By taking the above two steps, the problem of drivers having to fill out various forms of documents has been effectively solved, as well as the problem that ordinary OCR cannot automatically and accurately fill out forms. This has greatly improved the drivers' work efficiency and saved the number of secondary reviews and labor costs.

[0190] Drivers no longer need to manually select invoice types or enter numbers; simply uploading a photo completes the form automatically. This reduces the difficulty of operating on small mobile screens, minimizes errors, and significantly improves ease of use for drivers. High recognition and filling accuracy greatly reduces the workload of manual review and correction of erroneous data in the back office, lowering enterprise operating and management costs. Combining visual and textual features, it is more adaptable and stable to real-world scenarios such as blurry images, insufficient lighting, tilted angles, and cluttered backgrounds. It enhances the overall intelligence level of the logistics management system and accelerates business processes such as expense reimbursement and document review.

[0191] The present invention has been described by the above-described embodiments; however, these embodiments are merely examples for implementing the present invention. It must be noted that the disclosed embodiments do not limit the scope of the present invention. Conversely, any modifications and refinements made without departing from the spirit and scope of the present invention are within the scope of patent protection of the present invention.

Claims

1. A method for automatic recognition and filling of driver's tickets based on multimodal neural networks, characterized in that: Includes the following steps: S1: Extract multimodal features; S11: Perform OCR text recognition on the original image of the invoice to generate text information; The original image of the invoice is input into a general OCR recognition tool, which scans the text area in the original image and recognizes the corresponding text information; each original image of the invoice corresponds to unique text information; the text information is in string format; S12: Perform word segmentation on the text information to obtain a word vector matrix; The text information in string form corresponding to the original image of each ticket is split into multiple independent and ordered words by a word segmentation tool, forming a word sequence corresponding to the original image of the ticket; Each word in the word sequence is converted into a set of numerical word vectors. All word vectors are combined in the order of the words to obtain the corresponding word vector matrix. S13: Obtain the text feature representation of the original image of each ticket; The internal state of the RNN network in the multimodal neural network is initialized beforehand, and the initial internal state values ​​are set. , Within the interval [0, 1]; let the total number of word vectors in each word vector matrix be n, where n is a positive integer; The multimodal neural network includes an RNN network, a CNN network, and a feature fusion module; For each original ticket image, the word vectors in the word vector matrix are sequentially input into the RNN network. The RNN network outputs its internal state value at the current time step using the following formula: ； in, Let be the internal state value of the (t-1)th word vector; W is the preset weight matrix; b is the preset offset term; tanh is the hyperbolic tangent activation function; Let t be the t-th word vector in the word vector matrix; Concatenate all the internal state values ​​obtained from each word vector matrix in word vector order to obtain the corresponding one-dimensional vector. ,Will This serves as a textual feature representation of the original image of the ticket; S14: Input the original image of the ticket into a CNN network to form a feature vector; The original image of the ticket is preprocessed, including size standardization and color standardization. The preprocessed original image of the ticket is input into a CNN network. The CNN network performs 2D convolutional feature extraction to obtain a feature vector representing the visual features of the original ticket image, denoted as . ; S15: Obtain the multimodal features of the original image of the invoice; The text feature representation of the original ticket image output in step S13 is... The feature vector of the visual features of the original image of the ticket obtained in step S14 The input is fed into the feature fusion module, which performs a weighted fusion of the text feature representation weight coefficients and the visual feature vector weight coefficients based on pre-set weights to obtain the multimodal features of the original image of the ticket. ; S2: Train the bill category model; S21: Multimodal feature classification and combination; S22: Calculate the correlation of multimodal features; S23: Train the RNN network; S3: Predict the type of newly uploaded invoice; S4: Extract the target information to be filled in from the original image of the invoice; S41: Obtain the corresponding information for text blocks and text boxes; S42: Find the tag text block; S43: Get the coordinates of the label text box; S44: Find other nearby text boxes and retrieve the target text box; Based on the target text box, obtain the corresponding text block, retrieve the content of the text block, and fill the text block content into the corresponding specific data item to be filled.

2. The method for automatic driver ticket recognition and filling based on multimodal neural networks as described in claim 1, characterized in that: The specific details of step S2 are as follows: S21: Multimodal feature classification and combination; Collect various known categories of tickets in the freight scenario, and obtain multimodal features of the known categories of tickets based on the method in step S1; classify and combine the multimodal features to obtain a multimodal feature set for each category of tickets, denoted as . ; ； Where i represents the type of bill and M represents the number of bills in this type. This is the i-th multimodal feature of bill category i; S22: Calculate the correlation of multimodal features; Element-wise, a combined inner product operation is performed on all multimodal features in the multimodal feature set of each type of ticket, and the result is used as the exponent. Substitute into the formula =1 / (1+ In this context, the characteristic relevance of this type of bill is obtained. ; The feature relevance of each category of collected tickets By summing the results, we obtain the comprehensive multimodal feature relevance of all types of tickets. ; ； in, The result of the multimodal feature element-level comprehensive inner product operation for bill category i is given, where n is the total number of bill categories. S23: Train the RNN network; The training set is composed of all the ticket categories and their corresponding multimodal feature sets obtained in step S21. Will As the training objective, the RNN network is iteratively trained using the training set; during training, the weight matrix W and the bias term b are continuously adjusted; when When the RNN network converges to its maximum value within a preset number of training rounds, the training is complete. At this point, the weight matrix W and the offset term b are the optimal parameters of the RNN network.

3. The method for automatic driver ticket recognition and filling based on multimodal neural networks as described in claim 1, characterized in that: The specific details of step S3 are as follows: For newly uploaded ticket images by drivers, their multimodal features T are extracted using step S1. The multimodal features T are then combined with the M multimodal features of each category of tickets in the training set using element-wise inner product operations. The results are then substituted into... =1 / (1+ In the process, the predicted Probability of Return (POR) value for each category corresponding to the ticket image is obtained, and the ticket category with the largest predicted POR value is selected as the ticket category to which the ticket image belongs.

4. The method for automatic driver ticket recognition and filling based on multimodal neural networks as described in claim 1, characterized in that: The specific details of step S4 are as follows: S41: Obtain the corresponding information for text blocks and text boxes; For invoice images that have completed invoice category prediction, text blocks and text boxes are extracted using an OCR recognition tool; each text block is matched with a unique text box. Each text block and its corresponding text box form a set of TEXT-BOX key-value pairs. All TEXT-BOX key-value pairs of each ticket image that has completed category prediction are integrated to obtain a list of TEXT-BOX key-value pairs containing the ticket image. S42: Find the tag text block; Based on the business needs of freight industry invoice filling, the specific data items to be extracted and entered into the system for each invoice category are set in advance and denoted as the target values ​​to be extracted; each target value to be extracted corresponds to a unique target label. Based on the ticket category to which the ticket image belongs, the system retrieves the set of tags and keywords corresponding to the target value to be extracted; the system contains the set of tags and keywords corresponding to each ticket category; the set of tags and keywords contains all possible expressions of the target tag and configures the matching priority for each expression; Traverse all text blocks in the TEXT-BOX key-value pairs obtained in step S41. After symbol filtering for each text block, match the text block with all possible expressions of the target tag in turn based on the matching priority order in the tag keyword set. When the corresponding content of the text block completely matches a certain expression in the tag keyword set, stop traversing and matching, determine the text block as the tag text block corresponding to the current target value to be extracted, and proceed to step S43. If no matching expression is found after traversing all text blocks, it is determined that there is no valid target label in the ticket image, and the extraction process of the current target value is terminated. S43: Get the coordinates of the label text box; Based on the one-to-one correspondence between text blocks and text boxes in the TEXT-BOX key-value pair list, the text box corresponding to the tag text block is retrieved, and the coordinates of the four vertices of the text box are obtained. The text box is denoted as the tag text box S. S={A( , ),B( , ),C( , ),D( , )}; Among them, A, B, C, and D are the top left, top right, bottom right, and bottom left vertices of the label text box S, respectively; S44: Find other nearby text boxes and retrieve the target text box; Search for other nearby text boxes in the order of right side first, then bottom side; The text box corresponding to the feature value to be extracted is designated as the target text box, and the target text box is obtained.

5. The method for automatic driver ticket recognition and filling based on multimodal neural networks as described in claim 4, characterized in that: The specific details of step S44 are as follows: S441: Search for other text boxes to the right of label text box S, and determine whether the target text box exists; Let the other text boxes be labeled T; T={E( , ),F( , ),G( , ),H( , )}; Among them, E, F, G, and H are the top left, top right, bottom right, and bottom left vertices of the other text box T, respectively; Match the label text box S against the other text boxes on its right in turn, and determine whether the following two conditions are met simultaneously: (1) > and > ; (2) > and < ; The intersection point of the extension of line segment FE of other text boxes T and the extension of line segment DB of label text box S is M1. , The intersection point M2 is the extension of line segment GH of other text box T and the extension of line segment BD of label text box S. , The midpoint of line segment BD is M. , ); When both conditions are met, other corresponding text boxes are identified as target text boxes, and step S443 is initiated. If the above two conditions cannot be met at the same time, then there is no target text box in the other text boxes to the right of the label text box S, and proceed to step S442; S442: Search for other text boxes below the label text box S and determine if the target text box exists; Match the label text box S sequentially with the other text boxes below it, and determine whether the following two conditions are met simultaneously: (1) > and > ; (2) < and > ; The intersection point of the extension of line segment HE of other text box T and the extension of line segment CD of label text box S is... The intersection point of the extension of line segment GF of other text box T and the extension of line segment DC of label text box S is... The midpoint of line segment DC is ; When both conditions are met, other corresponding text boxes are identified as target text boxes, and step S443 is initiated. If both of the above conditions cannot be met simultaneously, then the target text box will not exist in any of the other text boxes below the label text box S; the process will end. S443: Get the text block corresponding to the target text box; Based on the TEXT-BOX key-value pair list of the ticket image, obtain the text block corresponding to each of the identified target text boxes, and retrieve the content of the text block. S444: Verify the content of the text block; Based on the business requirements for invoice filling in the freight industry, corresponding content verification rules are formulated for each target value to be extracted in advance; the content verification rules include the usage requirements of numbers, letters and symbols. The text block content extracted in step S443 is matched and verified with the content verification rules of the corresponding target value to be extracted. If the content of the text block conforms to the content validation rules, then the content of the text block is determined to be a valid target input value, and the extraction is complete; If the content of a text block does not conform to the content validation rules, the content of the text block is determined to be an invalid target input value. Steps S441-S444 are repeated until a valid target input value is obtained, at which point the extraction is complete. The valid target input value is then filled into the corresponding specific data item in the system. Alternatively, if no valid target input value is obtained after traversing all nearby text boxes, an "Extraction Failed" message is displayed.

Images