A lightweight official document intelligent circulation method based on artificial intelligence technology

Abstract

The application discloses a kind of lightweight official document intelligent flow based on artificial intelligence technology, belong to office automation and artificial intelligence field, the official document file is first classified heterogeneous pretreatment, unified extraction text data are combined with optical character recognition and format conversion;Then category data is vectorized, and text data is subjected to latent Dirichlet allocation model and is subjected to theme vectorization, and fusion feature vector is obtained by splicing;Next, a prediction model is constructed to output the next processing node and responsibility, and the receiver is obtained through the dynamically maintained responsibility-personnel mapping table;Finally, the entire process is packaged as a containerized web application programming interface;The application realizes the unified processing of multi-source official document data, dynamic adaptation of organizational structure, lightweight deployment and plug-in integration, significantly improves the accuracy and automation level of official document distribution.

Description

Technical Field

[0001] This invention belongs to the technical field of intelligent workflow based on machine learning, specifically relating to a lightweight intelligent document workflow method based on artificial intelligence technology. Background Technology

[0002] In China, a company, especially one with standardized management, often needs to handle numerous official documents from external sources, including announcements, notices, requests, approvals, opinions, and letters. These documents originate from various sources such as document exchange platforms, email, and internal communication systems. Due to the widespread adoption of information technology, many companies have built their own Office Automation (OA) systems to standardize the processing of received documents. A typical process involves: after a document is registered or signed for, it enters the OA system, is passed to a processing node, and the company office selects the next processing node and the appropriate handler based on the document's content. To ensure accurate document processing, the corresponding office staff need a clear understanding of the company's document classification system, the responsibilities and responsible persons of each department, and then make appropriate selections based on the document's content. However, for large, rapidly growing companies with diverse operations, the number of documents received annually is substantial and continuously increasing. For example, a design institute saw its document volume increase from 2,842 in 2020 to 3,388 in 2024, resulting in immense pressure on the office's document processing capabilities.

[0003] To alleviate this problem, some researchers [CN 110489667 A, CN 110490547 A] have proposed using intelligent document processing technology based on user profiles. This technology calculates the similarity between processed and pending documents, adjusts the similarity weights using user profiles, and recommends the best personnel to handle the target document based on collaborative filtering. Other researchers [CN118333559 A] have aligned and encoded business input data and organizational structure data, and then used a Transformer model to predict the responsible personnel. However, current methods generally suffer from the following problems: ① Insufficient processing capabilities for multi-source heterogeneous official document data: Existing technologies often directly extract text data from editable files, business systems, and databases. However, analysis of historical incoming document data reveals diverse formats for the main text and attachments. In addition to common, editable formats such as DOCX, XLSX, PPTX, PDF, and TXT, there are also outdated formats like DOC, XSL, and PPT, as well as uneditable images and scanned PDFs. Existing solutions cannot address the challenges of parsing multi-source heterogeneous official document files.

[0004] ② Inability to handle personnel changes: Changes in personnel responsibilities, departmental transfers, onboarding, and offboarding are relatively common within an organization. Existing technologies that directly predict the person handling the documents will fail in such situations. Furthermore, for employees with short tenures, existing methods cannot effectively predict their actions due to a lack of historical document processing data.

[0005] ③ The established document processing flow is not reasonable enough: The existing method implicitly assumes that there is only one next node in the document flow and that it is only necessary to predict the recipient. However, in the actual document receiving process, there are multiple possibilities for the next node. It is necessary to select the appropriate node and select the appropriate recipient according to the nature of the node.

[0006] ④ Insufficient information utilization during vectorization: [CN 110489667 A, CN 110490547 A] integrates all information into text and uses natural language processing technologies such as Doc2Vec to vectorize official document-related data, capturing the semantic information of the documents, but it does not fully utilize the distinguishing ability inherent in existing structured data (such as incoming document types); [CN 118333559 A] organizes business input data into structured data and then encodes it into vectors, losing the details and semantics of the accompanying documents in the process.

[0007] Furthermore, methods that use complex neural networks such as Transformers often require dedicated computing hardware (such as graphics processing units, GPUs) and computing environments, resulting in high deployment costs and difficulties, thus hindering their implementation and promotion. Summary of the Invention

[0008] To address the aforementioned problems, this invention proposes a lightweight intelligent document processing method based on artificial intelligence technology. This invention enables unified processing of multi-source heterogeneous document data, vectorization of both structured and unstructured data, prediction of the next node and recipient unaffected by personnel changes, lightweight system deployment, and plug-in integration.

[0009] The technical solution for implementing the present invention is as follows: A lightweight intelligent document processing method based on artificial intelligence technology includes the following steps: S1. Obtaining input data in the document processing flow, the input data including form business data and document files; S2. Preprocessing the document files with multi-source heterogeneous data, extracting text content, and concatenating it with non-category data in the form business data to obtain text data, while simultaneously extracting category data from the form business data; S3. Vectorizing the category data to obtain category vectors, vectorizing the text data to obtain text vectors, and concatenating the category vectors and text vectors to obtain feature vectors; S4. Constructing a prediction model based on historical document processing data, the input of the prediction model being the feature vectors, and the output being the next processing node and responsibility; S5. Establishing and dynamically maintaining a responsibility-personnel mapping table, mapping the predicted responsibilities to specific processing personnel; S6. Encapsulating the processing flow of the above steps into a web application programming interface for use by office automation systems to achieve intelligent document processing.

[0010] Furthermore, the output of the prediction model is expressed as: [N, D] = f pred,2 (C, M) Where N is the next processing node, D is the responsibility, C is the content transmitted by the current node, M is auxiliary information, and f pred,2 This is a prediction algorithm.

[0011] Furthermore, the process of mapping the predicted responsibilities to the personnel involved is represented as follows: P = f map (D) Where P represents the processing personnel, f map This is a dynamically maintained responsibility-person mapping table, where D represents responsibility.

[0012] Furthermore, the feature vector is obtained by concatenating the features using the following formula: V = [Vc, Vt] = [f e,c (Cc), f e,t (Ct)] Where V is the feature vector, Vc is the category vector, Vt is the text vector, Cc is the category data, Ct is the concatenated text data, and f e,c f is a vectorization function for categorical data. e,t This is a function for vectorizing text data.

[0013] Furthermore, the preprocessing of multi-source heterogeneous data for official documents includes: for editable documents, directly extracting the text; for outdated format documents, converting them to editable format using open-source tools before extracting the text; for images or scanned PDFs, using lightweight optical character recognition tools for text recognition; and for compressed files, extracting only the filename.

[0014] Furthermore, the text data is vectorized using the latent Dirichlet allocation model, which vectorizes the text data of a document process into a distribution across K topics; where the number of topics K is automatically determined using the perplexity elbow method.

[0015] Furthermore, one-hot encoding, label encoding, or sequence encoding are used to vectorize categorical data; while latent Dirichlet allocation model or Doc2Vec model are used to vectorize text data.

[0016] Furthermore, the prediction model is implemented using a classifier, which is selected from one or more of the following: K-nearest neighbor classifier, multilayer perceptron, support vector machine, Naive Bayes, decision tree, random forest, gradient boosting classifier, lightweight gradient booster; and the optimal hyperparameters are selected through k-fold cross-validation and grid search.

[0017] Furthermore, when encapsulating the processing flow into a network application programming interface, the FastAPI framework is used, and Docker container technology is employed to package the service into an image, thereby achieving containerized deployment.

[0018] Furthermore, the method does not depend on a graphics processor and runs only in a central processing unit environment.

[0019] Beneficial effects: 1. This invention adopts a hierarchical modeling architecture of "predicting responsibilities + mapping personnel," decoupling the business logic of document circulation from specific personnel entities. When employees join, leave, or change positions, only the responsibility-personnel mapping table needs to be dynamically maintained, without retraining the prediction model. This effectively solves the cold start and model failure problems of traditional end-to-end prediction methods in personnel mobility scenarios.

[0020] 2. In view of the diverse formats of official documents, this invention adopts different processing strategies, such as direct parsing of editable documents, conversion of outdated formats through open source tools, and lightweight OCR recognition of images and scanned PDFs. It can uniformly extract text content from various heterogeneous files and overcome the technical obstacles of existing methods that cannot directly process scanned documents, images and outdated format files.

[0021] 3. This invention vectorizes explicit category data (such as incoming document type and incoming document type) and concatenates it with semantic vectors of official document text extracted through models such as LDA to form a fused feature vector. This method takes into account both the distinguishing ability of structured data and the semantic information of unstructured text, making up for the shortcomings of losing category features in single text vectorization or losing semantic details in single category vectors, and significantly improving the accuracy of the model in judging the intention of official document circulation.

[0022] 4. This invention uses the perplexity elbow method to automatically determine the number of LDA topics K, and combines a domain-specific terminology dictionary and an optimized stop word list for Chinese word segmentation. This enables the model to adapt to the characteristics of long official documents, avoiding the problem of key information being diluted when processing long texts in traditional word vector models, and ensuring the fidelity of text vectorization to the core content of official documents.

[0023] 5. This invention encapsulates data preprocessing, feature extraction, model prediction, and post-processing into independent HTTP API services, and packages them into an image using Docker container technology. This service does not rely on a graphics processing unit (GPU) and can run on ordinary CPU servers. It can be quickly decoupled and integrated with existing office automation systems, solving the problems of difficult deployment and high migration costs of AI models in legacy office systems.

[0024] 6. Using 2,945 valid documents received by a design institute company in 2024 as a test sample, the method of this invention achieved an accuracy of 74.5% in predicting the next node and an F1 score of 73.3% on the test set, and an accuracy of 61.3% in predicting the receiving department and an F1 score of 60.0%. It can effectively assist office staff in completing document approval work, significantly reducing the burden of manual processing and adapting to the needs of the continuously increasing number of documents received by enterprises. Attached Figure Description

[0025] Figure 1 This is a flowchart of the method of the present invention.

[0026] Figure 2 This is a schematic diagram of the multi-source heterogeneous document data processing flow.

[0027] Figure 3 This is a schematic diagram of the feature extraction process.

[0028] Figure 4 A schematic diagram illustrating the selection and training process of the classification model.

[0029] Figure 5 This is a schematic diagram of the prediction process. Detailed Implementation

[0030] This invention provides a lightweight intelligent document processing method based on artificial intelligence technology. Based on optical character recognition (OCR), natural language processing, machine learning, and other AI technologies, this invention first uses OCR to preprocess document data after remodeling the problem. Then, it uses natural language processing to vectorize the data. Next, it combines historical processing results to train a machine learning model to predict the next processing node and the next responsibility, and obtains the recipient through a responsibility-person mapping table. Finally, the entire processing flow is transformed into streaming processing, providing services to the OA system in the form of a web application programming interface (HTTP API), and encapsulated as a mirror image.

[0031] like Figure 1 As shown, the specific process of the method of the present invention is as follows: (1) Problem description and modeling For many large and medium-sized enterprises, document processing is handled by a workflow within the internal OA system. This workflow contains multiple nodes, but each workflow has only one start and one end node. A node can have multiple parent nodes but only one child node. Documents are passed between different nodes and processed by relevant personnel. The workflow ends when the end node is reached. For the personnel receiving a document at a particular node, their job is to receive information from the previous node, primarily including form data, previous node's processing comments, the document body, and attachments. Based on the node's functional positioning, the document itself, the previous node's processing comments, the responsibilities of various departments within the company, and department heads, they make a comprehensive judgment and then pass the document to the next node, assigning a handler. A node can have multiple handlers (corresponding to overlapping responsibilities in reality), and these individuals take turns processing the document; once one person processes it and passes it to the next node, others no longer need to or can process it.

[0032] Existing methods directly predict the processor P based on the content C passed from the previous node, as well as other information M such as user profile. P = f pred,1 (C, M) Among them, f pred,1 The prediction algorithm can be either a classic machine learning algorithm such as KNN (k-Nearest Neighbors) or SVM (Support Vector Machine), or a deep learning algorithm customized for the problem. Clearly, this modeling approach cannot handle changes in personnel responsibilities and ignores the possibility of multiple child nodes N. In reality, the assignment of document processors is based on their responsibilities; therefore, responsibilities D can be predicted and then mapped to personnel P, where the mapping table f between responsibilities and personnel... mapDynamic maintenance is performed based on personnel changes. The modeling method at this time is: [N, D] = f pred,2 (C, M) P = f map (D) Where, f pred,2 It is similar to f pred,1 The prediction algorithm outputs the next node N and responsibility D, instead of personnel P. By decoupling responsibility and personnel, this method can effectively solve problems such as "cold start" caused by new employee onboarding and historical data invalidation caused by employee transfers or departures.

[0033] (2) Data preprocessing like Figure 2 As shown, the data used in this invention mainly consists of two parts: form business data and official documents. The former can be directly extracted from the database, while the latter requires file parsing.

[0034] 1) Form business data Usable form business data includes titles, dates, document types, document issuing units, and previous approval comments. This data often exists in multiple relational database tables and is retrieved through related queries using document numbers. Form business data can be further categorized into two types: ① Categorical data: such as document type, which can only take one of a few preset values. Since categorical data itself has strong distinguishing capabilities, it can be used as part of the feature vector after encoding.

[0035] ② Non-categorical data: Fields in the form with no restrictions on values, such as titles. These are concatenated with text extracted from official documents and then vectorized using natural language processing techniques.

[0036] 2) Official documents Official documents include the main text and attachments, and multiple documents may be processed in a single document processing flow. Different processing strategies are adopted for different document types.

[0037] ① Regular DOCX, XLSX, PPTX, editable PDF, TXT: Extract the text content from the files using existing software packages (such as python-docx and pandas in the Python language).

[0038] ② Outdated DOC, XLS, and PPT files: These formats are obsolete and cannot be directly parsed by existing software packages. They need to be converted to DOCX, XLSX, PPTX, or other formats using tools such as Office or WPS. To facilitate application deployment, open-source tools (such as LibreOffice and OpenOffice) can be used to convert the formats of these documents.

[0039] ③ Images and scanned PDF files: These are commonly found in official documents stamped with the official seal of external organizations. They are text recognition and processing based on lightweight open-source OCR tools such as RapidOCR, which can perform OCR model inference with only CPU.

[0040] ④ Other files: Files of types such as .zip, .rar, .7z, .ceb, .docm, etc. are difficult to process, occur infrequently, and are mostly found in attachments. Therefore, only the file names are extracted, and the file content is not extracted.

[0041] All documents in the same document process are processed by extracting their text using the method described above. This text is then concatenated with non-categorical data from the form business data and sent to the feature extraction module for vectorization.

[0042] It is worth noting that attachments are generally supplementary documents such as registration forms, filling templates, and instructions, and do not play a significant role in the processing of official documents. They can be used as needed.

[0043] In actual official document processing, the first three types of documents account for the vast majority, so the proposed document processing methods have strong universality.

[0044] (3) Feature extraction After data preprocessing, the resulting data consists of two parts: category data Cc and concatenated text data Ct. These are vectorized separately to obtain category vector Vc and text vector Vt, and then concatenated to obtain the feature vector V. This approach leverages the discriminative power of the category data while also capturing semantic information within the file.

[0045] V = [Vc, Vt] = [f e,c (Cc), f e,t (Ct)] Among them, f e,c Common methods for vectorizing categorical data include one-hot encoding and label encoding; f e,t Common methods for converting text vector data include Doc2Vec and LDA (Latent Dirichlet Allocation).

[0046] 1) Vectorization of categorical data like Figure 3 As shown, for categorical data, one-hot encoding, label encoding, or ordinal encoding can be used to vectorize and obtain the category vector Vc. Furthermore, for classifiers such as Lightweight Gradient Booster (LightGBM) that can directly process categorical data, the original data can be used directly.

[0047] 2) Text data vectorization The preprocessed text data is extensive, necessitating the selection of a text vectorization model capable of handling long texts. Considering resource constraints, availability, and generalizability, a lightweight model is chosen. Possible models include Doc2Vec and LDA (Latent Dirichlet Allocation). While both Doc2Vec and LDA are unsupervised methods, Doc2Vec's sliding window approach to long texts may dilute key information, making its performance less stable than LDA for long texts. Given the often lengthy nature of official documents, LDA is the preferred choice.

[0048] The LDA model posits that a document is composed of a mixture of multiple topics, and each topic is composed of a mixture of words. A trained LDA model can vectorize a document into its distribution across K topics. A single official document, after preprocessing, is considered a document.

[0049] The LDA model training process is as follows: ① Chinese word segmentation When using open-source word segmentation tools such as jieba, SnowNLP, and HanLP for Chinese word segmentation, it is necessary to optimize the built-in dictionary and stop word list to increase the accuracy of word segmentation.

[0050] Optimize the dictionary: Add internal proper nouns to avoid splitting a noun into two parts. Common internal proper nouns include names of counterparties, department names, department abbreviations, and document number prefixes. First, query all counterparties in official documents from the historical business form database and add them to a custom dictionary. Second, query all department names and their abbreviations within the company from the internal OA system database and add them to a custom stop word list. Third, query all received document numbers from the historical business form database, extract the initial abbreviation of the document number using regular expressions, and add it to a custom dictionary.

[0051] Optimize the stop word list: Add common auxiliary words, modal particles, Chinese and English punctuation marks, special symbols, etc. to the stop word list to prevent meaningless word segmentation results.

[0052] ②Word frequency statistics Based on the word segmentation results, the frequency of each word in each document is counted to obtain the word frequency vector. The word frequency vectors of all documents are summed to obtain the document-word frequency matrix.

[0053] ③LDA model training By setting hyperparameters such as the number of topics K, maximum number of iterations, fitting method, batch size, evaluation frequency, and random number seed, the LDA model parameters are fitted based on the document-term frequency matrix. The trained model can calculate the distribution of a document across K topics, i.e., a K-dimensional vector, which serves as the document's feature vector Vt.

[0054] To ensure that Vt effectively preserves the document's information while avoiding overfitting, a suitable K needs to be chosen: gradually increase K from 2 to a specified maximum value, train an LDA model for each K value, and calculate the corresponding perplexity; plot K on the horizontal axis and perplexity on the vertical axis on a single graph; select an appropriate K value using the "elbow method." Perplexity represents the model's ability to reconstruct the input corpus, and a smaller value is better.

[0055] 3) Feature vector concatenation Following the aforementioned formula, the category vector and text vector are concatenated to obtain the feature vector of one document flow. The feature vectors of all historical document flows constitute the feature dataset, which, together with the corresponding annotations (i.e., the next node and responsibility), forms the dataset required for training the prediction model. Among them, the responsibility needs to be decomposed and determined according to the actual situation of the flow, commonly by dividing responsibilities by department or by position.

[0056] (4) Search and training of the best prediction model Treating next node and responsibility prediction as a classification problem, with labels "next node" and "responsibility," the steps include: vectorization, dataset partitioning, classifier selection, and classifier fitting. The input is the feature vector obtained from the feature extraction step, and the output is the next node and responsibility.

[0057] First, the dataset is divided into training and test sets according to a certain ratio.

[0058] Next, select a classifier and set the value ranges for the corresponding important hyperparameters. Optional classifiers and hyperparameters include, but are not limited to:

[0059] Next, we search for the optimal hyperparameters. For example... Figure 4 As shown, on the training set, for each classifier, k-fold cross-validation and grid search are used, combined with evaluation metrics (such as the accuracy of predicting the next node), to search for the optimal combination of hyperparameters. The classifier with the best evaluation metric is selected as the optimal classifier.

[0060] Finally, train the classifiers. Using all the training data, train the optimal classifier with the best hyperparameter combination to obtain the final classification model. Note that two classifiers need to be trained, both with the same inputs and hyperparameters; one outputs the next node, and the other outputs the responsibility.

[0061] (5) Model reasoning and deployment 1) Service-oriented model inference First, a post-processing procedure for the prediction results is added. Based on the division of labor among departments and personnel within the company, a responsibility-person mapping table is established and dynamically maintained to map the predicted responsibilities to the recipients.

[0062] Then, as Figure 5 As shown, based on the trained LDA model weights and classification model weights, the data preprocessing, feature extraction, classification model prediction, and post-processing processes are connected into a streaming processing method and encapsulated as a web application programming interface (HTTP API) for the OA system to call. A simple interface call log is added to facilitate problem tracking and usage statistics. The OA system receives the prediction results returned in the same format via HTTP requests, carrying the document's title, category, body, and other input information in JSON, XML, or other formats. After parsing, the prediction results are displayed on the corresponding document workflow form.

[0063] 2) Containerized deployment of model inference service To ensure rapid deployment, migration, startup, shutdown, and fault recovery of the service, Docker container technology is used to encapsulate the model inference service, packaging dependencies, models, etc. into an image and placing it on a suitable server to start as a container.

[0064] Example This invention takes the document receipt batching of a design institute company as an example and implements it using the Python language and related software packages.

[0065] (1) Problem and Data Description After receiving and approving official documents, a design institute has five next steps to choose from: departmental comments, review and approval, final approval, instructions, and document summary. Preliminary data analysis shows that the recipients are mostly the same department heads. Considering personnel changes and concurrent positions, it is confirmed that all recipients are the same department heads. Therefore, responsibilities are divided according to departments, i.e., predicting the receiving departments of official documents and mapping them to department heads. There are 15 departments in total, including the Organization and Personnel Department and the Digital Development Department.

[0066] A dataset was constructed using all official documents received in 2024, totaling 3388 documents. Duplicate documents and processes were excluded based on document number and title. Processes that did not undergo document batching were also excluded, leaving 2945 valid documents that can be used for training and evaluation.

[0067] (2) Data preprocessing 1) Form business data The form service data used in this embodiment includes the title, incoming document reference number, incoming document entity, incoming document type, received document type, received document remarks, official document body file name, and official document attachment file name, which are obtained through associated query from multiple relational databases by the official document number. In actual implementation, appropriate service form data can be selected according to the content of the enterprise's internal service forms, and is not limited to the content listed in this embodiment.

[0068] ① Category data: including incoming document type and received document type.

[0069] ② Non-category data: including title, incoming document reference number, incoming document entity, received document remarks, official document body file name, and official document attachment file name. A regular expression is used to remove meaningless prefixes of the file, for example, "83770FD331E5EE9A48258A98000F1281 Meeting Notice Module 3.doc" is changed to "Meeting Notice Module 3.doc".

[0070] 2) Official document file The official document files include the official document body and attachments, and there may be multiple files in one official document processing flow. Different processing strategies are adopted for different file types.

[0071] ① Conventional DOCX, XLSX, PPTX, editable PDF, TXT: the text content in the file is extracted by using existing software packages, specifically, docx2txt is used for DOCX, pandas is used for XLSX, rapidocr-pdf is used for editable PDF, python-pptx is used for PPTX, and a custom function is used for TXT.

[0072] ② Outdated DOC, XLS, PPT files: the open-source tool LibreOffice is used to convert them into DOCX, XLSX, PPTX, and then the method described in ① is used for reading.

[0073] ③ Image, scanned PDF files: text recognition and processing are performed based on the lightweight open-source OCR tool RapidOCR.

[0074] ④ Other files: such as compressed files like ZIP and RAR, only the file name is extracted, and the file content is not extracted.

[0075] The text extracted from all files in the same official document process by the above method is spliced with the non-category data in the form service data, and sent to the feature extraction module for vectorization.

[0076] In implementation, software packages or tools that satisfy the same function can be selected, and the software packages used in the embodiment are not limited.

[0077] (3) Feature extraction 1) Category data vectorization For incoming document types and received document types, one-hot encoding and label encoding were tested in the implementation. Subsequent classification results show that one-hot encoding achieves better performance, because the maximum value of the vector obtained by one-hot encoding is 1 (while the maximum value of label encoding is the number of labels minus one), which has a smaller difference from the value range (0~1) of the vector obtained by LDA, so that the text vector will not be ignored when calculating the distance between vectors. There are 6 types of incoming documents, such as administrative and Party committee documents, and 11 types of received documents, such as request for instruction, circular, and meeting minutes. Therefore, the vector dimensions after one-hot encoding are 6 and 11 respectively.

[0078] 2) Text data vectorization The LDA model is trained for text vectorization, and the training is carried out based on the built-in LDA model of sklearn: ① Chinese word segmentation Jieba is used for word segmentation, and optimization of the dictionary and stop word list is required.

[0079] Optimizing the dictionary: add internal proper nouns to prevent a noun from being split into two parts. First, query all official document exchange units from the database tables of historical business forms and add them to the custom dictionary. Second, query all department names and their abbreviations within the company from the internal OA system database, and add them to the custom stop word list. Third, query all official document numbers of received documents (such as "Shuidianguisigui

[2025] No.9") from the historical business form database, extract the prefix abbreviation of the official document number (such as "Shuidianguisigui") through a regular expression, and add it to the custom dictionary.

[0080] Optimizing the stop word list: add common auxiliary words, modal particles, Chinese and English punctuation marks, special symbols, etc. to the stop word list to prevent meaningless word segmentation results.

[0081] ② Word frequency statistics According to the word segmentation results, the occurrence frequency of each word in each document is counted to obtain a word frequency vector. The word frequency vectors of all documents are superimposed to obtain a document-word frequency matrix. Here, one document refers to the text obtained by concatenating non-category data and file content in one received document process.

[0082] ③ LDA model training The maximum number of iterations is set to 100, the fitting method is online learning, the batch size is 256, the evaluation frequency is once every 10 rounds, and the random seed is 42. The LDA model parameters are fitted based on the document-word frequency matrix. The trained model can calculate the distribution of a document on K topics, that is, a K-dimensional vector, which is used as the feature vector Vt of the document.

[0083] To ensure that Vt effectively preserves the document's information while avoiding overfitting, a suitable K needs to be chosen: gradually increase K from 2 to a specified maximum value of 60. For each K value, train an LDA model and calculate the corresponding perplexity. Plot K on the horizontal axis and perplexity on the vertical axis on a graph, as shown in the figure below. According to the "elbow method," K=30 is more suitable, meaning that the text, after passing through the LDA model, obtains a distribution across 30 topics, forming a 30-dimensional vector.

[0084] 3) Feature vector concatenation According to the aforementioned formula, by concatenating the category vector and the text vector, a 47-dimensional feature vector for a document flow is obtained, including 6 dimensions of incoming document type, 11 dimensions of receiving document type, and 30 dimensions of concatenated text.

[0085] (4) Search and training of the best prediction model Treating the prediction of the next node and responsibility as a classification problem, the labels are "next node" and "responsibility". The steps include: vectorization, dataset partitioning, classifier selection, and classifier fitting. The input is the feature vector obtained from the feature extraction step, and the output is the next node and the receiving department.

[0086] First, the dataset was divided into a training set and a test set at an 8:2 ratio, containing 2356 received documents and 589 received documents, respectively.

[0087] Next, select a classifier and set the value ranges for the corresponding important hyperparameters. The classifier and its hyperparameters selected in the implementation example are as follows:

[0088] Next, the optimal hyperparameters are sought. On the training set, for each classifier, 10-fold cross-validation and grid search are used, with the accuracy of predicting the next node and the F1 score as evaluation metrics to find the optimal combination of hyperparameters. Among them, the k-nearest neighbor classifier has the best overall performance, and the optimal hyperparameters are: 17 nearest neighbors and distance weights. The lightweight gradient booster is implemented using LightGBM, and the other classifiers are implemented using sklearn. In practice, implementers can choose other software packages to implement the classifiers according to their personal preferences.

[0089] Finally, train the classifier. Using all the training data, train the optimal classifier with the best hyperparameter combination to obtain the final classification model. Note that two classifiers need to be trained, both with the same input and hyperparameters; one outputs the next node, and the other outputs the receiver.

[0090] The trained classification model, on the test set, achieved an accuracy of 0.745 and an F1 score of 0.733 for predicting the next node, and an accuracy of 0.613 and an F1 score of 0.600 for predicting the receiving department. The confusion matrix for the next node prediction results is shown in the table below. Since the next node is rarely the approval process for incoming documents, there may be instances where no samples of this type are found after dividing the test set.

[0091]

[0092] (5) Model reasoning and deployment 1) Service-oriented model inference First, add a post-processing procedure for the prediction results. Based on the division of labor among departments and personnel within the company, establish and dynamically maintain a department-responsible person mapping table to map the predicted receiving department to the recipient.

[0093] Then, based on the trained LDA model weights and classification model weights, the data preprocessing, feature extraction, classification model prediction, and post-processing processes are connected into a streaming processing method, and encapsulated into a web application programming interface (HTTP API) using the FastAPI framework for the OA system to call. A simple interface call log is added to facilitate problem tracking and usage statistics. The OA system receives the prediction results returned in JSON format via HTTP requests, carrying the document's title, category, body, and other input information, and after parsing, displays them on the corresponding document workflow form.

[0094] During implementation, implementers can choose other web application frameworks and communication formats based on personal preferences and enterprise technical architecture.

[0095] 2) Containerized deployment of model inference service To ensure rapid deployment, migration, startup, shutdown, and fault recovery of the service, Docker container technology is used to encapsulate the model inference service. Dependencies, models, and other components are packaged into an image and placed on a server to be started as a container. The proposed method does not have high computational requirements and can run on ordinary servers. In the implementation example, the packaged image runs normally on Intel CPUs and Ubuntu operating systems, as well as on Hygon CPUs and Kylin operating systems.

[0096] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A lightweight intelligent document processing method based on artificial intelligence technology, characterized in that, Includes the following steps: S1. Obtain input data from the document processing flow, wherein the input data includes form business data and document files; S2. Perform multi-source heterogeneous data preprocessing on the official document, extract the text content, and concatenate it with non-category data in the form business data to obtain text data, while extracting category data from the form business data; S3. Vectorize the category data to obtain category vectors, vectorize the text data to obtain text vectors, and concatenate the category vectors and text vectors to obtain feature vectors; S4. Construct a prediction model based on historical document processing data. The input of the prediction model is a feature vector, and the output is the next processing node and responsibility. S5. Establish and dynamically maintain a responsibility-personnel mapping table to map the predicted responsibilities to specific processing personnel. S6. Encapsulate the processing flow of the above steps into a web application programming interface for office automation systems to call, thereby realizing intelligent document circulation.

2. The method according to claim 1, characterized in that, The output of the prediction model is expressed as follows: [N, D] = f pred,2 (C, M) Where N is the next processing node, D is the responsibility, C is the content transmitted by the current node, M is auxiliary information, and f pred,2 This is a prediction algorithm.

3. The method according to claim 1, characterized in that, The process of mapping predicted responsibilities to personnel is represented as follows: P = f map (D) Where P represents the processing personnel, f map This is a dynamically maintained responsibility-person mapping table, where D represents responsibility.

4. The method according to claim 1, characterized in that, The feature vector is obtained by concatenating the following formula: V = [Vc, Vt] = [f e,c (Cc), f e,t (Ct)] Where V is the feature vector, Vc is the category vector, Vt is the text vector, Cc is the category data, Ct is the concatenated text data, and f e,c f is a vectorization function for categorical data. e,t This is a function for vectorizing text data.

5. The method according to claim 1, characterized in that, Preprocessing of multi-source heterogeneous data for official documents includes: for editable documents, directly extracting text; for outdated format documents, converting them to editable format using open-source tools before extracting text; for images or scanned PDFs, using lightweight optical character recognition tools for text recognition; and for compressed files, extracting only the filename.

6. The method according to claim 1, characterized in that, The text data is vectorized using the latent Dirichlet allocation model, which vectorizes the text data of a document process into a distribution across K topics; where the number of topics K is automatically determined using the perplexity elbow method.

7. The method according to claim 1, characterized in that, Vectorization of categorical data can be performed using one-hot encoding, label encoding, or sequence encoding; vectorization of text data can be performed using the Latent Dirichlet Allocation Model or the Doc2Vec Model.

8. The method according to claim 1, characterized in that, The prediction model is implemented using a classifier, which is selected from one or more of the following: K-nearest neighbor classifier, multilayer perceptron, support vector machine, Naive Bayes, decision tree, random forest, gradient boosting classifier, and lightweight gradient booster; and the optimal hyperparameters are selected through k-fold cross-validation and grid search.

9. The method according to claim 1, characterized in that, When encapsulating the processing flow into a network application interface, the FastAPI framework is used, and Docker container technology is used to package the service into an image to achieve containerized deployment.

10. The method according to any one of claims 1-9, characterized in that, The method does not depend on a graphics processor and runs only in a central processing unit environment.

Images