Intelligent file sorting method and device for generating understanding ability based on large model

Abstract

The invention discloses an intelligent file sorting method and device for generating understanding ability based on a large model, and is applied to a server. The method comprises the following steps: receiving a file and extracting a plain text, and extracting a picture text by using an OCR technology; splicing the plain text and a preset prompt template, sending the spliced plain text and the preset prompt template to a large model engine, and extracting a file type and a business association field to generate JSON (JavaScript Object Notation) data; and querying in a business system based on the business association field, directly associating and sorting the file if the business association field is queried, temporarily storing the file if the business association field is not queried, and associating and sorting after the business data is maintained. The preprocessing comprises format conversion, deduplication and integrity verification. The large model engine adopts a Transform architecture, and the method further relates to named entity recognition, message queue monitoring, cross entropy training, confusion matrix verification and the like. The method realizes intelligent file sorting, improves efficiency and accuracy, and reduces labor cost.

Description

Technical Field

[0001] The present invention relates to the technical field of file sorting, and in particular to an intelligent file sorting method and device based on large model generation and comprehension capabilities. Background Art

[0002] In traditional business systems, linking tasks and files typically involves customizing the business information management and maintenance interface. The process involves manually uploading the task-related files to the system via a form. Specifically, to process a task-related file, one must first locate the corresponding task row in the system and then manually upload the file via the form interface. This entire linking process involves multiple steps.

[0003] When faced with large volumes of documents requiring sorting and association, traditional technologies require significant manual effort. This is because each document must be manually processed according to the aforementioned multi-step process, which is not only time-consuming and labor-intensive, but also significantly increases a company's labor costs. In traditional manual processes, human error can easily lead to file association errors. For example, operators may accidentally upload a file to an inappropriate business record, or make an error when filling out a form, resulting in inaccurate association between the file and the business.

[0004] Furthermore, traditional technologies for document sorting and association rely primarily on manual operations, which presents significant limitations. On the one hand, they are unable to efficiently handle the large volumes of documents processed, resulting in high labor costs. On the other hand, they struggle to ensure the accuracy of document associations, making them prone to errors. These issues severely impact the efficiency and quality of business processing, necessitating an urgent need for a technical solution that can achieve automated and intelligent document sorting and association, reducing labor costs and improving operational accuracy and efficiency. Summary of the Invention

[0005] To this end, the present invention provides an intelligent file sorting method and device based on large model generation and understanding capabilities, which solves the problems in traditional business systems where file sorting and association require manual operation, large-scale processing is labor-intensive and error-prone.

[0006] To achieve the above objectives, the present invention provides the following technical solution: an intelligent file sorting method based on large model generation and understanding capabilities, applied to a server, comprising the following steps:

[0007] The server receives the file uploaded by the user and recognizes and extracts the plain text in the file. If the file contains an image, it uses OCR technology to recognize and extract the text in the image.

[0008] The server assembles the extracted plain text with the preset prompt template;

[0009] The server sends the assembled prompt to the large model engine, which uses the content generation capability of the large model engine to extract the file type and business-related fields to form JSON format data;

[0010] The server performs a query in the business system based on the extracted business-related fields;

[0011] If the corresponding business data is found, the server will directly associate and sort the file with the corresponding business data; if the corresponding business data is not found, the server will temporarily store the file and associate and sort it after the business data is maintained.

[0012] As a preferred solution for the intelligent file sorting method based on the large model's ability to generate and understand, the prompt template contains instruction information for instructing the large model to extract file types and business-related fields. The large model engine adopts the Transformer architecture, and the mathematical model formula is:

[0013]

[0014] Where Q is the query vector, which is used to query relevant information in the attention mechanism; K is the key vector, which is used to identify the characteristics of the information; V is the value vector, which is used to provide specific information content; d k is the dimension of the key vector, which is used to scale the attention scores to stabilize the training process.

[0015] As a preferred solution for the intelligent file sorting method based on the ability to generate and understand large models, after the server receives the files uploaded by the user, the server also preprocesses the files. The preprocessing includes at least one of file format conversion, file deduplication, and file integrity verification. File deduplication uses the SimHash algorithm. The specific steps are as follows:

[0016] Segment the document into words and assign weights to each word;

[0017] Perform hash operation on each word to obtain hash value;

[0018] The SimHash value of the file is obtained by weighted combination of hash values. The Hamming distance of the SimHash value is compared to determine whether the file is duplicated. The Hamming distance calculation formula is:

[0019]

[0020] Where a and b are two SimHash values ​​used to represent the feature code of the file; a i and b iThe i-th bit of a and b respectively, corresponding to each feature in the code; It is an XOR operation, used to determine whether two features are different.

[0021] As a preferred solution for the intelligent document sorting method based on the large model generation and understanding capability, the business-related fields include at least one of the contract number, business type, project name, and date. When extracting the business-related fields, a named entity recognition algorithm is used. The probability calculation formula of the conditional random field model of the named entity recognition algorithm is:

[0022]

[0023] In the formula, x is the input text, that is, the content of the file to be processed; y is the label sequence, which is used to represent the entity category corresponding to each part of the text; t k is the transfer feature function, which is used to describe the transfer relationship between labels; s l is the state feature function, which is used to describe the matching relationship between the label and the text content; k and μ l is the feature weight, which is used to adjust the importance of different features; Z(x) is the normalization factor, which is used to ensure the correctness of the probability calculation.

[0024] As a preferred solution for the intelligent file sorting method based on the ability to generate and understand large models, after temporarily storing the files, the server monitors the temporarily stored files. When it detects that the corresponding business data is maintained in the business system, it automatically triggers the associated sorting operation of the temporarily stored files. The monitoring process adopts the message queue mechanism. The mathematical model of the message queue mechanism is:

[0025] Q(t)=Q(t-1)+I(t)-O(t)

[0026] Where Q(t) is the number of messages in the message queue at time t, which reflects the current amount of tasks in the queue; I(t) is the number of messages entering the queue at time t, which is the number of newly generated monitoring tasks; O(t) is the number of messages processed at time t, which is the number of completed monitoring tasks.

[0027] As a preferred solution for the intelligent file sorting method based on the ability of large-scale model generation and understanding, the large-scale model engine is a natural language processing model based on deep learning. The cross entropy loss function is used in the training process, and the calculation formula is:

[0028]

[0029] Where N is the number of samples, that is, the number of files involved in training; C is the number of categories, that is, the number of entity types that need to be classified; y i,c is the true label of sample i, which is used to indicate the category to which the sample actually belongs; pi,c The model predicts the probability that sample i belongs to category c, which is used to measure the prediction accuracy of the model.

[0030] As a preferred solution for intelligent file sorting based on the ability to generate and understand large models, after directly associating and sorting files with corresponding business data, the server verifies and corrects the association results. The verification process uses a confusion matrix evaluation method. The indicator calculation of the confusion matrix evaluation method includes:

[0031] Accuracy:

[0032]

[0033] Recall:

[0034]

[0035] F1 score:

[0036]

[0037] Where TP is the true positive, i.e., the number of files correctly identified and associated; FP is the false positive, i.e., the number of files incorrectly identified and associated; and FN is the false negative, i.e., the number of files that were not correctly identified but should actually be associated.

[0038] The present invention also provides an intelligent file sorting device based on large model generation and understanding capability, comprising:

[0039] The file processing module is used to receive files uploaded by users and identify and extract the plain text in the files. If the files contain pictures, the OCR technology is used to identify and extract the text in the pictures.

[0040] The prompt assembly module is used to assemble the extracted plain text with the preset prompt template;

[0041] The large model processing module is used to send the assembled prompt to the large model engine, extract the file type and business-related fields through the content generation capability of the large model engine, and form JSON format data;

[0042] A business query module, configured to query the business system based on the extracted business-related fields;

[0043] The sorting processing module is used to associate and sort the file directly with the corresponding business data if the corresponding business data is queried; if the corresponding business data is not queried, the server will temporarily store the file and associate and sort it after the business data is maintained.

[0044] As a preferred solution for an intelligent file sorting device based on the ability to generate and understand large models, in the prompt assembly module, the prompt template contains instruction information for instructing the large model to extract file types and business-related fields. The large model engine adopts the Transformer architecture, and the mathematical model formula is:

[0045]

[0046] Where Q is the query vector, which is used to query relevant information in the attention mechanism; K is the key vector, which is used to identify the characteristics of the information; V is the value vector, which is used to provide specific information content; d k is the dimension of the key vector, which is used to scale the attention scores to stabilize the training process;

[0047] The large model engine is a natural language processing model based on deep learning. The cross entropy loss function is used in the training process. The calculation formula is:

[0048]

[0049] Where N is the number of samples, that is, the number of files involved in training; C is the number of categories, that is, the number of entity types that need to be classified; y i,c is the true label of sample i, which is used to indicate the category to which the sample actually belongs; p i,c The model predicts the probability that sample i belongs to category c, which is used to measure the prediction accuracy of the model.

[0050] As a preferred solution for an intelligent document sorting device based on the ability to generate and understand large models, in the large model processing module, the business-related fields include at least one of contract number, business type, project name, and date. When extracting business-related fields, a named entity recognition algorithm is used. The probability calculation formula of the conditional random field model of the named entity recognition algorithm is:

[0051]

[0052] In the formula, x is the input text, that is, the content of the file to be processed; y is the label sequence, which is used to represent the entity category corresponding to each part of the text; t k is the transfer feature function, which is used to describe the transfer relationship between labels; s l is the state feature function, which is used to describe the matching relationship between the label and the text content; k and μ l is the feature weight, which is used to adjust the importance of different features; Z(x) is the normalization factor, which is used to ensure the correctness of the probability calculation.

[0053] As a preferred solution for the intelligent file sorting device based on the ability to generate and understand large models, in the sorting processing module, after temporarily storing the files, the server monitors the temporarily stored files. When it detects that the corresponding business data is maintained in the business system, it automatically triggers the associated sorting operation of the temporarily stored files. The monitoring process adopts a message queue mechanism. The mathematical model of the message queue mechanism is:

[0054] Q(t)=Q(t-1)+I(t)-O(t)

[0055] Where Q(t) is the number of messages in the message queue at time t, which reflects the current amount of tasks in the queue; I(t) is the number of messages entering the queue at time t, which is the number of newly generated monitoring tasks; O(t) is the number of messages processed at time t, which is the number of completed monitoring tasks.

[0056] As a preferred solution for the intelligent document sorting device based on the ability to generate and understand large models, it also includes:

[0057] The file preprocessing module is used to preprocess the file. The preprocessing includes at least one of file format conversion, file deduplication and file integrity verification. The file deduplication adopts the SimHash algorithm, which is specifically:

[0058] Segment the document into words and assign weights to each word;

[0059] Perform hash operation on each word to obtain hash value;

[0060] The SimHash value of the file is obtained by weighted combination of hash values. The Hamming distance of the SimHash value is compared to determine whether the file is duplicated. The Hamming distance calculation formula is:

[0061]

[0062] Where a and b are two SimHash values ​​used to represent the feature code of the file; a i and b i The i-th bit of a and b respectively, corresponding to each feature in the code; It is an XOR operation, used to determine whether two features are different.

[0063] As a preferred solution of the intelligent file sorting device based on the large model generation and understanding capability, it also includes: an association correction module, which is used to verify and correct the association results after the files are directly associated and sorted with the corresponding business data. The verification process adopts the confusion matrix evaluation method. The indicator calculation of the confusion matrix evaluation method includes:

[0064] Accuracy:

[0065]

[0066] Recall:

[0067]

[0068] F1 score:

[0069]

[0070] Where TP is the true positive, i.e., the number of files correctly identified and associated; FP is the false positive, i.e., the number of files incorrectly identified and associated; and FN is the false negative, i.e., the number of files that were not correctly identified but should actually be associated.

[0071] The present invention has the following advantages:

[0072] First, the server automatically handles document uploads and text extraction, and combines OCR technology to achieve image text recognition, replacing traditional manual uploading operations, thereby improving the efficiency of large-scale file processing and significantly reducing manpower investment.

[0073] Second, through the intelligent interaction between the prompt template and the large model engine, document types and business-related fields are automatically extracted and JSON data is generated. Compared with manual form filling, the accuracy of key information extraction is improved and association errors are greatly reduced.

[0074] Third, the automatic query and sorting mechanism based on business fields enables intelligent matching of files and business data. The temporary storage mechanism when no data is found can avoid omissions. The overall sorting process is highly automated, effectively solving the inefficiency problem of traditional multi-step operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are merely exemplary, and those skilled in the art can, without inventive effort, derive other implementation drawings based on the provided drawings.

[0076] The structures, proportions, sizes, etc. illustrated in this specification are intended solely to complement the contents disclosed herein and to facilitate understanding and reading by persons skilled in the art. They are not intended to limit the conditions under which the present invention may be implemented and therefore have no substantive technical significance. Any structural modifications, changes in proportions, or adjustments in sizes, without affecting the efficacy and objectives of the present invention, shall remain within the scope of the technical contents disclosed herein.

[0077] Figure 1This is a flow chart of the intelligent file sorting method based on large model generation and understanding capability provided in Example 1 of the present invention;

[0078] Figure 2 This is a technical diagram of the intelligent file sorting method based on large model generation and understanding capabilities provided in Example 1 of the present invention;

[0079] Figure 3 This is a schematic diagram of the architecture of an intelligent file sorting device based on large model generation and understanding capabilities provided in Example 2 of the present invention. DETAILED DESCRIPTION

[0080] The following describes the implementation of the present invention using specific embodiments. Those skilled in the art will readily understand the other advantages and benefits of the present invention from the disclosure herein. Obviously, the embodiments described are only a portion of the present invention, not all of it. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are intended to fall within the scope of protection of the present invention.

[0081] Example 1

[0082] See also Figure 1 and Figure 2 The embodiment of the present invention provides an intelligent file sorting method based on large model generation and understanding capability, which is applied to a server and includes the following steps:

[0083] S1. The server receives the file uploaded by the user and recognizes and extracts the plain text in the file. If the file contains an image, it uses OCR technology to recognize and extract the text in the image.

[0084] Specifically, the server receives user-uploaded files through a network interface. These files may contain a variety of content, including text and images. Plain text can be directly extracted using text parsing techniques. However, text within images cannot be directly read, so OCR (Optical Character Recognition) technology is employed. This technology uses image processing and pattern recognition algorithms to convert text within images into editable text data. The core principle of OCR technology is to pre-process the image through grayscale conversion and binarization, then extract features and match them with templates in a character library to identify the text content.

[0085] S2. The server assembles the extracted plain text with the preset prompt template.

[0086] Specifically, a prompt template is a pre-designed text framework containing specific instructions. Its function is to guide the large model to process the input text according to the specified requirements. The extracted plain text is combined with the prompt template to provide the large model with clear task instructions and contextual information. For example, the prompt may contain instructions such as "Please extract the contract number, business type, and other information from the following text," allowing the large model to extract the required key information in a targeted manner during processing. This approach takes advantage of the conditional generation characteristics of the large model, guiding the model to produce the desired output by constructing an appropriate input prompt.

[0087] S3. The server sends the assembled prompt to the large model engine, and extracts the file type and business-related fields through the content generation capability of the large model engine to form JSON format data.

[0088] Specifically, large model engines are typically natural language processing models based on deep learning, such as models using the Transformer architecture. Their core principle is to use the attention mechanism to capture long-range dependencies in text, thereby understanding the text's semantics. When the assembled prompt is input into the model, the model generates the corresponding file type and business-related fields based on the instructions and text content in the prompt through calculations using a multi-layer neural network. Specifically, the model encodes the input text to obtain a semantic representation for each word, and then generates the required information through a decoding process. The generated information is organized in JSON format to facilitate subsequent storage and processing. The JSON format is structured and easy to parse, enabling convenient data exchange with other business systems.

[0089] S4. The server performs a query in the business system based on the extracted business-related fields.

[0090] Specifically, business systems store vast amounts of business data, organized according to specific structures and indexes. After extracting business-related fields (such as contract numbers and project names), the server uses these fields as query criteria to search the business system's database. Database queries are typically based on indexing techniques, such as B-tree indexes and hash indexes, to improve query efficiency. By matching business-related fields with data in the business system, corresponding business records can be found, providing a basis for subsequent file association sorting.

[0091] S5. If the corresponding business data is found, the server will directly associate and sort the file with the corresponding business data; if the corresponding business data is not found, the server will temporarily store the file and associate and sort it after the business data is maintained.

[0092] Specifically, when matching business data is found, the server will establish an association relationship between the file and the business data based on preset rules and data structures, and sort the file into the corresponding business category. This process usually involves updating the database to associate the file's identification information with the business record. If the corresponding business data is not found, it means that there is no business record related to the file in the current business system. At this time, the file will be temporarily stored in the designated storage location. After temporary storage, the server will monitor the temporary file, and when it detects that the corresponding business data is maintained in the business system, it will automatically trigger the association sorting operation. This mechanism ensures that files will not be lost due to delayed entry of business data, and ensures the accurate association between files and business data.

[0093] In this embodiment, in step S1, after the server receives the file uploaded by the user, the server also preprocesses the file. The preprocessing includes at least one of file format conversion, file deduplication, and file integrity verification. File deduplication uses the SimHash algorithm. The specific steps are as follows:

[0094] Segment the document into words and assign weights to each word;

[0095] Perform hash operation on each word to obtain hash value;

[0096] The SimHash value of the file is obtained by weighted combination of hash values; the Hamming distance of the SimHash value is compared to determine whether the file is duplicated. The Hamming distance calculation formula is:

[0097]

[0098] Where a and b are two SimHash values ​​used to represent the feature code of the file; a i and b i The i-th bit of a and b respectively, corresponding to each feature in the code; It is an XOR operation, used to determine whether two features are different.

[0099] Specifically, file preprocessing aims to improve the efficiency and accuracy of subsequent processing. File format conversion converts files of different formats into a unified format to facilitate subsequent text extraction and processing. File integrity verification ensures that files are not damaged during transmission and storage, guaranteeing data accuracy. File deduplication uses the SimHash algorithm, which converts file content into a fixed-length binary code (SimHash value) that reflects the characteristics of the file. Specifically, the file is first segmented and each word is assigned a weight based on its importance (such as the TF-IDF weight, which measures the importance of a word in the file). Then, a hash operation is performed on each word to obtain a hash value. These hash values ​​are then weighted and combined according to the word weight to obtain the file's SimHash value. Duplicates are determined by comparing the Hamming distance (i.e., the number of bits that differ between the two codes) between the SimHash values ​​of two files. The smaller the Hamming distance, the more similar the files. This algorithm can quickly identify duplicate or similar files in large volumes of files, reducing data redundancy.

[0100] In this embodiment, in step S2, the prompt template includes instruction information for instructing the large model to extract file types and business-related fields. The large model engine adopts the Transformer architecture, and the mathematical model formula is:

[0101]

[0102] Where Q is the query vector, which is used to query relevant information in the attention mechanism; K is the key vector, which is used to identify the characteristics of the information; V is the value vector, which is used to provide specific information content; d k is the dimension of the key vector, which is used to scale the attention scores to stabilize the training process.

[0103] Specifically, the instruction information in the prompt template is to let the large model know clearly what information needs to be extracted from the input text. The core of the Transformer architecture is the self-attention mechanism, which calculates the similarity between the query vector (Q) and the key vector (K) to determine the degree of attention to each position in the input sequence when generating the output, and then multiplies this degree of attention with the value vector (V) to obtain the final output. T What is calculated is the dot product similarity between Q and K, is the scaling factor used to prevent kWhen V is large, the dot product is too large, causing the softmax function gradient to vanish. The softmax function converts similarity into a probability distribution, resulting in a weight for each position. This weight is then multiplied by V to produce the final attention output. This mechanism enables the model to efficiently capture long-range dependencies in text, thereby better understanding text semantics and completing information extraction tasks.

[0104] In this embodiment, in step S3, the business-related fields include at least one of a contract number, a business type, a project name, and a date. When extracting the business-related fields, a named entity recognition algorithm is used. The probability calculation formula of the conditional random field model of the named entity recognition algorithm is:

[0105]

[0106] In the formula, x is the input text, that is, the content of the file to be processed; y is the label sequence, which is used to represent the entity category corresponding to each part of the text; t k is the transfer feature function, which is used to describe the transfer relationship between labels; s l is the state feature function, which is used to describe the matching relationship between the label and the text content; k and μ l is the feature weight, which is used to adjust the importance of different features; Z(x) is the normalization factor, which is used to ensure the correctness of the probability calculation.

[0107] Specifically, named entity recognition (NER) refers to identifying entities with specific meanings from text, such as contract numbers and project names. The conditional random field (CRF) model is a commonly used NER algorithm. Its principle is to take a text sequence as input and predict the entity label sequence corresponding to each position. The probability calculation formula of the model takes into account two parts of features: the transfer feature function t k and state characteristic function s l The transition feature function describes the transition relationship between labels, for example, the probability of "date" appearing after "contract number" is higher; the state feature function describes the matching relationship between labels and text content, for example, whether a text segment conforms to the format characteristics of the contract number. By learning the weights λ of these features k and μ l , the model can calculate the most likely label sequence y based on the input text x. The normalization factor Z(x) is used to convert the exponential sum into a probability distribution, ensuring the correctness of the probability calculation. This model can fully utilize the contextual information of the text and improve the accuracy of named entity recognition.

[0108] In this embodiment, in step S5, after temporarily storing the file, the server monitors the temporarily stored file. When it detects that the corresponding business data is maintained in the business system, it automatically triggers the associated sorting operation of the temporarily stored file. The monitoring process adopts a message queue mechanism. The mathematical model of the message queue mechanism is:

[0109] Q(t)=Q(t-1)+I(t)-O(t)

[0110] Where Q(t) is the number of messages in the message queue at time t, which reflects the current amount of tasks in the queue; I(t) is the number of messages entering the queue at time t, which is the number of newly generated monitoring tasks; O(t) is the number of messages processed at time t, which is the number of completed monitoring tasks.

[0111] Specifically, the message queue mechanism is used to implement asynchronous management of monitoring and task processing of temporary files. When a file is temporarily stored, the system generates a monitoring task and places it in the message queue. The number of messages in the message queue changes over time. Q(t) represents the number of messages in the queue at time t, which is equal to the number at the previous moment Q(t-1) plus the number of newly entered messages I(t) minus the number of processed messages O(t). The server will continuously obtain monitoring tasks from the message queue and check whether the corresponding business data is maintained in the business system. When the existence of business data is detected, the associated sorting operation is triggered, and the task is removed from the queue (that is, O(t) increases). This mechanism enables monitoring tasks to be processed in an orderly manner, avoiding the confusion caused by concurrent processing, while ensuring timely monitoring and processing of temporary files, improving the reliability and stability of the system.

[0112] In a possible embodiment, the large model engine is a natural language processing model based on deep learning, and a cross entropy loss function is used in the training process, and the calculation formula is:

[0113]

[0114] Where N is the number of samples, that is, the number of files involved in training; C is the number of categories, that is, the number of entity types that need to be classified; y i,c is the true label of sample i, which is used to indicate the category to which the sample actually belongs; p i,c The model predicts the probability that sample i belongs to category c, which is used to measure the prediction accuracy of the model.

[0115] Specifically, the training process of a large model engine is to optimize the model parameters through a large amount of labeled data so that it can better complete tasks such as information extraction. The cross entropy loss function is used to measure the difference between the model prediction results and the true label. For each sample i, the model predicts the probability p of it belonging to each category. i,c , and the true label yi,c Indicates the actual category to which the sample belongs (typically one-hot encoded, meaning only the correct category has a value of 1, and all others are 0). The cross-entropy loss function calculates the logarithmic difference between the true label and the predicted probability and averages this across all samples to obtain the model's loss. During training, the backpropagation algorithm continuously adjusts the model parameters to minimize the loss, thereby improving the model's predictive accuracy. This loss function is suitable for multi-classification problems and can effectively guide the model to learn the correct classification boundaries and semantic features.

[0116] In one possible embodiment, after directly associating and sorting the files with the corresponding business data, the server verifies and corrects the association results. The verification process uses a confusion matrix evaluation method. The indicator calculation of the confusion matrix evaluation method includes:

[0117] Accuracy:

[0118]

[0119] Recall:

[0120]

[0121] F1 score:

[0122]

[0123] Where TP is the true positive, i.e., the number of files correctly identified and associated; FP is the false positive, i.e., the number of files incorrectly identified and associated; and FN is the false negative, i.e., the number of files that were not correctly identified but should actually be associated.

[0124] Specifically, the confusion matrix evaluation method is used to objectively evaluate the accuracy of file association sorting. True positives (TP) represent the number of files correctly identified and associated, false positives (FP) represent the number of files incorrectly identified and associated, and false negatives (FN) represent the number of files that should have been associated but were incorrectly identified. Precision (P) measures the proportion of correct associations, while recall (R) measures the proportion of files that should have been associated that were correctly identified. The F1 score is the harmonic mean of precision and recall and is used to comprehensively evaluate model performance. By calculating these metrics, the server can identify problems in the association sorting process. For example, a low precision may indicate a high number of incorrect associations, while a low recall may indicate a high number of incorrectly identified files. Based on the evaluation results, the server can correct the association results, such as manually adjusting incorrect associations or optimizing model parameters to improve subsequent association accuracy. This verification and correction mechanism ensures the quality of file association sorting, improving system reliability and user satisfaction.

[0125] The application scenarios of the present invention are as follows:

[0126] Enterprise contract management scenario: A large number of contract documents are generated in the daily operation of an enterprise, such as purchase contracts, sales contracts, etc. The present invention can be applied to the enterprise contract management system. After the server receives the contract document uploaded by the user, it identifies and extracts the plain text in the document. If it contains pictures, it uses OCR technology to extract the text. The extracted plain text is assembled with the preset prompt template and sent to the large model engine to extract business-related fields such as contract number, business type, project name, date, etc. Based on these fields, the corresponding business data is queried in the business system to realize intelligent association and sorting of contract documents with business data such as purchase and sales. For example, a large manufacturing enterprise signs hundreds of purchase contracts every month. Through the present invention, the contract documents can be automatically associated with business data such as purchase orders and supplier information, thereby improving contract management efficiency and reducing manual errors.

[0127] Financial institution document processing scenarios: Financial institutions such as banks and securities firms process large volumes of documents, including customer information and transaction vouchers. When applying this invention, a server receives uploaded financial documents, extracts and processes the text, and then uses a large model engine to extract business-related fields such as customer information, transaction type, transaction amount, and date. The server then queries the financial business system for corresponding business data, including customer accounts and transaction records, to accurately associate and sort documents with business data. For example, when a bank processes customer loan applications, it can automatically associate the documents with the customer's loan business records, facilitating business review and management.

[0128] Government document management scenarios: Government departments generate a large number of documents during the government affairs processing process, such as approval documents and policy documents. Utilizing this invention, the server extracts and processes the text of received government documents. Using a large model engine, it extracts business-related fields such as file type, project name, approval number, and date. It then queries the government affairs business system for corresponding business data such as approval processes and policy execution, enabling intelligent sorting and association of documents. For example, when a government approval department processes enterprise qualification approval documents, it can automatically associate the documents with the approval business process, improving government affairs processing efficiency.

[0129] Medical record management scenario: Hospitals generate a large number of patient medical records, diagnostic reports, and other documents. This invention can be applied to hospital medical record management systems. After receiving uploaded medical record files, the server extracts the text information and assembles it with a prompt template. Using a large model engine, it extracts business-related fields such as the patient's name, medical record number, disease type, and diagnosis date. The corresponding business data, such as patient visit records and treatment plans, is then searched in the hospital information system, enabling intelligent sorting and association of medical record files. This helps hospitals improve medical record management efficiency and facilitates doctors' access to and reference of patient medical histories.

[0130] Educational institution file management scenario: Educational institutions such as schools manage a large number of files, including student files and course materials. Using this invention, a server receives uploaded student file files, extracts and processes the text, and then uses a large model engine to extract business-related fields such as student name, student ID, major, and enrollment date. The server then searches the school management system for corresponding student information, course records, and other business data, enabling intelligent file sorting and association. For example, when processing new student admission files, a school can automatically associate the files with the student's enrollment information, facilitating file management and query.

[0131] It should be noted that the method of the embodiment of the present disclosure can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied in a distributed scenario, where multiple devices cooperate to perform the method. In such a distributed scenario, one of the multiple devices may only perform one or more steps of the method of the embodiment of the present disclosure. The multiple devices will interact with each other to complete the intelligent file sorting method based on the large model generation and understanding capability.

[0132] It should be noted that the above description is limited to some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in an order different from that described in the above embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0133] Example 2

[0134] See also Figure 3 Embodiment 2 of the present invention further provides an intelligent file sorting device based on large model generation and understanding capability, comprising:

[0135] The file processing module 100 is used to receive files uploaded by users and identify and extract the plain text in the files. If the files contain pictures, the OCR technology is used to identify and extract the text in the pictures.

[0136] The prompt assembly module 200 is used to assemble the extracted plain text with the preset prompt template;

[0137] The large model processing module 300 is used to send the assembled prompt to the large model engine, extract the file type and business-related fields through the content generation capability of the large model engine, and form JSON format data;

[0138] A business query module 400 is used to query the business system based on the extracted business-related fields;

[0139] The sorting processing module 500 is used to, if the corresponding business data is found, the server directly associates and sorts the file with the corresponding business data; if the corresponding business data is not found, the server temporarily stores the file and associates and sorts it after the business data is maintained.

[0140] In this embodiment, in the prompt assembly module 200, the prompt template includes instruction information for instructing the large model to extract file types and business-related fields. The large model engine adopts the Transformer architecture, and the mathematical model formula is:

[0141]

[0142] Where Q is the query vector, which is used to query relevant information in the attention mechanism; K is the key vector, which is used to identify the characteristics of the information; V is the value vector, which is used to provide specific information content; d k is the dimension of the key vector, which is used to scale the attention scores to stabilize the training process;

[0143] The large model engine is a natural language processing model based on deep learning. The cross entropy loss function is used in the training process. The calculation formula is:

[0144]

[0145] Where N is the number of samples, that is, the number of files involved in training; C is the number of categories, that is, the number of entity types that need to be classified; y i,c is the true label of sample i, which is used to indicate the category to which the sample actually belongs; p i,c The model predicts the probability that sample i belongs to category c, which is used to measure the prediction accuracy of the model.

[0146] In this embodiment, in the large model processing module 300, the business-related fields include at least one of a contract number, a business type, a project name, and a date. When extracting the business-related fields, a named entity recognition algorithm is used. The probability calculation formula of the conditional random field model of the named entity recognition algorithm is:

[0147]

[0148] In the formula, x is the input text, that is, the content of the file to be processed; y is the label sequence, which is used to represent the entity category corresponding to each part of the text; t k is the transfer feature function, which is used to describe the transfer relationship between labels; s lis the state feature function, which is used to describe the matching relationship between the label and the text content; k and μ l is the feature weight, which is used to adjust the importance of different features; Z(x) is the normalization factor, which is used to ensure the correctness of the probability calculation.

[0149] In this embodiment, in the sorting processing module 500, after temporarily storing the file, the server monitors the temporarily stored file. When it detects that the corresponding business data is maintained in the business system, it automatically triggers the associated sorting operation of the temporarily stored file. The monitoring process adopts a message queue mechanism. The mathematical model of the message queue mechanism is:

[0150] Q(t)=Q(t-1)+I(t)-O(t)

[0151] Where Q(t) is the number of messages in the message queue at time t, which reflects the current amount of tasks in the queue; I(t) is the number of messages entering the queue at time t, which is the number of newly generated monitoring tasks; O(t) is the number of messages processed at time t, which is the number of completed monitoring tasks.

[0152] In this embodiment, it also includes:

[0153] The file preprocessing module 600 is used to preprocess files. The preprocessing includes at least one of file format conversion, file deduplication, and file integrity verification. File deduplication uses the SimHash algorithm, specifically:

[0154] Segment the document into words and assign weights to each word;

[0155] Perform hash operation on each word to obtain hash value;

[0156] The SimHash value of the file is obtained by weighted combination of hash values. The Hamming distance of the SimHash value is compared to determine whether the file is duplicated. The Hamming distance calculation formula is:

[0157]

[0158] Where a and b are two SimHash values ​​used to represent the feature code of the file; a i and b i The i-th bit of a and b respectively, corresponding to each feature in the code; It is an XOR operation, used to determine whether two features are different.

[0159] This embodiment further includes: an association correction module 700, which is used to verify and correct the association result after the file is directly associated and sorted with the corresponding business data. The verification process uses a confusion matrix evaluation method. The indicator calculation of the confusion matrix evaluation method includes:

[0160] Accuracy:

[0161]

[0162] Recall:

[0163]

[0164] F1 score:

[0165]

[0166] Where TP is the true positive, i.e., the number of files correctly identified and associated; FP is the false positive, i.e., the number of files incorrectly identified and associated; and FN is the false negative, i.e., the number of files that were not correctly identified but should actually be associated.

[0167] It should be noted that the information interaction, execution process, etc. between the modules of the above-mentioned device are based on the same concept as the method embodiment in Example 1 of the present application, and the technical effects they bring are the same as those of the method embodiment of the present application. For specific contents, please refer to the description in the method embodiment shown above in the present application, and will not be repeated here.

[0168] Example 3

[0169] Embodiment 3 of the present invention provides a non-transitory computer-readable storage medium, in which the program code of the intelligent file sorting method based on the large model generation and understanding capability is stored. The program code includes instructions for executing the intelligent file sorting method based on the large model generation and understanding capability of embodiment 1 or any possible implementation thereof.

[0170] Computer-readable storage media can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

[0171] Example 4

[0172] Embodiment 4 of the present invention provides an electronic device, including: a memory and a processor;

[0173] The processor and the memory communicate with each other via a bus; the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the intelligent file sorting method based on large model generation and understanding capabilities of Example 1 or any possible implementation thereof.

[0174] Specifically, the processor can be implemented by hardware or by software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc.; when implemented by software, the processor can be a general-purpose processor, which is implemented by reading software code stored in a memory. The memory can be integrated into the processor or located outside the processor and exist independently.

[0175] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode.

[0176] Obviously, those skilled in the art will appreciate that the various modules or steps of the present invention described above can be implemented using a general-purpose computing device, centralized on a single computing device, or distributed across a network of multiple computing devices. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. In some cases, the steps shown or described can be performed in a different order than that shown, or can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0177] Although the present invention has been described in detail above using general descriptions and specific embodiments, it will be apparent to those skilled in the art that modifications and improvements may be made thereto. Therefore, such modifications and improvements, without departing from the spirit of the present invention, are intended to be within the scope of protection claimed herein.

Claims

1. An intelligent file sorting method based on large model generation and understanding capabilities, applied to the server, characterized by: The following steps are involved: The server receives the file uploaded by the user and recognizes and extracts the plain text in the file. If the file contains an image, it uses OCR technology to recognize and extract the text in the image. The server assembles the extracted plain text with the preset prompt template; The server sends the assembled prompt to the large model engine, which uses the content generation capability of the large model engine to extract the file type and business-related fields to form JSON format data; The server performs a query in the business system based on the extracted business-related fields; If the corresponding business data is found, the server will directly associate and sort the file with the corresponding business data; if the corresponding business data is not found, the server will temporarily store the file and associate and sort it after the business data is maintained.

2. The intelligent file sorting method based on large model generation and understanding ability according to claim 1 is characterized in that: The prompt template contains instruction information for instructing the large model to extract file types and business-related fields. The large model engine adopts the Transformer architecture, and the mathematical model formula is: Where Q is the query vector, which is used to query relevant information in the attention mechanism; K is the key vector, which is used to identify the characteristics of the information; V is the value vector, which is used to provide specific information content; d k is the dimension of the key vector, which is used to scale the attention scores to stabilize the training process.

3. The intelligent file sorting method based on large model generation and understanding ability according to claim 1 is characterized in that: After receiving the file uploaded by the user, the server also pre-processes the file. The pre-processing includes at least one of file format conversion, file deduplication, and file integrity verification. File deduplication uses the SimHash algorithm. The specific steps are as follows: Segment the document into words and assign weights to each word; Perform hash operation on each word to obtain hash value; The SimHash value of the file is obtained by weighted combination of hash values. The Hamming distance of the SimHash value is compared to determine whether the file is duplicated. The Hamming distance calculation formula is: Where a and b are two SimHash values ​​used to represent the feature code of the file; a i and b i The i-th bit of a and b respectively, corresponding to each feature in the code; It is an XOR operation, used to determine whether two features are different.

4. The intelligent file sorting method based on large model generation and understanding capability according to claim 1 is characterized in that: The business-related fields include at least one of a contract number, a business type, a project name, and a date. When extracting the business-related fields, a named entity recognition algorithm is used. The probability calculation formula of the conditional random field model of the named entity recognition algorithm is: In the formula, x is the input text, that is, the content of the file to be processed; y is the label sequence, which is used to represent the entity category corresponding to each part of the text; t k is the transfer feature function, which is used to describe the transfer relationship between labels; s l is the state feature function, which is used to describe the matching relationship between the label and the text content; k and μ l is the feature weight, which is used to adjust the importance of different features; Z(x) is the normalization factor, which is used to ensure the correctness of the probability calculation.

5. The intelligent file sorting method based on large model generation and understanding ability according to claim 1 is characterized in that: After temporarily storing the files, the server monitors the temporarily stored files. When it detects that the corresponding business data is maintained in the business system, it automatically triggers the associated sorting operation of the temporarily stored files. The monitoring process adopts the message queue mechanism. The mathematical model of the message queue mechanism is: Q(t)=Q(t-1)+I(t)-O(t) Where Q(t) is the number of messages in the message queue at time t, which reflects the current amount of tasks in the queue; I(t) is the number of messages entering the queue at time t, which is the number of newly generated monitoring tasks; O(t) is the number of messages processed at time t, which is the number of completed monitoring tasks.

6. The intelligent file sorting method based on large model generation and understanding capability according to claim 1 is characterized in that: The large model engine is a natural language processing model based on deep learning. The cross entropy loss function is used in the training process. The calculation formula is: Where N is the number of samples, that is, the number of files involved in training; C is the number of categories, that is, the number of entity types that need to be classified; y i,c is the true label of sample i, which is used to indicate the category to which the sample actually belongs; p i,c The model predicts the probability that sample i belongs to category c, which is used to measure the prediction accuracy of the model.

7. The intelligent file sorting method based on large model generation and understanding capability according to claim 1 is characterized in that: After directly associating and sorting the files with the corresponding business data, the server verifies and corrects the association results. The verification process uses the confusion matrix evaluation method. The indicator calculation of the confusion matrix evaluation method includes: Accuracy: Recall: F1 score: Where TP is the true positive, i.e., the number of files correctly identified and associated; FP is the false positive, i.e., the number of files incorrectly identified and associated; and FN is the false negative, i.e., the number of files that were not correctly identified but should actually be associated.

8. An intelligent document sorting device based on large model generation and understanding capabilities, characterized by: Including deployed on the server: The file processing module is used to receive files uploaded by users and identify and extract the plain text in the files. If the files contain pictures, the OCR technology is used to identify and extract the text in the pictures. The prompt assembly module is used to assemble the extracted plain text with the preset prompt template; The large model processing module is used to send the assembled prompt to the large model engine, extract the file type and business-related fields through the content generation capability of the large model engine, and form JSON format data; A business query module, configured to query the business system based on the extracted business-related fields; The sorting processing module is used to associate and sort the file directly with the corresponding business data if the corresponding business data is queried; if the corresponding business data is not queried, the server will temporarily store the file and associate and sort it after the business data is maintained.

9. The intelligent document sorting device based on large model generation and understanding capability according to claim 8 is characterized in that: In the prompt assembly module, the prompt template contains instruction information for instructing the large model to extract file types and business-related fields. The large model engine adopts the Transformer architecture, and the mathematical model formula is: Where Q is the query vector, which is used to query relevant information in the attention mechanism; K is the key vector, which is used to identify the characteristics of the information; V is the value vector, which is used to provide specific information content; d k is the dimension of the key vector, which is used to scale the attention scores to stabilize the training process; The large model engine is a natural language processing model based on deep learning. The cross entropy loss function is used in the training process. The calculation formula is: Where N is the number of samples, that is, the number of files involved in training; C is the number of categories, that is, the number of entity types that need to be classified; y i,c is the true label of sample i, which is used to indicate the category to which the sample actually belongs; p i,c The model predicts the probability that sample i belongs to category c, which is used to measure the prediction accuracy of the model; In the large model processing module, the business-related fields include at least one of a contract number, a business type, a project name, and a date. When extracting the business-related fields, a named entity recognition algorithm is used. The probability calculation formula of the conditional random field model of the named entity recognition algorithm is: In the formula, x is the input text, that is, the content of the file to be processed; y is the label sequence, which is used to represent the entity category corresponding to each part of the text; t k is the transfer feature function, which is used to describe the transfer relationship between labels; s l is the state feature function, which is used to describe the matching relationship between the label and the text content; k and μ l is the feature weight, which is used to adjust the importance of different features; Z(x) is the normalization factor, which is used to ensure the correctness of probability calculation; In the sorting processing module, after temporarily storing the files, the server monitors the temporarily stored files. When it detects that the corresponding business data is maintained in the business system, it automatically triggers the associated sorting operation of the temporarily stored files. The monitoring process adopts the message queue mechanism. The mathematical model of the message queue mechanism is: Q(t)=Q(t-1)+I(t)-O(t) Where Q(t) is the number of messages in the message queue at time t, which reflects the current amount of tasks in the queue; I(t) is the number of messages entering the queue at time t, which is the number of newly generated monitoring tasks; O(t) is the number of messages processed at time t, which is the number of completed monitoring tasks.

10. The intelligent document sorting device based on large model generation and understanding capability according to claim 8, characterized in that: Also includes: The file preprocessing module is used to preprocess the file. The preprocessing includes at least one of file format conversion, file deduplication and file integrity verification. The file deduplication adopts the SimHash algorithm, which is specifically: Segment the document into words and assign weights to each word; Perform hash operation on each word to obtain hash value; The SimHash value of the file is obtained by weighted combination of hash values. The Hamming distance of the SimHash value is compared to determine whether the file is duplicated. The Hamming distance calculation formula is: Where a and b are two SimHash values ​​used to represent the feature code of the file; a i and b i The i-th bit of a and b respectively, corresponding to each feature in the code; It is an XOR operation, used to determine whether two features are different; The association correction module is used to verify and correct the association results after directly associating and sorting the files with the corresponding business data. The verification process uses the confusion matrix evaluation method. The indicator calculation of the confusion matrix evaluation method includes: Accuracy: Recall: F1 score: Where TP is the true positive, i.e., the number of files correctly identified and associated; FP is the false positive, i.e., the number of files incorrectly identified and associated; and FN is the false negative, i.e., the number of files that were not correctly identified but should actually be associated.

Images