Text management method and device, computer device and storage medium

Abstract

The application relates to a text management method and device, computer equipment and a storage medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: obtaining a target text, performing data preprocessing on the target text to obtain a semantic vector corresponding to the target text; classifying the target text according to the semantic vector and a trained general language model to obtain a target classification result of the text type of the target text; determining a target storage path corresponding to the target classification result according to a mapping relationship between a storage path and a classification result; and storing the target text according to the target storage path. The method can improve the document management efficiency.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a text management method, apparatus, computer device, and storage medium. Background Technology

[0002] As digital transformation continues to deepen, the number of electronic documents in enterprises is showing a rapid growth trend. Customer information, business records, policy documents and meeting minutes, which were traditionally stored in paper form, are gradually being converted into electronic storage.

[0003] In related technologies, electronic documents are usually classified and managed manually. With the rapid increase in the number of electronic documents, this requires a lot of human resources and the document management efficiency is low. Summary of the Invention

[0004] Therefore, it is necessary to provide a text management method, device, computer equipment, computer-readable storage medium, and computer program product that can improve document management efficiency in response to the above-mentioned technical problems.

[0005] Firstly, this application provides a text management method. The method includes:

[0006] Obtain the target text, perform data preprocessing on the target text, and obtain the semantic vector corresponding to the target text;

[0007] The target text is classified based on semantic vectors and a trained general language model to obtain a target classification result regarding the text type of the target text;

[0008] Based on the mapping relationship between storage paths and classification results, determine the target storage path corresponding to the target classification result;

[0009] Store the target text according to the target storage path.

[0010] In one embodiment, the construction of the trained general language model includes:

[0011] Obtain the classification training set and the question-answering training set; the classification training set includes training text and text type labels; the question-answering training set includes training questions and question answers.

[0012] The training text is input into the initial general language model, and the text type label corresponding to the training text is used as the target output to train the initial general language model and obtain the first intermediate language model.

[0013] The training question is input into the first intermediate language model, and the answer to the training question is used as the target output to train the first intermediate language model, thus obtaining the second intermediate language model.

[0014] Based on the classification validation set and the question-answering validation set, the parameters of the second intermediate language model are tuned to obtain the trained general language model.

[0015] In one embodiment, the method further includes:

[0016] Obtain user question-and-answer requests; user question-and-answer requests include question description information;

[0017] Input the problem description information into a trained general language model to obtain the target problem answer based on the problem description information;

[0018] Based on the answer to the target question, generate the user question and answer response corresponding to the user question and answer request.

[0019] In one embodiment, after storing the target text, the method further includes:

[0020] Obtain user feedback information; user feedback information includes target text identifier and target text type;

[0021] Adjust the type of the target text corresponding to the target text identifier based on the target text type.

[0022] In one embodiment, the method further includes:

[0023] When the number of target texts reaches the first threshold or the number of user feedback information reaches the second threshold, the trained general language model is updated and trained based on the target texts and user feedback information to obtain the updated general language model.

[0024] If the user traffic does not exceed the traffic threshold, the trained general language model is updated based on the updated general language model.

[0025] In one embodiment, the method further includes:

[0026] Retrieve the text query request; the text query request includes query keywords;

[0027] Generate a query index value based on the query keywords, and determine the query results based on the query index value; the query results include the target text corresponding to the query keywords;

[0028] Generate a text query response based on the query results.

[0029] Secondly, this application also provides a text management device. The device includes:

[0030] The preprocessing module is used to acquire the target text, perform data preprocessing on the target text, and obtain the semantic vector corresponding to the target text.

[0031] The classification module is used to classify target text based on semantic vectors and a trained general language model to obtain target classification results regarding the text type of the target text;

[0032] The mapping module is used to determine the target storage path corresponding to the target classification result based on the mapping relationship between the storage path and the classification result;

[0033] The storage module is used to store the target text according to the target storage path.

[0034] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0035] Obtain the target text, perform data preprocessing on the target text, and obtain the semantic vector corresponding to the target text;

[0036] The target text is classified based on semantic vectors and a trained general language model to obtain a target classification result regarding the text type of the target text;

[0037] Based on the mapping relationship between storage paths and classification results, determine the target storage path corresponding to the target classification result;

[0038] Store the target text according to the target storage path.

[0039] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0040] Obtain the target text, perform data preprocessing on the target text, and obtain the semantic vector corresponding to the target text;

[0041] The target text is classified based on semantic vectors and a trained general language model to obtain a target classification result regarding the text type of the target text;

[0042] Based on the mapping relationship between storage paths and classification results, determine the target storage path corresponding to the target classification result;

[0043] Store the target text according to the target storage path.

[0044] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0045] Obtain the target text, perform data preprocessing on the target text, and obtain the semantic vector corresponding to the target text;

[0046] The target text is classified based on semantic vectors and a trained general language model to obtain a target classification result regarding the text type of the target text;

[0047] Based on the mapping relationship between storage paths and classification results, determine the target storage path corresponding to the target classification result;

[0048] Store the target text according to the target storage path.

[0049] The aforementioned text management methods, devices, computer equipment, storage media, and computer program products first acquire the target text, preprocess the target text to obtain the semantic vector corresponding to the target text, and classify the target text according to the semantic vector and a trained general language model to obtain a classification result of the text type of the target text. Then, based on the mapping relationship between the storage path and the classification result, the target storage path corresponding to the target classification result is determined, and the target text is stored according to the target storage path. In this way, text can be automatically classified and stored according to the trained general language model, thereby improving document management efficiency. Attached Figure Description

[0050] Figure 1 This is a diagram illustrating the application environment of a text management method in one embodiment;

[0051] Figure 2 This is a flowchart illustrating a text management method in one embodiment;

[0052] Figure 3 This is a flowchart illustrating the text management method in another embodiment;

[0053] Figure 4 This is a structural block diagram of a text management device in one embodiment;

[0054] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0056] The text management method provided in this application embodiment can be applied to, for example, Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located in the cloud or on other network servers. Terminal 102 can upload target text to server 104, and server 104 can automatically classify and store the received target text. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices, such as smartwatches, smart bracelets, and head-mounted devices. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0057] In one embodiment, such as Figure 2 As shown, a text management method is provided, which can be applied to... Figure 1 Taking server 104 as an example, the following steps are included:

[0058] S202: Obtain the target text, perform data preprocessing on the target text, and obtain the semantic vector corresponding to the target text.

[0059] The target text refers to the working documents that need to be retained during the production and operation process, such as technical documents, system documents, financial reports, meeting minutes, etc., without specific limitations.

[0060] Optionally, during the text management process, the server receives target text uploaded by the user via a webpage, terminal application, or email.

[0061] Upon receiving the target text, the server first cleans it using preset regular expressions to remove meaningless content such as punctuation and special characters. Then, the server segments the cleaned target text, for example, using a general word segmentation tool to obtain the first segmentation result, and removes stop words from the first segmentation result to obtain the second segmentation result. From the second segmentation result, keywords are extracted from the target text; for example, keywords can be determined based on word frequency or part-of-speech, without specific limitations here. Finally, the target text is vectorized based on the keywords to obtain the corresponding semantic vector.

[0062] S204: Classify the target text based on the semantic vector and the trained general language model to obtain the target classification result regarding the text type of the target text.

[0063] Among them, the trained general language model can classify texts based on semantic vectors and determine the text type corresponding to the input text from a preset set of text types.

[0064] Optionally, after obtaining the semantic vector, the server inputs the semantic vector into a trained general language model, and determines the text type of the target text based on the output of the trained general language model.

[0065] S206: Determine the target storage path corresponding to the target classification result based on the mapping relationship between the storage path and the classification result.

[0066] The server pre-configures the storage path for each text type in the text type set, and this configuration can be updated according to changes in the text type set.

[0067] Optionally, after determining the target classification result of the target text, the server matches and obtains the storage path of the target text based on the target classification result of the target text and the mapping relationship between the storage path and the classification result. S208: Store the target text according to the target storage path.

[0068] Optionally, after determining the target storage path, the server stores the target text in the storage space corresponding to the target storage path.

[0069] In the above text management method, the target text is first acquired, and then preprocessed to obtain the semantic vector corresponding to the target text. The target text is then classified according to the semantic vector and a trained general language model to obtain the classification result of the text type. Then, based on the mapping relationship between the storage path and the classification result, the target storage path corresponding to the target classification result is determined. The target text is then stored according to the target storage path. In this way, the text can be automatically classified and stored according to the trained general language model, thereby improving the efficiency of document management.

[0070] In one embodiment, the construction of a trained general language model includes: obtaining a classification training set and a question-answering training set; the classification training set includes training text and text type labels; the question-answering training set includes training questions and question answers; inputting the training text into an initial general language model, using the text type labels corresponding to the training text as the target output, training the initial general language model to obtain a first intermediate language model; inputting the training questions into the first intermediate language model, using the question answers corresponding to the training questions as the target output, training the first intermediate language model to obtain a second intermediate language model; and fine-tuning the parameters of the second intermediate language model based on the classification validation set and the question-answering validation set to obtain the trained general language model.

[0071] The initial general-purpose language model can be Tsinghua GLM-6B or other applicable general-purpose language models, without any specific restrictions.

[0072] Optionally, during the training of the initial general language model, the server first acquires a classification training set and a question-answering training set. The classification training set includes multiple sets of classification training samples, each consisting of a set of training text and text type labels. The question-answering training set is constructed based on stored historical target files and includes multiple sets of question-answering training samples, each consisting of a set of question text and target answer text.

[0073] After obtaining the classification training set, the server inputs it into the initial general language model for training, obtaining text type prediction values ​​for each training text. Then, based on the text type prediction values, text type labels, and a pre-configured loss function, the model parameters of the initial general language model are adjusted, and training is repeated until a preset stopping condition is reached. When the preset stopping condition is reached, the first intermediate language model is generated based on the model parameters with the minimum loss value or the best robustness during training.

[0074] After obtaining the first intermediate language model, the server inputs the question-answering training set into it to obtain the predicted answer text for each question. Then, based on the predicted answer text, the target answer text, and the pre-configured loss function, the model parameters of the first intermediate language model are adjusted, and training is repeated until a preset stopping condition is reached. When the preset stopping condition is reached, the second intermediate language model is generated based on the model parameters with the minimum loss value or the best robustness during training.

[0075] After obtaining the second intermediate language model, the server inputs the classification validation set and the question-answer validation set into the second intermediate language model to obtain the text type prediction value corresponding to each validation text and the predicted answer text corresponding to each validation question text. Based on the consistency between the text type label and the text type prediction value of each validation text, as well as the consistency between the preset answer text and the target answer text corresponding to each validation question text, the model parameters of the second intermediate language model are adjusted to obtain the trained general language model.

[0076] In this embodiment, by acquiring a classification training set, the training text is input into the initial general language model, and the text type label corresponding to the training text is used as the target output to train the initial general language model, thereby obtaining an intermediate language model. Based on the classification validation set, the parameters of the intermediate language model are tuned to obtain a trained general language model, which can accurately classify the target text and generate relevant answers to user questions.

[0077] In one embodiment, the method further includes: obtaining a user question-and-answer request; the user question-and-answer request includes question description information; inputting the question description information into a trained general language model to obtain a target question answer about the question description information; and generating a user question-and-answer response corresponding to the user question-and-answer request based on the target question answer.

[0078] Optionally, during the question-and-answer process, the server receives the user's question-and-answer request sent by the terminal, inputs the question description information in the user's question-and-answer request into the trained general language model, obtains the target question answer corresponding to the question description information, generates a user question-and-answer response based on the target question answer, and sends the user question-and-answer response to the terminal.

[0079] In this embodiment, by first obtaining the user's question-and-answer request and inputting the question description information in the user's question-and-answer request into a trained general language model, a target question answer based on the question description information is obtained. Then, based on the target question answer, a user question-and-answer response corresponding to the user's question-and-answer request is generated. In this way, the user's question can be responded to in a timely manner, and the efficiency of information acquisition can be improved.

[0080] In one embodiment, after storing the target text, the method further includes: obtaining user feedback information; the user feedback information includes a target text identifier and a target text type; and adjusting the type of the target text corresponding to the target text identifier according to the target text type.

[0081] Optionally, after storing the target text, the server sends a text classification message to the terminal, indicating that the target text belongs to the automatically classified text type. After receiving the text classification message through the terminal's interactive interface, if the user determines that the classification result is incorrect, they can make adjustments and send user feedback information to the server through the terminal.

[0082] After receiving user feedback, the server determines the target text to be adjusted based on the target text identifier, and then adjusts the text type of the target text to the target text type.

[0083] In this embodiment, by obtaining user feedback information and adjusting the type of the target text corresponding to the target text identifier according to the target text type, it is possible to determine the text classification error through user feedback information and correct it in a timely manner, thereby improving the classification accuracy of the target text.

[0084] In one embodiment, the method further includes: when the number of target texts reaches a first quantity threshold or the number of user feedback information reaches a second quantity threshold, updating the trained general language model based on the target texts and user feedback information to obtain an updated general language model; and updating the trained general language model based on the updated general language model when the user traffic is not greater than a traffic threshold.

[0085] User traffic refers to the number of business accesses to the server's text management service.

[0086] Optionally, during text management, the server counts the number of target texts and the number of user feedback messages. When the number of new target texts reaches a first threshold, or the number of new user feedback messages reaches a second threshold, the server updates the trained general language model based on the new target texts and new user feedback messages, determines new model parameters, and recounts the number of target texts and the number of user feedback messages.

[0087] If the user traffic does not exceed the traffic threshold, the server updates the trained general language model based on the new model parameters.

[0088] In this embodiment, when the number of target texts reaches a first threshold or the number of user feedback messages reaches a second threshold, the trained general language model is updated and trained based on the target texts and user feedback messages to obtain an updated general language model. In this way, self-learning and optimization can be performed based on the target texts and user feedback messages to improve the accuracy of text classification. Furthermore, when the user traffic does not exceed the traffic threshold, the trained general language model is updated based on the updated general language model, thus achieving smooth migration and reducing the impact of the update process on users.

[0089] In one embodiment, the method further includes: obtaining a text query request; the text query request includes query keywords; generating a query index value based on the query keywords; determining a query result based on the query index value; the query result includes target text corresponding to the query keywords; and generating a text query response based on the query result.

[0090] Optionally, during the storage of the target file, the server generates a query index based on the target file. When a user queries the target file, the server obtains the text query request uploaded by the terminal, generates a query index value based on the query keywords in the text query request, determines the target text corresponding to the query keywords based on the query index value and the query index, and generates a text query response based on the target text and sends it to the terminal.

[0091] In this embodiment, by obtaining a text query request, generating a query index value based on the query keywords in the text query request, determining the query result based on the query index value, and generating a text query response based on the query result, the target text can be retrieved quickly.

[0092] In one embodiment, a text management method, taking its application in a text management system as an example, such as... Figure 3As shown, the system includes a user interface component, a file upload component, a trained GLM model, a file storage and management component, a file search and index management component, a training and update component, a user interaction component, a database management component, a load balancing component, a backup and recovery component, a monitoring and alarm component, and a data security component, among which:

[0093] The user interface component is responsible for receiving documents uploaded by users via web pages, office software embedded in company terminals, and email. During the document upload process, users can include a description of the document and training requirements, allowing the system to better understand the document's content. Documents can also be uploaded by dragging and dropping them into the chat box, making user interaction more convenient. Training results will be provided as natural language descriptions and notified to users via the interactive interface, email, etc., allowing users to further adjust the results.

[0094] The file upload component is responsible for receiving documents from the user interface, along with the document description and training requirements, and then sending the documents to the trained GLM model and the file storage and management component.

[0095] The trained GLM model is responsible for receiving documents from the file upload component, understanding and classifying the document content based on the document's content, user descriptions, and training requirements, and answering user questions. The file storage and management component is responsible for receiving documents forwarded by the file upload component and storing them based on the classification results of the trained GLM model. Furthermore, the file storage and management component also has an API interface that allows files to be uploaded to third-party file management platforms, such as WPS Cloud Platform.

[0096] The document search and index management component is responsible for creating document indexes, enabling rapid document searching, and helping users quickly locate the files they need through keyword searches. The index is updated regularly as documents are updated and added. Combined with a trained GLM model, it allows users to search for files using natural language.

[0097] The training and update component is responsible for training and updating the model based on categorized documents and user feedback. Training is triggered when a certain amount of new documents accumulate in the system or when users provide a certain amount of new feedback. The model training considers not only new data but also past training history for continuous optimization.

[0098] The user interaction component is responsible for receiving user feedback and passing it to the training and update component; converting the output of the trained GLM model into easily understandable text descriptions using natural language processing and sending them to the user; notifying the user immediately via email, SMS, or in-system notifications when model training or query tasks are completed; providing charts or other visualization elements for complex data or multidimensional results; allowing users to adjust or ask further questions after receiving preliminary results; saving the user's query history and results for comparison or further analysis; and providing options to download results in common formats (such as CSV, PDF, etc.).

[0099] The database management component is responsible for managing relational databases and vector databases. Relational databases are used to store structured data, such as user information and document classification information; vector databases are used to store unstructured data, such as the text content of documents and the training history of GLM models.

[0100] The load balancing component is responsible for distributing tasks according to the workload of each component in the system to ensure the efficient operation of the system.

[0101] The backup and recovery component is responsible for performing regular data backups and data recovery when data problems occur.

[0102] The monitoring and alarm component is responsible for monitoring the system's operational status and issuing alarms when problems occur.

[0103] The data security component is responsible for data encryption, access control, intrusion detection, audit logging, and data privacy protection.

[0104] The following provides further explanation regarding GLM model training:

[0105] 1. Data Preprocessing

[0106] (1) Text cleaning: Use regular expressions to remove non-letters, numbers and common punctuation marks.

[0107] (2) Word segmentation: Ready-made word segmentation tools such as NLTK or spaCy can be used for English word segmentation, while jieba can be used for Chinese word segmentation.

[0108] (3) Word embedding: Use GLoVe or Word2Vec for pre-training, or directly use the pre-trained word embedding of the GLM model.

[0109] 2. Model Structure

[0110] (1) Number and size of network layers: Assume 12 layers are used, with 768 neurons in each layer.

[0111] (2) Activation function: Use the ReLU (Rectified Linear Unit) activation function.

[0112] (3) Regularization method: Use L2 regularization and a dropout ratio of 0.5.

[0113] 3. Training Strategies

[0114] (1) Learning rate: Use an initial learning rate of 0.001 and apply learning rate decay.

[0115] (2) Optimizer: Use the Adam optimizer.

[0116] (3) Batch size: Use a batch size of 32.

[0117] (4) Number of training rounds: Set a maximum of 50 training cycles.

[0118] 4. Verification and Testing

[0119] (1) Data partitioning: 80% of the data is used for training, 10% for validation, and 10% for testing.

[0120] (2) Performance metrics: F1 score and accuracy are used as the main performance metrics.

[0121] (3) Early stopping strategy: If the performance does not improve within 10 consecutive validation cycles, then stop training.

[0122] 5. Model Update Strategy

[0123] (1) Triggering conditions: When the accumulated new data reaches 10,000 samples or more than 50 user feedbacks are received.

[0124] (2) Version control: Use Git for version control of the model.

[0125] (3) Smooth migration: Update the model during periods of low traffic to reduce the impact on users.

[0126] 6. Transfer learning

[0127] (1) Pre-trained model: Tsinghua GLM-6B or other applicable GLM models are used as pre-trained models.

[0128] (2) Data adaptation: Fine-tuning using a small amount of labeled domain-specific data.

[0129] (3) Data augmentation: Use back translation (translating the text into another language and then back) and synonym replacement; oversample or weight the minority class samples.

[0130] 7. Hyperparameter tuning scheme

[0131] (1) Preliminary screening: Use mini-batch data for rapid experiments and perform preliminary screening through random search.

[0132] (2) Core parameter optimization

[0133] Learning rate: Use grid search to select the optimal learning rate from the values ​​of 0.001, 0.01, and 0.1; Regularization parameters: Try L1, L2 regularization, or a combination thereof, to prevent overfitting.

[0134] (3) Model complexity adjustment: Based on the preliminary results, the number of layers or hidden units of the model needs to be fine-tuned.

[0135] (4) Advanced optimization: Based on the optimization of core parameters, Bayesian optimization is further used to find the optimal parameters.

[0136] (5) Early stopping and cross-validation: Early stopping strategy and 5-fold cross-validation are used to accurately evaluate model performance.

[0137] (6) Learning rate scheduling: Implement learning rate decay or periodic adjustment in the later stages of training.

[0138] (7) Experiment tracking and version control: Use tools such as TensorBoard or MLflow to track all experiments; use Git for parameter and model version control.

[0139] (8) Evaluation and feedback: Evaluate the model on an independent test set and conduct feedback loops with the business and R&D teams.

[0140] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0141] Based on the same inventive concept, this application also provides a text management device for implementing the text management method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more text management device embodiments provided below can be found in the limitations of the text management method described above, and will not be repeated here.

[0142] In one embodiment, such as Figure 4 As shown, a text management device is provided, including: a preprocessing module 410, a classification module 420, a mapping module 430, and a storage module 440, wherein:

[0143] The preprocessing module 410 is used to acquire the target text, perform data preprocessing on the target text, and obtain the semantic vector corresponding to the target text.

[0144] The classification module 420 is used to classify the target text based on the semantic vector and the trained general language model to obtain the target classification result about the text type of the target text.

[0145] The mapping module 430 is used to determine the target storage path corresponding to the target classification result based on the mapping relationship between the storage path and the classification result.

[0146] Storage module 440 is used to store target text according to the target storage path.

[0147] In one embodiment, the classification module 420 is further configured to obtain a classification training set and a question-answering training set; the classification training set includes training text and text type labels; the question-answering training set includes training questions and question answers;

[0148] The training text is input into the initial general language model, and the text type label corresponding to the training text is used as the target output to train the initial general language model, resulting in the first intermediate language model; the training question is input into the first intermediate language model, and the question answer corresponding to the training question is used as the target output to train the first intermediate language model, resulting in the second intermediate language model; the parameters of the second intermediate language model are tuned based on the classification validation set and the question answer validation set to obtain the trained general language model.

[0149] In one embodiment, the classification module 420 is further configured to obtain a user question-and-answer request; the user question-and-answer request includes question description information; input the question description information into a trained general language model to obtain a target question answer about the question description information; and generate a user question-and-answer response corresponding to the user question-and-answer request based on the target question answer.

[0150] In one embodiment, the storage module 440 is further configured to acquire user feedback information; the user feedback information includes a target text identifier and a target text type; and adjust the type of the target text corresponding to the target text identifier according to the target text type.

[0151] In one embodiment, the classification module 420 is further configured to update the trained general language model based on the target text and user feedback information when the number of target texts reaches a first threshold or the number of user feedback information reaches a second threshold, thereby obtaining an updated general language model; and to update the trained general language model based on the updated general language model when the user traffic does not exceed a traffic threshold.

[0152] In one embodiment, the storage module 440 is further configured to obtain a text query request; the text query request includes query keywords; generate a query index value based on the query keywords; determine the query result based on the query index value; the query result includes the target text corresponding to the query keywords; and generate a text query response based on the query result.

[0153] Each module in the aforementioned text management device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can invoke and execute the operations corresponding to each module.

[0154] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores target text and other business data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements a text management method.

[0155] Those skilled in the art will understand that Figure 5The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0156] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: acquiring target text; performing data preprocessing on the target text to obtain a semantic vector corresponding to the target text; classifying the target text according to the semantic vector and a trained general language model to obtain a target classification result regarding the text type of the target text; determining a target storage path corresponding to the target classification result according to the mapping relationship between the storage path and the classification result; and storing the target text according to the target storage path.

[0157] In one embodiment, when the processor executes the computer program, it further performs the following steps: acquiring a classification training set and a question-answering training set; the classification training set includes training text and text type labels; the question-answering training set includes training questions and question answers; inputting the training text into an initial general language model, using the text type labels corresponding to the training text as the target output, training the initial general language model to obtain a first intermediate language model; inputting the training questions into the first intermediate language model, using the question answers corresponding to the training questions as the target output, training the first intermediate language model to obtain a second intermediate language model; and fine-tuning the parameters of the second intermediate language model based on the classification validation set and the question-answering validation set to obtain a trained general language model.

[0158] In one embodiment, when the processor executes the computer program, it further performs the following steps: obtaining a user question-and-answer request; the user question-and-answer request includes question description information; inputting the question description information into a trained general language model to obtain a target question answer about the question description information; and generating a user question-and-answer response corresponding to the user question-and-answer request based on the target question answer.

[0159] In one embodiment, when the processor executes the computer program, it further performs the following steps: obtaining user feedback information; the user feedback information includes a target text identifier and a target text type; and adjusting the type of the target text corresponding to the target text identifier according to the target text type.

[0160] In one embodiment, when the processor executes the computer program, it further performs the following steps: when the number of target texts reaches a first threshold or the number of user feedback information reaches a second threshold, the trained general language model is updated and trained based on the target texts and user feedback information to obtain an updated general language model; when the user traffic is not greater than a traffic threshold, the trained general language model is updated based on the updated general language model.

[0161] In one embodiment, when the processor executes the computer program, it further performs the following steps: obtaining a text query request; the text query request includes query keywords; generating a query index value based on the query keywords; determining the query result based on the query index value; the query result includes target text corresponding to the query keywords; and generating a text query response based on the query result.

[0162] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it performs the following steps: acquiring target text; performing data preprocessing on the target text to obtain a semantic vector corresponding to the target text; classifying the target text according to the semantic vector and a trained general language model to obtain a target classification result regarding the text type of the target text; determining the target storage path corresponding to the target classification result according to the mapping relationship between the storage path and the classification result; and storing the target text according to the target storage path.

[0163] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: obtaining a classification training set and a question-answering training set; the classification training set includes training text and text type labels; the question-answering training set includes training questions and question answers; inputting the training text into an initial general language model, using the text type labels corresponding to the training text as the target output, training the initial general language model to obtain a first intermediate language model; inputting the training questions into the first intermediate language model, using the question answers corresponding to the training questions as the target output, training the first intermediate language model to obtain a second intermediate language model; and fine-tuning the parameters of the second intermediate language model based on the classification validation set and the question-answering validation set to obtain a trained general language model.

[0164] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: obtaining a user question-and-answer request; the user question-and-answer request includes question description information; inputting the question description information into a trained general language model to obtain a target question answer about the question description information; and generating a user question-and-answer response corresponding to the user question-and-answer request based on the target question answer.

[0165] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: obtaining user feedback information; the user feedback information includes a target text identifier and a target text type; and adjusting the type of the target text corresponding to the target text identifier according to the target text type.

[0166] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: when the number of target texts reaches a first threshold or the number of user feedback information reaches a second threshold, the trained general language model is updated and trained based on the target texts and user feedback information to obtain an updated general language model; when the user traffic is not greater than a traffic threshold, the trained general language model is updated based on the updated general language model.

[0167] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: obtaining a text query request; the text query request includes query keywords; generating a query index value based on the query keywords; determining the query result based on the query index value; the query result includes target text corresponding to the query keywords; and generating a text query response based on the query result.

[0168] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps: acquiring target text; preprocessing the target text to obtain a semantic vector corresponding to the target text; classifying the target text based on the semantic vector and a trained general language model to obtain a target classification result regarding the text type of the target text; determining a target storage path corresponding to the target classification result based on the mapping relationship between the storage path and the classification result; and storing the target text according to the target storage path.

[0169] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: obtaining a classification training set and a question-answering training set; the classification training set includes training text and text type labels; the question-answering training set includes training questions and question answers; inputting the training text into an initial general language model, using the text type labels corresponding to the training text as the target output, training the initial general language model to obtain a first intermediate language model; inputting the training questions into the first intermediate language model, using the question answers corresponding to the training questions as the target output, training the first intermediate language model to obtain a second intermediate language model; and fine-tuning the parameters of the second intermediate language model based on the classification validation set and the question-answering validation set to obtain a trained general language model.

[0170] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: obtaining a user question-and-answer request; the user question-and-answer request includes question description information; inputting the question description information into a trained general language model to obtain a target question answer about the question description information; and generating a user question-and-answer response corresponding to the user question-and-answer request based on the target question answer.

[0171] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: obtaining user feedback information; the user feedback information includes a target text identifier and a target text type; and adjusting the type of the target text corresponding to the target text identifier according to the target text type.

[0172] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: when the number of target texts reaches a first threshold or the number of user feedback information reaches a second threshold, the trained general language model is updated and trained based on the target texts and user feedback information to obtain an updated general language model; when the user traffic is not greater than a traffic threshold, the trained general language model is updated based on the updated general language model.

[0173] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: obtaining a text query request; the text query request includes query keywords; generating a query index value based on the query keywords; determining the query result based on the query index value; the query result includes target text corresponding to the query keywords; and generating a text query response based on the query result.

[0174] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0175] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0176] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0177] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A text management method, characterized in that, The method includes: Obtain the target text, perform data preprocessing on the target text to obtain the semantic vector corresponding to the target text, wherein the data preprocessing includes at least one of the following: regular expression cleaning, word segmentation, stop word removal, keyword extraction and vectorization processing; The target text is classified based on the semantic vector and the trained general language model to obtain a target classification result regarding the text type of the target text; Based on the mapping relationship between storage path and classification result, determine the target storage path corresponding to the target classification result; The target text is stored according to the target storage path; The construction method of the trained general language model includes: obtaining a classification training set and a question-answering training set; the classification training set includes training text and text type labels; the question-answering training set includes training questions and question answers; inputting the training text into an initial general language model, using the text type labels corresponding to the training text as the target output, training the initial general language model to obtain a first intermediate language model; inputting the training questions into the first intermediate language model, using the question answers corresponding to the training questions as the target output, training the first intermediate language model to obtain a second intermediate language model; and fine-tuning the parameters of the second intermediate language model based on the classification validation set and the question-answering validation set to obtain the trained general language model.

2. The method according to claim 1, characterized in that, The method further includes: Obtain user question-and-answer requests; the user question-and-answer requests include question description information; The problem description information is input into the trained general language model to obtain the target question answer based on the problem description information; Based on the answer to the target question, generate a user question and answer response corresponding to the user question and answer request.

3. The method according to claim 1, characterized in that, After storing the target text, the process further includes: Obtain user feedback information; the user feedback information includes the target text identifier and the target text type; Based on the target text type, the type of the target text corresponding to the target text identifier is adjusted.

4. The method according to claim 3, characterized in that, The method further includes: If the number of target texts reaches a first threshold or the number of user feedback information reaches a second threshold, the trained general language model is updated and trained based on the target texts and the user feedback information to obtain an updated general language model. If the user traffic does not exceed the traffic threshold, the trained general language model is updated according to the updated general language model.

5. The method according to claim 1, characterized in that, The method further includes: Obtain a text query request; the text query request includes query keywords; A query index value is generated based on the query keywords, and the query result is determined based on the query index value; the query result includes the target text corresponding to the query keywords; Generate a text query response based on the query results.

6. A text management device, characterized in that, The device includes: The preprocessing module is used to acquire target text, perform data preprocessing on the target text, and obtain the semantic vector corresponding to the target text. The data preprocessing includes at least one of the following: regular expression cleaning, word segmentation, stop word removal, keyword extraction, and vectorization. A classification module is used to classify the target text based on the semantic vector and a trained general language model to obtain a target classification result regarding the text type of the target text; The mapping module is used to determine the target storage path corresponding to the target classification result based on the mapping relationship between the storage path and the classification result; A storage module is used to store the target text according to the target storage path; The construction method of the trained general language model includes: obtaining a classification training set and a question-answering training set; the classification training set includes training text and text type labels; the question-answering training set includes training questions and question answers; inputting the training text into an initial general language model, using the text type labels corresponding to the training text as the target output, training the initial general language model to obtain a first intermediate language model; inputting the training questions into the first intermediate language model, using the question answers corresponding to the training questions as the target output, training the first intermediate language model to obtain a second intermediate language model; and fine-tuning the parameters of the second intermediate language model based on the classification validation set and the question-answering validation set to obtain the trained general language model.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images