A lightweight work order service type identification model training method, identification method and device

Abstract

The application provides a lightweight work order service type identification model training method, an identification method and an apparatus. The method cleans and textually enhances and expands the sample quantity of work order sample data. On the basis of word embedding of the text content of the work order sample data, the generation date and the generation time are also vectorized to mine the relationship between the work order classification result and the generation time. After training a large-scale parameter teacher model dominated by a linear full connection layer, a student model dominated by a long short-term memory network and an attention network is trained based on a knowledge distillation method. In the parameter updating process, the hard loss of the prediction result, the distillation loss of the prediction distribution difference between the student model and the teacher model, and the intermediate feature alignment loss are combined to ensure the model recognition accuracy on the basis of obtaining a lightweight work order service classification identification model, save the computing power resources, and reduce the demand for hardware capacity.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of work order data processing, and in particular to a lightweight work order business type identification model training method and device. BACKGROUND

[0002] In the prior art, the government service platform undertakes a large amount of citizen appeal and distribution work, but relying on manual sorting and classification cannot meet the efficient processing demand. The traditional text classification method mainly includes two steps of feature extraction and machine learning model classification, which is efficient but has low accuracy and poor generalization ability. In recent years, deep learning models such as BERT represented by the Transformer model are widely used in text classification, but they require high-performance computing environments with GPUs and a large number of labeled data, which is difficult to achieve in resource-constrained application scenarios. Government work order data has the characteristics of unstructured, colloquial expression, and large topic span, and the type of work order is affected by the time of its generation, such as government work orders that are mostly concentrated in working hours on weekdays. The existing technology is difficult to fully utilize the advantages of pre-training models and combine the characteristics of government work order data to achieve efficient and accurate government text classification in resource-constrained application scenarios. Therefore, there is an urgent need for a new work order type identification scheme. SUMMARY

[0003] In view of this, the embodiments of the present application provide a lightweight work order business type identification model training method and device to eliminate or improve one or more defects in the prior art, and solve the problem that the prior art is difficult to accurately and efficiently classify government work orders in a resource-constrained situation.

[0004] Therefore, the present application provides a lightweight work order business type identification model training method, which comprises the following steps:

[0005] Data cleaning is performed on a plurality of work order sample data, and the work order sample data includes time information and text content;

[0006] The sample expansion is performed on the work order sample data in a text enhancement manner, and the work order type is added as a label of the classification task to construct a training sample set; the work order type includes government work order and non-government work order;

[0007] The text content of the work order sample data in the training sample set is processed by a preset word embedding model to obtain a first embedding vector, a second embedding vector indicating whether it belongs to a working day is obtained by looking up a table according to the generation date in the time information, and a third embedding vector indicating the formation time period is obtained by looking up a table according to the generation time in the time information; and the first embedding vector, the second embedding vector and the third embedding vector are connected as an overall embedding vector;

[0008] The training sample set is used to train a linear full connection layer dominant teacher model, the teacher model takes the overall embedding vector as input and outputs a first prediction value of the classification task, a first classification loss is constructed based on the deviation of the first prediction value and the label, and the teacher model is updated in parameters combined with a time constraint loss; the calculation formula of the time constraint loss is:

[0009] ;

[0010] wherein, denotes the time constraint loss, denotes the third embedding vector, is the Frobenius norm, is used to constrain the similarity of the third embedding vector corresponding to adjacent generation time;

[0011] The training sample set is used to train a long short-term memory network and attention network dominant student model, the student model takes the overall embedding vector as input and outputs a second prediction value of the classification task, a second classification loss is constructed based on the deviation of the second prediction value and the label, and the student model is updated in parameters combined with a distillation loss of the difference between the prediction distribution of the student model and the updated teacher model and an intermediate feature alignment loss, to obtain a work order business type recognition model.

[0012] In some embodiments, the data cleaning includes removing special characters, invalid content, duplicate text, Chinese word segmentation, and stop words;

[0013] The sample expansion is performed in a text enhancement manner for the work order sample data, including:

[0014] Based on the work order sample data, the sample expansion is performed in a synonym replacement and sentence transformation manner based on HowNet synonym dictionary;

[0015] By constructing a guide word, a preset large language model is used to generate similar sample expansion based on the work order sample data, the large language model adopts ChatGPT or Deepseek;

[0016] The word embedding model is a RoBERTa model.

[0017] In some embodiments, the teacher model includes a first full connection layer, an activation function layer, a Dropout layer, a second full connection layer, and a first softmax layer; the activation function layer adopts a Tanh function;

[0018] The student model comprises a bidirectional long short-term memory network, an attention layer, a third fully connected layer, and a second softmax layer.

[0019] In some embodiments, the output of the teacher model is:

[0020] ;

[0021] ;

[0022] wherein, represents the first predicted value; E represents the overall embedding vector, , is a parameter of the first fully connected layer, , is a parameter of the second fully connected layer; and z represents an output of the second fully connected layer.

[0023] The calculation formula of the first classification loss is:

[0024] ;

[0025] wherein, represents the first classification loss, and BCELoss is a binary cross-entropy loss function; and y represents the label.

[0026] The total loss calculation formula of the teacher model combined with the time constraint loss is:

[0027] ;

[0028] wherein, is the total loss of the teacher model, is a weight of the time constraint loss.

[0029] In some embodiments, the calculation formula of the second classification loss is:

[0030] ;

[0031] wherein, represents the second classification loss. represents the second predicted value; and y represents the label.

[0032] The calculation formula of the distillation loss is:

[0033] ;

[0034] wherein, represents the distillation loss, This represents the output probability distribution of the second fully connected layer in the teacher model. The output probability distribution of the third fully connected layer in the student model is represented by KL; KL represents the calculation of KL divergence. For fixed parameters;

[0035] The formula for calculating the intermediate feature alignment loss is:

[0036] ;

[0037] in, This represents the intermediate feature alignment loss. This represents the output feature of the Dropout layer in the teacher model. This represents the output features of the student model;

[0038] The total loss of the student model, calculated by combining the second classification loss, the distillation loss, and the intermediate feature alignment loss, is as follows:

[0039] ;

[0040] in, The total loss of the student model is represented by β and γ, which are weighting coefficients.

[0041] In some embodiments, the method further employs the Adma optimizer to update the parameters of the teacher model and the student model.

[0042] On the other hand, the present invention also provides a method for identifying work order business types, the method comprising the following steps:

[0043] Obtain the work order data to be identified, wherein the work order data includes time information and text content;

[0044] The text content of the work order data is processed through a preset word embedding model to obtain a first embedding vector. A second embedding vector is obtained by looking up the date of generation in the time information to determine whether it belongs to a workday. A third embedding vector is formed by looking up the time of generation in the time information to determine the time period. The first embedding vector, the second embedding vector, and the third embedding vector are then connected to form a whole embedding vector.

[0045] The overall embedding vector is input into the work order business type recognition model in the above-mentioned lightweight work order business type recognition model training method, and the recognition result of the work order category is output.

[0046] On the other hand, the present invention also provides a work order business type identification device, including a processor, a memory, and a computer program / instructions stored in the memory. The processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the device implements the steps of the above method.

[0047] On the other hand, the present invention also provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0048] On the other hand, the present invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0049] The lightweight work order business type recognition model training method, recognition method, and apparatus described in this invention expands the sample size by cleaning and text augmenting the work order sample data. In addition to word embedding of the text content of the work order sample data, the generation date and generation time are vectorized to explore the relationship between the work order classification result and the generation time. After training a large-scale parameter teacher model dominated by a linear fully connected layer, a student model dominated by a long short-term memory network and an attention network is trained based on knowledge distillation. During parameter update, the hard loss of the prediction result, the distillation loss due to the difference in prediction distribution between the student model and the teacher model, and the intermediate feature alignment loss are combined. This ensures the model's recognition accuracy while obtaining a lightweight work order business type classification and recognition model, saving computing resources and reducing hardware requirements.

[0050] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the description, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the description and drawings.

[0051] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0052] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, are not intended to limit the scope of the invention. In the drawings:

[0053] Figure 1 This is a flowchart illustrating the lightweight work order business type identification model training method according to an embodiment of the present invention.

[0054] Figure 2 This is a flowchart illustrating a lightweight automatic classification scheme for government work orders based on BERT, as described in another embodiment of the present invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0056] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0057] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0058] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0059] With the digital transformation of cities and the widespread deployment of government service platforms, citizens can submit various types of information, including requests, suggestions, inquiries, and complaints, through multiple channels such as telephone, apps, WeChat mini-programs, and websites. The platforms quickly and accurately distribute these textual messages to the relevant functional departments, ensuring timely responses and processing. However, with the continuous increase in citizen participation, the number of requests received daily by government service platforms is rising exponentially, and manual sorting and classification can no longer meet the demand for efficient processing. The work order text data generated by government service platforms is characterized by unstructured, colloquial expressions, and a wide range of topics. Traditional text classification methods mainly consist of two steps: first, extracting text features such as BoW, N-gram, and TF-IDF to represent the text data; and then using machine learning models such as logistic regression, support vector machines, and tree models represented by XGBoost for classification. While these methods are relatively efficient, they suffer from low accuracy and poor generalization ability. In recent years, end-to-end deep learning models, represented by the Transformer model, and pre-trained models such as BERT, have been widely used in text classification. However, pre-trained models like BERT have high requirements for hardware resources and the number of training data samples, typically requiring a high-performance computing environment with GPUs and a large amount of labeled data. Furthermore, unlike ordinary text, the time of generation of government service orders also affects the type of order. For example, government service orders are generally concentrated during working hours on weekdays, such as between 8 am and 5 pm. Therefore, in resource-constrained practical application scenarios, such as CPU environments and insufficient training samples, how to fully leverage the advantages of pre-trained models and combine them with the characteristics of government service order data to achieve efficient and accurate government text classification is the problem this invention aims to solve.

[0060] Specifically, this invention provides a lightweight work order business type recognition model training method, such as... Figure 1 As shown, the method includes the following steps S101~S105:

[0061] Step S101: Perform data cleaning on multiple work order sample data, which includes time information and text content.

[0062] Step S102: Expand the work order sample data by using text augmentation, and add work order type as a label for the classification task to build a training sample set; the work order type includes government work orders and non-government work orders.

[0063] Step S103: Process the text content of the work order sample data in the training sample set through a preset word embedding model to obtain the first embedding vector; look up the table according to the generation date in the time information to retrieve the second embedding vector indicating whether it belongs to a working day; look up the table according to the generation time in the time information to retrieve the marker to form the third embedding vector of the time period; connect the first embedding vector, the second embedding vector and the third embedding vector into the overall embedding vector.

[0064] Step S104: Train the teacher model dominated by the linear fully connected layer using the training sample set. The teacher model takes the overall embedding vector as input and outputs the first predicted value of the classification task. Construct the first classification loss based on the deviation between the first predicted value and the label, and update the parameters of the teacher model in combination with the time constraint loss.

[0065] Step S105: Train the student model dominated by the long short-term memory network and the attention network using the training sample set. The student model takes the overall embedding vector as input and outputs the second predicted value of the classification task. Construct the second classification loss based on the deviation between the second predicted value and the label. Combine the distillation loss of the difference in the predicted distribution between the student model and the updated teacher model and the intermediate feature alignment loss to update the parameters of the student model and obtain the work order business type recognition model.

[0066] In step S101, the purpose of data cleaning is to remove noise and extract clean, valid text data, laying the foundation for subsequent processing. In some embodiments, data cleaning includes: removing special characters, invalid content, duplicate text, Chinese word segmentation, and stop words.

[0067] The purpose of removing special characters and invalid content is to clean up noise in the text, making it cleaner and easier for subsequent processing. Common special characters include newline characters (\n), tab characters (\t), non-breaking spaces (\u3000), and zero-width spaces (\u200b). In addition, the work order text may contain some irrelevant information, such as "work order source," "work order progress," "message title," and "message body," which usually do not help with the classification task and also need to be removed.

[0068] For example, the original text is: "Work order source: telephone voicemail\n\nMessage title: Road repair problem\n\nMessage body: Hello, I would like to report that there are many potholes on the road near my home, which affects the safety of travel."

[0069] The text after cleaning reads: "Hello, I would like to report that there are many potholes on the road near my home, which affects travel safety."

[0070] Due to the possibility that the same user may repeatedly submit the same question or multiple users may file complaints about the same matter, there may be texts that are exactly the same or highly similar in the work orders. To reduce redundancy, improve data quality and processing efficiency, it is necessary to remove duplicate texts. For exactly duplicate texts, the duplicate items can be directly deleted. For texts that are highly similar in content, techniques such as MinHash (Minimum Hash Algorithm) can be used for deduplication.

[0071] Chinese word segmentation is to split continuous Chinese text into individual independent lexical units because Chinese expresses meaning based on words as the basic unit. Accurate word segmentation helps with subsequent text representation and analysis. Commonly used Chinese word segmentation tools include Jieba, etc., and the word segmentation mode can be selected as the "accurate mode" to ensure the maximum restoration of the user's expression intention.

[0072] Stop words refer to words that frequently appear in text but have little significance for differentiating the theme meaning of the text, such as "de", "le", "shi", "zai", "he", etc. Removing stop words can reduce the data dimension, decrease the complexity of model training, and at the same time help improve the performance and interpretability of the model. Usually, a common stop word list and a custom reserved word list are loaded to clean the word segmentation results and delete common stop words. In addition, high-frequency invalid words that appear in a large number of texts can be further removed according to the document frequency, and these high-frequency words can be extended and incorporated into the stop word list.

[0073] In step S102, the text enhancement technology can effectively alleviate the problem of insufficient training samples for government work orders. Specifically, the rule enhancement method is based on historical work order texts and uses synonym replacement and sentence pattern transformation to expand the work order texts, such as using HowNet's Chinese Thesaurus of Synonyms. In addition, using the Few-Shot text generation method, combined with Prompt design and a small amount of historical government work order texts, similar complaint texts for government work orders are generated with the help of large language models such as ChatGPT, Deepseek, Kimi, etc.

[0074] Specifically, the sample expansion is carried out by means of text enhancement for the work order sample data, including steps S1021 and S1022:

[0075] Step S1021: Based on the work order sample data, sample expansion is carried out by means of synonym replacement and sentence pattern transformation based on HowNet's Chinese Thesaurus of Synonyms.

[0076] HowNet is a large-scale Chinese lexical semantic knowledge base, and its thesaurus provides rich synonym information. By analyzing the vocabulary in the work order samples, and searching for synonyms in HowNet's thesaurus for key expressions, these synonyms are replaced. For example, "road potholes" is replaced with "road surface uneven," generating new samples in this way and enriching the text's expressive form. Based on the lexical semantic relationships and grammatical structure information provided by HowNet, the sentence structure of the work order text is adjusted. For example, "I discovered that the garbage at the entrance of the community is not cleaned up in time" is changed to "I discovered that the garbage at the entrance of the community is not cleaned up in time," further expanding the sample and enabling the model to learn the same semantic features under different sentence structures.

[0077] Step S1022: By constructing guide words, similar sample expansions are generated based on the work order sample data using a preset large language model. The large language model adopts ChatGPT or Deepseek.

[0078] Guide words are used to guide the large language model in generating text that meets the requirements. In the context of government service order text enhancement, guide words can be keywords related to the service order topic, common complaint and suggestion phrases, etc. The guide words and original service order sample data are fed into the large language model (such as ChatGPT or Deepseek). Based on the guidance words and the semantic information of the original samples, the model generates new samples that are similar to the original service orders in terms of semantics, style, etc., thereby expanding the dataset and increasing the diversity of samples.

[0079] Furthermore, step S102 defines the work order type as including government work orders and non-government work orders, thus defining the core objectives of the classification tasks. In other embodiments, government work orders can be further classified according to business domain into: government services, urban management, public services, and economic management.

[0080] Based on the level of urgency, government work orders can be further divided into: urgent work orders, relatively urgent work orders, and routine work orders.

[0081] Based on the type of complaint, government service orders can be further divided into: service quality complaints, service efficiency complaints, and facility and equipment complaints.

[0082] Based on the type of suggestion, government work orders can be further divided into: service optimization suggestions, policy formulation suggestions, and facility improvement suggestions.

[0083] Based on the scope of impact, government work orders can be further divided into: work orders with personal impact, work orders with partial impact, and work orders with widespread impact.

[0084] Based on the nature of the problem, government work orders can be further divided into: public safety, environmental protection, and social welfare.

[0085] In step S103, the text content of the work order sample data in the training sample set is processed by a preset word embedding model to obtain the first embedding vector. Here, the preset word embedding model can be a Chinese RoBERTa pre-trained model. The first token of the text (i.e., the vector corresponding to [CLS]) or the average pooling vector of the first and last tokens is extracted as the embedding vector of the main content field. According to the generation date in the time information, a table is looked up to retrieve the second embedding vector indicating whether it belongs to a working day. If it is a working day, it is marked as 1; otherwise, it is marked as 0. The corresponding embedding vector is obtained by looking up the table. According to the generation time in the time information, a table is looked up to retrieve the third embedding vector indicating the time period. The day is divided into 24 hours, and the embedding vector of the hour when the work order was generated is learned. The embedding vector of that hour is retrieved by looking up the table. Finally, the first, second, and third embedding vectors are concatenated into a whole embedding vector, which contains the features of the text content and the time information.

[0086] In step S104, the teacher model takes the overall embedding vector as input and outputs the first predicted value for the classification task. A first classification loss is constructed based on the deviation between the first predicted value and the label. The classification loss of the teacher model can be calculated using the binary cross-entropy loss function. To better learn the temporal embedding vector, a temporal constraint loss is also added. Considering that the temporal characteristics of two adjacent time slots are similar, the corresponding temporal embedding vectors should also be similar. A specific loss function is used to incorporate these temporal characteristics. The total loss of the teacher model is calculated, where the weight of the temporal constraint loss is 0.5 in this embodiment. The Adam optimizer is used to backpropagate the teacher model loss, updating the parameters of the teacher model. The optimizer learning rate can be set to 0.001, thus completing the parameter update of the teacher model.

[0087] In some embodiments, the teacher model includes a first fully connected layer, an activation function layer, a Dropout layer, a second fully connected layer, and a first softmax layer; the activation function layer uses the Tanh function.

[0088] In some embodiments, the output of the teacher model is:

[0089] ;

[0090] ;

[0091] in, E represents the first predicted value; E represents the overall embedding vector. , These are the parameters of the first fully connected layer. , represents the parameters of the second fully connected layer; z represents the output of the second fully connected layer.

[0092] The formula for calculating the first classification loss is:

[0093] ;

[0094] in, The first classification loss is represented by BCELoss, which is the binary classification cross-entropy loss function; y represents the label.

[0095] The formula for calculating the time constraint loss is:

[0096] ;

[0097] in, Indicates time-constrained loss. Represents the third embedding vector. It is the Frobenius norm. This is used to constrain the similarity of the third embedding vectors corresponding to adjacent generation times;

[0098] ;

[0099] The total loss of the teacher model, which combines the first-class loss with the time-constraint loss, is calculated as follows:

[0100] ;

[0101] in, The total loss of the teacher model, The weights for time-constrained losses.

[0102] In step S105, the training set samples are predicted using the fine-tuned teacher model to obtain the teacher model's classification score, which is then smoothed and used as the soft label for the student model. After text preprocessing, the student model randomly initializes word vectors, first passing them through a Bidirectional Long Short-Term Memory (BiLSTM) network to obtain hidden representations, and then entering an attention layer to obtain the text representation. The hard loss of the student model is calculated, which is the loss constructed by the deviation between the student model's classification score (after passing through layers to obtain the probability distribution of the student model's predicted categories) and the label. Distillation loss is calculated based on the difference between the probability distribution predicted by the student model and the teacher model's soft label, measured by KL divergence. The intermediate feature alignment loss is calculated using the cosine similarity function of the fine-tuned teacher model's text embedding vector and the student model's text representation. The total loss of the student model is obtained by weighted summing of the hard loss, distillation loss, and intermediate feature alignment loss, where the hard loss weight can be set to 0.3 and the distillation loss weight can be set to 0.7. The Adam optimizer is used to backpropagate the total loss of the student model to update the parameters of the student model. The optimizer learning rate is set to 0.001, and finally a work order business type identification model is obtained, which can predict and judge the business type of newly added government work orders every day.

[0103] In some embodiments, the student model includes a bidirectional long short-term memory network, an attention layer, a third fully connected layer, and a second softmax layer.

[0104] In some embodiments, the second classification loss is calculated as follows:

[0105] ;

[0106] in, Indicates the second category loss; y represents the second predicted value; y represents the label.

[0107] The formula for calculating distillation loss is:

[0108] ;

[0109] in, Indicates distillation loss, This represents the output probability distribution of the second fully connected layer in the teacher model. The output probability distribution of the third fully connected layer in the student model is represented by KL; KL represents the calculation of the KL divergence. These are fixed parameters.

[0110] The formula for calculating the intermediate feature alignment loss is:

[0111] ;

[0112] in, This represents the intermediate feature alignment loss. This represents the output features of the Dropout layer in the teacher model. This represents the output features of the student model.

[0113] The total loss of the student model, which combines the second classification loss, distillation loss, and intermediate feature alignment loss, is calculated as follows:

[0114] ;

[0115] in, This represents the total loss of the student model, where β and γ are weighting coefficients.

[0116] On the other hand, the present invention also provides a method for identifying work order business types, the method comprising the following steps S201~S203:

[0117] Step S201: Obtain the work order data to be identified. The work order data includes time information and text content.

[0118] Step S202: Process the text content of the work order data through a preset word embedding model to obtain the first embedding vector; look up the table according to the date of generation in the time information to retrieve the second embedding vector indicating whether it belongs to a working day; look up the table according to the time of generation in the time information to retrieve the marker to form the third embedding vector of the time period; connect the first embedding vector, the second embedding vector and the third embedding vector into a whole embedding vector.

[0119] Step S203: Input the overall embedding vector into the work order business type recognition model in the above lightweight work order business type recognition model training method, and output the recognition result of the work order category.

[0120] On the other hand, the present invention also provides a work order business type identification device, including a processor, a memory, and a computer program / instructions stored in the memory. The processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the device implements the steps of the above method.

[0121] On the other hand, the present invention also provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0122] On the other hand, the present invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0123] The present invention will now be described with reference to a specific embodiment:

[0124] This embodiment proposes a lightweight automatic classification scheme for government work orders based on BERT, which specifically includes the following modules:

[0125] Work order text preprocessing module: This module is designed for the characteristics of government work orders and performs preprocessing operations on the government work order text, including steps such as removing special characters and invalid content, removing duplicate text, Chinese word segmentation, and removing stop words.

[0126] Work order text enhancement module: For situations where the number of government work orders is relatively small, it uses rule enhancement and generation techniques based on pre-trained language models to generate artificially synthesized text.

[0127] The government work order embedding vector initialization module first uses the Chinese RoBERTa pre-trained model to obtain the embedding vector of the work order text data. Then, based on the time field generated by the work order, it determines whether it is a working day and extracts its hourly features, thereby obtaining the embedding vector of the working day feature and the hourly embedding vector respectively. These three are added together to obtain the final initial embedding vector of the government work order.

[0128] Teacher model training module: During the training phase of the teacher model, the initial embedding vectors of government work orders are input into the BERT model, and then a two-layer feedforward neural network is used to train the labeled training samples.

[0129] Student Model Training Module: During the training phase of the student model, knowledge from the BERT model is transferred to the lightweight student model through knowledge distillation. The student model uses a bidirectional long short-term memory model based on an attention mechanism.

[0130] Government work order classification prediction model: For government work orders that need to be predicted, the text is first preprocessed, and then a pre-trained student model is used for prediction.

[0131] like Figure 2 As shown, the specific implementation steps are as follows:

[0132] 1. Work order text preprocessing: Extract the main content fields from government work orders and perform preprocessing operations on the text, specifically including:

[0133] 1.1 Removal of special characters and invalid content: The original text contains a variety of special characters and symbols, such as \n, \t, \u3000, \u200b, etc. In addition, many message texts contain information such as "work order source", "work order flow", "message title", and "message body" at the beginning and end. Regular expressions and string replacement are used to remove the above noise data. After removing the above noise, work orders with less than a certain number of characters are directly rejected to improve the accuracy of subsequent processing. In this example, the number of characters is 10.

[0134] 1.2 Duplicate text removal: Since there are cases where the same user submits repeatedly or multiple people complain about the same matter, there is text that is exactly the same or highly similar in the work orders. For approximately duplicate text, the minhash technique is used for deduplication.

[0135] 1.3 Chinese text word segmentation: After completing text cleaning, the Jieba word segmentation tool is used to segment the text. The text is segmented using the "accurate mode", and the entity reserved word list is imported to ensure the maximum degree of restoring the user's expression intention.

[0136] 1.4 Stop word removal: First, load the common stop word list and the custom reserved word list, clean the word segmentation results, and delete common stop words such as "de", "le", "qingkuang", "yixia", etc. At the same time, to further improve the feature sparsity and distinguishability, by calculating the document frequency of each word, the high-frequency invalid words that appear in a large number of texts are removed. Then, the high-frequency words are extended and incorporated into the stop word list to form a reusable text preprocessing component.

[0137] 2. Work order text enhancement: In view of the situation where the number of government work orders is small, text enhancement technology is used to generate artificial synthetic texts, specifically including:

[0138] 2.1 Rule enhancement method: Based on the historical work order text, the HowNet synonym thesaurus in Chinese is used to expand the work order text by means of synonym replacement and sentence pattern transformation;

[0139] 2.2 Few-Shot text generation method: Combining Prompt design and a small amount of historical government work order text, large language models such as ChatGPT, Deepseek, Kimi, etc. are used to generate complaint texts for similar government work orders.

[0140] The design of the guiding word Prompt can refer to the example "Please imitate the complaint message styles of the following three citizens in the '12345 hotline' to generate a new citizen's government affairs appeal text. The topic should be close to public services and livelihood issues, with a natural tone, clear logic, and the content should be no less than 150 words. Example 1:..., Example 2:..., Example 3:...".

[0141] 3. Initialization of the embedded vector of government work orders:

[0142] 3.1 First, extract the main content field in the government work order, that is, the complaint text, load the output layer of the Chinese RoBERTa pre-trained model, and take the first token (the vector corresponding to [CLS]) or the average pooling vector of the first and last tokens of the text as the embedded vector of the main content field ;

[0143] 3.2 Determine whether the work order was generated on a working day based on its time. Learn the embedding vector for whether the work order was generated on a working day; if it was, set the value to 1; otherwise, set it to 0. Retrieve the embedding vector for whether it was generated on a working day using a lookup table. ;

[0144] 3.3 Divide a day into 24 hours and learn the hourly embedding vectors generated by the work orders. The hourly embedding vectors are shown here. Given a 2D matrix, each row represents the hour. Extract the hours generated by government work orders, ranging from 0 to 23, and retrieve the embedding vector for each hour using a lookup table. .

[0145] 3.4 Adding the three embedding vectors above yields the final embedding vector for the government work order, expressed as: .

[0146] 4. For the embedding vector E of the work order obtained in step (3), it first goes through a fully connected layer, then is activated by the Tanh function, then through a Dropout layer, and then through another fully connected layer to achieve the classification function, outputting the relative scores of each type of label. Finally, the scores are converted into probabilities through the Softmax function. In the implementation example of this invention, the output dimension of the last fully connected layer is 2, mainly distinguishing between government work orders and non-government work orders. The final output of the model for:

[0147] ;

[0148] ;

[0149] in, , , , These are learnable parameters.

[0150] 5. Calculate the classification loss of the teacher model. , where y is the actual label of the work order, and BCELoss is the binary cross-entropy loss function.

[0151] 6. Calculate the time constraint loss of the teacher model. To better learn the temporal embedding vector This embodiment incorporates a time-constrained loss, meaning that two adjacent time slots (such as 9:00-10:00 AM and 10:00-11:00 AM) should have similar time characteristics, and therefore their corresponding time embedding vectors should also be similar. This invention uses the following loss function. To incorporate the aforementioned time characteristics:

[0152] ;

[0153] in, This represents the time constraint loss. This represents the third embedding vector. It is the Frobenius norm. This is used to constrain the similarity of the third embedding vector corresponding to adjacent generation times;

[0154] ;

[0155] 7. Calculate the total loss of the teacher model:

[0156] ;

[0157] in, The total loss of the teacher model, As the weight for the time constraint loss, in this embodiment The value is 1.

[0158] 8. Fine-tuning the teacher model training. The Adam optimizer is used to backpropagate the loss of the teacher model, updating the parameters of the Chinese RoBERTa pre-trained model. In this implementation example, the optimizer learning rate is set to 0.001, and the number of model iterations is set to 50.

[0159] 9. Using the finely tuned teacher model, make predictions on the training set samples to obtain the classification score of the teacher model. Then, smoothing is performed as soft labeling for the student model. :

[0160] ;

[0161] in, This is the temperature coefficient, primarily responsible for controlling the degree of distillation in the model. The higher the value, the flatter the probability prediction distribution, and the richer the knowledge information the student model learns from the teacher model. In this implementation example... .

[0162] 10. Student Model Training. After text preprocessing, word vectors are randomly initialized from the training samples. These vectors are then passed through a bidirectional long short-term memory (BiLSTM) network to obtain the hidden representation h. i Then it enters an attention layer to obtain the text representation h. s The attention weights are calculated as follows:

[0163] ;

[0164] Then the text is represented by h s It is obtained from the following formula:

[0165] ;

[0166] Where W and b are the parameters that the model needs to learn.

[0167] 11. Calculate the hard loss of the student model. The text representation h s Input a fully connected layer to obtain the classification score z of the student model. student Then, after passing through a softmax layer, the probability distribution of the student model's predicted class is obtained. Then the hard loss of the student model :

[0168] ;

[0169] 12. Calculate the distillation loss in the student model. First, calculate the soft labels of the probability distribution predicted by the student model. :

[0170] ;

[0171] Calculation of distillation loss based on KL divergence :

[0172] ;

[0173] in, Distribution of teachers and students The KL divergence.

[0174] 13. Calculate the intermediate feature alignment loss between the student model and the teacher model. The finely tuned embedding vector of the teacher model text. ,but , where cosine is the cosine similarity function.

[0175] 14. Calculate the total loss of the student model. :

[0176] ;

[0177] Where β and γ are weighting coefficients. In the embodiments of this invention... , .

[0178] 15. Student Model Training. The Adam optimizer is used to optimize the total loss of the student model. Backpropagation is performed to update the parameters of the student model. In this implementation example, the optimizer learning rate is set to 0.001, and the number of model iterations is set to 50.

[0179] 16. Classification of Government Service Orders. For each new government service order added daily, it is input into a pre-trained student model for prediction to determine whether its business type is government service.

[0180] Corresponding to the above method, the present invention also provides an apparatus / system including a computer device, the computer device including a processor and a memory, the memory storing computer instructions, the processor executing the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the apparatus / system performs the steps of the method as described above.

[0181] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned edge computing server deployment method. The computer-readable storage medium can be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disks, hard disks, removable storage disks, CD-ROMs, or any other form of storage medium known in the art.

[0182] In summary, the lightweight work order business type recognition model training method, recognition method, and apparatus of this invention expands the sample size by cleaning and text augmenting the work order sample data. In addition to word embedding of the text content of the work order sample data, it also vectorizes the generation date and generation time to explore the relationship between the work order classification result and the generation time. After training a large-scale parameter teacher model dominated by a linear fully connected layer, a student model dominated by a long short-term memory network and an attention network is trained based on knowledge distillation. During parameter update, the hard loss of the prediction results, the distillation loss due to the difference in prediction distribution between the student model and the teacher model, and the intermediate feature alignment loss are combined. This ensures the model's recognition accuracy while obtaining a lightweight work order business type classification and recognition model, saving computing resources and reducing hardware requirements.

[0183] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0184] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0185] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0186] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A lightweight work order business type recognition model training method, characterized in that, The method includes the following steps: Data cleaning is performed on multiple work order sample data, which include time information and text content; The work order sample data is augmented using text enhancement, and work order type is added as a label for the classification task to construct a training sample set; the work order type includes government work orders and non-government work orders. The text content of the work order sample data in the training sample set is processed by a preset word embedding model to obtain a first embedding vector. A second embedding vector is obtained by looking up the date of generation in the time information to determine whether it belongs to a workday. A third embedding vector is formed by looking up the time of generation in the time information to form a time period. The first embedding vector, the second embedding vector and the third embedding vector are connected to form a whole embedding vector. The training sample set is used to train a teacher model dominated by a linear fully connected layer. The teacher model takes the overall embedding vector as input and outputs a first predicted value for the classification task. A first classification loss is constructed based on the deviation between the first predicted value and the label, and the teacher model's parameters are updated by incorporating a time constraint loss. The time constraint loss is calculated as follows: ； in, This represents the time constraint loss. This represents the third embedding vector. It is the Frobenius norm. This is used to constrain the similarity of the third embedding vector corresponding to adjacent generation times; ； The training sample set is used to train a student model dominated by a long short-term memory network and an attention network. The student model takes the overall embedding vector as input and outputs a second predicted value for the classification task. A second classification loss is constructed based on the deviation between the second predicted value and the label. The student model is then updated with parameters by combining the distillation loss of the difference in predicted distribution between the student model and the updated teacher model and the intermediate feature alignment loss, thus obtaining a work order business type recognition model.

2. The lightweight work order business type identification model training method according to claim 1, characterized in that, The data cleaning includes: removing special characters, invalid content, duplicate text, Chinese word segmentation, and stop words; The work order sample data is augmented using text enhancement techniques, including: Based on the work order sample data, the sample was expanded using synonym replacement and sentence transformation based on HowNet thesaurus. By constructing guiding words, similar sample expansions are generated based on the work order sample data using a preset large language model. The large language model uses ChatGPT or Deepseek. The word embedding model is the RoBERTa model.

3. The lightweight work order business type identification model training method according to claim 1, characterized in that, The teacher model includes a first fully connected layer, an activation function layer, a Dropout layer, a second fully connected layer, and a first softmax layer; the activation function layer uses the Tanh function. The student model includes a bidirectional long short-term memory network, an attention layer, a third fully connected layer, and a second softmax layer.

4. The lightweight work order business type identification model training method according to claim 3, characterized in that, The output of the teacher model is: ； ； in, Let E represent the first predicted value; let E represent the overall embedding vector. , These are the parameters of the first fully connected layer. , Here, z represents the parameters of the second fully connected layer; z represents the output of the second fully connected layer. The formula for calculating the first classification loss is: ； in, The first classification loss is represented by BCELoss, which is the binary cross-entropy loss function; y represents the label. The total loss of the teacher model, calculated by combining the first classification loss with the time constraint loss, is as follows: ； in, The total loss of the teacher model is... The weight is the time constraint loss.

5. The lightweight work order business type identification model training method according to claim 3, characterized in that, The formula for calculating the second classification loss is: ； in, This indicates the second category loss; y represents the second predicted value; y represents the label; The formula for calculating the distillation loss is: ； in, This indicates the distillation loss. This represents the output probability distribution of the second fully connected layer in the teacher model. The output probability distribution of the third fully connected layer in the student model is represented by KL; KL represents the calculation of KL divergence. For fixed parameters; The formula for calculating the intermediate feature alignment loss is: ； in, This represents the intermediate feature alignment loss. This represents the output feature of the Dropout layer in the teacher model. This represents the output features of the student model; The total loss of the student model, calculated by combining the second classification loss, the distillation loss, and the intermediate feature alignment loss, is as follows: ； in, The total loss of the student model is represented by β and γ, which are weighting coefficients.

6. The lightweight work order business type identification model training method according to claim 3, characterized in that, The method also employs the Adma optimizer to update the parameters of the teacher model and the student model.

7. A method for identifying work order business types, characterized in that, The method includes the following steps: Obtain the work order data to be identified, wherein the work order data includes time information and text content; The text content of the work order data is processed through a preset word embedding model to obtain a first embedding vector. A second embedding vector is obtained by looking up the date of generation in the time information to determine whether it belongs to a working day. A third embedding vector is formed by looking up the time of generation in the time information to determine the time period. The first embedding vector, the second embedding vector, and the third embedding vector are then connected to form a whole embedding vector. The overall embedding vector is input into the work order business type recognition model in the lightweight work order business type recognition model training method according to any one of claims 1 to 6, and the recognition result of the work order category is output.

8. A work order service type identification device, comprising a processor, a memory, and a computer program / instructions stored in the memory, characterized in that, The processor is configured to execute the computer program / instructions, and when the computer program / instructions are executed, the device implements the steps of the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method as described in any one of claims 1 to 7.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 7.

Images