Inspection auditing large model based on multi-modal supervised learning and intelligent chemical order auditing method and system

Abstract

The invention discloses an inspection and auditing large model based on multi-modal supervised learning and an intelligent chemical order auditing method, which are characterized by comprising the following steps of: performing data cleaning and data annotation on acquired power marketing inspection data to form a marketing inspection field instruction data set; the method comprises the following steps: extracting sequence features of a marketing inspection field instruction data set by using multi-modal coding, inputting the extracted sequence features into a base model taking a multi-layer Transform model as a framework for deep feature extraction, and constructing a marketing inspection large model by using a retrieval enhancement generation method; pre-training and instruction fine tuning are carried out on the constructed marketing inspection large model, unified modeling is carried out on different types of data, tasks are understood and generated, and a PPO algorithm based on different strategy optimization is adopted to adjust instruction fine tuning model parameters; and based on an improved adaptive low-rank adaptation technology, a layer weight matrix of the model part is approximated to a product of a low-rank matrix, and an API interface service is provided to carry out anomaly detection on input marketing inspection industry work order data.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and more particularly, to a marketing inspection large model based on multi-modal and supervised learning algorithm and an intelligent work order auditing method and system. BACKGROUND

[0002] With the advancement of digital transformation, as the all-around supervision and control of marketing business, the inspection work urgently needs to rely on informationization and intelligent means to promote its high-quality development and adapt to the needs of the new era.

[0003] In recent years, with the rapid development of power marketing business, the types of business involved and the amount of data have increased dramatically, and the policy specifications for inspection have also evolved continuously. Inspectors not only need to be familiar with the latest inspection specifications, but also need to master the constantly increasing, iterative marketing business policies, industry standards and regulations. The traditional marketing inspection work mainly carries out inspection work through online distribution of inspection work orders. Due to the large amount of work orders and extensive tasks, manual auditing has problems such as low efficiency and insufficient accuracy. In addition, individual differences, lack of experience and subjective judgment of inspectors may lead to deviations in accuracy and fairness during the auditing process. In order to solve the above problems, improve work efficiency and reduce potential errors, it is urgent to introduce informationization and intelligent means to promote the efficient and accurate development of power marketing inspection.

[0004] In recent years, machine learning technology has made significant progress in marketing inspection. For example, existing technology proposes the necessity of applying power big data technology to power marketing audit, especially for anomaly detection in power data, which can effectively improve the quality of power marketing audit. Existing technology suggests using big data algorithms to analyze customer illegal electricity use behavior in power marketing inspection. In addition, existing technology designs an intelligent power marketing audit model based on knowledge graph, which solves the limitations of traditional audit models. Existing technology focuses on marketing audit label library technology based on multi-data fusion, emphasizing the importance of data integration and mining for efficient audit work. Although existing research has made some progress in the intelligent auditing technology of power marketing inspection work orders, there are still some challenges. The current models generally have small scale and overfitting problems, and cannot fully utilize the complexity of multi-modal data. Small models can only handle local information and are difficult to cope with complex and variable power marketing environments, and their generalization ability is weak, which can easily lead to performance degradation when dealing with heterogeneous data. In addition, the existing small models have poor robustness in data noise and deformation, which limits their universality and application effect in practical scenarios.

[0005] With the rapid development of the power industry, marketing inspection work has gradually accumulated massive data containing rich information. Massive information promotes the scale of artificial intelligence technology model from quantitative change to qualitative change, and promotes the transformation of artificial intelligence technology in the power field from small model technology to large model. Large language model (LLM) refers to a neural network model with more than 100 million parameters based on the core architecture of Transformer, which is trained on massive text data in an unsupervised manner. It has strong multi-modal learning ability and 100 million data fitting ability in processing natural language tasks. Large models have significant advantages in natural language processing, especially in zero-shot learning, few-shot learning, and noisy data processing. They can effectively improve the generalization ability of models and solve the limitations of small models in handling complex data. Large models have shown great potential in multiple fields and cross-scenario applications. For example, the prior art proposes a cloud ERP community domain problem classification method based on BERT-TextCNN, which inputs cloud community problem text vectors into a pre-trained model to extract deep features. However, this method lacks pre-training of power professional knowledge, and has problems such as insufficient model generalization and poor domain adaptability. The prior art developed the first financial large model in the industry, which can provide customized financial professional services for different enterprises and promote the development of financial intelligence. The prior art developed a customized Chinese legal large model LawLLM, which provides intelligent services in deep application scenarios such as legal information extraction, judgment prediction, and super-long judgment processing for super-long text processing in the legal field. In the power grid field, small model artificial intelligence represented by deep learning and reinforcement learning has been widely applied, but large model technology is still in its infancy and needs to be developed and popularized.

[0006] To solve the above problems, there is an urgent need for a marketing inspection large model based on multi-modal and supervised learning algorithm and an intelligent work order auditing method and system. SUMMARY

[0007] To solve the problems in the prior art, the present application provides a marketing inspection large model based on multi-modal and supervised learning algorithm and an intelligent work order auditing method and system.

[0008] The present application adopts the following technical solutions.

[0009] The first aspect of the application relates to a multi-modal supervised learning-based inspection audit large model and an intelligent work order audit method, the method comprising the following steps: collecting power marketing inspection data by a marketing business support system, performing data cleaning and data labeling on the collected power marketing inspection data, constructing an inspection field instruction data set, and inputting a model pre-training and fine-tuning module; fully extracting sequence features of the inspection field instruction data set using multi-modal encoding, inputting the extracted sequence features into a base model with a multi-layer Transformer model as the architecture for deep feature extraction, introducing domain expert knowledge through a retrieval enhancement generation method to construct a marketing inspection large model; pre-training and instruction fine-tuning the constructed marketing inspection large model using the inspection field instruction data set, performing unified modeling, understanding and generation tasks on text, time sequence and other data types in the power marketing inspection scene, and adjusting the instruction fine-tuning model parameters using a PPO algorithm based on heterogeneous strategy optimization; based on the improved adaptive low-rank adaptation technology, the model part layer weight matrix is approximated as the product of a low-rank matrix, and an API interface service is provided to perform abnormality detection on the input marketing inspection industry work order data.

[0010] The marketing business support system collects power marketing inspection data, performs data cleaning and data labeling on the collected power marketing inspection data, and constructs an inspection field instruction data set, which is input into a model pre-training and fine-tuning module, including: obtaining marketing inspection industry global data measured by power detection equipment, the marketing inspection industry global data being multi-source multi-modal information including inspection work orders, regulations and systems, policy reports, and work order daily quantities; preprocessing the collected global data, and obtaining power marketing inspection instruction fine-tuning data set using data labeling.

[0011] Fully extracting sequence features of the inspection field instruction data set using multi-modal encoding, including: performing key feature extraction, feature alignment and feature reconstruction on multi-modal data including marketing inspection industry work order data, regulations and systems, policy reports, and work order daily quantities through a multi-modal encoding-decoding structure.

[0012] Through a multimodal encoding-decoding structure, key features are extracted, aligned, and reconstructed for multimodal data including work order data, rules and regulations, policy reports, and daily work order quantities in the marketing audit industry. This includes: using a large model's word segmenter to segment the original text of unstructured text data including audit work orders and rules and regulations, constructing a numerical vector representation of each word according to the vocabulary, and extracting text features through a text encoder; dividing structured time series data including daily and monthly work order quantities into multiple patches, converting time series features into text features through a multi-head attention mechanism, and using a linear mapping layer to output a time series representation with the same dimension as the text vector; using a multimodal encoder to map different modal data into a unified feature space, and using a fully connected layer to map different modal data in the unified feature space to the text embedding space of the large model.

[0013] The extracted sequence features are input into a base model based on a multi-layer Transformer model for deep feature extraction. Domain expert knowledge is introduced through a retrieval-enhanced generation method to construct a marketing audit model, including: using a multi-layer Transformer architecture to perform deep nonlinear operations on multimodal features to achieve deep feature representation of global marketing audit data; building a multimodal decoder to reconstruct text and time series data of the marketing audit industry to obtain multimodal output; using retrieval-enhanced generation technology to externally expand the knowledge base of the marketing audit model, and automatically expand the prompt words that match the problem without updating the parameters of the large model.

[0014] The constructed marketing audit model is pre-trained and fine-tuned using the audit field instruction dataset, and unified modeling, understanding and generation tasks are performed on text, time series and other data types in the power marketing audit scenario, including: using the base model to perform deep feature representation of unified multimodal features, and reconstructing the original input output through the trained text decoder and time series decoder.

[0015] Based on the improved adaptive low-rank adaptation technology, the weight matrices of some layers of the model are approximated as the product of low-rank matrices, including: adaptive low-rank adaptation technology improved based on singular value decomposition, freezing the remaining parameters of the large model, and only updating the Transformer network bias parameters or linear layer parameters.

[0016] Based on the improved adaptive low-rank adaptation technology, the weight matrices of some layers of the model are approximated as the product of low-rank matrices, and an API interface service is provided to perform anomaly detection on the input marketing audit industry work order data, including: calling the marketing audit big model API to generate audit work order data, using power marketing audit work order data to test the model, and performing intelligent judgment and analysis of abnormal work orders.

[0017] The second aspect of the application relates to a multi-modal supervised learning-based inspection and review large model and an intelligent work order review system using the method in the first aspect of the application, the system comprising a multi-modal instruction data set construction module, an inspection and review large model research and development module, a model pre-training and fine-tuning module, and a lightweight deployment and application module; the multi-modal instruction data set construction module collects power marketing inspection data from a marketing business support system, performs data cleaning and data labeling on the collected power marketing inspection data, and forms an inspection field instruction data set, which is input into the model pre-training and fine-tuning module; the inspection and review large model research and development module fully extracts sequence features of the inspection field instruction data set using multi-modal coding, inputs the extracted sequence features into a base model with a multi-layer Transformer model as the architecture for deep feature extraction, introduces field expert knowledge through a retrieval enhancement generation method, and constructs a marketing inspection large model; the model pre-training and fine-tuning module pre-trains and fine-tunes the constructed marketing inspection large model using the inspection field instruction data set, uniformly models, understands and generates tasks for text, time sequence and other data types in the power marketing inspection scene, and adjusts instruction fine-tuning model parameters using a PPO algorithm based on heterogeneous strategy optimization; the lightweight deployment and application module approximates part of the layer weight matrix of the model to the product of a low-rank matrix based on an improved adaptive low-rank adaptation technology, provides API interface services, and performs abnormality detection on input marketing inspection industry work order data.

[0018] The third aspect of the application relates to a terminal comprising a processor and a storage medium; the storage medium is used to store instructions; the processor is used to operate according to the instructions to perform the steps of the method in the first aspect of the application.

[0019] The fourth aspect of the application relates to a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method in the first aspect of the application.

[0020] The beneficial effects of the present application are that, compared with the prior art, the marketing inspection large model based on multi-modal and supervised learning algorithm and intelligent work order auditing method and system in the present application fuse the global data of the power marketing inspection industry through multi-modal data, and perform data labeling to make instruction fine-tuning data set, provide large-scale professional field data support for model pre-training and fine-tuning, adopt multi-modal encoder to extract key features of different modal data, perform feature alignment to convert into unified sequence form, further construct a base model with multi-layer Transformer architecture as the core, introduce expert knowledge into the model performance through the retrieval augmented generation (RAG) module, use the pre-training-fine-tuning paradigm of the large model, adopt autoregressive prediction optimization target on the multi-modal feature sequence, combine supervised instruction fine-tuning and reinforcement learning algorithm to enhance the ability of the model to understand and follow human instructions in specific scenarios, align the output to human values and preferences through human feedback, improve the quality and reliability of the model answer, compress the parameter update amount in the model inference process through the low-rank adaptation (LoRA) technology based on singular value decomposition improvement, realize the lightweight deployment of the large model, and provide API interface service to realize the calling of the inspection auditing large model and intelligent work order auditing.

[0021] The beneficial effects of the present application also include: 1. The inspection auditing large model based on multi-modal supervised learning and intelligent work order auditing method provided by the present application solves the technical problems such as abnormal checking of marketing inspection work orders in the power industry, realizes effective detection and identification of abnormal work orders, has high operation efficiency, high detection precision, can fully capture work order text features, etc. The present application can provide a method for marketing inspection work order auditing, provide ideas and theoretical basis for future power system control, and the ultimate goal is to accurately control the inspection work order, determine whether it meets the industry standards, realize the intelligent level of marketing inspection work order quality inspection and trend prediction, and has certain reference value and actual economic benefits.

[0022] 2. The base model of the Transformer architecture based on the attention mechanism is pre-trained and fine-tuned on the global data set in the field of power marketing inspection to realize the development of the marketing inspection large model, solves the problems of too large difference between general data distribution and multi-modal data in the inspection field and low professional degree. At the same time, the retrieval augmented generation is introduced to introduce professional power knowledge, and the perplexity score of the large language model is used as a supervision signal to fine-tune the retriever parameters, solves the dynamic adaptation problem of the large model in the marketing inspection field, improves the inference ability of the large model in complex downstream tasks, and enhances the generalization of the model.

[0023] 3. Through multi-modal encoding-decoding operation on unstructured text such as inspection work order, policy report and structured time series data set such as daily work order quantity, the multi-head attention mechanism is used to convert the time series characteristics into text characteristics, enhance the cross-modal and reasoning ability of the model to the time series data, and make work order data multi-dimensional evolution trend prediction in time domain scale to meet the needs of marketing policy changes and business sustainable development, help continuous optimization of inspection business process, improve the intelligent level of marketing inspection, and develop more accurate and effective inspection strategies.

[0024] 4. By introducing a low-rank adaptation method based on singular value decomposition, the problem of model performance and computational efficiency decline caused by fixed rank of parameter matrix of each layer of LoRA is solved. In addition, the redundant LoRA rank will cause the degradation of model performance and efficiency, and the importance of weights will differ in different layers of the Transformer model during fine-tuning. The parameter update matrix is singular value decomposed, and the rank of each layer is dynamically adjusted through singular value pruning during training, so that the expression ability of different layers is flexibly optimized according to the actual task requirements. This method effectively improves the parameter efficiency and task adaptability of model fine-tuning, reduces the model storage occupation and computational overhead without sacrificing the overall performance of the model. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 A schematic diagram of the inspection and audit large model based on multi-modal supervised learning and the intelligent work order audit method of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical scheme and advantages of the present application clearer and more accurate, the technical scheme of the present application is described in detail through multiple specific embodiments below. The embodiments adopted by the present application are only used to explain the present application and are not used to limit the content of the present application.

[0027] In the first aspect of the present application, an inspection and audit large model based on multi-modal supervised learning and an intelligent work order audit method are provided, and the method comprises the following steps: Step 1: Collecting power marketing inspection data from the marketing business support system, and performing data cleaning and data labeling on the collected power marketing inspection data to form an inspection field instruction data set and input the model pre-training and fine-tuning module.

[0028] The marketing inspection large model and intelligent work order audit method based on multi-modal supervised learning need to collect data information related to industry work orders, mainly including the following data, namely rules and regulations, inspection work order, policy report, and time series data. For example Figure 1The multi-modal supervised learning-based marketing inspection large model and the intelligent work order auditing method shown comprise industry inspection global data acquisition, data preprocessing, model development, model expansion, model pre-training, model fine-tuning, model deployment, and abnormal work order identification and analysis.

[0029] The industry inspection global data acquisition needs to pull the unstructured text data related to the industry, including regulations, inspection work orders, policy reports, and structured time series data such as daily work order quantity; the data preprocessing includes low-quality filtering, redundancy removal, privacy elimination, and data labeling.

[0030] First, collect industry-wide data for marketing inspection, including inspection work orders, regulations, policy reports, and daily work order quantities, then perform a series of preprocessing steps on the collected unstructured text information, including low-quality filtering, redundancy removal, and privacy elimination, and finally provide a scenario-rich inspection field instruction dataset for downstream tasks through data labeling. Low-quality filtering refers to deleting low-quality content from the collected data. By using a linear classifier based on feature hashing to evaluate the text, the classifier is trained on high-quality text to identify and remove low-quality data, achieving data set quality optimization and providing more reliable input for subsequent model training. Redundancy removal aims to identify and remove duplicate content at different granularities (such as sentences, paragraphs, and documents). The presence of duplicate information may affect the diversity of training data, further increasing the instability of the training process. Therefore, the de-duplication operation is crucial to ensure the richness of the data set and the stability of the model. Privacy elimination mainly involves removing any data that may contain personal sensitive information from the dataset, such as user names, addresses, and phone numbers. To reduce the risk of privacy leakage, these sensitive data will be deleted to ensure that the data processing process complies with privacy protection regulations and safeguards data security. Data labeling aims to build an instruction fine-tuning dataset and fine-tune the model on this dataset to adapt to downstream tasks. Typically, instruction instances consist of instructions (task descriptions), input-output pairs, questions, and answers. By using templates to convert labeled natural language datasets into instruction format <input, output> pairs, structured data is provided for model fine-tuning.

[0031] Model development uses a multi-modal encoder to extract key features from input data and align multi-modal features, converting them into a unified sequence representation. The model base is a Transformer with an attention mechanism at its core.

[0032] Step 2, use multi-modal encoding to fully extract sequence features of the inspection field instruction dataset, and input the extracted sequence features into a base model with a multi-layer Transformer model architecture for deep feature extraction. By introducing domain expert knowledge through the retrieval enhancement generation method, the marketing inspection large model is constructed.

[0033] A large-scale power inspection auditing model is constructed to provide a unified and general solution for multi-modal and multi-scenario task processing in the field of power marketing inspection.

[0034] The process mainly involves converting data of different modalities into a unified feature space, and then implementing deep extraction and fusion of features through a Transformer-based architecture. Specifically, first, various forms of data such as text, time series data, etc. are converted into a unified sequence representation through a multi-modal encoder, and the features of these data are aligned. Then, based on these multi-modal data, a Transformer architecture based on a self-attention mechanism is used for deep modeling to extract key features from various data. Finally, the multi-modal decoder is used for feature reconstruction to ensure that text and time series data can be efficiently processed and obtain accurate output.

[0035] The multi-modal encoder is used to extract features from the input multi-modal data and align the data of different modalities in a unified feature space, providing a unified input representation for subsequent deep modeling. For input text data, a WordPiece-based tokenizer is used to split it into token sequences, and a large model word embedding matrix is used to map it to a numerical vector representation. Each text token is mapped to an embedding vector , which constitutes the text embedding , where M is the total number of tokens. Time series data usually has strong local stationarity, and sliding window technology is used to divide it into several patches, and each patch is treated as a time series token for processing. These time series features are converted into time series embeddings , where P is the length of each window, , and the embedding space dimension of the large model. Since there are modal differences between text data and time series data, the multi-head attention mechanism is used to convert time series features into text features to enhance the cross-modal understanding and reasoning ability of the large model.

[0036] The multi-head self-attention mechanism (MHSA) is used to fuse cross-modal features. The query matrix of the time series features is defined as , the key matrix , and the value matrix , where E is the embedding matrix of the large model. The attention score matrix S is calculated to fuse time series information and text information:

[0037] Further, a fully connected layer The fused cross-modal features are mapped to the text embedding space of the large model. Through the above multi-modal feature extraction and alignment steps, the model can effectively process data from different modalities such as text and time series, and unify them into the same feature space.

[0038] The GPT2-based deep feature extraction uses a GPT-2 model based on the Transformer architecture to perform deep feature extraction on multi-modal features. The GPT-2 model is stacked with multiple Transformer decoders, where each layer of decoder consists of multi-head self-attention mechanism (MHSA), feed-forward neural network (FFN), and layer normalization (LN).

[0039] First, define the input representation of the model as , which is the concatenation of text and time series embeddings. Through each layer i of the model, the input is processed through the multi-head self-attention module, layer normalization, and feed-forward neural network in turn to generate the next layer input representation . The mathematical expression of this process is:

[0040]

[0041] Through the step-by-step stacking of multiple Transformer decoders, the model can extract rich contextual information from the input multi-modal features and form deep feature representations.

[0042] Finally, after N layers of processing, the output representation of the model is which will be used for subsequent decoding tasks, and its formal expression is: where is the intermediate state after the multi-head self-attention mechanism and the first normalization layer.

[0043] The model reconstructs the features through the multi-modal decoder to ensure that the output text and time series data can be restored to the same format as the input data. The decoding process includes two main modules: the text decoder and the time series decoder. Each decoder restores the original data by mapping its input features back to the target data space. The text decoder is based on the Transformer decoder structure and generates text features step by step. We define the input of the text decoder as , and the generation formula of the th step is where is the output of the previous time. The text decoder uses a masked multi-head self-attention mechanism to ensure that each generated token can only rely on previous tokens. This process can be represented as:

[0044] ( )

[0045] wherein denotes the hidden layer state of the step layer, denotes the intermediate state after the multi-head self-attention layer and the first normalization layer of the step, denotes a vocabulary mapping matrix, denotes the size of the vocabulary, denotes a bias term.

[0046] Considering the local stationarity and long-term dependency of time series, the time series decoder combines causal convolution operations to capture temporal dependencies. Specifically, the input of the time series decoder is , and its output is processed by an expanded causal convolution: , wherein the convolution operation captures local temporal patterns through a convolution kernel of size 7. To enhance the expressive power of the time series decoder, the local patterns and the output of the time series encoder are fused by combining a gating mechanism:

[0047] wherein , denotes the Hadamard product, denotes the concatenation operation, is a sigmoid gating function, denotes the local contextual features extracted by the time series decoder at the current time step t, denotes the global contextual representation output by the time series encoder, which is finally mapped back to the original time series through a linear layer: . Based on the retrieval-enhanced generation of expert knowledge injection, the retrieval-enhanced generation technology is introduced, which expands the knowledge boundary of the model through an external knowledge base, enhances the professionalism and timeliness of the generated model without modifying the parameters of the large model. The retriever encodes the query and retrieves the relevant documents from the external document library, and uses cosine similarity to evaluate the similarity between the query and the document: , wherein is the query embedding, d is the document embedding. The retriever is fine-tuned so that it can retrieve documents that effectively reduce the perplexity of the large model. Specifically, the optimization objective of the retriever is to minimize the following KL-divergence loss:

[0048]

[0049]

[0050] where, is the document distribution generated by the retriever, is the probability distribution output by the large model, and the minimization target of the KL-divergence is to make the retrieved documents and the output generated by the model more consistent. and are model parameters.

[0051] In the retrieval stage, the retriever selects the top documents that are most relevant to the query and inputs them to the large model together with the query. The final generated prediction probability is the weighted average of all documents and queries:

[0052] where is the probability output by the large model given the query and the document.

[0053] Model expansion integrates domain expert knowledge by integrating retrieval enhancement generation modules and fine-tunes the retriever parameters using the perplexity score, freezes the large model parameters to expand the knowledge boundary of the large model and alleviate the large model hallucination phenomenon.

[0054] Step 3, use the inspection field instruction data set to pre-train and fine-tune the constructed marketing inspection large model, and use the PPO algorithm based on heterogeneous strategy optimization to adjust the instruction fine-tuning model parameters. The text, time sequence and other data types in the power marketing inspection scene are uniformly modeled, understood and generated, and the PPO algorithm based on heterogeneous strategy optimization is used to adjust the instruction fine-tuning model parameters.

[0055] The process of pre-training the power multi-modal inspection and review large model on the global data set in the power inspection field aims to realize unified modeling, understanding and generation of multi-modal data (including text, time sequence, etc.) in the power inspection scene. Specifically, this step contains two main sub-steps: training multi-modal segmentation encoder and reconstruction module and pre-training power inspection and review large model. Through the above two steps, the model can process data from different modalities, and through the large model, it can extract deeper feature representations from the data, realizing deep semantic understanding of work order text.

[0056] The training of the multi-modal segmentation encoder and the reconstruction module first preprocesses the multi-modal data (including text and time series data), converts the text data into word embeddings in a high-dimensional vector space using segmentation techniques and embedding methods, and segments the time series data. The training process of the encoder can be formally represented as follows: wherein is the input data token, is the feature vector after encoder processing. The role of the reconstruction module is to restore the extracted feature vector to the original sequence. The training process of the decoder is as follows: wherein is the reconstructed data sequence. The reconstruction module is trained by minimizing the reconstruction error, wherein the loss function combines the cross-entropy loss of the text data and the mean square error loss of the time series data:

[0057] is the error loss weight, is the model prediction.

[0058] Through this collaborative optimization process, it is ensured that the model not only effectively extracts multi-modal features, but also maintains the structure and information of the data in the reconstruction process, thereby improving the performance of the model in downstream tasks.

[0059] Model pre-training first trains the segmentation encoder and decoder on the input inspection global dataset, encodes the multi-modal data into continuous feature vectors, and uses the decoder to reconstruct them; then pre-train the marketing inspection large model, apply a unified autoregressive prediction optimization objective to the multi-modal interleaved sequence to achieve unified multi-modal data understanding and generation tasks.

[0060] Pre-training the power inspection audit large model, the power marketing inspection large model is unsupervised trained on large-scale high-quality power marketing inspection global dataset. Specifically, the unified feature sequence obtained after inputting multi-modal data after segmentation encoding is subjected to deep feature extraction by the GPT-2 model, undergoes complex nonlinear changes, and then extracts higher-level abstract features. Then, these feature sequences are reconstructed by the multi-modal decoder trained above. The model uses a unified autoregressive prediction for optimization to achieve unified multi-modal data understanding and generation tasks.

[0061] The model was fine-tuned using instruction fine-tuning and an improved PPO algorithm on the instruction fine-tuning dataset. This combination of instruction fine-tuning and the improved PPO algorithm aims to enhance the pre-trained marketing audit model's ability to understand and execute natural language processing task instructions, while ensuring that the model's output aligns with human values ​​and preferences. This process optimizes model behavior by combining supervised fine-tuning, reward model training, and PPO fine-tuning based on off-policy optimization, resulting in more accurate and reliable performance in practical tasks in the field of power marketing auditing.

[0062] To fine-tune the pre-trained model, we first performed supervised fine-tuning on a dataset of manually labeled audit-related instructions. This dataset contains specific task instructions and their correct outputs. The model learns the mapping between these instructions and outputs, improving its adaptability to specific tasks. During fine-tuning, the model adjusts parameters by minimizing the loss between the instruction output and the true label. The mathematical expression for this is:

[0063] in, is the predicted output generated by the model at time step t, The model is based on the input instructions and model parameters Generate predicted probability, T is the length of the output sequence; Use command fine-tuning Model training reward model To evaluate the quality and conformity of the model's generated output. The output value of the reward model reflects the degree of match between the generated results and human expectations. The higher the reward, the more consistent the model's output is with human preferences and values. The training process of the reward model is based on a manually annotated dataset and uses a loss function improved by imitation learning. It calculates the difference between the generated output and the expected output and introduces an autoregressive language model loss to the output for optimization. The loss function used in reward model training is:

[0064] in Represents the empirical distribution of the training data set, In the model, Represents a given input prompt and preferred output Likelihood probability under the conditions of . and is the loss weight, To reject the output, is the Sigmoid function.

[0065] The PPO fine-tuning instruction fine-tunes the marketing inspection large model based on heterogeneous policy optimization, so that the model generates output that conforms to human preferences and values when given task instructions, thereby improving the reliability and generalization ability of the model. PPO algorithm is a reinforcement learning method based on heterogeneous policy optimization, which limits the amplitude of each policy update to maintain the stability of training, and uses importance sampling to evaluate the difference between the current policy and the old policy. The basic form of the policy gradient of the PPO algorithm is as follows:

[0066]

[0067] Model fine-tuning combines supervised instruction fine-tuning and PPO algorithm with heterogeneous policy optimization to enhance the model's ability to understand and follow human instructions in specific scenarios, while aligning the output to human values and preferences through human feedback, improving the quality and reliability of the model's answers.

[0068] Step 4: Based on the improved adaptive low-rank adaptation technology, the model part layer weight matrix is approximated as the product of low-rank matrices, and API interface services are provided to detect abnormalities in input marketing inspection industry work order data.

[0069] The low-rank adaptation method improved based on singular value decomposition compresses the parameter quantity updated when the model adapts to downstream tasks, and realizes the strong quantization deployment and application of the inspection and audit large model by calling the API interface.

[0070] The low-rank adaptation method improved based on singular value decomposition compresses the parameters of the model to reduce the storage and computing requirements of the model. Specifically, the weight matrix is decomposed using singular value decomposition, and then the rank of the LoRA module in different layers is dynamically adjusted according to the gradient information. The parameter update process can be formalized as:

[0071] where, and are orthogonal, is a diagonal matrix containing singular values. In the actual training process, the importance of singular values is adjusted according to the gradient change of weights. By calculating the contribution degree of each singular value and pruning the singular values according to the size of the contribution degree, dynamic rank adjustment is realized. The goal of this process is to minimize redundant parameter updates while maintaining the effectiveness and robustness of the model.

[0072] The inspection work order data is applied to downstream task application by calling the marketing inspection large model API. By testing the model by using the power marketing inspection work order data, intelligent research and analysis of abnormal work orders are performed, so as to help business personnel to more efficiently identify potential abnormal or irregular work orders, and significantly improve the single auditing efficiency.

[0073] The abnormal work order identification research result analysis is based on the work order text of the power industry user, and outputs the classification and abnormal check result. The probability of 1 indicates that the work order meets the industry regulations, and the lower the probability, the more the model judges that the work order does not meet the industry regulations.

[0074] In the second aspect of the present application, an inspection and audit large model based on multi-modal supervised learning and an intelligent work order audit system are provided. The system is implemented by the method of the first aspect of the present application. The system comprises a multi-modal instruction data set construction module, an inspection and audit large model research and development module, a model pre-training and fine-tuning module, and a lightweight deployment and application module. The multi-modal instruction data set construction module collects power marketing inspection data from a marketing business support system, performs data cleaning and data labeling on the collected power marketing inspection data, and forms an inspection field instruction data set, which is input into the model pre-training and fine-tuning module. The inspection and audit large model research and development module fully extracts the sequence features of the inspection field instruction data set using multi-modal coding, and inputs the extracted sequence features into a base model with a multi-layer Transformer model architecture for deep feature extraction. The marketing inspection large model is constructed by introducing domain expert knowledge through a retrieval enhancement generation method. The model pre-training and fine-tuning module pre-trains and fine-tunes the constructed marketing inspection large model using the inspection field instruction data set, and performs unified modeling, understanding and generation tasks on text, time sequence and other data types in the power marketing inspection scene, and adjusts the instruction fine-tuning model parameters using the PPO algorithm based on the heterogeneous strategy optimization. The lightweight deployment and application module approximates part of the layer weight matrix of the model to the product of low-rank matrices based on the improved adaptive low-rank adaptation technology, and provides API interface services for abnormal detection of input marketing inspection industry work order data.

[0075] In the third aspect of the present application, a terminal is provided, which comprises a processor and a storage medium. The storage medium is used to store instructions, and the processor is used to operate according to the instructions to perform the steps of the method of the first aspect of the present application.

[0076] In the fourth aspect of the present application, a computer readable storage medium is provided, which stores a computer program. When the program is executed by a processor, the steps of the method of the first aspect of the present application are implemented.

[0077] It should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present application but not to limit the present application. Although the present application has been described in detail with reference to the above embodiments, it should be understood by those skilled in the art that the technical solutions of the present application still include the modifications or equivalent replacements of the specific embodiments of the present application. Any modification or equivalent replacement without departing from the spirit and scope of the present application should be covered within the protection scope of the claims of the present application.

Claims

1. A large audit model and intelligent work order audit method based on multimodal supervised learning, characterized by: The method comprises the following steps: The marketing business support system collects power marketing audit data, cleans and labels the collected power marketing audit data, and forms an audit field instruction dataset, which is input into the model pre-training and fine-tuning module; Multimodal coding is used to extract sequence features from the audit domain instruction dataset. These features are then fed into a base model based on a multi-layer Transformer model for deep feature extraction. Domain expert knowledge is introduced through a retrieval-enhanced generation method to construct a large marketing audit model. The constructed marketing audit model is pre-trained and fine-tuned using the audit field instruction dataset. This allows for unified modeling, understanding, and generation of different types of data in power marketing audit scenarios. The PPO algorithm based on heterogeneous strategy optimization is then used to adjust the instruction fine-tuning model parameters. Based on the improved adaptive low-rank adaptation technology, the weight matrices of some layers of the model are approximated as the product of low-rank matrices, and an API interface service is provided to perform anomaly detection on the input marketing audit work order data.

2. The large audit and review model and intelligent work order review method based on multimodal supervised learning according to claim 1 is characterized by: The marketing business support system collects power marketing audit data, cleans and labels the collected power marketing audit data, forms an audit field instruction data set, and inputs it into the model pre-training and fine-tuning module, including: Obtain the marketing audit industry-wide data involved in the marketing business support system. This data is multi-source and multi-modal information, including audit work orders, rules and regulations, policy reports, and the daily number of work orders. The collected global data is preprocessed, and the power marketing audit instruction fine-tuning dataset is obtained by data annotation.

3. The large audit and review model and intelligent work order review method based on multimodal supervised learning according to claim 2 is characterized by: The use of multimodal coding to fully extract the sequence characteristics of the audit field instruction dataset includes: Through the multimodal encoding-decoding structure, key features are extracted, aligned and reconstructed from multimodal data including work order data, rules and regulations, policy reports and daily number of work orders in the marketing audit industry.

4. The large audit and review model and intelligent work order review method based on multimodal supervised learning according to claim 3 is characterized by: The multimodal encoding-decoding structure is used to extract key features, align features, and reconstruct features from multimodal data including work order data, rules and regulations, policy reports, and daily work order quantities in the marketing audit industry, including: For unstructured text data such as audit work orders and rules and regulations, a large-scale word segmenter is used to segment the original text. A numerical vector representation of each word is constructed according to the vocabulary, and text features are extracted using a text encoder. The structured time series data, including daily and monthly work order volumes, is split into multiple patches. The time series features are converted into text features using a multi-head attention mechanism. A linear mapping layer is used to output a time series representation with the same dimension as the text vector. A multimodal encoder is used to map different modal data into a unified feature space, and a fully connected layer is used to map the different modal data in the unified feature space into the text embedding space of the large model.

5. The large audit and review model and intelligent work order review method based on multimodal supervised learning according to claim 4 is characterized by: The extracted sequence features are input into a base model based on a multi-layer Transformer model for deep feature extraction. Domain expert knowledge is introduced through a retrieval-enhanced generation method to build a large marketing audit model, including: Using a multi-layer Transformer architecture to perform deep nonlinear operations on multimodal features, we can achieve deep feature representation of global marketing audit data. Construct a multimodal decoder to reconstruct text and time series data in the marketing audit industry to obtain multimodal output; The marketing audit big model knowledge base is externally expanded using retrieval enhancement generation technology, and prompt words that match the questions are automatically expanded without updating the big model parameters.

6. The large audit model and intelligent work order audit method based on multimodal supervised learning according to claim 5 is characterized by: The aforementioned use of the audit field instruction dataset to pre-train and fine-tune the constructed marketing audit model, and to perform unified modeling, understanding, and generation tasks for text and time series data types in power marketing audit scenarios, including: The base model is used to perform deep feature representation on the unified multimodal features, and the original input is reconstructed and outputted through the trained text decoder and temporal decoder.

7. The large audit and review model and intelligent work order review method based on multimodal supervised learning according to claim 6 is characterized by: The improved adaptive low-rank adaptation technology approximates the weight matrix of some layers of the model as the product of low-rank matrices, including: Based on the adaptive low-rank adaptation technology improved by singular value decomposition, the remaining parameters of the large model are frozen, and only the bias parameters or linear layer parameters of the Transformer network are updated.

8. The large audit model and intelligent work order audit method based on multimodal supervised learning according to claim 7 is characterized by: The improved adaptive low-rank adaptation technology approximates the weight matrix of some layers of the model as the product of low-rank matrices, and provides an API interface service to perform anomaly detection on input marketing audit industry work order data, including: Call the marketing audit model API to generate audit work order data, use the power marketing audit work order data to test the model, and conduct intelligent research and analysis of abnormal work orders.

9. A large audit model and intelligent work order audit system based on multimodal supervised learning using the method according to any one of claims 1 to 8, characterized in that: The system includes a multimodal instruction dataset construction module, an audit and review large model development module, a model pre-training and fine-tuning module, and a lightweight deployment and application module; The multimodal instruction dataset construction module collects power marketing audit data from the marketing business support system, cleans and annotates the collected power marketing audit data, and constructs an audit field instruction dataset, which is then input into the model pre-training and fine-tuning module. The audit and review large-scale model development module uses multimodal coding to extract sequence features from the audit domain instruction dataset. These extracted sequence features are then input into a base model based on a multi-layer Transformer model for deep feature extraction. Domain expert knowledge is introduced through a retrieval-enhanced generation method to build a marketing audit large-scale model. The model pre-training and fine-tuning module uses the audit field instruction dataset to pre-train and fine-tune the constructed marketing audit model. It performs unified modeling, understanding, and generation tasks for different types of data in the power marketing audit scenario, and uses the PPO algorithm based on different strategy optimization to adjust the instruction fine-tuning model parameters. The lightweight deployment and application module approximates the weight matrices of some layers of the model as the product of low-rank matrices based on the improved adaptive low-rank adaptation technology, and provides API interface services to perform anomaly detection on the input marketing audit industry work order data.

10. A terminal comprising a processor and a storage medium; characterized in that: The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images