Language model construction method and device based on data enhancement, equipment and medium

Abstract

The invention relates to the technical field of data enhancement, and discloses a language model construction method based on data enhancement, which comprises the following steps: classifying acquired large-scale corpus data to obtain text data, audio data and video data; enhancing the text data, the audio data and the video data to obtain enhanced text data, enhanced audio data and enhanced video data; extracting an image of each frame in the enhanced video data to obtain enhanced image data, and respectively standardizing the enhanced image data to obtain standard text data, standard audio data and standard image data; selecting a corresponding feature conversion method based on the data category, converting the data into feature vectors, and obtaining a text feature vector, an audio feature vector and an image feature vector; and carrying out model construction through a Transform model and each vector to obtain a large language model. The invention further provides a language model construction device and equipment based on data enhancement and a storage medium. According to the invention, the performance of the constructed language model can be improved.

Description

Technical Field

[0001] The present invention relates to the field of data enhancement technology, and in particular to a method, device, equipment and medium for constructing a language model based on data enhancement. Background Art

[0002] In recent years, power companies have accumulated a lot of data related to power user services within their service scope, which provides a data foundation for the study of big language. It can extract a large amount of corpus and semantic information from the user power service process, thereby supporting the construction of pre-trained language models for power service.

[0003] However, the current data on power user services may be mainly concentrated in a few aspects, and may lack data of different types or fields, resulting in insufficient representation of the data in some aspects and failure to fully cover all possible scenarios or situations. In addition, a large amount of data is usually required for the training of deep learning and large language models. Insufficient data may cause the model to be unable to fully learn the characteristics of the data, affecting the generalization ability of the model.

[0004] The above defects deserve improvement. Summary of the invention

[0005] The present invention provides a method, device, equipment and medium for building a language model based on data enhancement, the main purpose of which is to improve the efficiency of data management.

[0006] To achieve the above object, the present invention provides a method for constructing a language model based on data enhancement, comprising:

[0007] Acquire large-scale corpus data, and classify the large-scale corpus data to obtain text corpus data, audio corpus data, and video corpus data;

[0008] Extracting text information from the audio corpus data and the video corpus data to obtain supplementary text data, and obtaining comprehensive text data based on the supplementary text data and the text corpus data;

[0009] Constructing a PLM model according to the enhanced text data, acquiring text data from a preset library through the PLM model to obtain supplementary text data, and obtaining expanded text data according to the supplementary text data and the comprehensive text data;

[0010] Extracting entity relationships in the expanded text data, audio corpus data, and video corpus data, constructing a knowledge graph based on the entity relationships, and enhancing the expanded text data, audio corpus data, and video corpus data based on connection relationships in the knowledge graph to obtain enhanced text data, enhanced audio data, and enhanced video data;

[0011] Extracting an image of each frame in the enhanced video data to obtain enhanced image data, and standardizing the enhanced text data, enhanced audio data, and enhanced image data to obtain standard text data, standard audio data, and standard image data;

[0012] Based on the data categories of the standard text data, the standard audio data, and the standard image data, a corresponding feature conversion method is selected to convert the standard text data, the standard audio data, and the standard image data into feature vectors to obtain text feature vectors, audio feature vectors, and image feature vectors;

[0013] The model is constructed by using the Transformer model and the text feature vector, the audio feature vector and the image feature vector to obtain a large language model.

[0014] Optionally, the large-scale corpus data is classified to obtain text corpus data, audio corpus data and video corpus data, including:

[0015] Querying whether there is a recognizable extension in the large-scale corpus data;

[0016] If there is a recognizable extension in the large-scale corpus data, classifying the large-scale corpus data according to the attribute of the recognizable extension to obtain the text corpus data, the audio corpus data and the video corpus data;

[0017] If there is no recognizable extension in the large-scale corpus data, the large-scale corpus data is classified through manual annotation or script program to obtain the text corpus data, audio corpus data and video corpus data.

[0018] Optionally, the extracting text information from the audio corpus data and the video corpus data to obtain the supplementary text data includes:

[0019] Converting the spoken content in the audio corpus data into text information through automatic speech recognition to obtain audio supplementary text;

[0020] Acquire image data in the video corpus data, and convert recognizable text in the image data into text information through optical character recognition to obtain video supplementary text;

[0021] The supplementary text data is obtained according to the audio supplementary text and the video supplementary text.

[0022] Optionally, constructing a PLM model according to the enhanced text data includes:

[0023] Selecting a suitable pre-training model architecture based on the data category of the enhanced text data to obtain a screening model;

[0024] Performing mask training on the screening model using the enhanced text data to obtain a training model;

[0025] A downstream task is constructed according to the enhanced text data, and the training model is fine-tuned through the downstream task to obtain the PLM model.

[0026] Optionally, acquiring text data from a preset library through the PLM model to obtain supplementary text data includes:

[0027] Cleaning the data obtained from the library to obtain cleaned library data;

[0028] Extracting features of the cleaning library data through the PLM model to obtain cleaning library features, and screening the cleaning library features based on preset requirements to obtain screening library features;

[0029] The text data corresponding to the screening library feature is obtained from the library to obtain supplementary text data.

[0030] Optionally, extracting entity relationships in the expanded text data, audio corpus data, and video corpus data, and constructing a knowledge graph according to the entity relationships includes:

[0031] Transcribing the audio corpus data and the video corpus data into text form, and combining them with the expanded text data to obtain comprehensive text data;

[0032] Identify entities in the comprehensive text data by word segmentation, part-of-speech tagging and named entity recognition methods to obtain text entities;

[0033] Based on a preset semantic analysis model, the relationship between the text entities is identified to obtain the entity relationship, and the text entities and the entity relationship are integrated to form a graph structure to obtain the knowledge graph.

[0034] Optionally, the step of enhancing the expanded text data, audio corpus data, and video corpus data according to the connection relationship in the knowledge graph to obtain enhanced text data, enhanced audio data, and enhanced video data includes:

[0035] Recommending related terms or concepts for the expanded text data through the connection relationship in the knowledge graph, enhancing the text data, and obtaining the enhanced text data;

[0036] By adding relevant terms or concepts in the knowledge graph to the transcribed text of the audio corpus data, the information content of the audio corpus data is increased to obtain enhanced audio data;

[0037] Based on the knowledge graph, metadata tags or letters are added to the video corpus data to enhance the video corpus data to obtain enhanced video data.

[0038] In order to solve the above problems, the present invention also provides a language model construction device based on data enhancement, the device comprising:

[0039] A data acquisition module is used to acquire large-scale corpus data and classify the large-scale corpus data to obtain text corpus data, audio corpus data and video corpus data; obtain supplementary text data by extracting text information from the audio corpus data and the video corpus data, and obtain comprehensive text data based on the supplementary text data and the text corpus data;

[0040] A data enhancement module is used to construct a PLM model according to the enhanced text data, obtain text data from a preset library through the PLM model to obtain supplementary text data, and obtain extended text data according to the supplementary text data and the comprehensive text data; extract entity relationships in the extended text data, audio corpus data, and video corpus data, construct a knowledge graph according to the entity relationships, and enhance the extended text data, audio corpus data, and video corpus data according to the connection relationships in the knowledge graph to obtain enhanced text data, enhanced audio data, and enhanced video data;

[0041] A feature extraction module is used to extract the image of each frame in the enhanced video data to obtain enhanced image data, and to standardize the enhanced text data, enhanced audio data and enhanced image data to obtain standard text data, standard audio data and standard image data; based on the data categories of the standard text data, standard audio data and standard image data, a corresponding feature conversion method is selected to convert the standard text data, standard audio data and standard image data into a feature vector to obtain a text feature vector, an audio feature vector and an image feature vector;

[0042] The model building module is used to build a model through a Transformer model and the text feature vector, the audio feature vector and the image feature vector to obtain a large language model.

[0043] In order to solve the above problem, the present invention further provides an electronic device, the electronic device comprising:

[0044] at least one processor; and,

[0045] a memory communicatively connected to the at least one processor; wherein,

[0046] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the language model construction method based on data enhancement as described above.

[0047] In order to solve the above problems, the present invention also provides a computer-readable storage medium, including a data storage area and a program storage area, the data storage area stores created data, and the program storage area stores a computer program; wherein, when the computer program is executed by a processor, the language model construction method based on data enhancement as described above is implemented.

[0048] The embodiment of the present invention obtains large-scale corpus data and classifies the large-scale corpus data to obtain text corpus data, audio corpus data and video corpus data; obtains supplementary text data by extracting text information from the audio corpus data and the video corpus data, and obtains comprehensive text data based on the supplementary text data and the text corpus data; constructs a PLM model based on the enhanced text data, obtains text data from a preset library through the PLM model to obtain supplementary text data, and obtains extended text data based on the supplementary text data and the comprehensive text data; extracts entity relationships in the extended text data, audio corpus data and video corpus data, constructs a knowledge graph based on the entity relationships, and classifies the extended text data, audio corpus data and video corpus data based on the connection relationships in the knowledge graph. The audio corpus data and the video corpus data are enhanced to obtain enhanced text data, enhanced audio data and enhanced video data; the image of each frame in the enhanced video data is extracted to obtain enhanced image data, and the enhanced text data, enhanced audio data and enhanced image data are respectively standardized to obtain standard text data, standard audio data and standard image data; based on the data category of the standard text data, standard audio data and standard image data, the corresponding feature conversion method is selected to convert the standard text data, standard audio data and standard image data into feature vectors to obtain text feature vectors, audio feature vectors and image feature vectors; the model is constructed by the Transformer model and the text feature vectors, audio feature vectors and image feature vectors to obtain a large language model. Therefore, the language model construction method, device, electronic device and computer-readable storage medium based on data enhancement proposed in the present invention expand and enhance the existing large-scale corpus data in a variety of ways, so that the language model effect constructed according to the data later better reflects the performance of the language model and enhances the efficiency of data management. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A flow chart of a method for constructing a language model based on data enhancement provided by one embodiment of the present invention;

[0050] Figure 2 A schematic diagram of a module of a language model construction device based on data enhancement provided by an embodiment of the present invention;

[0051] Figure 3 A schematic diagram of the internal structure of an electronic device for implementing a method for building a language model based on data enhancement provided by an embodiment of the present invention.

[0052] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0053] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0054] The embodiment of the present application provides a method for constructing a language model based on data enhancement. The execution subject of the method for constructing a language model based on data enhancement includes but is not limited to at least one of the electronic devices such as a server, a terminal, etc. that can be configured to execute the method provided by the embodiment of the present application. Among them, the server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms. In other words, the method for constructing a language model based on data enhancement can be executed by software or hardware installed on a remote device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc.

[0055] Reference Figure 1 FIG. 1 is a flow chart of a method for constructing a language model based on data enhancement provided by an embodiment of the present invention. In this embodiment, the method for constructing a language model based on data enhancement includes the following steps S1-S7:

[0056] S1. Acquire large-scale corpus data, and classify the large-scale corpus data to obtain text corpus data, audio corpus data, and video corpus data.

[0057] In an embodiment of the present invention, the large-scale forecast data is applied to the power system. In the field of power systems, large-scale corpus data refers to various types of data sets including text, audio, image, video and structured data. These data come from multiple aspects of the power system, such as technical documents, communication records, equipment images, monitoring videos and real-time sensor data. Using these data, artificial intelligence models can be developed and trained to improve the intelligence level of the power system, including improving operational efficiency, optimizing maintenance strategies, improving customer service and enhancing decision support.

[0058] Furthermore, the large-scale corpus data is classified to obtain text corpus data, audio corpus data and video corpus data, including:

[0059] Querying whether there is a recognizable extension in the large-scale corpus data;

[0060] If there is a recognizable extension in the large-scale corpus data, classifying the large-scale corpus data according to the attribute of the recognizable extension to obtain the text corpus data, the audio corpus data and the video corpus data;

[0061] If there is no recognizable extension in the large-scale corpus data, the large-scale corpus data is classified through manual annotation or script program to obtain the text corpus data, audio corpus data and video corpus data.

[0062] In the embodiments of the present invention, large-scale corpus data, including text, audio and video, is acquired and classified, which can enrich data resources, improve data quality, and provide a basis for building multimodal knowledge graphs and training efficient artificial intelligence models, thereby enhancing the machine's ability to understand human language and behavior and expanding the scope of application.

[0063] S2. Obtain supplementary text data by extracting text information from the audio corpus data and the video corpus data, and obtain comprehensive text data based on the supplementary text data and the text corpus data.

[0064] In the embodiment of the present invention, the process of extracting text information from audio corpus data and video corpus data to obtain supplementary text data can be understood as converting non-text data (such as voice and video content) into text form to expand the data source for further analysis and processing. For example, the spoken content in the audio data is converted into written content through automatic speech recognition (ASR) to finally obtain text data, and the text information in the image of the video is converted into text data through image recognition.

[0065] In the embodiment of the present invention, the step of extracting text information from the audio corpus data and the video corpus data to obtain supplementary text data includes:

[0066] Converting the spoken content in the audio corpus data into text information through automatic speech recognition to obtain audio supplementary text;

[0067] Acquire image data in the video corpus data, and convert recognizable text in the image data into text information through optical character recognition to obtain video supplementary text;

[0068] The supplementary text data is obtained according to the audio supplementary text and the video supplementary text.

[0069] In the embodiments of the present invention, Automatic Speech Recognition (ASR) is a technology that converts human speech into written text. It involves multiple steps, including audio signal processing, feature extraction, acoustic model matching and language model application, to achieve automatic recognition and transcription of speech.

[0070] Among them, Optical Character Recognition (OCR) is a technology that can recognize and extract machine-readable text from image files. In short, OCR technology can convert text content in scanned documents, photos, or PDF images into editable and searchable electronic text data.

[0071] Furthermore, text information is extracted from audio and video corpus data and combined with existing text data to form comprehensive text data. This can enhance data diversity, expand the corpus, improve the model's understanding and processing capabilities of language, and support multimodal learning and cross-modal interaction, thereby improving the performance of artificial intelligence systems in various natural language processing tasks.

[0072] For example, the maintenance records of power systems may be recorded in the form of videos. Through automatic speech recognition technology, the verbal instructions during the maintenance process can be converted into text, and then combined with the text information appearing in the video content (such as equipment labels, operating steps, etc.) to construct a comprehensive text dataset. This dataset not only contains the specific steps of the operation, but also may include problems and solutions found during the maintenance process, providing rich information resources for the maintenance and troubleshooting of power systems.

[0073] S3. Construct a PLM model according to the enhanced text data, obtain text data from a preset library through the PLM model to obtain supplementary text data, and obtain extended text data according to the supplementary text data and the comprehensive text data.

[0074] In the embodiment of the present invention, the PLM model refers to a pre-trained language model (Pre-trained Language Model), which is a model used to process and understand human language in the field of natural language processing (NLP). PLM learns the common features and patterns of language by pre-training on a large amount of text data, and then can be fine-tuned on specific tasks to improve performance. These models perform well in various NLP tasks, such as text classification, sentiment analysis, machine translation, text summarization, and question-answering systems.

[0075] In the field of power systems, PLM models can help process and understand large amounts of text data such as technical documents, operating manuals, fault reports, and maintenance records. By extracting valuable information from these unstructured texts, PLM models can support applications such as intelligent monitoring, fault diagnosis, maintenance decision-making, and knowledge management of power systems. In addition, PLM models can also be combined with other data types of power systems (such as sensor data, log files, etc.) to achieve more comprehensive data analysis and decision support.

[0076] Furthermore, the preset library is a text data set for the field of electric power technology, which is used to train and build pre-trained models in the field of electric power technology. The library contains electric power technology-related text data collected from different sources, such as scientific papers, patents, scientific project reports, etc.

[0077] Further, constructing a PLM model according to the enhanced text data includes:

[0078] Selecting a suitable pre-training model architecture based on the data category of the enhanced text data to obtain a screening model;

[0079] Performing mask training on the screening model using the enhanced text data to obtain a training model;

[0080] A downstream task is constructed according to the enhanced text data, and the training model is fine-tuned through the downstream task to obtain the PLM model.

[0081] Furthermore, the downstream task is a task constructed based on the enhanced text data and used to fine-tune the model, including a term classification task or a text classification task constructed based on the enhanced text data.

[0082] In the embodiment of the present invention, the step of acquiring text data from a preset library through the PLM model to obtain supplementary text data includes:

[0083] Cleaning the data obtained from the library to obtain cleaned library data;

[0084] Extracting features of the cleaning library data through the PLM model to obtain cleaning library features, and screening the cleaning library features based on preset requirements to obtain screening library features;

[0085] The text data corresponding to the screening library feature is obtained from the library to obtain supplementary text data.

[0086] S4. Extract entity relationships in the extended text data, audio corpus data, and video corpus data, construct a knowledge graph based on the entity relationships, and enhance the extended text data, audio corpus data, and video corpus data based on the connection relationships in the knowledge graph to obtain enhanced text data, enhanced audio data, and enhanced video data.

[0087] In an embodiment of the present invention, the entity relationship in the knowledge graph of the electric power technology field refers to the semantic ties that describe the mutual connections between entities such as experts, results, and terms. These relationships help to clarify the research fields of experts, organize their research results, and reveal the connections between terms, thereby forming a structured knowledge system to support various applications, such as expert matching and knowledge retrieval.

[0088] Furthermore, in the power system, the knowledge graph is a structured semantic knowledge base that graphically represents entities (such as equipment, components, events) and their relationships (such as attribution, function, causality). By extracting entities and relationships from extended text data, audio corpus data, and video corpus data, the constructed knowledge graph can help power system operators and analysts understand complex relationships, optimize decisions, and enhance data analysis.

[0089] For example, a power system knowledge graph may contain entities such as substations, transmission lines, and power generation equipment, as well as relationships between them, such as control relationships, maintenance history, and fault records. Through this knowledge graph, the expanded text, audio, and video data can be contextualized and enhanced to make them richer and more valuable.

[0090] In the embodiment of the present invention, the extracting entity relationships in the expanded text data, audio corpus data, and video corpus data, and constructing a knowledge graph according to the entity relationships, includes:

[0091] Transcribing the audio corpus data and the video corpus data into text form, and combining them with the expanded text data to obtain comprehensive text data;

[0092] Identify entities in the comprehensive text data by word segmentation, part-of-speech tagging and named entity recognition methods to obtain text entities;

[0093] Based on a preset semantic analysis model, the relationship between the text entities is identified to obtain the entity relationship, and the text entities and the entity relationship are integrated to form a graph structure to obtain the knowledge graph.

[0094] Furthermore, the expanded text data, audio corpus data and video corpus data are enhanced according to the connection relationship in the knowledge graph to obtain enhanced text data, enhanced audio data and enhanced video data, including:

[0095] Recommending related terms or concepts for the expanded text data through the connection relationship in the knowledge graph, enhancing the text data, and obtaining the enhanced text data;

[0096] By adding relevant terms or concepts in the knowledge graph to the transcribed text of the audio corpus data, the information content of the audio corpus data is increased to obtain enhanced audio data;

[0097] Based on the knowledge graph, metadata tags or letters are added to the video corpus data to enhance the video corpus data to obtain enhanced video data.

[0098] In an embodiment of the present invention, by constructing a knowledge graph and using it to enhance the original data, the power system can achieve structuring of data and information richness, thereby improving intelligent analysis capabilities, assisting decision-making, improving operational efficiency, promoting innovation, managing risks, and serving as a tool for training new employees, thereby enhancing the intelligence and automation level of the entire system.

[0099] For example, by building a knowledge graph that contains power system components (such as generators, transformers) and the relationships between them (such as connections, dependencies), the monitoring system can enhance its understanding of the system status, so that it can quickly locate the problem and take measures when a failure occurs. In addition, the knowledge graph can also be used to train new operators by visually displaying the structure of the system and the interactions between components.

[0100] S5. Extract the image of each frame in the enhanced video data to obtain enhanced image data, and standardize the enhanced text data, enhanced audio data and enhanced image data to obtain standard text data, standard audio data and standard image data.

[0101] In the embodiment of the present invention, the video data is essentially a sequence composed of a series of static images (frames), and these images are played quickly and continuously in time order to create a dynamic visual effect, so images can be extracted from the video data.

[0102] In power systems, extracted video frames can also be used to monitor the operating status of equipment, detect abnormal behavior, or be used for training and simulation of operation scenarios. For example, by analyzing frames of substation video, unauthorized intrusion or equipment failure can be detected.

[0103] Further, extracting the image of each frame in the enhanced video data to obtain enhanced image data includes:

[0104] The enhanced video data is read through a preset video processing library, and each frame of the enhanced video data is accessed according to an interface of the video processing library to obtain the enhanced image data.

[0105] In the embodiment of the present invention, the enhanced text data, enhanced audio data and enhanced image data are respectively standardized to obtain standard text data, standard audio data and standard image data, including:

[0106] Performing word segmentation, stop word removal, part-of-speech restoration and part-of-speech tagging on the enhanced text data to obtain tagged text data, and performing denoising and error correction on the tagged text data to obtain standard text data;

[0107] quantizing the audio signal and removing background noise from the audio signal to obtain standard audio data;

[0108] The enhanced image data is scaled and cropped to obtain cropped image data, and the cropped text data is rotated and grayed to obtain standard image data.

[0109] In an embodiment of the present invention, extracting and standardizing video frames into images, and standardizing text, audio, and video data can improve the consistency and quality of power system data, facilitate cross-modal analysis and training of machine learning models, and thus improve the accuracy and efficiency of monitoring, diagnosis, and predictive maintenance.

[0110] For example, in the power system, after the frames of the surveillance video are extracted and standardized, they can be used to monitor the operating status of power equipment in real time, detect abnormal behavior, or be used for training and simulation of operation scenarios. At the same time, standardized text data can be used to improve the accuracy of the fault diagnosis system, and standardized audio data can be used to analyze the sound patterns of power equipment during operation to identify potential faults.

[0111] S6. Based on the data categories of the standard text data, standard audio data and standard image data, select a corresponding feature conversion method to convert the standard text data, standard audio data and standard image data into feature vectors to obtain text feature vectors, audio feature vectors and image feature vectors.

[0112] In an embodiment of the present invention, the preprocessed corpus is converted into a feature vector, which can standardize different types of data, extract key information, reduce data dimensions, improve computing efficiency, and help improve the performance and generalization ability of the machine learning model, making it more accurate and efficient in various AI tasks.

[0113] For example, in the power system, converting equipment monitoring reports into text feature vectors, converting equipment operation sounds into audio feature vectors, and converting images captured by surveillance cameras into image feature vectors can make the monitoring and diagnosis of the power system more automated and intelligent.

[0114] Furthermore, based on the data categories of the standard text data, standard audio data and standard image data, a corresponding feature conversion method is selected to convert the standard text data, standard audio data and standard image data into feature vectors. Text data can be converted using text conversion methods such as bag-of-words model, TF-IDF, N-gram or Word2Vec to obtain a feature vector; for audio data, Mel spectrum graph, MFCC (Mel frequency cepstral coefficient) and audio energy extraction methods can be used for conversion to obtain a feature vector; for image data, color histogram, texture features, shape descriptors, and convolutional features extracted by convolutional neural network (CNN) can be used to obtain a feature vector.

[0115] S7. Building a model using a Transformer model and the text feature vector, the audio feature vector, and the image feature vector to obtain a large language model.

[0116] In the embodiment of the present invention, the Transformer model is a deep learning model based on the self-attention mechanism, which is mainly used to process sequence-to-sequence (Seq2Seq) tasks, such as machine translation, text summarization, and question-answering systems.

[0117] Furthermore, the application of the Transformer model in the power system is mainly reflected in its ability to efficiently process and analyze large-scale multimodal data, achieve real-time monitoring and fault prediction, and use transfer learning to adapt to specific tasks, thereby improving the intelligence level of the power system, optimizing operational efficiency, and providing personalized service experience for staff and users.

[0118] In an embodiment of the present invention, for a large-scale language model architecture suitable for electricity business, a recurrent neural network (RNN) or Transformer architecture can be considered for text information, and a convolutional neural network (CNN) or Transformer architecture can be considered for large-scale language models of audio and images. Since the data in the present invention is power industry data, covering text, audio, images or video, the Transformer model can be used to build a large language model.

[0119] Furthermore, after obtaining the large language model, the large language model can be used to identify the user's questions and respond according to the model information provided by the large language model.

[0120] The embodiment of the present invention obtains large-scale corpus data and classifies the large-scale corpus data to obtain text corpus data, audio corpus data and video corpus data; obtains supplementary text data by extracting text information from the audio corpus data and the video corpus data, and obtains comprehensive text data based on the supplementary text data and the text corpus data; constructs a PLM model based on the enhanced text data, obtains text data from a preset library through the PLM model to obtain supplementary text data, and obtains extended text data based on the supplementary text data and the comprehensive text data; extracts entity relationships in the extended text data, audio corpus data and video corpus data, constructs a knowledge graph based on the entity relationships, and classifies the extended text data, audio corpus data and video corpus data based on the connection relationships in the knowledge graph. The audio corpus data and the video corpus data are enhanced to obtain enhanced text data, enhanced audio data and enhanced video data; the image of each frame in the enhanced video data is extracted to obtain enhanced image data, and the enhanced text data, enhanced audio data and enhanced image data are respectively standardized to obtain standard text data, standard audio data and standard image data; based on the data category of the standard text data, standard audio data and standard image data, the corresponding feature conversion method is selected to convert the standard text data, standard audio data and standard image data into feature vectors to obtain text feature vectors, audio feature vectors and image feature vectors; the model is constructed by the Transformer model and the text feature vectors, audio feature vectors and image feature vectors to obtain a large language model. Therefore, the language model construction method, device, electronic device and computer-readable storage medium based on data enhancement proposed in the present invention expand and enhance the existing large-scale corpus data in a variety of ways, so that the language model effect constructed according to the data later better reflects the performance of the language model and enhances the efficiency of data management.

[0121] like Figure 2 , which is a module schematic diagram of a language model construction device based on data enhancement of the present invention.

[0122] The language model construction device 100 based on data enhancement of the present invention can be installed in an electronic device. According to the functions to be implemented, the language model construction device based on data enhancement can include a data acquisition module 101, a data enhancement module 102, a feature extraction module 103 and a model construction module 104. The module of the present invention can also be called a unit, which refers to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, which are stored in the memory of the electronic device.

[0123] In this embodiment, the functions of each module / unit are as follows:

[0124] The data acquisition module 101 is used to acquire large-scale corpus data and classify the large-scale corpus data to obtain text corpus data, audio corpus data and video corpus data; obtain supplementary text data by extracting text information from the audio corpus data and the video corpus data, and obtain comprehensive text data based on the supplementary text data and the text corpus data;

[0125] The data enhancement module 102 is used to construct a PLM model according to the enhanced text data, obtain text data from a preset library through the PLM model to obtain supplementary text data, and obtain extended text data according to the supplementary text data and the comprehensive text data; extract entity relationships in the extended text data, audio corpus data, and video corpus data, construct a knowledge graph according to the entity relationships, and enhance the extended text data, audio corpus data, and video corpus data according to the connection relationships in the knowledge graph to obtain enhanced text data, enhanced audio data, and enhanced video data;

[0126] The feature extraction module 103 is used to extract the image of each frame in the enhanced video data to obtain enhanced image data, and to standardize the enhanced text data, enhanced audio data and enhanced image data to obtain standard text data, standard audio data and standard image data; based on the data categories of the standard text data, standard audio data and standard image data, select a corresponding feature conversion method to convert the standard text data, standard audio data and standard image data into a feature vector to obtain a text feature vector, an audio feature vector and an image feature vector;

[0127] The model building module 104 is used to build a model through the Transformer model and the text feature vector, the audio feature vector and the image feature vector to obtain a large language model.

[0128] In detail, each module described in the data-enhanced language model construction device 100 in the embodiment of the present invention is used in the same manner as described above. Figure 1The above-mentioned language model construction method based on data enhancement has the same technical means and can produce the same technical effects, so it will not be repeated here.

[0129] like Figure 3 , which is a schematic diagram of the structure of an electronic device for implementing a method for constructing a language model based on data enhancement according to the present invention.

[0130] The electronic device may include a processor 10, a memory 11, a communication bus 12, and a communication interface 13, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as a language model building program based on data enhancement.

[0131] In some embodiments, the processor 10 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor 10 is the control core (ControlUnit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes or executes programs or modules stored in the memory 11 (for example, executing a language model construction program based on data enhancement, etc.), and calls data stored in the memory 11 to execute various functions of the electronic device and process data.

[0132] The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of an electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory 11 may also be an external storage device of an electronic device, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Further, the memory 11 may also include both an internal storage unit of the electronic device and an external storage device. The memory 11 can not only be used to store application software and various types of data installed in the electronic device, such as the code for building a program based on a data-enhanced language model, but also can be used to temporarily store data that has been output or is to be output.

[0133] The communication bus 12 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory 11 and at least one processor 10, etc.

[0134] The communication interface 13 is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a display (Display), an input unit (such as a keyboard (Keyboard)), and optionally, the user interface may also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface.

[0135] Figure 3 Only an electronic device with components is shown, and those skilled in the art will understand that Figure 3 The structure shown does not constitute a limitation on the electronic device, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0136] For example, although not shown, the electronic device may also include a power source (such as a battery) for supplying power to each component. Preferably, the power source may be logically connected to the at least one processor 10 through a power management device, so that the power management device can realize functions such as charging management, discharging management, and power consumption management. The power source may also include one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0137] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0138] The language model building program based on data enhancement stored in the memory 11 in the electronic device is a combination of multiple computer programs. When running in the processor 10, it can achieve:

[0139] Acquire large-scale corpus data, and classify the large-scale corpus data to obtain text corpus data, audio corpus data, and video corpus data;

[0140] Extracting text information from the audio corpus data and the video corpus data to obtain supplementary text data, and obtaining comprehensive text data based on the supplementary text data and the text corpus data;

[0141] Constructing a PLM model according to the enhanced text data, acquiring text data from a preset library through the PLM model to obtain supplementary text data, and obtaining expanded text data according to the supplementary text data and the comprehensive text data;

[0142] Extracting entity relationships in the expanded text data, audio corpus data, and video corpus data, constructing a knowledge graph based on the entity relationships, and enhancing the expanded text data, audio corpus data, and video corpus data based on connection relationships in the knowledge graph to obtain enhanced text data, enhanced audio data, and enhanced video data;

[0143] Extracting an image of each frame in the enhanced video data to obtain enhanced image data, and standardizing the enhanced text data, enhanced audio data, and enhanced image data to obtain standard text data, standard audio data, and standard image data;

[0144] Based on the data categories of the standard text data, the standard audio data, and the standard image data, a corresponding feature conversion method is selected to convert the standard text data, the standard audio data, and the standard image data into feature vectors to obtain text feature vectors, audio feature vectors, and image feature vectors;

[0145] The model is constructed by using the Transformer model and the text feature vector, the audio feature vector and the image feature vector to obtain a large language model.

[0146] Specifically, the specific implementation method of the processor 10 for the above computer program can refer to Figure 1 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0147] Furthermore, if the module / unit integrated in the electronic device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0148] The present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor of an electronic device, the computer program can implement:

[0149] Acquire large-scale corpus data, and classify the large-scale corpus data to obtain text corpus data, audio corpus data, and video corpus data;

[0150] Extracting text information from the audio corpus data and the video corpus data to obtain supplementary text data, and obtaining comprehensive text data based on the supplementary text data and the text corpus data;

[0151] Constructing a PLM model according to the enhanced text data, acquiring text data from a preset library through the PLM model to obtain supplementary text data, and obtaining expanded text data according to the supplementary text data and the comprehensive text data;

[0152] Extracting entity relationships in the expanded text data, audio corpus data, and video corpus data, constructing a knowledge graph based on the entity relationships, and enhancing the expanded text data, audio corpus data, and video corpus data based on connection relationships in the knowledge graph to obtain enhanced text data, enhanced audio data, and enhanced video data;

[0153] Extracting an image of each frame in the enhanced video data to obtain enhanced image data, and standardizing the enhanced text data, enhanced audio data, and enhanced image data to obtain standard text data, standard audio data, and standard image data;

[0154] Based on the data categories of the standard text data, the standard audio data, and the standard image data, a corresponding feature conversion method is selected to convert the standard text data, the standard audio data, and the standard image data into feature vectors to obtain text feature vectors, audio feature vectors, and image feature vectors;

[0155] The model is constructed by using the Transformer model and the text feature vector, the audio feature vector and the image feature vector to obtain a large language model.

[0156] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0157] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0158] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0159] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0160] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0161] The blockchain referred to in this invention is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0162] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0163] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The second and other words are used to indicate names, but not to indicate any particular order.

[0164] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for constructing a language model based on data enhancement, characterized in that: The method comprises: Acquire large-scale corpus data, and classify the large-scale corpus data to obtain text corpus data, audio corpus data, and video corpus data; Extracting text information from the audio corpus data and the video corpus data to obtain supplementary text data, and obtaining comprehensive text data based on the supplementary text data and the text corpus data; Constructing a PLM model according to the enhanced text data, acquiring text data from a preset library through the PLM model to obtain supplementary text data, and obtaining expanded text data according to the supplementary text data and the comprehensive text data; Extracting entity relationships in the expanded text data, audio corpus data, and video corpus data, constructing a knowledge graph based on the entity relationships, and enhancing the expanded text data, audio corpus data, and video corpus data based on connection relationships in the knowledge graph to obtain enhanced text data, enhanced audio data, and enhanced video data; Extracting an image of each frame in the enhanced video data to obtain enhanced image data, and standardizing the enhanced text data, enhanced audio data, and enhanced image data to obtain standard text data, standard audio data, and standard image data; Based on the data categories of the standard text data, the standard audio data, and the standard image data, a corresponding feature conversion method is selected to convert the standard text data, the standard audio data, and the standard image data into feature vectors to obtain text feature vectors, audio feature vectors, and image feature vectors; The model is constructed by using the Transformer model and the text feature vector, the audio feature vector and the image feature vector to obtain a large language model.

2. The method for constructing a language model based on data enhancement according to claim 1, characterized in that: The large-scale corpus data is classified to obtain text corpus data, audio corpus data and video corpus data, including: Querying whether there is a recognizable extension in the large-scale corpus data; If there is a recognizable extension in the large-scale corpus data, classifying the large-scale corpus data according to the attribute of the recognizable extension to obtain the text corpus data, the audio corpus data and the video corpus data; If there is no recognizable extension in the large-scale corpus data, the large-scale corpus data is classified through manual annotation or script program to obtain the text corpus data, audio corpus data and video corpus data.

3. The method for constructing a language model based on data enhancement according to claim 1, characterized in that: The method of extracting text information from the audio corpus data and the video corpus data to obtain supplementary text data includes: Converting the spoken content in the audio corpus data into text information through automatic speech recognition to obtain audio supplementary text; Acquire image data in the video corpus data, and convert recognizable text in the image data into text information through optical character recognition to obtain video supplementary text; The supplementary text data is obtained according to the audio supplementary text and the video supplementary text.

4. The method for constructing a language model based on data enhancement according to claim 1, characterized in that: The step of constructing a PLM model according to the enhanced text data comprises: Selecting a suitable pre-training model architecture based on the data category of the enhanced text data to obtain a screening model; Performing mask training on the screening model using the enhanced text data to obtain a training model; A downstream task is constructed according to the enhanced text data, and the training model is fine-tuned through the downstream task to obtain the PLM model.

5. The method for constructing a language model based on data enhancement according to claim 1, characterized in that: The acquiring text data from a preset library through the PLM model to obtain supplementary text data includes: Cleaning the data obtained from the library to obtain cleaned library data; Extracting features of the cleaning library data through the PLM model to obtain cleaning library features, and screening the cleaning library features based on preset requirements to obtain screening library features; The text data corresponding to the screening library feature is obtained from the library to obtain supplementary text data.

6. The method for constructing a language model based on data enhancement according to claim 1, characterized in that: The extracting entity relationships in the expanded text data, audio corpus data, and video corpus data, and constructing a knowledge graph according to the entity relationships, includes: Transcribing the audio corpus data and the video corpus data into text form, and combining them with the expanded text data to obtain comprehensive text data; Identify entities in the comprehensive text data by word segmentation, part-of-speech tagging and named entity recognition methods to obtain text entities; Based on a preset semantic analysis model, the relationship between the text entities is identified to obtain the entity relationship, and the text entities and the entity relationship are integrated to form a graph structure to obtain the knowledge graph.

7. The method for constructing a language model based on data enhancement according to any one of claims 1 to 6, characterized in that: The method of enhancing the expanded text data, audio corpus data, and video corpus data according to the connection relationship in the knowledge graph to obtain enhanced text data, enhanced audio data, and enhanced video data includes: Recommending related terms or concepts for the expanded text data through the connection relationship in the knowledge graph, enhancing the text data, and obtaining the enhanced text data; By adding relevant terms or concepts in the knowledge graph to the transcribed text of the audio corpus data, the information content of the audio corpus data is increased to obtain enhanced audio data; Based on the knowledge graph, metadata tags or letters are added to the video corpus data to enhance the video corpus data to obtain enhanced video data.

8. A language model construction device based on data enhancement, characterized in that: The device comprises: A data acquisition module is used to acquire large-scale corpus data and classify the large-scale corpus data to obtain text corpus data, audio corpus data and video corpus data; obtain supplementary text data by extracting text information from the audio corpus data and the video corpus data, and obtain comprehensive text data based on the supplementary text data and the text corpus data; A data enhancement module is used to construct a PLM model according to the enhanced text data, obtain text data from a preset library through the PLM model to obtain supplementary text data, and obtain extended text data according to the supplementary text data and the comprehensive text data; extract entity relationships in the extended text data, audio corpus data, and video corpus data, construct a knowledge graph according to the entity relationships, and enhance the extended text data, audio corpus data, and video corpus data according to the connection relationships in the knowledge graph to obtain enhanced text data, enhanced audio data, and enhanced video data; A feature extraction module is used to extract the image of each frame in the enhanced video data to obtain enhanced image data, and to standardize the enhanced text data, enhanced audio data and enhanced image data to obtain standard text data, standard audio data and standard image data; based on the data categories of the standard text data, standard audio data and standard image data, a corresponding feature conversion method is selected to convert the standard text data, standard audio data and standard image data into a feature vector to obtain a text feature vector, an audio feature vector and an image feature vector; The model building module is used to build a model through a Transformer model and the text feature vector, the audio feature vector and the image feature vector to obtain a large language model.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the language model construction method based on data enhancement as described in any one of claims 1 to 7.

10. A computer-readable storage medium, comprising a data storage area and a program storage area, wherein the data storage area stores created data and the program storage area stores a computer program; wherein: When the computer program is executed by a processor, the method for constructing a language model based on data enhancement is implemented as described in any one of claims 1 to 7.

Images