Data augmentation based language model construction method and device, equipment and medium

Abstract

The application relates to the technical field of data enhancement, and discloses a language model construction method based on data enhancement, which comprises the following steps: classifying large-scale corpus data obtained, to obtain text data, audio data and video data; enhancing the text data, the audio data and the video data, to obtain enhanced text data, enhanced audio data and enhanced video data; extracting images of each frame in the enhanced video data, to obtain enhanced image data, and respectively performing standardization, to obtain standard text data, standard audio data and standard image data; based on data categories, selecting corresponding feature conversion methods to convert the data into feature vectors, to obtain text feature vectors, audio feature vectors and image feature vectors; and performing model construction through a Transformer model and the vectors, to obtain a large language model. The application further discloses a language model construction device and equipment based on data enhancement and a storage medium. The application can improve the performance of the constructed language model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data enhancement, in particular to a language model construction method and device based on data enhancement, equipment and medium. BACKGROUND

[0002] In recent years, power companies have accumulated a lot of power user service related data within their service range, providing a data basis for large language research, which can extract a large amount of corpus and semantic information from the user service process, thereby supporting the construction of power service pre-training language models.

[0003] However, the current power user service data may mainly focus on a few aspects, and the data may lack data of different types or fields, resulting in insufficient representativeness of the data in some aspects and inability to comprehensively cover all possible scenarios or situations. In addition, for deep learning and large language model training, a large amount of data is usually required. Insufficient data may prevent the model from fully learning the characteristics of the data, affecting the generalization ability of the model.

[0004] The above defects are worthy of improvement. SUMMARY

[0005] The present application provides a language model construction method and device based on data enhancement, equipment and medium, which mainly aims to improve the efficiency of data governance.

[0006] To achieve the above purpose, the present application provides a language model construction method based on data enhancement, which comprises:

[0007] Obtain large-scale corpus data and classify the large-scale corpus data to obtain text corpus data, audio corpus data and video corpus data;

[0008] Extract text information from the audio corpus data and video corpus data to obtain supplementary text data, and obtain comprehensive text data according to the supplementary text data and the text corpus data;

[0009] Construct a PLM model according to the augmented text data, obtain text data from a predetermined library through the PLM model to obtain supplementary text data, and obtain expanded text data according to the supplementary text data and the comprehensive text data;

[0010] Extract entity relationships in the expanded text data, audio corpus data and video corpus data, construct a knowledge graph according to the entity relationships, and enhance the expanded text data, audio corpus data and video corpus data according to the connection relationships in the knowledge graph to obtain augmented text data, augmented audio data and augmented video data;

[0011] extracting images of each frame in the enhanced video data to obtain enhanced image data, and normalizing the enhanced text data, the enhanced audio data and the enhanced image data to obtain standard text data, standard audio data and standard image data;

[0012] selecting a corresponding feature conversion method based on the data categories of the standard text data, the standard audio data and the standard image data, and converting the standard text data, the standard audio data and the standard image data into feature vectors to obtain text feature vectors, audio feature vectors and image feature vectors;

[0013] performing model construction through a Transformer model and the text feature vectors, the audio feature vectors and the image feature vectors to obtain a large language model.

[0014] Optionally, the classifying the large-scale corpus data to obtain text corpus data, audio corpus data and video corpus data comprises:

[0015] querying whether there is an identifiable extension name in the large-scale corpus data;

[0016] if there is an identifiable extension name in the large-scale corpus data, classifying the large-scale corpus data through the attributes of the identifiable extension name to obtain the text corpus data, the audio corpus data and the video corpus data;

[0017] if there is no identifiable extension name in the large-scale corpus data, classifying the large-scale corpus data through manual annotation or a script program to obtain the text corpus data, the audio corpus data and the video corpus data.

[0018] Optionally, the obtaining supplementary text data by extracting text information from the audio corpus data and the video corpus data comprises:

[0019] converting spoken content in the audio corpus data into text information through automatic speech recognition to obtain audio supplementary text;

[0020] obtaining image data in the video corpus data, and converting recognizable text in the image data into text information through optical character recognition to obtain video supplementary text;

[0021] obtaining the supplementary text data according to the audio supplementary text and the video supplementary text.

[0022] Optionally, the constructing a PLM model according to the enhanced text data comprises:

[0023] selecting a suitable pre-training model architecture based on the data category of the enhanced text data to obtain a screening model;

[0024] mask training the screening model through the enhanced text data, to obtain a training model;

[0025] constructing a downstream task according to the enhanced text data, and fine-tuning the training model through the downstream task, to obtain the PLM model.

[0026] Optionally, the obtaining of the supplementary text data from a preset library through the PLM model comprises:

[0027] cleaning the data obtained from the library to obtain cleaned library data;

[0028] extracting features from the cleaned library data through a PLM model to obtain cleaned library features, and screening the cleaned library features based on a preset requirement to obtain screened library features;

[0029] obtaining text data corresponding to the screened library features from the library to obtain the supplementary text data.

[0030] Optionally, the extracting of the entity relationship in the expanded text data, audio corpus data and video corpus data, and the construction of a knowledge graph according to the entity relationship comprise:

[0031] transcribing the audio corpus data and the video corpus data into text form, and combining the expanded text data to obtain comprehensive text data;

[0032] identifying entities in the comprehensive text data through a word segmentation, part-of-speech tagging and named entity recognition method to obtain text entities;

[0033] identifying relationships between the text entities based on a preset semantic analysis model to obtain entity relationships, and integrating the text entities and the entity relationships to form a graph structure to obtain the knowledge graph.

[0034] Optionally, the enhancing of the expanded text data, audio corpus data and video corpus data according to the connection relationship in the knowledge graph to obtain enhanced text data, enhanced audio data and enhanced video data comprises:

[0035] recommending related terms or concepts for the expanded text data through the connection relationship in the knowledge graph to enhance the text data to obtain the enhanced text data;

[0036] adding related terms or concepts in the knowledge graph to the transcribed text of the audio corpus data to improve the information quantity of the audio corpus data to obtain enhanced audio data;

[0037] Metadata tags or letters are added to the video corpus data based on the knowledge graph, the video corpus data is enhanced, and enhanced video data is obtained.

[0038] To solve the above problems, the application further provides a language model construction device based on data enhancement, which comprises:

[0039] A data acquisition module is configured to acquire large-scale corpus data, classify the large-scale corpus data to obtain text corpus data, audio corpus data and video corpus data, extract text information from the audio corpus data and the video corpus data to obtain supplementary text data, and obtain comprehensive text data according to the supplementary text data and the text corpus data.

[0040] A data enhancement module is configured to construct a PLM model according to the enhanced text data, acquire text data from a preset library through the PLM model to obtain supplementary text data, and obtain expanded text data according to the supplementary text data and the comprehensive text data, extract entity relationships in the expanded text data, the audio corpus data and the video corpus data, construct a knowledge graph according to the entity relationships, and enhance the expanded text data, the audio corpus data and the video corpus data according to the connection relationships in the knowledge graph to obtain enhanced text data, enhanced audio data and enhanced video data.

[0041] A feature extraction module is configured to extract images of each frame in the enhanced video data to obtain enhanced image data, and normalize the enhanced text data, the enhanced audio data and the enhanced image data to obtain standard text data, standard audio data and standard image data, select a corresponding feature conversion method based on the data categories of the standard text data, the standard audio data and the standard image data, and convert the standard text data, the standard audio data and the standard image data into feature vectors to obtain text feature vectors, audio feature vectors and image feature vectors.

[0042] A model construction module is configured to construct a large language model through a Transformer model and the text feature vectors, the audio feature vectors and the image feature vectors.

[0043] To solve the above problems, the application further provides an electronic device, which comprises:

[0044] at least one processor; and

[0045] a memory in communication with the at least one processor; wherein

[0046] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the data enhancement-based language model construction method as described above.

[0047] To solve the above problems, the application further provides a computer-readable storage medium, comprising a storage data area and a storage program area, the storage data area stores created data, and the storage program area stores a computer program; wherein the computer program is executed by a processor to implement the data enhancement-based language model construction method as described above.

[0048] The embodiment of the application obtains large-scale corpus data, classifies the large-scale corpus data to obtain text corpus data, audio corpus data and video corpus data, extracts text information from the audio corpus data and the video corpus data to obtain supplementary text data, and obtains comprehensive text data according to the supplementary text data and the text corpus data, constructs a PLM model according to the enhanced text data, obtains text data from a preset library through the PLM model to obtain supplementary text data, and obtains expanded text data according to the supplementary text data and the comprehensive text data, extracts entity relationships in the expanded text data, the audio corpus data and the video corpus data, constructs a knowledge graph according to the entity relationships, and enhances the expanded text data, the audio corpus data and the video corpus data according to the connection relationships in the knowledge graph to obtain enhanced text data, enhanced audio data and enhanced video data, extracts images of each frame in the enhanced video data to obtain enhanced image data, and respectively standardizes the enhanced text data, the enhanced audio data and the enhanced image data to obtain standard text data, standard audio data and standard image data, selects a corresponding feature conversion method based on the data categories of the standard text data, the standard audio data and the standard image data, converts the standard text data, the standard audio data and the standard image data into feature vectors to obtain text feature vectors, audio feature vectors and image feature vectors, and constructs a large language model through a Transformer model and the text feature vectors, the audio feature vectors and the image feature vectors. Therefore, the data enhancement-based language model construction method, device, electronic equipment and computer-readable storage medium provided by the application expand and enhance the existing large-scale corpus data in multiple ways, so that the language model constructed based on the data in the subsequent process better reflects the performance of the language model and enhances the efficiency of data governance. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 A flowchart of a data enhancement-based language model construction method provided by an embodiment of the application is shown in the figure;

[0050] Figure 2 A module schematic diagram of a language model construction device based on data enhancement provided by an embodiment of the present application is provided.

[0051] Figure 3 An internal structure schematic diagram of an electronic device for implementing a language model construction method based on data enhancement provided by an embodiment of the present application is provided.

[0052] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0053] It should be understood that the specific embodiments described herein are merely intended to explain the present application and are not intended to limit the present application.

[0054] Embodiments of the present application provide a language model construction method based on data enhancement. The execution subject of the language model construction method based on data enhancement includes but is not limited to at least one of the electronic devices such as a server and a terminal which can be configured to execute the method provided by the embodiments of the present application. The server can be an independent server, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms. In other words, the language model construction method based on data enhancement can be executed by software or hardware installed in a remote device or a server device, and the software can be a blockchain platform. The server includes but is not limited to a single server, a server cluster, a cloud server, or a cloud server cluster.

[0055] Referring to Figure 1 A flowchart of a language model construction method based on data enhancement provided by an embodiment of the present application is shown. In the embodiment, the language model construction method based on data enhancement includes the following steps S1-S7:

[0056] S1, obtaining large-scale corpus data, and classifying the large-scale corpus data to obtain text corpus data, audio corpus data and video corpus data.

[0057] In the embodiment of the present application, the large-scale expected data is applied to the power system, and the large-scale corpus data in the field of the power system refers to various types of data sets including text, audio, image, video and structured data. These data come from various aspects of the power system, such as technical documents, communication records, equipment images, monitoring videos and real-time sensor data. By using these data, artificial intelligence models can be developed and trained to improve the intelligent level of the power system, including improving operation efficiency, optimizing maintenance strategy, improving customer service and enhancing decision support.

[0058] Further, the large-scale corpus data is classified to obtain text corpus data, audio corpus data and video corpus data, comprising:

[0059] Querying whether there is an identifiable extension name in the large-scale corpus data;

[0060] If there is an identifiable extension name in the large-scale corpus data, the large-scale corpus data is classified by the attributes of the identifiable extension name to obtain the text corpus data, the audio corpus data and the video corpus data;

[0061] If there is no identifiable extension name in the large-scale corpus data, the large-scale corpus data is classified by manual annotation or script program to obtain the text corpus data, the audio corpus data and the video corpus data.

[0062] In the embodiment of the present application, the large-scale corpus data including text, audio and video is obtained and classified, which can enrich data resources, improve data quality, and provide a basis for constructing a multi-modal knowledge graph and training an efficient artificial intelligence model, thereby enhancing the understanding ability of machines for human language and behavior and expanding the application range.

[0063] S2, supplementary text data is obtained by extracting text information from the audio corpus data and the video corpus data, and comprehensive text data is obtained according to the supplementary text data and the text corpus data.

[0064] In the embodiment of the present application, the process of extracting text information from the audio corpus data and the video corpus data to obtain supplementary text data can be understood as converting non-text form data (such as speech and video content) into text form, expanding data sources for further analysis and processing. For example, through automatic speech recognition (ASR), spoken content in audio data is converted into written content, and finally text data is obtained, and through image recognition, the text information in the image of the video is converted into text data.

[0065] In the embodiment of the present application, the supplementary text data obtained by extracting text information from the audio corpus data and the video corpus data comprises:

[0066] convert the spoken content in the audio corpus data into text information through automatic speech recognition, obtaining audio supplementary text;

[0067] obtain image data in the video corpus data, and convert recognizable text in the image data into text information through optical character recognition, obtaining video supplementary text;

[0068] obtain the supplementary text data according to the audio supplementary text and the video supplementary text.

[0069] In the embodiment of the application, automatic speech recognition (ASR) is a technology for converting human speech into written text. It involves multiple steps, including audio signal processing, feature extraction, acoustic model matching and language model application, to achieve automatic recognition and transcription of speech.

[0070] Optical character recognition (OCR) is a technology that can identify and extract machine-readable text from image files. In short, OCR technology can convert text content in scanned documents, photos or PDF images into editable and searchable electronic text data.

[0071] Further, text information is extracted from audio and video corpus data and combined with existing text data to form comprehensive text data. This can enhance data diversity, expand the corpus, improve the understanding and processing capabilities of the model for language, and support multi-modal learning and cross-modal interaction, thereby improving the performance of artificial intelligence systems in various natural language processing tasks.

[0072] For example, maintenance records of power systems may be recorded in video form. Through automatic speech recognition technology, oral instructions during maintenance can be converted into text, and text information appearing in the video content (such as equipment labels, operation step instructions, etc.) can be combined to construct a comprehensive text data set. This data set not only contains specific operation steps, but also may contain problems and solutions found during maintenance, providing rich information resources for the maintenance and fault handling of power systems.

[0073] S3, constructing a PLM model according to the enhanced text data, obtaining text data from a pre-set library through the PLM model, obtaining supplementary text data, and obtaining expanded text data according to the supplementary text data and the comprehensive text data.

[0074] In the embodiments of the present application, the PLM model refers to a pre-trained language model, which is a model used in the field of natural language processing (NLP) to process and understand human language. PLM learns the general features and patterns of language by pre-training on a large amount of text data, and then can be fine-tuned on specific tasks to improve performance. These models perform well in various NLP tasks, such as text classification, sentiment analysis, machine translation, text summarization, and question answering systems, etc.

[0075] In the field of power systems, PLM models can help process and understand a large amount of technical documents, operation manuals, fault reports, and maintenance records, etc. By extracting valuable information from these unstructured texts, PLM models can support intelligent monitoring, fault diagnosis, maintenance decision-making, and knowledge management, etc. in power systems. In addition, PLM models can also be combined with other data types of power systems (such as sensor data, log files, etc.) to achieve more comprehensive data analysis and decision support.

[0076] Further, the pre-set library is a collection of text data in the field of power technology, which is used to train and build pre-trained models in the field of power technology. The library contains power technology-related text data collected from different sources, such as technical papers, patents, and technical project reports, etc.

[0077] Further, the PLM model is constructed according to the enhanced text data, comprising:

[0078] Selecting a suitable pre-trained model architecture based on the data category of the enhanced text data to obtain a filtered model;

[0079] Mask training the filtered model through the enhanced text data to obtain a trained model;

[0080] Constructing a downstream task according to the enhanced text data, and fine-tuning the trained model through the downstream task to obtain the PLM model.

[0081] Further, the downstream task is a task for fine-tuning the model, which is constructed according to the enhanced text data, including a term classification task or a text classification task constructed based on the enhanced text data.

[0082] In the embodiments of the present application, the PLM model is used to obtain text data from the pre-set library to obtain supplementary text data, comprising:

[0083] Cleaning the data obtained from the library to obtain cleaned library data;

[0084] Feature extraction is performed on the cleaning library data through a PLM model to obtain cleaning library features, and the cleaning library features are filtered based on a preset requirement to obtain filtered library features.

[0085] Text data corresponding to the filtered library features is obtained from the library to obtain supplementary text data.

[0086] S4, extract the entity relationship in the expanded text data, audio corpus data and video corpus data, construct a knowledge graph according to the entity relationship, and enhance the expanded text data, audio corpus data and video corpus data according to the connection relationship in the knowledge graph to obtain enhanced text data, enhanced audio data and enhanced video data.

[0087] In the embodiment of the application, the entity relationship in the power technology field knowledge graph refers to the semantic link describing the mutual connection between entities such as experts, achievements and terms. These relationships help to clarify the research field of experts, organize their research achievements, and reveal the connection between terms, thereby forming a structured knowledge system to support various applications such as expert matching and knowledge retrieval.

[0088] Further, in the power system, the knowledge graph is a structured semantic knowledge base that represents entities (such as devices, components, events) and their relationships (such as ownership, function, causality) in a graphical manner. By extracting entities and relationships from expanded text data, audio corpus data and video corpus data, the constructed knowledge graph can help power system operators and analysts understand complex relationships, optimize decisions, and enhance data analysis.

[0089] For example, a power system knowledge graph may include entities such as substations, transmission lines, and power generation equipment, as well as their relationships such as control relationships, maintenance history, and fault records. Through this knowledge graph, the expanded text, audio and video data can be enhanced with cultural and contextualization, making it more rich and valuable.

[0090] In the embodiment of the application, the entity relationship in the expanded text data, audio corpus data and video corpus data is extracted, and the knowledge graph is constructed according to the entity relationship, comprising:

[0091] Transcribe the audio corpus data and video corpus data into text form, and combine the expanded text data to obtain comprehensive text data;

[0092] Identify the entities in the comprehensive text data through word segmentation, part-of-speech tagging and named entity recognition methods to obtain text entities;

[0093] The relationship between the text entities is identified based on a preset semantic analysis model to obtain entity relationships, and the text entities and the entity relationships are integrated to form a graph structure to obtain the knowledge graph.

[0094] Further, the augmented text data, audio corpus data and video corpus data are enhanced according to the connection relationship in the knowledge graph to obtain enhanced text data, enhanced audio data and enhanced video data, including:

[0095] The related terms or concepts are recommended for the augmented text data through the connection relationship in the knowledge graph to enhance the text data and obtain the enhanced text data.

[0096] The information amount of the audio corpus data is improved by adding the related terms or concepts in the knowledge graph in the transcription text of the audio corpus data to obtain enhanced audio data.

[0097] The video corpus data is enhanced by adding metadata labels or letters based on the knowledge graph to obtain enhanced video data.

[0098] In the embodiment of the application, by constructing a knowledge graph and using it to enhance the original data, the power system can realize the structuring of data and the richness of information, thereby improving the intelligent analysis capability, assisting decision making, improving the operation efficiency, promoting innovation, managing risks, and serving as a tool for training new employees to enhance the intelligent and automated level of the entire system.

[0099] For example, by constructing a knowledge graph containing power system components (such as generators, transformers) and their relationships (such as connections, dependencies), the monitoring system can enhance its understanding of the system state, thereby quickly locating problems and taking measures when a fault occurs. In addition, the knowledge graph can also be used to train new operators by visually displaying the structure of the system and the interaction between components.

[0100] S5, extract the image of each frame in the enhanced video data to obtain enhanced image data, and standardize the enhanced text data, enhanced audio data and enhanced image data to obtain standard text data, standard audio data and standard image data.

[0101] In the embodiment of the application, video data is essentially a sequence composed of a series of static images (frames), which are played quickly and continuously in time sequence to create dynamic visual effects, so images can be extracted from video data.

[0102] In a power system, the extracted video frames can also be used to monitor the operating status of equipment, detect abnormal behavior, or for training and simulation of operating scenarios. For example, by analyzing the frames of a substation video, unauthorized intrusion or equipment failure can be detected.

[0103] Further, the extracting the image of each frame in the enhanced video data to obtain enhanced image data comprises:

[0104] The enhanced video data is read through a preset video processing library, and each frame of the enhanced video data is accessed according to an interface of the video processing library to obtain the enhanced image data.

[0105] In the embodiment of the application, the standardizing the enhanced text data, enhanced audio data and enhanced image data to obtain standard text data, standard audio data and standard image data comprises:

[0106] The enhanced text data is segmented, stop words are removed, part-of-speech is restored and part-of-speech is labeled to obtain labeled text data, and the labeled text data is denoised and error-corrected to obtain the standard text data.

[0107] The audio signal is quantized, and background noise in the audio signal is removed to obtain the standard audio data.

[0108] The enhanced image data is scaled and cropped to obtain cropped image data, and the cropped text data is rotated and grayscale to obtain the standard image data.

[0109] In the embodiment of the application, the extraction and standardization of video frames into images, as well as the standardization of text, audio and video data, can improve the consistency and quality of power system data, facilitate cross-modal analysis and training of machine learning models, and thus improve the accuracy and efficiency of monitoring, diagnosis and predictive maintenance.

[0110] For example, in a power system, after the frames of the monitoring video are extracted and standardized, they can be used to monitor the operating status of power equipment in real time, detect abnormal behavior, or for training and simulation of operating scenarios. At the same time, the standardized text data can be used to improve the accuracy of the fault diagnosis system, and the standardized audio data can be used to analyze the sound patterns of power equipment during operation to identify potential faults.

[0111] S6, based on the data categories of the standard text data, standard audio data and standard image data, selecting a corresponding feature conversion method to convert the standard text data, standard audio data and standard image data into feature vectors to obtain text feature vectors, audio feature vectors and image feature vectors.

[0112] In the embodiment of the present application, the preprocessed corpus is converted into a feature vector, which can standardize different types of data, extract key information, reduce data dimension, improve computing efficiency, and help improve the performance and generalization ability of the machine learning model, making it more accurate and efficient in various AI tasks.

[0113] For example, in the power system, converting device monitoring reports into text feature vectors, converting device operation sounds into audio feature vectors, and converting images captured by surveillance cameras into image feature vectors can make the monitoring and diagnosis of the power system more automated and intelligent.

[0114] Further, based on the data categories of the standard text data, standard audio data and standard image data, the corresponding feature conversion method is selected to convert the standard text data, standard audio data and standard image data into feature vectors. The text data can be converted by text conversion methods such as bag-of-words model, TF-IDF, N-gram or Word2Vec to obtain feature vectors; for audio data, Mel spectrogram, MFCC (Mel frequency cepstrum coefficient) and audio energy extraction methods can be used for conversion to obtain feature vectors; for image data, color histogram, texture feature, shape descriptor, and convolutional features extracted by convolutional neural network (CNN) can be applied to obtain feature vectors.

[0115] S7, model construction is performed through a Transformer model and the text feature vector, the audio feature vector and the image feature vector to obtain a large language model.

[0116] In the embodiment of the present application, the Transformer model is a deep learning model based on self-attention mechanism, mainly used for processing sequence-to-sequence (Seq2Seq) tasks such as machine translation, text summarization and question answering systems.

[0117] Further, the application of the Transformer model in the power system mainly reflects its ability to efficiently process and analyze large-scale multi-modal data, realize real-time monitoring and fault prediction, and adapt to specific tasks using transfer learning, thereby improving the intelligent level of the power system, optimizing the operation efficiency, and providing personalized service experience for staff and users.

[0118] In the embodiment of the present application, for suitable large language model architecture for power business, text information can consider using recurrent neural network (RNN) or Transformer architecture, and for large language model of audio and image, convolutional neural network (CNN) or Transformer architecture can be considered. Since the data in the present application is power industry data, covering text, audio, image or video, the Transformer model is selected to construct a large language model.

[0119] Further, after obtaining the large language model, the large language model can be used to identify the user's question, and a reply can be provided according to the model information provided by the large language model.

[0120] The embodiment of the application obtains large-scale corpus data, classifies the large-scale corpus data, and obtains text corpus data, audio corpus data, and video corpus data; extracts text information from the audio corpus data and the video corpus data to obtain supplementary text data, and obtains comprehensive text data according to the supplementary text data and the text corpus data; constructs a PLM model according to the enhanced text data, obtains text data from a preset library through the PLM model, obtains supplementary text data, and obtains expanded text data according to the supplementary text data and the comprehensive text data; extracts entity relationships in the expanded text data, the audio corpus data, and the video corpus data, constructs a knowledge graph according to the entity relationships, and enhances the expanded text data, the audio corpus data, and the video corpus data according to the connection relationships in the knowledge graph to obtain enhanced text data, enhanced audio data, and enhanced video data; extracts images of each frame in the enhanced video data to obtain enhanced image data, and respectively standardizes the enhanced text data, the enhanced audio data, and the enhanced image data to obtain standard text data, standard audio data, and standard image data; selects a corresponding feature conversion method based on the data categories of the standard text data, the standard audio data, and the standard image data, converts the standard text data, the standard audio data, and the standard image data into feature vectors to obtain text feature vectors, audio feature vectors, and image feature vectors; constructs a model through a Transformer model and the text feature vectors, the audio feature vectors, and the image feature vectors to obtain a large language model. Therefore, the language model construction method, device, electronic equipment, and computer readable storage medium based on data enhancement proposed by the application expand and enhance the existing large-scale corpus data in multiple ways, so that the language model constructed based on the data later can better reflect the performance of the language model and enhance the efficiency of data governance.

[0121] As shown in Figure 2 Fig. 1 is a module schematic diagram of the language model construction device based on data enhancement.

[0122] The data enhancement-based language model construction device 100 can be installed in an electronic device. According to the functions implemented, the data enhancement-based language model construction device can include a data acquisition module 101, a data enhancement module 102, a feature extraction module 103, and a model construction module 104. The modules of the present application can also be referred to as units, which refer to a series of computer program segments that can be executed by an electronic device processor and can complete a fixed function, which are stored in the memory of the electronic device.

[0123] In the present embodiment, the functions of each module / unit are as follows:

[0124] The data acquisition module 101 is configured to acquire large-scale corpus data, classify the large-scale corpus data to obtain text corpus data, audio corpus data, and video corpus data, extract text information from the audio corpus data and the video corpus data to obtain supplementary text data, and obtain comprehensive text data based on the supplementary text data and the text corpus data.

[0125] The data enhancement module 102 is configured to construct a PLM model based on the enhanced text data, acquire text data from a predetermined library through the PLM model to obtain supplementary text data, and obtain expanded text data based on the supplementary text data and the comprehensive text data. The data enhancement module 102 is further configured to extract entity relationships in the expanded text data, audio corpus data, and video corpus data, construct a knowledge graph based on the entity relationships, and enhance the expanded text data, audio corpus data, and video corpus data based on the connection relationships in the knowledge graph to obtain enhanced text data, enhanced audio data, and enhanced video data.

[0126] The feature extraction module 103 is configured to extract images of each frame in the enhanced video data to obtain enhanced image data, and normalize the enhanced text data, enhanced audio data, and enhanced image data to obtain standard text data, standard audio data, and standard image data. The feature extraction module 103 is further configured to select a corresponding feature conversion method based on the data categories of the standard text data, standard audio data, and standard image data, convert the standard text data, standard audio data, and standard image data into feature vectors to obtain text feature vectors, audio feature vectors, and image feature vectors.

[0127] The model construction module 104 is configured to construct a large language model by using a Transformer model and the text feature vectors, audio feature vectors, and image feature vectors.

[0128] In detail, each module in the data enhancement-based language model construction device 100 in the present embodiment uses the above-described Figure 1The data augmentation-based language model construction method has the same technical means and can produce the same technical effects, and thus will not be described here.

[0129] As Figure 3 Fig. 1 is a structural schematic diagram of an electronic device for implementing the data augmentation-based language model construction method according to the present application.

[0130] The electronic device can include a processor 10, a memory 11, a communication bus 12, and a communication interface 13, and can further include a computer program stored in the memory 11 and executable on the processor 10, such as a data augmentation-based language model construction program.

[0131] In some embodiments, the processor 10 can be composed of integrated circuits, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same function or different functions, including one or more combinations of central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control core of the electronic device, which connects all components of the electronic device through various interfaces and lines, executes programs or modules stored in the memory 11 (such as the data augmentation-based language model construction program), and calls data stored in the memory 11 to perform various functions and process data of the electronic device.

[0132] The memory 11 includes at least one type of readable storage medium, including flash memories, mobile hard disks, multimedia cards, card-type memories (such as SD or DX memories, etc.), magnetic memories, magnetic disks, optical disks, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory 11 can also be an external storage device of the electronic device, such as a plug-in mobile hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 11 can include both an internal storage unit and an external storage device of the electronic device. The memory 11 can be used not only to store application software and various data installed in the electronic device, such as the code of the data augmentation-based language model construction program, but also to temporarily store data that has been output or will be output.

[0133] The communication bus 12 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize the connection and communication between the memory 11 and the at least one processor 10, etc.

[0134] The communication interface 13 is configured to realize the communication between the electronic device and other devices, and includes a network interface and a user interface. Optionally, the network interface can include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), and is usually configured to establish a communication connection between the electronic device and other electronic devices. The user interface can be a display (Display), an input unit (such as a keyboard (Keyboard)), and optionally, the user interface can also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display can be an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (Organic Light-Emitting Diode) touch screen, etc. The display can also be appropriately referred to as a display screen or a display unit, and is configured to display information processed in the electronic device and display a visualized user interface.

[0135] Figure 3 Only the electronic device with components is shown, and those skilled in the art can understand that, Figure 3 The structure shown does not constitute a limitation on the electronic device, and can include fewer or more components than shown, or combine certain components, or different component arrangements.

[0136] For example, although not shown, the electronic device can further include a power supply (such as a battery) for supplying power to each component. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, so as to realize functions such as charge management, discharge management and power consumption management through the power management device. The power supply can also include one or more direct current or alternating current power supplies, a recharging device, a power supply fault detection circuit, a power supply converter or inverter, a power supply status indicator, etc. The electronic device can also include various sensors, a Bluetooth module, a Wi-Fi module, etc., which are not described here.

[0137] It should be understood that the embodiments are only for illustration, and the scope of the patent application is not limited by the structure.

[0138] The data augmentation-based language model construction program stored in the memory 11 in the electronic device is a combination of multiple computer programs, which, when running in the processor 10, can realize:

[0139] Obtaining large-scale corpus data, and classifying the large-scale corpus data to obtain text corpus data, audio corpus data and video corpus data;

[0140] Obtaining supplementary text data by extracting text information from the audio corpus data and the video corpus data, and obtaining comprehensive text data according to the supplementary text data and the text corpus data;

[0141] Constructing a PLM model according to the augmented text data, obtaining supplementary text data from a preset library through the PLM model, and obtaining expanded text data according to the supplementary text data and the comprehensive text data;

[0142] Extracting entity relationships in the expanded text data, audio corpus data and video corpus data, constructing a knowledge graph according to the entity relationships, and enhancing the expanded text data, audio corpus data and video corpus data according to the connection relationships in the knowledge graph to obtain enhanced text data, enhanced audio data and enhanced video data;

[0143] Extracting images of each frame in the enhanced video data to obtain enhanced image data, and respectively normalizing the enhanced text data, enhanced audio data and enhanced image data to obtain standard text data, standard audio data and standard image data;

[0144] Based on the data categories of the standard text data, standard audio data and standard image data, selecting a corresponding feature conversion method to convert the standard text data, standard audio data and standard image data into feature vectors to obtain text feature vectors, audio feature vectors and image feature vectors;

[0145] Model construction is performed through a Transformer model and the text feature vectors, audio feature vectors and image feature vectors to obtain a large language model.

[0146] Specifically, the specific implementation method of the processor 10 for the above computer programs can refer to Figure 1 The descriptions of related steps in the corresponding embodiments are not repeated here.

[0147] Further, the modules / units integrated in the electronic device, if realized in the form of software function units and sold or used as independent products, can be stored in a nonvolatile computer readable storage medium. The computer readable storage medium can be volatile or nonvolatile. For example, the computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM).

[0148] The application further provides a computer readable storage medium, which stores a computer program, and the computer program can realize the following when executed by a processor of an electronic device:

[0149] Obtaining large-scale corpus data, classifying the large-scale corpus data to obtain text corpus data, audio corpus data and video corpus data;

[0150] Obtaining supplementary text data by extracting text information from the audio corpus data and the video corpus data, and obtaining comprehensive text data according to the supplementary text data and the text corpus data;

[0151] Constructing a PLM model according to the enhanced text data, obtaining text data from a preset library through the PLM model to obtain supplementary text data, and obtaining expanded text data according to the supplementary text data and the comprehensive text data;

[0152] Extracting entity relationships in the expanded text data, the audio corpus data and the video corpus data, constructing a knowledge graph according to the entity relationships, and enhancing the expanded text data, the audio corpus data and the video corpus data according to the connection relationships in the knowledge graph to obtain enhanced text data, enhanced audio data and enhanced video data;

[0153] Extracting images of each frame in the enhanced video data to obtain enhanced image data, and standardizing the enhanced text data, the enhanced audio data and the enhanced image data respectively to obtain standard text data, standard audio data and standard image data;

[0154] Based on the data categories of the standard text data, the standard audio data and the standard image data, selecting a corresponding feature conversion method to convert the standard text data, the standard audio data and the standard image data into feature vectors to obtain text feature vectors, audio feature vectors and image feature vectors;

[0155] Model construction through a Transformer model and the text feature vectors, the audio feature vectors and the image feature vectors to obtain a large language model.

[0156] In several embodiments provided by the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the above-described device embodiments are merely illustrative, and the division of the modules is merely a logical function division. Actual implementation can have another division manner.

[0157] The modules described as separate components can or can not be physically separated, and the components displayed as modules can or can not be physical units, i.e., can be located in one place or distributed to multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment.

[0158] In addition, each functional module in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of hardware plus software functional modules.

[0159] It is obvious for those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application.

[0160] Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present application is defined by the appended claims rather than the above description, and therefore all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present application. Any additional reference signs in the claims should not be considered as limiting the claims involved.

[0161] The blockchain referred to in the present application is a new application mode of distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm and other computer technologies. Blockchain, in essence, is a decentralized database, which is a series of data blocks associated using cryptography. Each data block contains information about a batch of network transactions, which is used to verify the validity (anti-fake) of the information and generate the next block. The blockchain can include a blockchain underlying platform, a platform product service layer, and an application service layer.

[0162] Embodiments of the present application can acquire and process related data based on artificial intelligence technology. Among them, artificial intelligence (AI) is to use digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0163] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in a system claim can also be implemented by one unit or device through software or hardware. The second word is used to indicate the name, not any specific order.

[0164] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present application.

Claims

1. A method for constructing a language model based on data enhancement, characterized in that, The method comprises: acquiring large-scale corpus data and classifying the large-scale corpus data to obtain text corpus data, audio corpus data and video corpus data; extracting text information from the audio corpus data and video corpus data to obtain supplementary text data, and obtaining comprehensive text data according to the supplementary text data and the text corpus data; constructing a PLM model according to enhanced text data, obtaining text data from a preset library through the PLM model to obtain supplementary text data, and obtaining expanded text data according to the supplementary text data obtained from the preset library and the comprehensive text data; extracting entity relationships in the expanded text data, audio corpus data and video corpus data, constructing a knowledge graph according to the entity relationships, and enhancing the expanded text data, audio corpus data and video corpus data according to the connection relationships in the knowledge graph to obtain enhanced text data, enhanced audio data and enhanced video data; wherein the enhancing of the expanded text data, audio corpus data and video corpus data according to the connection relationships in the knowledge graph to obtain enhanced text data, enhanced audio data and enhanced video data comprises: recommending related terms or concepts for the expanded text data through the connection relationships in the knowledge graph, enhancing the text data to obtain the enhanced text data; adding related terms or concepts in the knowledge graph to the transcription text of the audio corpus data to improve the information quantity of the audio corpus data to obtain enhanced audio data; adding metadata tags or letters to the video corpus data based on the knowledge graph to enhance the video corpus data to obtain enhanced video data; extracting images of each frame in the enhanced video data to obtain enhanced image data, and standardizing the enhanced text data, enhanced audio data and enhanced image data to obtain standard text data, standard audio data and standard image data; based on the data categories of the standard text data, standard audio data and standard image data, selecting a corresponding feature conversion method to convert the standard text data, standard audio data and standard image data into feature vectors to obtain text feature vectors, audio feature vectors and image feature vectors; constructing a model through a Transformer model and the text feature vectors, audio feature vectors and image feature vectors to obtain a large language model. 2.The data augmentation based language model construction method of claim 1, wherein, The classifying of the large-scale corpus data to obtain text corpus data, audio corpus data and video corpus data comprises: querying whether there is a recognizable extension name in the large-scale corpus data; if there is a recognizable extension name in the large-scale corpus data, classifying the large-scale corpus data through the attributes of the recognizable extension name to obtain the text corpus data, audio corpus data and video corpus data; If there is no identifiable extension in the large-scale corpus data, the large-scale corpus data is classified by manual labeling or a script program to obtain the text corpus data, the audio corpus data and the video corpus data. 3.The data augmentation based language model construction method of claim 1, wherein, The obtaining of the supplementary text data from the audio corpus data and the video corpus data comprises: converting the spoken content in the audio corpus data into text information by automatic speech recognition to obtain audio supplementary text; obtaining image data in the video corpus data and converting the identifiable text in the image data into text information by optical character recognition to obtain video supplementary text; obtaining the supplementary text data according to the audio supplementary text and the video supplementary text. 4.The data augmentation based language model construction method of claim 1, wherein, The construction of the PLM model according to the enhanced text data comprises: selecting a suitable pre-training model architecture based on the data category of the enhanced text data to obtain a screening model; mask training the screening model by the enhanced text data to obtain a training model; constructing a downstream task according to the enhanced text data and fine-tuning the training model by the downstream task to obtain the PLM model. 5.The data augmentation based language model construction method of claim 1, wherein, The obtaining of the supplementary text data from the preset library by the PLM model comprises: cleaning the data obtained from the library to obtain cleaned library data; extracting features of the cleaned library data by the PLM model to obtain cleaned library features, and screening the cleaned library features based on a preset requirement to obtain screened library features; obtaining text data corresponding to the screened library features from the library to obtain the supplementary text data. 6.The data augmentation based language model construction method of claim 1, wherein, The extraction of the entity relationship in the expanded text data, the audio corpus data and the video corpus data and the construction of the knowledge graph according to the entity relationship comprise: transcribing the audio corpus data and the video corpus data into text form and combining the expanded text data to obtain comprehensive text data; identifying entities in the comprehensive text data by a word segmentation, part-of-speech tagging and named entity recognition method to obtain text entities; identifying the relationship between the text entities based on a preset semantic analysis model to obtain entity relationships, and integrating the text entities and the entity relationships to form a graph structure to obtain the knowledge graph. 7.A data augmentation based language model construction apparatus, characterized in that, The device for implementing the data enhancement-based language model construction method according to any one of claims 1 to 6 comprises: a data acquisition module configured to acquire large-scale corpus data, classify the large-scale corpus data to obtain text corpus data, audio corpus data and video corpus data, extract text information from the audio corpus data and the video corpus data to obtain supplementary text data, and obtain comprehensive text data according to the supplementary text data and the text corpus data; The data enhancement module is configured to construct a PLM model according to the enhanced text data, acquire text data from a preset library through the PLM model, obtain supplementary text data, and obtain expanded text data according to the supplementary text data obtained from the preset library and the comprehensive text data; extract an entity relationship in the expanded text data, audio corpus data, and video corpus data, construct a knowledge graph according to the entity relationship, and enhance the expanded text data, the audio corpus data, and the video corpus data according to a connection relationship in the knowledge graph to obtain enhanced text data, enhanced audio data, and enhanced video data; The feature extraction module is configured to extract an image of each frame in the enhanced video data to obtain enhanced image data, and normalize the enhanced text data, the enhanced audio data, and the enhanced image data to obtain standard text data, standard audio data, and standard image data; select a corresponding feature conversion method based on a data category of the standard text data, the standard audio data, and the standard image data, and convert the standard text data, the standard audio data, and the standard image data into feature vectors to obtain text feature vectors, audio feature vectors, and image feature vectors; The model construction module is configured to construct a large language model by using a Transformer model and the text feature vectors, the audio feature vectors, and the image feature vectors.

8. An electronic device, comprising: The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the language model construction method based on data enhancement according to any one of claims 1 to 6.

9. A computer readable storage medium comprising a storage data area and a storage program area, the storage data area storing created data, the storage program area storing a computer program; wherein, The computer program is executed by the processor to implement the language model construction method based on data enhancement according to any one of claims 1 to 6.

Images