Data set construction method and apparatus, and computer device

Abstract

The application relates to a data set construction method, device and computer equipment. The method comprises the following steps: acquiring heterogeneous power equipment data to be processed, performing standardization processing on the heterogeneous power equipment data to obtain standardized heterogeneous data, performing question and answer pair labeling on the standardized heterogeneous data to obtain labeled data sets, mapping the standardized heterogeneous data to the same vector space for alignment processing to obtain aligned heterogeneous data, and taking the aligned heterogeneous data and corresponding labeled data in the labeled data sets as target data sets. The method can improve the reliability of the data sets.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a dataset construction method, apparatus, and computer device. Background Technology

[0002] As the power system transforms towards intelligence and digitalization, the reliance on large models in power transmission and distribution production command scenarios is deepening, making the performance evaluation and iterative optimization of large models a key requirement.

[0003] However, the power transmission and distribution production domain has characteristics such as multi-source heterogeneous data, dynamic updates of professional knowledge, and complex multimodal data fusion, which leads to low reliability of the dataset constructed from the status data of power transmission and distribution equipment, making it difficult to apply to the training of large models. Summary of the Invention

[0004] Therefore, it is necessary to provide a dataset construction method, apparatus, and computer equipment that can improve the reliability of datasets in response to the above-mentioned technical problems.

[0005] Firstly, this application provides a method for constructing a dataset, including:

[0006] Acquire data from heterogeneous power equipment to be processed;

[0007] Standardize the data from heterogeneous power equipment to obtain standardized heterogeneous data.

[0008] Standardized heterogeneous data is labeled with question-answer pairs to obtain a labeled dataset;

[0009] Standardized heterogeneous data is mapped to the same vector space and aligned to obtain aligned heterogeneous data.

[0010] Align heterogeneous data with the corresponding labeled data in the labeled dataset as the target dataset.

[0011] In one embodiment, heterogeneous power equipment data is standardized to obtain standardized heterogeneous data, including:

[0012] Identify heterogeneous power equipment data and determine its characteristic dimensions;

[0013] Adjust the dimensions of the input layer of the specified feature extraction network according to the feature dimensions;

[0014] Based on the dimension-adjusted feature extraction network, feature extraction is performed on heterogeneous power equipment data to obtain a power feature matrix;

[0015] Based on the power characteristic matrix, standardized heterogeneous data are determined.

[0016] In one embodiment, standardized heterogeneous data is determined based on the power feature matrix, including:

[0017] Data type identification is performed on the power feature matrix to determine the edge data matrix within the power feature matrix;

[0018] Data augmentation processing is performed on the edge data matrix to obtain the augmented data matrix;

[0019] The power characteristic matrix and the enhanced data matrix were identified as standardized heterogeneous data.

[0020] In one embodiment, the edge data matrix is ​​subjected to data augmentation processing to obtain an augmented data matrix, including:

[0021] Random noise is added to the edge data matrix to obtain the enhanced data matrix. The noise intensity of the random noise is determined based on the standard deviation of the edge data matrix.

[0022] In one embodiment, question-answer pair annotation is performed on standardized heterogeneous data to obtain an annotated dataset, including:

[0023] Based on the specified annotation model, question-answer pair annotations are performed on standardized heterogeneous data to obtain an initial annotation set;

[0024] The initial annotation set is validated, and the annotation dataset that meets the preset quality requirements is selected.

[0025] In one embodiment, before obtaining the initial annotation set by annotating standardized heterogeneous data with question-answer pairs based on a specified annotation model, the process includes:

[0026] Obtain test data samples and labeled question-and-answer pairs from the labeled sample library, wherein the labeled question-and-answer pairs are constructed based on entities and entity relationships at each level in the specified power knowledge graph;

[0027] Based on the labeled question-answer pairs, the initial labeled model is trained with the first learning rate to obtain the undetermined labeled model;

[0028] Based on the undetermined annotation model, the test data samples are predicted, and the predicted annotation data with a prediction confidence level greater than or equal to a predetermined confidence threshold are selected.

[0029] Based on the labeled question-answer pairs and the predicted labeled data, the specified labeled model is trained with a second learning rate to obtain the specified labeled model, wherein the second learning rate is less than the first learning rate.

[0030] In one embodiment, normalized heterogeneous data is mapped to the same vector space for alignment processing to obtain aligned heterogeneous data, including:

[0031] Based on the data type of each data sample in the standardized heterogeneous data, the data samples are encoded by the corresponding modal encoders to obtain the modal feature vectors of each data type.

[0032] Modal feature vectors of various data types are mapped to the same vector space through a shared encoder;

[0033] Alignment processing is performed on the modal feature vectors within the same vector space to obtain aligned modal feature vectors, which are then identified as aligned heterogeneous data.

[0034] In one embodiment, the method further includes:

[0035] Incremental update operation is triggered when the number of newly added heterogeneous power equipment data exceeds the specified growth amount, or when the time difference between the last update time and the current time exceeds the specified duration threshold.

[0036] In response to the incremental update operation, the aligned heterogeneous data and labeled data corresponding to the newly added heterogeneous power equipment data are added as incremental data to the target dataset.

[0037] Secondly, this application also provides a dataset construction apparatus, comprising:

[0038] The data acquisition module is used to acquire data from heterogeneous power equipment to be processed.

[0039] The standardization module is used to standardize heterogeneous power equipment data to obtain standardized heterogeneous data.

[0040] The data annotation module is used to annotate standardized heterogeneous data with question-answer pairs to obtain an annotated dataset;

[0041] The data alignment module is used to map standardized heterogeneous data to the same vector space for alignment processing, resulting in aligned heterogeneous data.

[0042] The dataset determination module is used to align heterogeneous data with the corresponding labeled data in the labeled dataset as the target dataset.

[0043] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0044] Acquire data from heterogeneous power equipment to be processed;

[0045] Standardize the data from heterogeneous power equipment to obtain standardized heterogeneous data.

[0046] Standardized heterogeneous data is labeled with question-answer pairs to obtain a labeled dataset;

[0047] Standardized heterogeneous data is mapped to the same vector space and aligned to obtain aligned heterogeneous data.

[0048] Align heterogeneous data with the corresponding labeled data in the labeled dataset as the target dataset.

[0049] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0050] Acquire data from heterogeneous power equipment to be processed;

[0051] Standardize the data from heterogeneous power equipment to obtain standardized heterogeneous data.

[0052] Standardized heterogeneous data is labeled with question-answer pairs to obtain a labeled dataset;

[0053] Standardized heterogeneous data is mapped to the same vector space and aligned to obtain aligned heterogeneous data.

[0054] Align heterogeneous data with the corresponding labeled data in the labeled dataset as the target dataset.

[0055] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0056] Acquire data from heterogeneous power equipment to be processed;

[0057] Standardize the data from heterogeneous power equipment to obtain standardized heterogeneous data.

[0058] Standardized heterogeneous data is labeled with question-answer pairs to obtain a labeled dataset;

[0059] Standardized heterogeneous data is mapped to the same vector space and aligned to obtain aligned heterogeneous data.

[0060] Align heterogeneous data with the corresponding labeled data in the labeled dataset as the target dataset.

[0061] The aforementioned dataset construction method, apparatus, and computer equipment acquire heterogeneous power equipment data to be processed, standardize the heterogeneous power equipment data to obtain standardized heterogeneous data, perform question-and-answer pair annotation on the standardized heterogeneous data to obtain an annotated dataset, map the standardized heterogeneous data to the same vector space for alignment processing to obtain aligned heterogeneous data, and use the aligned heterogeneous data and the corresponding annotated data in the annotated dataset as the target dataset. Thus, this application eliminates the original noise and format inconsistencies of heterogeneous power equipment data at the data level through standardization, achieves semantic alignment of different data types from multiple sources at the feature level through alignment, realizes semantic consistency of heterogeneous power equipment data, and provides clearly supervised learning information at the label level through question-and-answer pair annotation. This systematically solves the problems of format inconsistency, noise, semantic gaps, and semantic ambiguity existing in multi-source heterogeneous power equipment data, and helps improve the reliability of the constructed target dataset. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 This is a diagram illustrating the application environment of a dataset construction method in one embodiment.

[0064] Figure 2 This is a flowchart illustrating a dataset construction method in one embodiment;

[0065] Figure 3 This is a schematic diagram of an alignment process in one embodiment;

[0066] Figure 4 This is a flowchart illustrating the dataset construction method in another embodiment;

[0067] Figure 5 This is a schematic diagram of the model structure for standardized processing in one embodiment;

[0068] Figure 6 This is a flowchart illustrating the dataset construction method in yet another embodiment;

[0069] Figure 7 This is a flowchart illustrating the dataset construction method in a specific embodiment of this application;

[0070] Figure 8 Here is a structural block diagram of a dataset construction apparatus in one embodiment;

[0071] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0072] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0073] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0074] The dataset construction method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on the cloud or other network servers. Taking server 104 as the execution entity as an example, server 104 can request heterogeneous power equipment data to be processed from terminal 102. After server 104 obtains the heterogeneous power equipment data, it can standardize the data to obtain standardized heterogeneous data, perform question-and-answer pair annotation on the standardized heterogeneous data to obtain an annotated dataset, map the standardized heterogeneous data to the same vector space for alignment processing to obtain aligned heterogeneous data, and use the aligned heterogeneous data and the corresponding annotated data in the annotated dataset as the target dataset. Terminal 102 can be, but is not limited to, various power transmission and distribution equipment, personal computers, laptops, smartphones, tablets, IoT devices, and other terminal devices. Server 104 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0075] In one exemplary embodiment, such as Figure 2 As shown, a dataset construction method is provided, which can be applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps S110 to S150. Wherein:

[0076] Step S110: Obtain the heterogeneous power equipment data to be processed.

[0077] Among them, heterogeneous power equipment data can include text data of power transmission and distribution equipment (such as equipment fault records and test reports), image data (such as equipment appearance diagrams, internal structure diagrams, and fault feature images), tabular data (such as equipment maintenance process tables and parameter configuration tables), and business data (such as scheduling instruction records and operation status monitoring data in production command scenarios).

[0078] For example, this embodiment can collect heterogeneous power equipment status data from multiple sources covering power transmission and distribution equipment in business domains such as power generation, substation, transmission, and distribution. For instance, this embodiment can use methods such as exporting structured equipment status data (e.g., technical standard documents) from a structured database, such as batch extraction of data from an SQL (Structured Query Language) database, and collecting unstructured equipment status data through methods such as web crawling and real-time uploading of data from equipment terminals (e.g., sensor monitoring data).

[0079] Step S120: Standardize the heterogeneous power equipment data to obtain standardized heterogeneous data.

[0080] Standardization processing involves pre-setting operations to eliminate differences in the format of status data for different devices in heterogeneous power equipment data, such as cleaning abnormal data and completing missing data.

[0081] This embodiment preprocesses the heterogeneous power equipment data by performing abnormal data cleaning and missing data completion on the equipment status data. Then, it extracts features from each preprocessed equipment status data to obtain standardized heterogeneous data. Furthermore, since real data often has a high proportion of marginal data, this embodiment employs a three-stage standardization process—feature extraction, data classification, and enhancement—to obtain standardized heterogeneous data. This ensures that the standardized heterogeneous data not only maintains consistency in data format but also achieves a balance between different data types (such as different equipment or data sources) through data classification and enhancement. For example, this embodiment can use an orthogonal basis feedforward neural network for feature extraction, with the input layer dimension of the orthogonal basis feedforward neural network matching the feature dimension of each equipment status data. For instance, this embodiment can perform feature dimension statistics on equipment status data (such as voltage, current, temperature, and equipment status codes) to obtain the number of effective features of the equipment status data as the feature dimension. The input layer dimension of the orthogonal basis feedforward neural network is adaptively adjusted according to the feature dimension of each equipment status data, ensuring consistency between the input layer dimension and the feature dimension of the equipment status data. For common equipment types in the power transmission and distribution field, such as transformers, switchgear, and circuit breakers, standard feature dimension templates corresponding to these equipment types are predefined. When the equipment type corresponding to the equipment status data is a known equipment type, the standard feature dimension template corresponding to that equipment type can be loaded to quickly configure the input layer dimension of the orthogonal basis feedforward neural network. In cases where equipment status data has missing features, mean imputation, linear interpolation, or feature completion algorithms can be used for completion processing to ensure that the input dimension is consistent with the input layer dimension. This embodiment can identify the data type of the feature samples obtained after feature extraction from each equipment status data to obtain the feature sample distribution. Based on the feature sample distribution, marginal samples are selected from the feature samples. For example, feature samples with a mean ± 1 standard deviation are selected as marginal samples from the feature sample distribution. This embodiment can perform data augmentation processing on the marginal samples to obtain augmented samples. This embodiment can use feature samples and augmented samples as data samples. The set of each data sample is standardized heterogeneous data, which reduces the reliability of the dataset caused by sample imbalance.

[0082] Step S130: Perform question-answer pair annotation on the standardized heterogeneous data to obtain an annotated dataset.

[0083] This embodiment can use a specified annotation model to perform question-and-answer pair annotations on each data sample in standardized heterogeneous data, obtaining initial annotations for each data sample. The set of these initial annotations constitutes the initial annotation set. The specified annotation model can be a pre-trained large language model or a neural network model constructed based on the annotation question-and-answer pairs built from entities and entity relationships at each level of a specified power knowledge graph. This embodiment can directly use the initial annotation set as the annotation dataset, or it can validate the initial annotation set and select annotation datasets that meet preset quality requirements to improve the quality of the annotation dataset.

[0084] In some embodiments, question-answer pair annotation is performed on standardized heterogeneous data to obtain an annotated dataset, including:

[0085] Step S131: Based on the specified annotation model, perform question-answer pair annotation on the standardized heterogeneous data to obtain the initial annotation set.

[0086] Step S132: Verify the initial annotation set and select the annotation dataset that meets the preset quality requirements.

[0087] The specified annotation model can be a pre-trained large language model or any neural network model constructed based on the annotated question-answer pairs built from entities and entity relationships at each level of the specified power knowledge graph. The preset quality requirements are the quality requirements set in advance for the annotated dataset, such as semantic consistency, graph-text matching, and completeness.

[0088] This embodiment can perform question-and-answer pair annotation on each data sample in standardized heterogeneous data based on a specified annotation model to obtain the initial annotations for each data sample. The set of all initial annotations is called the initial annotation set. Then, this embodiment can perform at least one of the following checks on the initial annotation set: semantic consistency, image-text matching, and completeness, to filter out the annotated dataset that meets the preset quality requirements. For example, for text samples in standardized heterogeneous data, this embodiment compares the initial annotations of the text samples with the standard text data in the vector knowledge base. If the obtained text similarity (e.g., cosine similarity) is greater than or equal to a predetermined similarity threshold (e.g., 0.75, 0.8, 0.85), it is determined to have passed the semantic consistency check. For image samples in standardized heterogeneous data, this embodiment compares the initial annotations of the image samples with the image samples. If the obtained image-text similarity is greater than or equal to a predetermined similarity threshold (e.g., 0.75, 0.8, 0.85), it is determined to have passed the image-text matching check. If the precision of the initial labeled set is greater than or equal to a predetermined precision threshold (e.g., 0.75, 0.8, 0.85), the recall is greater than or equal to a predetermined recall threshold (e.g., 0.75, 0.8, 0.85), and the F1 score (a statistical indicator used to measure the accuracy of binary classification models) is greater than or equal to a predetermined threshold (e.g., 0.75, 0.8, 0.85), the integrity check is deemed to have passed. This embodiment can mark the reasons for non-compliance with labeled data that does not meet the preset quality requirements, relabel the data, and transfer the labeled data and its corresponding data sample to a manual labeling queue if the relabeled data still does not meet the preset quality requirements.

[0089] Step S140: The standardized heterogeneous data is mapped to the same vector space for alignment processing to obtain aligned heterogeneous data.

[0090] This embodiment can convert each data sample in standardized heterogeneous data into feature vectors, map them to the same vector space, and then align the feature vectors in the unified vector space to obtain aligned feature vectors as aligned heterogeneous data. For example, this embodiment can encode each data sample in the standardized heterogeneous data using a corresponding modal encoder to obtain modal feature vectors for each data type. These modal feature vectors are then mapped to the same vector space using a shared encoder. Alignment is then performed on the modal feature vectors in the same vector space to obtain aligned modal feature vectors, which are then identified as aligned heterogeneous data.

[0091] In some embodiments, normalized heterogeneous data is mapped to the same vector space for alignment processing to obtain aligned heterogeneous data, including:

[0092] Step S141: Based on the data type of each data sample in the standardized heterogeneous data, the data samples are encoded by the corresponding modal encoders to obtain the modal feature vectors of each data type.

[0093] Step S142: The modal feature vectors of each data type are mapped to the same vector space through a shared encoder.

[0094] Step S143: Align the modal feature vectors in the same vector space to obtain aligned modal feature vectors, which are then identified as aligned heterogeneous data.

[0095] Modal encoders can include text encoders, image encoders, table encoders, and other encoders for data samples of different modalities, used to extract modal features from data samples of different modalities. A shared encoder is used to map various feature vectors to the same vector space, eliminating spatial heterogeneity between different modal features such as text, images, and tables, and providing a unified semantic representation and vector mapping space for various modal features.

[0096] This embodiment can encode each data sample in the standardized heterogeneous data according to its data type using a corresponding modal encoder to obtain modal feature vectors for each data type. It is understood that, for ease of subsequent alignment operations, the dimensions of the modal feature vectors for each data type are consistent. Then, the modal feature vectors of each data type are mapped to the same vector space using a common encoder. Alignment processing is then performed on the modal feature vectors within the same vector space to obtain aligned modal feature vectors, which are then identified as aligned heterogeneous data. For example, this embodiment can use modal feature vectors with the same semantics within the same vector space as positive sample pairs, and modal feature vectors with different semantics as negative sample pairs. Then, based on the first difference between positive sample pairs and the second difference between negative sample pairs, the parameters of the common encoder are iteratively updated. Based on the updated common encoder, new modal feature vectors are obtained until the first difference is minimized and the second difference is maximized, resulting in aligned modal feature vectors as aligned heterogeneous data. The loss function for the shared encoder can be the in-sample collaborative loss function: central moment loss × K1 + orthogonality loss × K2, where K1 is the weight of the central moment loss and K2 is the weight of the orthogonality loss. The central moment loss characterizes the differences in the first and second moments of the feature vectors of each modality, and the orthogonality loss is the difference between the Gram matrix of the feature matrix and the identity matrix. The Gram matrix is ​​used to describe the pairwise correlations between features. For example, such as... Figure 3As shown, the text encoder can use the RoBERTa (Robustly Optimized BERT Pretraining Approach) model, outputting a 768-dimensional feature vector. The image encoder can use the ResNet (Residual Network) model, outputting a 2048-dimensional feature vector. The table encoder can use the TabNet (Tabular Data Network) model, outputting a 512-dimensional feature vector. Fully connected layers can be added after each modality encoder to unify the feature dimensions to a specified dimension (e.g., 768 dimensions). The shared encoder can be a dual encoder based on the Transformer architecture. This neural network model can include 6 encoding layers, each containing a multi-head self-attention mechanism (the number of heads can be selected according to requirements, such as 6 or 8) and a feedforward neural network. This feedforward neural network can use LayerNorm (Layer Normalization) and Dropout to prevent overfitting.

[0097] one

[0098] Step S150: Align the heterogeneous data with the corresponding labeled data in the labeled dataset as the target dataset.

[0099] In this embodiment, the corresponding labeled data in the aligned heterogeneous data and labeled dataset can be used as the target dataset. This target dataset can be used for model training, model testing and other purposes in the field of power transmission and distribution.

[0100] This embodiment can also perform data balancing on the target dataset. Data balancing is a processing operation that balances the types of data. For example, resampling: oversampling is performed on sample data in business scenarios with a small sample size (aligning heterogeneous data with corresponding labeled data), that is, sampling processing with a sampling rate greater than 1 (e.g., 1.5-2 times). Undersampling is performed on sample data in scenarios with a large sample size, that is, sampling processing with a sampling rate less than 1 (e.g., 0.5-0.7 times). During the sampling process, the SMOTE (Synthetic Minority Oversampling Technique) algorithm can be used to avoid overfitting. This embodiment can also employ an adversarial generative model to synthesize sample data for business scenarios with limited sample sizes. The generator of this adversarial generative network uses a 4-layer deconvolutional network, and the discriminator uses a 4-layer convolutional network. The input is the feature vector of the real sample data, and the output is the generated simulated data. The training iterations are 50-100 rounds. The distribution difference between the generated simulated data and the real sample data is verified using KL (Kullback-Leible) divergence, where the KL divergence is ≤ a predetermined divergence threshold (e.g., 0.1). In this embodiment, the data in the target dataset exhibits uneven distribution, meaning the difference in the proportion of sample data for each type (e.g., business scenario type, device type, fault type) exceeds a specified difference threshold (e.g., 10%). Data balancing is performed on the target dataset, optimizing the distribution of sample data through at least one method: resampling and data generation via the adversarial generative network, so that the difference in the proportion of sample data for each type (e.g., business scenario type, device type, fault type) is less than or equal to the specified difference threshold.

[0101] In some embodiments, the method further includes:

[0102] Step S160: If the number of newly added heterogeneous power equipment data exceeds the specified growth amount, or if the time difference between the last update time and the current time exceeds the specified duration threshold, an incremental update operation is triggered.

[0103] In step S170, in response to the incremental update operation, the aligned heterogeneous data and labeled data corresponding to the newly added heterogeneous power equipment data are added as incremental data to the target dataset.

[0104] The specified duration threshold is a pre-set fixed period used to trigger updates, such as three days or one week. The specified growth amount is a pre-set amount of data used to trigger updates.

[0105] For updating the target dataset, this embodiment can use either timed updates or new triggering to incrementally update the target dataset. For example, this embodiment can acquire heterogeneous power equipment data for a batch within a fixed period (e.g., one week), standardize the heterogeneous power equipment data to obtain standardized heterogeneous data, perform question-and-answer pair annotation on the standardized heterogeneous data to obtain an annotated dataset, and map the standardized heterogeneous data to the same vector space for alignment processing to obtain aligned heterogeneous data. Then, the aligned heterogeneous data and annotated datasets corresponding to all heterogeneous power equipment data within that period are added to the target dataset. This embodiment can also standardize the newly added heterogeneous power equipment data when the number of new heterogeneous power equipment data exceeds a specified growth number, perform question-and-answer pair annotation on the standardized heterogeneous data to obtain an annotated dataset, and map the standardized heterogeneous data to the same vector space for alignment processing to obtain aligned heterogeneous data. Then, the aligned heterogeneous data and annotated datasets corresponding to the specified growth number of newly added heterogeneous power equipment data are added to the target dataset. This embodiment triggers an incremental update operation when the number of newly added heterogeneous power equipment data exceeds a specified growth amount, or when the time difference between the last update time and the current time exceeds a specified duration threshold. In response to the incremental update operation, the aligned heterogeneous data and labeled data corresponding to the newly added heterogeneous power equipment data are added as incremental data to the target dataset, thereby achieving periodic or quantitative incremental updates to the target dataset.

[0106] In the aforementioned dataset construction method, heterogeneous power equipment data to be processed is acquired, standardized to obtain standardized heterogeneous data, question-and-answer pair annotation is performed on the standardized heterogeneous data to obtain an annotated dataset, the standardized heterogeneous data is mapped to the same vector space for alignment to obtain aligned heterogeneous data, and the aligned heterogeneous data and the corresponding annotated data in the annotated dataset are used as the target dataset. Therefore, this embodiment eliminates the original noise and format inconsistencies of heterogeneous power equipment data at the data level through standardization, achieves semantic alignment of different data types from multiple sources through alignment at the feature level, realizes semantic consistency of heterogeneous power equipment data, and provides clearly supervised learning information through question-and-answer pair annotation at the label level. This systematically solves the problems of format inconsistency, noise, semantic gap, and semantic ambiguity in multi-source heterogeneous power equipment data, helping to improve the reliability of the constructed target dataset.

[0107] In one exemplary embodiment, such as Figure 4 As shown, the standardization process for heterogeneous power equipment data to obtain standardized heterogeneous data includes steps S210 to S240. Wherein:

[0108] Step S210: Identify the heterogeneous power equipment data and determine the feature dimensions of the heterogeneous power equipment data.

[0109] Step S220: Adjust the dimension of the input layer of the specified feature extraction network according to the feature dimension.

[0110] Step S230: Based on the dimension-adjusted feature extraction network, feature extraction is performed on the heterogeneous power equipment data to obtain the power feature matrix.

[0111] Step S240: Based on the power feature matrix, determine the standardized heterogeneous data.

[0112] For example, this embodiment can identify the status data of each device in heterogeneous power equipment data, determine the feature dimensions of each device status data, and adjust the dimension of the input layer of a specified feature extraction network according to the feature dimensions. Taking an orthogonal basis feedforward neural network as an example, the input layer dimension of the orthogonal basis feedforward neural network matches the feature dimensions of each device status data. For example, this embodiment can perform feature dimension statistics on device status data (such as voltage, current, temperature, device status code, etc.) to obtain the number of effective features of the device status data as the feature dimensions. The input layer dimension of the orthogonal basis feedforward neural network is adaptively adjusted according to the feature dimensions of each device status data, so that the input layer dimension is consistent with the feature dimensions of the device status data. For common equipment types in the power transmission and distribution field, such as transformers, switchgear, circuit breakers, etc., standard feature dimension templates corresponding to the equipment type are predefined. When the equipment type corresponding to the device status data belongs to a known equipment type, the standard feature dimension template corresponding to the equipment type can be loaded to quickly configure the input layer dimension of the orthogonal basis feedforward neural network, so that the input layer dimension matches the feature dimensions of each device status data.

[0113] Therefore, this embodiment can extract features from heterogeneous power equipment data based on a dimensionally adjusted feature extraction network to obtain various power feature matrices. Based on these power feature matrices, standardized heterogeneous data is determined. This embodiment can directly use the power feature matrices as standardized heterogeneous data, or it can identify data types within the power feature matrices, determine the edge data matrices, perform data augmentation on the edge data matrices to obtain augmented data matrices, and then define the power feature matrices and augmented data matrices as standardized heterogeneous data. This ensures data type balance within the standardized heterogeneous data.

[0114] This embodiment identifies heterogeneous power equipment data, determines the feature dimensions of the heterogeneous power equipment data, and adjusts the dimensions of the input layer of the specified feature extraction network according to the feature dimensions, so that the specified feature extraction network can be adapted to different heterogeneous power equipment data, meeting the feature extraction needs of different heterogeneous power equipment data while ensuring the format consistency of standardized heterogeneous data.

[0115] In some embodiments, standardized heterogeneous data is determined based on the power feature matrix, including:

[0116] Step S241: Identify the data types in the power feature matrix and determine the edge data matrix in the power feature matrix.

[0117] Step S242: Perform data augmentation processing on the edge data matrix to obtain the augmented data matrix.

[0118] Step S243: The power feature matrix and the enhanced data matrix are determined to be standardized heterogeneous data.

[0119] like Figure 5As shown, this embodiment employs an unsupervised learning sparse autoencoder network for data category identification. The unsupervised learning sparse autoencoder network includes a feedforward neural network and an autoencoder module. The autoencoder module includes an encoding layer and a decoding layer. The feedforward neural network includes an input layer, hidden layers, activation functions, a matrix transformation module, and input / output layers. The hidden layer can have 2-3 layers, with the number of neurons in each hidden layer being 1.5-2 times that of the input layer. The activation function uses the ReLU (Rectified Linear Unit) function. Through orthogonal basis functions, such as Legendre orthogonal bases, the input data undergoes a linear transformation, achieving both linear and non-linear feature extraction. The output dimension is uniformly set to a 256-dimensional feature vector. The matrix transformation module then converts the feature vector into an n×256-dimensional matrix, where n is the number of data samples. The input / output layer of the feedforward neural network inputs the matrix-transformed feature data into the autoencoder module. The number of neurons in the encoding layer of the autoencoder module is 0.5 times that of the input layer. The decoding layer has the same dimension as the input / output layer, and the sparsity penalty coefficient is set to 0.01-0.05. Data category identification is achieved by minimizing the reconstruction error. The principle behind minimizing reconstruction error to identify data types is that an unsupervised learning sparse autoencoder network, trained with known data from the power transmission and distribution field, has learned the feature distribution patterns of different data types. It exhibits low reconstruction deviation and error between data of the same type, but large reconstruction deviation and error between different types of data. Therefore, this embodiment can identify data types from different equipment types and data sources by calculating the reconstruction error of each sample in the power feature matrix and combining it with the statistical characteristics of the error distribution to define a threshold. Furthermore, this embodiment can use power feature matrices with reconstruction errors greater than the mean ± 1 standard deviation as marginal data matrices, and perform data augmentation using methods such as adding random noise and data interpolation (i.e., linear interpolation to supplement missing values).

[0120] In some embodiments, data augmentation processing is performed on the edge data matrix to obtain an augmented data matrix, including:

[0121] Step S310: Add random noise to the edge data matrix to obtain the enhanced data matrix. The noise intensity of the random noise is determined based on the standard deviation of the edge data matrix.

[0122] This embodiment adds random noise to the edge data matrix to obtain an enhanced data matrix. The noise intensity is determined based on the standard deviation of the edge data matrix, for example, 0.1-0.2 times the standard deviation. By adding random noise, this embodiment expands the sample diversity of the edge data without changing its core features and physical meaning. This allows the dataset to better reflect the distribution of equipment state characteristics in actual production, while also improving the robustness of the large model trained and evaluated on this dataset. This enables the large model to maintain stable performance even when faced with equipment state data exhibiting slight fluctuations in actual production, thus enhancing the robustness of the large model trained on the target dataset.

[0123] In one exemplary embodiment, such as Figure 6 The process shown involves steps S410 to S440 before obtaining the initial annotation set by annotating standardized heterogeneous data using a specified annotation model. Specifically:

[0124] Step S410: Obtain test data samples and labeled question-and-answer pairs from the labeled sample library, wherein the labeled question-and-answer pairs are constructed based on entities and entity relationships at each level in the specified power knowledge graph.

[0125] Step S420: Based on the labeled question-answer pairs, train the initial labeled model with the first learning rate to obtain the undetermined labeled model.

[0126] Step S430: Based on the undetermined annotation model, predict the test data samples and filter the predicted annotation data with a prediction confidence level greater than or equal to a predetermined confidence threshold.

[0127] Step S440: Based on the labeled question-answer pairs and the predicted labeled data, train the specified labeled model with a second learning rate to obtain the specified labeled model, wherein the second learning rate is less than the first learning rate.

[0128] The designated power knowledge graph refers to specific power-related knowledge graphs, such as power equipment ontology graphs and maintenance procedure knowledge graphs in the power transmission and distribution field. The initial annotation model can be a pre-trained model such as BERT or RoBERTa that can be used to predict labeled data. For example, the initial annotation model might use BERT-base as the backbone of the pre-trained model, freezing the weights of the first 8 layers during training and fine-tuning the last 4 layers and the classification head. The predetermined confidence threshold is a pre-set confidence level, such as 0.8 or 0.85.

[0129] To improve the accuracy of the specified annotation model, this embodiment can construct an annotation sample library based on a specified power knowledge graph. For example, this embodiment can first filter entities (such as equipment names, fault types, maintenance steps, etc.) and entity relationships (such as equipment-fault, fault-maintenance plan, etc.) at each level of the specified power knowledge graph, constructing an annotation sample library containing several annotation question-and-answer pairs. The format of the annotation question-and-answer pairs is uniformly "Question: [Professional Question]; Answer: [Standard Solution]". Since the number of annotation question-and-answer pairs in the annotation sample library is limited, this embodiment can first perform supervised pre-training on the initial annotation model with a first learning rate based on the annotation sample library to obtain a pending annotation model. Then, based on the pending annotation model, predictions are made on test data samples, and predicted annotation data with a prediction confidence greater than or equal to a predetermined confidence threshold are filtered. Finally, based on the annotation question-and-answer pairs and the predicted annotation data, the pending annotation model is trained with a second learning rate to obtain the specified annotation model, where the second learning rate is less than the first learning rate. Therefore, this embodiment can achieve fine-tuning of the labeled model by using a smaller second learning rate, resulting in smaller update steps and a more stable and refined training process. This prevents the disruption of features already learned by the labeled model based on the labeled sample library, while leveraging a larger dataset. For example, this embodiment can use a first learning rate (1×10⁻⁶). -5 Iterate 10-20 times to obtain the undetermined labeled model, using a second learning rate (5×10). -6 The process iterates for 5-10 rounds to obtain the specified annotation model. For iterative updates, the reward module for both the initial annotation model and the model to be labeled can employ a two-layer fully connected network. The input consists of the model's output annotations and factual data from the power industry (such as equipment category dictionaries, technical standard summaries, etc.). The reward module determines the reward value based on the degree of matching between the model's output annotations and the power industry factual data. An exemplary formula for calculating the reward value is: Reward Value = Semantic Similarity × 0.6 + Factual Consistency Score × 0.4, where the factual consistency score is verified by retrieving relevant factual data from the model's output annotations. After each training round, the model parameters are adjusted based on the reward value to mitigate annotation bias.

[0130] like Figure 7 As shown, the dataset construction method in this embodiment includes the following steps:

[0131] Multi-source heterogeneous data collection startup: By configuring the collection scope, subdividing business domains, data types and storage paths, the multi-source data collection process is started, automatically traversing the target data sources, collecting raw data and storing it in the distributed database. Data that fails to be collected, such as data that is marked as "to be retried" due to network interruption, will be automatically retried after 1 hour. If the retry fails 3 times, the log will be recorded and an alarm will be triggered.

[0132] Intelligent preprocessing execution: Read the raw data from the distributed database and automatically process it according to the process of "orthogonal basis feedforward neural network feature extraction - matrix transformation - sparse autoencoder data classification - edge data augmentation - outlier data removal - standardization". After the data is processed, it is stored in the preprocessing database after passing data verification to ensure the correctness of the format and the completeness of the dimensions.

[0133] Fine-grained annotation trigger: When the cumulative amount of data in the preprocessed database reaches a preset threshold (the threshold is configurable, with a default of 5000 records), the annotation process is automatically triggered. The trained semi-supervised annotation model is called to generate question-answer pair annotations in a unified format, forming an initial annotation set. Samples that fail to be annotated, such as semantically ambiguous data, are automatically transferred to the manual annotation queue.

[0134] Multimodal alignment execution: The text, image, and table data in the initial annotation set are separated by type and input into the corresponding encoders respectively. Feature alignment is completed through the modal shared encoder. The aligned dataset is stored in the alignment database. Data with alignment accuracy lower than the threshold (vector space similarity ≥ 0.7) is automatically returned to the preprocessing stage for reprocessing.

[0135] Automated quality verification: Read data from the alignment database, perform batch verification based on indicators such as text, image-text matching, and completeness, filter out qualified data that meet preset requirements, mark the reasons for unqualified data, and prioritize returning it to the fine-grained knowledge annotation module for automatic re-annotation; data that is still unqualified or has insufficient confidence after automatic re-annotation is transferred to the manual annotation queue or discarded directly;

[0136] Dynamic updates and load balancing:

[0137] If a new knowledge document is detected, the process of "slicing - vectorization - dense retrieval - knowledge atom integration - incremental update of dataset" is automatically executed. The updated dataset overwrites the old version and retains a backup of the historical version.

[0138] If the data after quality verification is unevenly distributed, i.e. the difference in the proportion of data volume of each business scenario type, each device type, and each fault type is greater than 10%, the data balancing process will be automatically started. The distribution will be optimized by resampling and generating data using GAN. The balanced dataset will be re-verified to ensure that it is qualified.

[0139] Dataset Output: The final qualified evaluation dataset is output in a standard format, namely JSONL, with fields including "data ID, modality type, question, answer, knowledge source, annotation time, and quality score". At the same time, a dataset description document is generated, including data distribution, construction process, and applicable scenarios. It supports direct calls to automated evaluation tools and large model fine-tuning frameworks.

[0140] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0141] Based on the same inventive concept, this application also provides a dataset construction apparatus for implementing the dataset construction method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more dataset construction apparatus embodiments provided below can be found in the limitations of the dataset construction method described above, and will not be repeated here.

[0142] In one exemplary embodiment, such as Figure 8 As shown, a dataset construction apparatus 600 is provided, including: a data acquisition module 610, a standardization module 620, a data annotation module 630, a data alignment module 640, and a dataset determination module 650, wherein:

[0143] Data acquisition module 610 is used to acquire data from heterogeneous power equipment to be processed;

[0144] Standardization module 620 is used to standardize heterogeneous power equipment data to obtain standardized heterogeneous data.

[0145] The data annotation module 630 is used to perform question-answer pair annotation on standardized heterogeneous data to obtain an annotated dataset.

[0146] The data alignment module 640 is used to map standardized heterogeneous data to the same vector space for alignment processing, thereby obtaining aligned heterogeneous data.

[0147] The dataset determination module 650 is used to align heterogeneous data with the corresponding labeled data in the labeled dataset as the target dataset.

[0148] In some embodiments, the standardization module 620 is further configured to:

[0149] The heterogeneous power equipment data is identified, and the feature dimensions of the heterogeneous power equipment data are determined. Based on the feature dimensions, the dimensions of the input layer of the specified feature extraction network are adjusted. Based on the feature extraction network with adjusted dimensions, features are extracted from the heterogeneous power equipment data to obtain the power feature matrix. Based on the power feature matrix, standardized heterogeneous data is determined.

[0150] In some embodiments, the standardization module 620 is further configured to:

[0151] Data types are identified in the power feature matrix, edge data matrices are determined, data augmentation is performed on the edge data matrices to obtain augmented data matrices, and the power feature matrix and augmented data matrices are determined to be standardized heterogeneous data.

[0152] In some embodiments, the standardization module 620 is further configured to:

[0153] Random noise is added to the edge data matrix to obtain the enhanced data matrix. The noise intensity of the random noise is determined based on the standard deviation of the edge data matrix.

[0154] In some embodiments, the data annotation module 630 is further configured to:

[0155] Based on the specified annotation model, question-answer pair annotations are performed on standardized heterogeneous data to obtain an initial annotation set. The initial annotation set is then validated, and the annotated dataset that meets the preset quality requirements is selected.

[0156] In some embodiments, the data annotation module 630 is further configured to:

[0157] The test data samples and labeled question-and-answer pairs from the labeled sample library are obtained. The labeled question-and-answer pairs are constructed based on entities and entity relationships at each level in the specified power knowledge graph. Based on the labeled question-and-answer pairs, the initial labeling model is trained with a first learning rate to obtain a pending labeling model. Based on the pending labeling model, the test data samples are predicted, and the predicted labeled data with a prediction confidence greater than or equal to a predetermined confidence threshold are selected. Based on the labeled question-and-answer pairs and the predicted labeled data, the pending labeling model is trained with a second learning rate to obtain a specified labeling model, wherein the second learning rate is less than the first learning rate.

[0158] In some embodiments, the data alignment module 640 is further configured to:

[0159] Based on the data type of each data sample in the standardized heterogeneous data, each data sample is encoded by a corresponding modal encoder to obtain modal feature vectors of each data type. The modal feature vectors of each data type are then mapped to the same vector space by a common encoder. The modal feature vectors in the same vector space are then aligned to obtain the aligned modal feature vectors, which are then identified as aligned heterogeneous data.

[0160] In some embodiments, the dataset determination module 650 is further configured to:

[0161] If the number of newly added heterogeneous power equipment data exceeds the specified growth amount, or if the time difference between the last update time and the current time exceeds the specified duration threshold, an incremental update operation is triggered. In response to the incremental update operation, the aligned heterogeneous data and labeled data corresponding to the newly added heterogeneous power equipment data are added as incremental data to the target dataset.

[0162] Each module in the aforementioned dataset construction device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can invoke and execute the operations corresponding to each module.

[0163] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 9 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data from heterogeneous power equipment. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements a dataset construction method.

[0164] Those skilled in the art will understand that Figure 9The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0165] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps described above.

[0166] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps described above.

[0167] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps described above.

[0168] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0169] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0170] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0171] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for constructing a dataset, characterized in that, The method includes: Acquire data from heterogeneous power equipment to be processed; The heterogeneous power equipment data is standardized to obtain standardized heterogeneous data; The standardized heterogeneous data is labeled with question-answer pairs to obtain a labeled dataset; The standardized heterogeneous data is mapped to the same vector space and aligned to obtain aligned heterogeneous data. The aligned heterogeneous data and the corresponding labeled data in the labeled dataset are used as the target dataset.

2. The method according to claim 1, characterized in that, The standardization process for the heterogeneous power equipment data to obtain standardized heterogeneous data includes: Identify the heterogeneous power equipment data and determine the feature dimensions of the heterogeneous power equipment data; Based on the aforementioned feature dimensions, the dimensions of the input layer of the specified feature extraction network are adjusted. Based on the dimension-adjusted feature extraction network, feature extraction is performed on the heterogeneous power equipment data to obtain a power feature matrix. Based on the power characteristic matrix, standardized heterogeneous data are determined.

3. The method according to claim 2, characterized in that, The process of determining standardized heterogeneous data based on the power feature matrix includes: Data type identification is performed on the power feature matrix to determine the edge data matrix in the power feature matrix; The edge data matrix is ​​subjected to data augmentation processing to obtain an augmented data matrix; The power feature matrix and the enhanced data matrix are determined to be standardized heterogeneous data.

4. The method according to claim 3, characterized in that, The step of performing data augmentation processing on the edge data matrix to obtain an augmented data matrix includes: Random noise is added to the edge data matrix to obtain an enhanced data matrix. The noise intensity of the random noise is determined based on the standard deviation of the edge data matrix.

5. The method according to claim 1, characterized in that, The process of annotating the standardized heterogeneous data with question-answer pairs to obtain an annotated dataset includes: Based on the specified annotation model, question-answer pair annotations are performed on the standardized heterogeneous data to obtain an initial annotation set; The initial annotation set is validated, and the annotation dataset that meets the preset quality requirements is selected.

6. The method according to claim 5, characterized in that, Before the step of annotating the standardized heterogeneous data with question-answer pairs based on a specified annotation model to obtain the initial annotation set, the following steps are included: Obtain test data samples and labeled question-and-answer pairs from the labeled sample library, wherein the labeled question-and-answer pairs are constructed based on entities and entity relationships at each level in the specified power knowledge graph; Based on the labeled question-answer pairs, the initial labeled model is trained with a first learning rate to obtain the undetermined labeled model; Based on the undetermined annotation model, the test data samples are predicted, and the predicted annotation data with a prediction confidence level greater than or equal to a predetermined confidence threshold are selected. Based on the labeled question-answer pair and the predicted labeled data, the undetermined labeled model is trained with a second learning rate to obtain a specified labeled model, wherein the second learning rate is less than the first learning rate.

7. The method according to claim 1, characterized in that, The step of mapping the standardized heterogeneous data to the same vector space for alignment processing to obtain aligned heterogeneous data includes: Based on the data type of each data sample in the standardized heterogeneous data, the data samples are encoded by the corresponding modal encoders to obtain the modal feature vectors of each data type. The modal feature vectors of each data type are mapped to the same vector space through a shared encoder; Alignment processing is performed on the modal feature vectors within the same vector space to obtain aligned modal feature vectors, which are then identified as aligned heterogeneous data.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes: Incremental update operation is triggered when the number of newly added heterogeneous power equipment data exceeds the specified growth amount, or when the time difference between the last update time and the current time exceeds the specified duration threshold. In response to the incremental update operation, the aligned heterogeneous data and labeled data corresponding to the newly added heterogeneous power equipment data are added as incremental data to the target dataset.

9. A dataset construction apparatus, characterized in that, The device includes: The data acquisition module is used to acquire data from heterogeneous power equipment to be processed. The standardization module is used to standardize the heterogeneous power equipment data to obtain standardized heterogeneous data. The data annotation module is used to annotate the standardized heterogeneous data with question-answer pairs to obtain an annotated dataset; The data alignment module is used to map the standardized heterogeneous data to the same vector space for alignment processing, thereby obtaining aligned heterogeneous data. The dataset determination module is used to select the aligned heterogeneous data and the corresponding labeled data in the labeled dataset as the target dataset.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

Images