Data text generation method and system based on feature extraction
By applying machine learning technology and natural language processing in entity extraction and text synthesis, the problem of processing complex data and lack of flexibility in the existing technology is solved, and higher extraction accuracy and text nature are achieved, satisfying professional and non-professional The needs of professional users.
Patent Information
- Application Number
- CN202411386384.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-30
- Publication Date
- 2025-02-14
AI Technical Summary
Existing entity extraction methods rely on predefined rules or simple pattern matching, cannot handle complex data and new entity types, and text synthesis systems lack flexibility and adaptability, making it difficult to generate natural and easy-to-understand text.
Using machine learning technologies such as deep learning and natural language processing, we improve the accuracy of extracting entities and relationships through feature extraction, and design flexible text templates and generation rules to adapt to different data structures and user needs.
Improves the accuracy of the extraction of entities and relationships, the generated text is more natural and easy to understand, meets the needs of professional standards and non-technical users, and improves the effectiveness of data analysis, reporting and decision support.
Smart Images
Figure CN119443101A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of information extraction and text synthesis, and in particular to a data text generation method and system based on feature extraction. Background Art
[0002] In the era of information explosion, entity and relationship extraction of structured data and text synthesis based on the extraction results have become key technologies to unlock the value of data. Structured data, such as database tables and spreadsheets, store rich information in a highly organized form. However, these data often require complex queries or professional data analysis to understand their connotations. Entity and relationship extraction can transform these data into easy-to-understand entities (such as names, places, organizations) and the connections between them (such as belonging to, located in, managed). This process not only improves the accessibility of data, but also lays the foundation for further analysis and application of data. Text synthesis based on extracted entities and relationships, that is, converting these data points into coherent narrative text, is crucial for end users to understand and make decisions. Text synthesis provides an intuitive way to present the relationship between data, so that non-technical users can quickly grasp the key points of the data. In addition, text synthesis plays an important role in automated report generation, data storytelling, information summarization and other fields. Through automated text synthesis, a lot of time for manual report writing can be saved while ensuring the consistency and accuracy of information transmission.
[0003] Although entity extraction and text synthesis technologies have made significant progress, there are still some shortcomings in the existing processes. First, traditional entity extraction methods rely on predefined rules or simple pattern matching, which limits their ability to handle complex data and new entity types. Second, existing text synthesis systems often lack sufficient flexibility and adaptability to handle diverse data structures and user needs. In addition, the generated text may be too technical or lack contextual information, making the text concatenation suboptimal and incomprehensible to non-professional users.
[0004] In view of this, a method and system for generating data text based on feature extraction is needed. Summary of the invention
[0005] In view of the fact that existing entity extraction methods in the prior art rely on predefined rules or simple pattern matching, are unable to handle complex data and new entity types, and text synthesis systems often lack sufficient flexibility and adaptability, the present invention provides a data text generation method and system based on feature extraction, which can use machine learning technologies, such as deep learning and natural language processing, to improve the extraction accuracy of entities and relationships. By designing flexible text templates and generation rules, the scheme can adapt to different data structures and user needs, and generate more natural and easy-to-understand text. In addition, the iterative optimization and manual review steps in the scheme ensure the quality and accuracy of the generated text, so that the final output text meets professional standards and is easy for non-technical users to understand. The specific technical solution is as follows:
[0006] A data text generation method based on feature extraction comprises the following steps:
[0007] Extracting entities and relationships from the data through information extraction and forming a first entity-relationship data set;
[0008] Extracting entities and relationships through the training model and forming a second entity-relationship dataset;
[0009] Separately storing the overlapping parts and non-overlapping parts of the first entity-relationship data set and the second entity-relationship data set to form a third entity-relationship data set;
[0010] Use a pre-set text generation template, fill it in according to data attributes based on the third entity-relationship data set, and finally generate data text.
[0011] Preferably, the generation process of the first entity-relationship dataset is as follows:
[0012] Input the original structured data set and perform data preprocessing;
[0013] List all field names and perform semantic analysis to determine the field type after understanding the meaning of each field, and identify possible entity fields and possible relationship fields;
[0014] Define the entity categories, create a list for each entity category, then iterate over each field and assign the value in the data row to the corresponding entity list based on the field name and type;
[0015] Define possible relationship types, identify relationships between fields, create relationship lists, record associations between entities, and use foreign keys or logical relationships between fields to determine relationships between entities;
[0016] The results of entity recognition and relationship extraction are obtained to construct the first entity-relationship dataset.
[0017] Preferably, the process of generating the first entity-relationship data set further includes the following steps:
[0018] Add attributes to each entity and relationship, including timestamps and descriptions;
[0019] Determine whether the field contains an enumeration value. If not, do not parse the enumeration value. If so, parse the enumeration value and add the enumeration value parsing result as an attribute to the entity and relationship.
[0020] Preferably, the generation process of the second entity-relationship dataset is as follows:
[0021] Obtain and store the operating data of power grid equipment as the original power grid data set. The operating data of power grid equipment includes equipment status, maintenance records, and fault reports.
[0022] Perform data preprocessing on the original power grid data set, and divide the preprocessed data into training set, validation set and test set;
[0023] Select a suitable machine learning algorithm or neural network architecture, use the training set data to train the model, adjust the model parameters during the process, and finally use the validation set for model selection and hyperparameter adjustment;
[0024] Use the test set to evaluate the model performance, calculate the model's accuracy, recall, F1 score and other indicators, and finally analyze the model's generalization ability and possible overfitting or underfitting problems;
[0025] Iteratively train and optimize machine learning algorithms or neural network architectures, use trained models to identify entities in power grid data, and use models to predict relationships between entities.
[0026] Preferably, the steps of generating data text are as follows:
[0027] Get the list of extracted entities and relationships, assign unique identifiers to each entity and relationship, and organize the list of entities and relationships;
[0028] Design text templates and formulate text generation rules, including grammar rules, word usage rules, and logic rules, to determine how to map entity and relationship identifiers to natural language descriptions;
[0029] According to the template and rules, a descriptive text is generated for each entity to obtain an entity description.
[0030] According to the templates and rules, descriptive text is generated for the relationship between each pair of entities to obtain a relationship description.
[0031] Get entity descriptions and relationship descriptions, and concatenate the entity descriptions and relationship descriptions in logical order.
[0032] Preferably, generating the data text further comprises the following steps:
[0033] The concatenated text is obtained, and the generated text is reviewed and edited, wherein the review and editing are completed by manual review or automated text quality inspection tools. If the review and editing fail, an alarm message is returned, and the current concatenated text is not used as the final output.
[0034] A data text generation system based on feature extraction, applied to the above method, comprises:
[0035] The data collection and storage unit collects the original structured data set for processing by the first data extraction unit, collects the operation data of the power grid equipment for processing by the second data extraction unit, and the operation data of the equipment includes equipment status, maintenance records, and fault reports;
[0036] A first data extraction unit extracts entities and relationships from the data through information extraction and forms a first entity-relationship data set;
[0037] A second data extraction unit extracts entities and relationships through the training model, and forms a second entity-relationship data set based on the trained model;
[0038] A third data synthesis unit stores the overlapping parts and non-overlapping parts of the first entity-relationship data set and the second entity-relationship data set separately to form a third entity-relationship data set;
[0039] A text synthesis unit uses a pre-set text generation template and fills it in according to data attributes based on a third entity-relationship data set to finally generate a data text;
[0040] The terminal receives the output data of the text synthesis unit, stores and displays it.
[0041] Preferably, it also includes an alarm unit, which is connected to the first data extraction unit, the second data extraction unit, the third data synthesis unit and the text synthesis unit, and is used to receive signals from the first data extraction unit, the second data extraction unit, the third data synthesis unit and the text synthesis unit and perform an alarm operation, and the alarm operation includes sound and light alarms and sending alarm information to the terminal.
[0042] A computer-readable storage medium includes a stored program, wherein when the program is run, the device where the computer-readable storage medium is located is controlled to execute the data text generation method based on feature extraction as described above.
[0043] A processor is used to run a program, wherein the program executes the data text generation method based on feature extraction as described above when running.
[0044] Compared with the prior art, the present invention has the following beneficial effects:
[0045] The present invention uses machine learning techniques, such as deep learning and natural language processing, to improve the accuracy of extracting entities and relationships. By designing flexible text templates and generation rules, the solution can adapt to different data structures and user needs, and generate more natural and easy-to-understand text. In addition, the iterative optimization and manual review steps in the solution ensure the quality and accuracy of the generated text, so that the final output text meets professional standards and is easy for non-technical users to understand. Through this method, not only can the accuracy and efficiency of data extraction be improved, but also more humane and easy-to-understand data texts can be generated, thereby better serving data analysis, reporting and decision support. This method of comprehensive application of machine intelligence and human review has demonstrated significant effects in improving data presentation quality and user experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for the specific embodiments or the description of the prior art. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn according to the actual scale.
[0047] Figure 1 is a flow chart of the method of the present invention;
[0048] Figure 2 It is a system principle diagram of the present invention. DETAILED DESCRIPTION
[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0050] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.
[0051] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0052] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0053] In one embodiment of the present invention, a method for generating data text based on feature extraction is provided. Figure 1 As shown, the following steps are included:
[0054] Step 1: Extract entities and relationships from the data through information extraction and form the first entity-relationship data set; this step can be divided into the following steps:
[0055] 1. Perform data preprocessing as follows:
[0056] Inputting an original structured data set, which may come from a database, a data warehouse, or a data set file;
[0057] Clean the data to remove invalid or erroneous records;
[0058] Standardize field names, for example, unify "Device Name", "Device_Name" and "Device Name" into "DeviceName";
[0059] To handle missing values, you can choose to delete records or fields with missing values, or fill in missing values.
[0060] Ensure that the data type is correct. For example, a date field should be of date type and a numeric field should be of numeric type. If the data type is incorrect, a warning message is returned and the field is not used as a field for subsequent processing.
[0061] 2. Perform field analysis as follows:
[0062] Input the preprocessed dataset, list all field names, and perform semantic analysis to understand the meaning of each field.
[0063] Determines the type of the field (for example, date, text, number).
[0064] Identify possible entity fields, such as "device name" and "manufacturer". The entity fields are determined by a pre-set entity dictionary, which stores all possible entities in the current field. The entity dictionary is pre-set by the staff.
[0065] Identify possible relationship fields, such as the relationship between "defect ID" and "equipment ID". The identification principle of relationship fields is the same as that of entity fields.
[0066] 3. Perform entity recognition, as follows:
[0067] Input field analysis results. Define entity categories, such as "equipment", "defect", "manufacturer".
[0068] For each entity class, create a list to collect the entity names.
[0069] Iterate over each field and assign the value in the data row to the corresponding entity list based on the field name and type.
[0070] Deduplicate and merge entities in an entity list to eliminate duplication.
[0071] 4. Perform relationship extraction, as follows:
[0072] Input entity recognition results. Define possible relationship types, such as "belongs to", "found in".
[0073] Identify the relationship between fields, for example, if "Defect ID" appears in the description of an "Equipment" entity, then it can be assumed that the "Defect" belongs to that "Equipment".
[0074] Create a relationship list to record the associations between entities. Use foreign keys or logical relationships between fields to determine the relationships between entities.
[0075] 5. Build an entity-relationship model as follows:
[0076] The results of entity recognition and relationship extraction are obtained, and an entity-relationship graph (the first entity-relationship dataset) is constructed, where nodes represent entities and edges represent relationships.
[0077] Add attributes to each entity and relationship, such as timestamp, description, etc. In this step, if the field contains enumeration values, the enumeration value content can be added to the entity and relationship as a further description through parsing the enumeration value. Correspondingly, templates containing enumeration values can also be added to the subsequent text generation templates, thereby further enriching the richness of data extraction and text generation and the accuracy of description.
[0078] Enumeration value analysis: By analyzing the enumeration values of selected key fields, you can gain a deeper understanding of the attribute distribution and category information of the data. With the help of detailed analysis of the enumeration values, you can effectively classify the field attributes. For example, through the different enumeration values of the "equipment status" field (such as "in operation", "retired", etc.), you can distinguish different status records of the equipment.
[0079] In addition, the consistency and integrity of the entity-relationship model (first entity-relationship data set) can be further verified, and entity-relationship models that do not meet the standards in terms of consistency and integrity can be given an alarm and not included in the data input range of subsequent steps.
[0080] 6. Verification and iteration:
[0081] Take the constructed entity-relationship model and verify the accuracy of entities and relationships through business experts or domain knowledge.
[0082] The definitions of entities and relationships are adjusted based on the feedback, and then the entity-relationship model is iteratively optimized.
[0083] Here is a simplified example:
[0084] Suppose we have a defect record dataset for power grid equipment, with fields including "equipment ID", "equipment name", "manufacturer", "defect ID", "defect description" and "defect discovery date".
[0085] 1. Data preprocessing: Clean the data and ensure that the "defect discovery date" is in the correct date format.
[0086] 2. Field analysis: Identify “Device Name” and “Manufacturer” as entity fields, and the relationship between “Defect ID” and “Device ID”.
[0087] 3. Entity recognition: Create an entity list, such as equipment {equipment ID: "001", name: "Transformer A", manufacturer: "Company X"}.
[0088] 4. Relationship extraction: Determine the relationship between "defect ID" and "device ID", such as defect {ID:"001", device ID:"001", description:"overheating", discovery date:"2024-09-27"}.
[0089] 5. Build entity-relationship model: Build the model, the equipment "Transformer A" has a defect "overheating", which was discovered on "2024-09-27".
[0090] 6. Verification and iteration: Confirm information with grid maintenance personnel and update the model.
[0091] Through these steps, we can extract entities and relationships from structured data and build a clear data model, providing a basis for subsequent data analysis and application.
[0092] Step 2: Extract entities and relationships through the training model and form a second entity-relationship dataset; this step can be divided into the following steps:
[0093] 1. Data collection, as follows:
[0094] Obtain the operating data of the power grid equipment, such as equipment status, maintenance records, fault reports, etc. These data can come from the database, sensors, smart meters or monitoring systems of the power grid company. Since the means of obtaining these data are common means in this field, they will not be elaborated here.
[0095] The data is collected and stored in a database or data warehouse, and the stored data is used as the original power grid data set.
[0096] 2. Data preprocessing, as follows:
[0097] Get the original power grid data set, then clean the data, remove noise, missing values, outliers, standardize field names and data formats, convert data types, such as converting date fields to date types, handle missing values, and choose to delete or fill them. Get the preprocessed data set.
[0098] 3. Data set division and model training evaluation, as follows:
[0099] Get the preprocessed data set and divide it into training set, validation set and test set. Usually the ratio is 60% training set, 20% validation set and 20% test set. In this process, stratified sampling can be used to ensure the consistency of data distribution.
[0100] Select a suitable machine learning algorithm or neural network architecture and use the training set data to train the model. In the process, adjust the model parameters, such as learning rate, number of layers, and number of hidden units. Finally, use the validation set to select the model and adjust the hyperparameters.
[0101] Use the test set to evaluate the model performance. Calculate the model's accuracy, recall, F1 score and other indicators. Finally, analyze the model's generalization ability and possible overfitting or underfitting problems.
[0102] 4Entity and relationship extraction:
[0103] Obtain model training results, use the trained model to perform entity recognition on power grid data, and use the model to predict the relationship between entities.
[0104] Here is a specific example:
[0105] Assume that the power grid dataset contains "equipment ID", "equipment name", "manufacturer", "defect ID", "defect description" and "defect discovery date".
[0106] 1. Data preprocessing: Clean the data and ensure that the "defect discovery date" is in the correct date format.
[0107] 2. Dataset division: Divide the dataset into 60% training set, 20% validation set, and 20% test set.
[0108] 3. Feature engineering: construct feature vectors, including equipment information and defect information.
[0109] 4. Model training: Use a neural network model (such as BERT for NER) to train the entity recognition model.
[0110] 5. Model evaluation: Evaluate the accuracy and recall of the model on the test set.
[0111] 6. Entity and relationship extraction: The model identifies entities such as "Transformer A", "Company X", "Defect 001", and the relationship between "Transformer A" and "Defect 001".
[0112] After training is completed, the model will output the results of entity recognition and the relationship between entities when used. The data that needs to be input is the real-time data of power grid equipment, and the model will output the categories of entities and the relationship between them.
[0113] Step 3: Separately store the overlapping parts and non-overlapping parts of the first entity-relationship data set and the second entity-relationship data set to form a third entity-relationship data set;
[0114] Obtain the first entity-relationship data set obtained in step 1 and the second entity-relationship data set obtained in step 2, and use the overlapping parts as the third entity-relationship data set. The purpose of this step is to obtain the entity-relationships identified in both step 1 and step 2 to avoid the errors of the two entity-relationship acquisition methods, and further refine the entity-relationship recognition results, thereby making the subsequent steps more accurate.
[0115] In addition, the non-overlapping parts of the first entity-relationship data set and the second entity-relationship data set are stored separately from the third entity-relationship data set, that is, there is room for flexible data set adjustment and use. For example, in step 4, if the user requires extremely high precision, only the third entity-relationship data set can be used to generate data texts. However, if the user requires high precision at this time, but also pursues the number of generated data texts, then, on the basis of the generation of the third entity-relationship data set, one or all of the first entity-relationship data set and the second entity-relationship data set can be selected to generate the final data text, thereby ensuring a certain degree of precision while also generating a larger number of texts. In this process, the benefit of separate storage is that these data sets and the data texts generated by them are also classified, stored and processed, and further distinction can make it more convenient in subsequent use and evaluation.
[0116] Step 4: Generate data text based on the third entity-relationship dataset. Follow the steps below:
[0117] Get the list of extracted entities and relationships (this example takes the third entity-relationship dataset as an example), assign a unique identifier to each entity and relationship, and organize the list of entities and relationships;
[0118] Design text templates to describe the attributes of entities and the relationships between entities. Templates should be flexible and can be designed in multiple different styles according to specific needs to adapt to different entity types and relationship types;
[0119] Obtain the organized entity and relationship lists and text templates, formulate text generation rules, including grammar rules, word usage rules, and logical rules, and determine how to map entity and relationship identifiers to natural language descriptions.
[0120] According to the template and rules, a descriptive text is generated for each entity, including the entity's name, type, attributes, and other information. The entity description is obtained.
[0121] According to the template and rules, a descriptive text is generated for each relationship between entities, including the type of relationship, the interaction between entities, etc., to obtain a relationship description.
[0122] Get entity descriptions and relationship descriptions, connect them in logical order, and finally use appropriate conjunctions and transition sentences to ensure the coherence and readability of the text.
[0123] In addition, the concatenated text can be obtained and the generated text can be reviewed and edited to improve the naturalness and accuracy of the language. This step can be done by manual review or automated text quality checking tools.
[0124] Here is a simple example:
[0125] Suppose we have extracted the following entities and relations from the power grid data:
[0126] Entity: Transformer A (equipment name), Company X (manufacturer);
[0127] Relationship: Transformer A is produced by Company X;
[0128] Therefore, do the following:
[0129] 1. Entity and relationship organization: Confirm that the information of Transformer A and Company X is accurate and complete.
[0130] 2. Text template design: Design the following template:
[0131] Entity template: {entity name} is {entity type}.
[0132] Relationship template: {Entity 1} is produced by {Entity 2}.
[0133] 3. Establishment of text generation rules: Determine the use of third-person narration and past tense to describe production relations.
[0134] 4. Entity description generation:
[0135] Transformer A: Transformer A is a power transformer.
[0136] Company X: Company X is a transformer manufacturer.
[0137] 5. Relationship description generation:
[0138] Transformer A is produced by Company X.
[0139] 6. Text concatenation:
[0140] Transformer A is a power transformer produced by Company X.
[0141] 7. Text optimization: Review the text to ensure the description is accurate.
[0142] 8. Text output: The final output text is: "Transformer A is an electric power transformer produced by company X."
[0143] Through these steps, we can generate clear and accurate data text from entities and relationships extracted from structured data. It should be appreciated that the above example is only used to illustrate the steps of text generation with a simple example, and is not intended to limit the technical solution of the present invention.
[0144] In one embodiment of the present invention, a data text generation system based on feature extraction is also provided. Figure 2 As shown, including:
[0145] The data collection and storage unit collects the original structured data set for processing by the first data extraction unit, collects the operation data of the power grid equipment for processing by the second data extraction unit, and the operation data of the equipment includes equipment status, maintenance records, and fault reports;
[0146] A first data extraction unit extracts entities and relationships from the data through information extraction and forms a first entity-relationship data set;
[0147] A second data extraction unit extracts entities and relationships through the training model, and forms a second entity-relationship data set based on the trained model;
[0148] A third data synthesis unit stores the overlapping parts and non-overlapping parts of the first entity-relationship data set and the second entity-relationship data set separately to form a third entity-relationship data set;
[0149] A text synthesis unit uses a pre-set text generation template and fills it in according to data attributes based on a third entity-relationship data set to finally generate a data text;
[0150] The terminal receives the output data of the text synthesis unit, stores and displays it.
[0151] An alarm unit is connected to the first data extraction unit, the second data extraction unit, the third data synthesis unit and the text synthesis unit, and is used to receive signals from the first data extraction unit, the second data extraction unit, the third data synthesis unit and the text synthesis unit and perform an alarm operation, the alarm operation including sound and light alarms and sending alarm information to the terminal.
[0152] In summary, the present invention uses machine learning techniques, such as deep learning and natural language processing, to improve the accuracy of extraction of entities and relationships. By designing flexible text templates and generation rules, the scheme can adapt to different data structures and user needs, and generate more natural and easy-to-understand text. In addition, the iterative optimization and manual review steps in the scheme ensure the quality and accuracy of the generated text, so that the final output text meets professional standards and is easy for non-technical users to understand. In this way, not only can the accuracy and efficiency of data extraction be improved, but also more humane and easy-to-understand data texts can be generated, thereby better serving data analysis, reporting and decision support. This method of comprehensive application of machine intelligence and human review has demonstrated significant effects in improving data presentation quality and user experience.
[0153] Those of ordinary skill in the art will appreciate that the units of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition of each example has been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0154] In the embodiments provided by the present invention, it should be understood that the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units can be combined into one unit, one unit can be split into multiple units, or some features can be ignored, etc.
[0155] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0156] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-0nlyMemory), random access memory (RAM, RandomAccessMemory), mobile hard disk, magnetic disk or optical disk, etc., which can store program code.
[0157] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.
Claims
1. A data text generation method based on feature extraction, characterized in that: The following steps are involved: Extracting entities and relationships from the data through information extraction and forming a first entity-relationship data set; Extracting entities and relationships through the training model and forming a second entity-relationship dataset; Separately storing the overlapping parts and non-overlapping parts of the first entity-relationship data set and the second entity-relationship data set to form a third entity-relationship data set; Use a pre-set text generation template, fill it in according to data attributes based on the third entity-relationship data set, and finally generate data text.
2. A data text generation method based on feature extraction according to claim 1, characterized in that: The generation process of the first entity-relationship dataset is as follows: Input the original structured data set and perform data preprocessing; List all field names and perform semantic analysis to determine the field type after understanding the meaning of each field, and identify possible entity fields and possible relationship fields; Define the entity categories, create a list for each entity category, then iterate over each field and assign the value in the data row to the corresponding entity list based on the field name and type; Define possible relationship types, identify relationships between fields, create relationship lists, record associations between entities, and use foreign keys or logical relationships between fields to determine relationships between entities; The results of entity recognition and relationship extraction are obtained to construct the first entity-relationship dataset.
3. A data text generation method based on feature extraction according to claim 2, characterized in that: The process of generating the first entity-relationship dataset also includes the following steps: Add attributes to each entity and relationship, including timestamps and descriptions; Determine whether the field contains an enumeration value. If not, do not parse the enumeration value. If so, parse the enumeration value and add the enumeration value parsing result as an attribute to the entity and relationship.
4. The method for generating data text based on feature extraction according to claim 1, characterized in that: The generation process of the second entity-relationship dataset is as follows: Obtain and store the operating data of power grid equipment as the original power grid data set. The operating data of power grid equipment includes equipment status, maintenance records, and fault reports. Perform data preprocessing on the original power grid data set, and divide the preprocessed data into training set, validation set and test set; Select a suitable machine learning algorithm or neural network architecture, use the training set data to train the model, adjust the model parameters during the process, and finally use the validation set for model selection and hyperparameter adjustment; Use the test set to evaluate the model performance, calculate the model's accuracy, recall, F1 score and other indicators, and finally analyze the model's generalization ability and possible overfitting or underfitting problems; Iteratively train and optimize machine learning algorithms or neural network architectures, use trained models to identify entities in power grid data, and use models to predict relationships between entities.
5. The method for generating data text based on feature extraction according to claim 1, characterized in that: The steps to generate data text are as follows: Get the list of extracted entities and relationships, assign unique identifiers to each entity and relationship, and organize the list of entities and relationships; Design text templates and formulate text generation rules, including grammar rules, word usage rules, and logic rules, to determine how to map entity and relationship identifiers to natural language descriptions; Generate descriptive text for each entity according to the template and rules to obtain entity description; Generate descriptive text for the relationship between each pair of entities according to the template and rules to obtain a relationship description; Get entity descriptions and relationship descriptions, and concatenate the entity descriptions and relationship descriptions in logical order.
6. A method for generating data text based on feature extraction according to claim 5, characterized in that: Generating data text also includes the following steps: The concatenated text is obtained, and the generated text is reviewed and edited, wherein the review and editing are completed by manual review or automated text quality inspection tools. If the review and editing fail, an alarm message is returned, and the current concatenated text is not used as the final output.
7. A data text generation system based on feature extraction, characterized in that: The method applied to any one of claims 1 to 6 comprises: The data collection and storage unit collects the original structured data set for processing by the first data extraction unit, collects the operation data of the power grid equipment for processing by the second data extraction unit, and the operation data of the equipment includes equipment status, maintenance records, and fault reports; A first data extraction unit extracts entities and relationships from the data through information extraction and forms a first entity-relationship data set; A second data extraction unit extracts entities and relationships through the training model, and forms a second entity-relationship data set based on the trained model; A third data synthesis unit stores the overlapping parts and non-overlapping parts of the first entity-relationship data set and the second entity-relationship data set separately to form a third entity-relationship data set; A text synthesis unit uses a pre-set text generation template and fills it in according to data attributes based on a third entity-relationship data set to finally generate a data text; The terminal receives the output data of the text synthesis unit, stores and displays it.
8. A data text generation system based on feature extraction according to claim 7, characterized in that: It also includes an alarm unit, which is connected to the first data extraction unit, the second data extraction unit, the third data synthesis unit and the text synthesis unit, and is used to receive signals from the first data extraction unit, the second data extraction unit, the third data synthesis unit and the text synthesis unit and perform alarm operations, and the alarm operations include sound and light alarms and sending alarm information to the terminal.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the data text generation method based on feature extraction as described in any one of claims 1 to 6.
10. A processor, characterized in that: The processor is used to run a program, wherein when the program is run, the method for generating data text based on feature extraction described in any one of claims 1 to 6 is executed.