Test identification task evaluation method and device
Through automated information extraction and semantic matching technology, the problem of time-consuming and lack of objectivity in experimental identification and evaluation is solved, and efficient and accurate experimental identification and evaluation results are achieved.
Patent Information
- Application Number
- CN202411123355.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-15
- Publication Date
- 2024-12-03
AI Technical Summary
Existing experimental identification and evaluation mainly relies on manual methods, is time-consuming and error-prone, lacks objectivity and consistency, and is difficult to accurately extract key information, which is affected by the subjective judgment of the appraiser.
By obtaining historical experimental identification data, structured and semi-structured data are extracted using preset data extraction rules, and inputting their input information extraction models for encoding conversion and semantic matching, generating experimental identification supervision data sets, combining real-time data and text Semantic matching model, automatic evaluation of experimental identification tasks.
It improves the efficiency and accuracy of experimental identification and evaluation, realizes automated information extraction and semantic matching, and provides objective and consistent evaluation results.
Smart Images

Figure CN119066097A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing, and in particular to a method and device for evaluating a test identification task. Background Art
[0002] Testing and certification is a systematic process that requires a series of indicators and standards to evaluate whether the performance of a product, system or equipment meets the established design requirements and specifications. This usually involves a series of testing and evaluation activities to ensure the safety, reliability and effectiveness of the product in actual use.
[0003] The evaluation of test and identification tasks is a complex system engineering with multiple levels and dimensions. It not only assesses the technical performance of the equipment, but also evaluates its use efficiency. It not only focuses on single performance indicators, but also attaches importance to the evaluation of overall efficiency. The test and identification industry spans many fields such as information, communications, big data, and artificial intelligence, and has high requirements for cross-border integration capabilities. With the demand for the development of test and identification targets in the directions of open sea, deep sea, deep space, unmanned, and intelligent, test and identification technologies need to be constantly updated and changed to adapt to development needs.
[0004] At present, test identification and evaluation are mainly performed by professionals who extract information based on established standards to conduct matching evaluation. However, test identification involves a large amount of text information. Traditional manual methods are often time-consuming and error-prone. It is difficult to accurately extract key information. Moreover, the results of test identification are often affected by the subjective judgment of the identification personnel, lacking objectivity and consistency. There is an urgent need for a test identification task evaluation method that can provide objective and consistent test identification results through automated information extraction and semantic matching, and improve the efficiency and accuracy of test identification and evaluation. Summary of the invention
[0005] In response to the problems in the prior art, the present application provides a test identification task evaluation method and device, which can improve the efficiency and accuracy of test identification evaluation based on automated information extraction and semantic matching.
[0006] In order to solve at least one of the above problems, the present application provides the following technical solutions:
[0007] In a first aspect, the present application provides a test identification task evaluation method, comprising:
[0008] Acquire a historical test identification data file, perform a data extraction operation on the historical test identification data file according to a preset data extraction rule, and determine corresponding structured test identification data and semi-structured test identification data;
[0009] Input the semi-structured test identification data into the input layer of the setting information extraction model for encoding conversion operation, determine the corresponding conversion code, input the conversion code into the bidirectional encoder layer of the setting information extraction model for semantic conversion operation, determine the corresponding semantic features, input the semantic features into the bidirectional long short-term memory network layer of the setting information extraction model for dependency capture operation, determine the corresponding semantic dependency features, input the semantic dependency features into the conditional random field layer of the setting information extraction model, determine the corresponding semi-structured test identification data label sequence, and determine the corresponding test identification supervision data set according to the semi-structured test identification data label sequence and the structured test identification data;
[0010] Acquire real-time test identification data, determine a corresponding matching score based on the real-time test identification data, the test identification supervision data set and a set text semantic matching model, perform a result evaluation operation on the test identification task based on the matching score, and determine a corresponding test identification task evaluation result.
[0011] Further, performing a data extraction operation on the historical test identification data file according to a preset data extraction rule to determine the corresponding structured test identification data and semi-structured test identification data includes:
[0012] Performing data extraction operations on the historical test identification data file according to preset key node extraction rules to determine corresponding structured test identification data;
[0013] A data extraction operation is performed on the historical test identification data file according to preset data cleaning rules to determine corresponding semi-structured test identification data.
[0014] Furthermore, the step of inputting the semi-structured test identification data into the input layer of the setting information extraction model for encoding conversion operation and determining the corresponding conversion code includes:
[0015] Performing a word segmentation operation on the semi-structured test identification data according to a preset word segmentation rule to determine a corresponding minimum word unit;
[0016] The minimum word unit is input into the input layer of the setting information extraction model to perform a coding conversion operation, and a corresponding conversion code is determined.
[0017] Furthermore, the step of inputting the conversion code into the bidirectional encoder layer of the setting information extraction model for semantic conversion operation to determine the corresponding semantic features includes:
[0018] Input the conversion code into the bidirectional encoder layer of the setting information extraction model, perform a vector conversion operation on the conversion code according to a preset word embedding algorithm, and determine a corresponding encoding vector;
[0019] A coding stacking operation is performed according to the coding vector and a preset number of coding structure layers to determine corresponding semantic features.
[0020] Furthermore, the step of inputting the semantic features into the bidirectional long short-term memory network layer of the setting information extraction model to perform a dependency capture operation and determine the corresponding semantic dependency features includes:
[0021] Inputting the semantic features into the bidirectional long short-term memory network layer of the setting information extraction model, performing dependency capture operations according to a preset gating mechanism, and determining a corresponding semantic feature matrix;
[0022] The corresponding semantic dependency features are determined according to the semantic feature matrix and the preset memory unit.
[0023] Furthermore, the determining of the corresponding matching score according to the real-time test identification data, the test identification supervision data set and the set text semantic matching model includes:
[0024] Inputting the test identification supervision data set into a preset text semantic matching model to perform a model training operation, thereby obtaining a set text semantic matching model after the model training operation;
[0025] The real-time test identification data is input into the set text semantic matching model to determine the matching score output by the set text semantic matching model.
[0026] Furthermore, the step of inputting the test identification supervision data set into a preset text semantic matching model for a model training operation to obtain a set text semantic matching model after the model training operation comprises:
[0027] Inputting the test identification supervision data set into a preset text semantic matching model, and determining the corresponding matching loss according to a pre-trained word embedding model and a preset cosine similarity algorithm;
[0028] The model parameter updating operation is performed according to the minimum matching loss and the back propagation algorithm to determine the corresponding set text semantic matching model.
[0029] In a second aspect, the present application provides a test identification task evaluation device, comprising:
[0030] A data extraction module is used to obtain a historical test identification data file, perform a data extraction operation on the historical test identification data file according to a preset data extraction rule, and determine the corresponding structured test identification data and semi-structured test identification data;
[0031] A supervised data set construction module, used for inputting the semi-structured test identification data into the input layer of the setting information extraction model for encoding conversion operation, determining the corresponding conversion code, inputting the conversion code into the bidirectional encoder layer of the setting information extraction model for semantic conversion operation, determining the corresponding semantic features, inputting the semantic features into the bidirectional long short-term memory network layer of the setting information extraction model for dependency capture operation, determining the corresponding semantic dependency features, inputting the semantic dependency features into the conditional random field layer of the setting information extraction model, determining the corresponding semi-structured test identification data label sequence, and determining the corresponding test identification supervision data set according to the semi-structured test identification data label sequence and the structured test identification data;
[0032] The test identification evaluation module is used to obtain real-time test identification data, determine the corresponding matching score based on the real-time test identification data, the test identification supervision data set and the set text semantic matching model, perform result evaluation operation on the test identification task based on the matching score, and determine the corresponding test identification task evaluation result.
[0033] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the test identification task evaluation method when executing the program.
[0034] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the test identification task evaluation method.
[0035] In a fifth aspect, the present application provides a computer program product, comprising a computer program / instruction, which implements the steps of the test identification task evaluation method when executed by a processor.
[0036] It can be seen from the above technical scheme that the present application provides a test identification task evaluation method and device, which obtains historical test identification data files, determines the corresponding structured test identification data and semi-structured test identification data according to preset data extraction rules, inputs the semi-structured test identification data into the conditional random field layer of the set information extraction model, determines the corresponding semi-structured test identification data tag sequence, determines the corresponding test identification supervision data set according to the semi-structured test identification data tag sequence and the structured test identification data, obtains real-time test identification data, determines the corresponding matching score according to the real-time test identification data, the test identification supervision data set and the set text semantic matching model, performs result evaluation operation on the test identification task according to the matching score, and determines the corresponding test identification task evaluation result, thereby being able to improve the efficiency and accuracy of test identification evaluation based on automated information extraction and semantic matching. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0038] Figure 1 This is one of the flow charts of the test identification task evaluation method in the embodiment of the present application;
[0039] Figure 2 This is a second flow chart of the test identification task evaluation method in the embodiment of the present application;
[0040] Figure 3 The third flowchart of the test identification task evaluation method in the embodiment of the present application;
[0041] Figure 4 This is a fourth flow chart of the test identification task evaluation method in the embodiment of the present application;
[0042] Figure 5 This is a fifth flow chart of the test identification task evaluation method in the embodiment of the present application;
[0043] Figure 6 This is a sixth flow chart of the test identification task evaluation method in the embodiment of the present application;
[0044] Figure 7 FIG7 is a flow chart of the test identification task evaluation method in the embodiment of the present application;
[0045] Figure 8 It is a structural diagram of the test identification task evaluation device in the embodiment of the present application;
[0046] Fig. 9 It is a schematic diagram of the structure of an electronic device in an embodiment of the present application.
[0047] Reference numerals:
[0048] Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver program storage unit 9144, antenna 9111, speaker 9131, microphone 9132. DETAILED DESCRIPTION
[0049] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0050] The acquisition, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.
[0051] Considering that at present, test identification evaluation is mainly performed by professionals based on established standards to evaluate the matching degree of the extracted information, but test identification involves a large amount of text information. The traditional manual method is often time-consuming and prone to errors. It is difficult to accurately extract key information and the results of test identification are often affected by the subjective judgment of the identification personnel, lacking objectivity and consistency. The present application provides a test identification task evaluation method and device, which obtains historical test identification data files, determines the corresponding structured test identification data and semi-structured test identification data according to preset data extraction rules, inputs the semi-structured test identification data into the conditional random field layer of the set information extraction model, determines the corresponding semi-structured test identification data tag sequence, determines the corresponding test identification supervision data set according to the semi-structured test identification data tag sequence and the structured test identification data, obtains real-time test identification data, determines the corresponding matching score according to the real-time test identification data, the test identification supervision data set and the set text semantic matching model, performs result evaluation operation on the test identification task according to the matching score, and determines the corresponding test identification task evaluation result, thereby improving the efficiency and accuracy of test identification evaluation based on automated information extraction and semantic matching.
[0052] In order to improve the efficiency and accuracy of test identification evaluation based on automated information extraction and semantic matching, the present application provides an embodiment of a test identification task evaluation method, see Figure 1 The test identification task evaluation method specifically includes the following contents:
[0053] Step S101: obtaining a historical test identification data file, performing a data extraction operation on the historical test identification data file according to a preset data extraction rule, and determining corresponding structured test identification data and semi-structured test identification data;
[0054] Optionally, in this embodiment, the purpose of this step is to classify the types of historical test identification data to facilitate subsequent data extraction.
[0055] Optionally, in this embodiment, the technical means adopted in this step is to extract the required data type files based on metadata from a large amount of text information involved in the test identification, and standardize the data structure in the required files to obtain structured test identification data and semi-structured test identification data.
[0056] Optionally, in this embodiment, the preset data extraction rules are divided into rules for directly extracting structured data, and rules for extracting semi-structured data after cleaning.
[0057] Optionally, in this embodiment, structured data is data from which relevant information can be directly extracted, and semi-structured data has a certain structure, but the data cannot be directly extracted. Therefore, it is necessary to pre-process the semi-structured data and then input it into the information extraction model for information extraction, so as to merge the extracted information with the information extracted from the structured data and construct a supervised data set.
[0058] Optionally, in this embodiment, before extracting data based on rules and dividing structured data and semi-structured data, it is necessary to first divide the types of historical test identification data files to obtain the files we need. This step can filter out unnecessary files and reduce data redundancy.
[0059] This step is divided based on metadata. Specifically, the metadata includes the file name, creator, creation time, last modification time, data format, data size, data source, etc.
[0060] For example, suppose a large number of data files are stored in a business management system. First, use metadata information to identify the file types containing key information and their relevance to obtain the file types we need. For example, by identifying the file name "outline", we can obtain performance test outlines, combat test outlines, in-service assessment outlines and other files. By tracing the original source in the metadata of the above outline files, we can obtain the system test overall plan file. We can also identify the timeliness of the data based on the time information in the metadata, such as screening data files in the past three years.
[0061] Optionally, in this embodiment, after the types of historical test identification data files are classified to obtain the files we need, these files are classified into structured data and semi-structured data based on rules.
[0062] For example, after obtaining the above-mentioned performance test outline, combat test outline, in-service assessment outline, system test overall plan and other documents, the data is divided into structured data (such as table data in the database) and semi-structured data (such as text data in PDF and Word documents).
[0063] For structured data, entity data in the structured data is directly extracted according to the needs of the scenario, such as the actual execution time and location of the test task, key nodes based on events (combat indicators), etc. After manual labeling of the entity labels, they are stored in the test identification supervision data set.
[0064] For semi-structured data, data preprocessing is required first. The purpose is to clean, convert and annotate the original corpus so that the subsequent machine learning model can annotate and learn based on the clean data to extract the entity data in the semi-structured data, such as the actual execution time and location of the test task, event-based key nodes (combat indicators), etc., and store them in the test identification supervision data set for fusion with the structured data.
[0065] Optionally, in this embodiment, the semi-structured data cleaning is mainly to process the impurity data contained in the data collected by the data collection part, and the tools mainly used in the processing process are some native libraries of Python such as re library, pandas library, etc. Pandas library can process tables and documents.
[0066] Data cleaning includes the following steps:
[0067] The data cleaning steps are as follows:
[0068] (1) Delete unnecessary columns and rows in DataFrame
[0069] DataFrame is a main data structure in the pandas library, used to store and manipulate structured data. It is similar to a table in Excel or a SQL table, but it has more powerful data processing and analysis capabilities. The data in a DataFrame can be of types such as numeric values, strings, Boolean values, and each column can be of a different data type.
[0070] (2) Rename columns to a set of more recognizable labels
[0071] ColumnsIn a DataFrame, columns are the vertical dimensions of data that define the fields or features of the data. Each column has a unique name (column name) that can be specified during the creation or modification of a DataFrame. Column names help users identify and access specific columns of data in a DataFrame.
[0072] (3) Change the data format
[0073] The data format may need to be adjusted according to the analysis requirements, such as converting a date string to a date type, or converting a text type number to a numeric type.
[0074] (4) Remove spaces or specific characters between data
[0075] When processing text data, it is often necessary to remove spaces, line breaks, special characters, etc. in the data to ensure the consistency and accuracy of the data.
[0076] (5) Remove duplicate rows
[0077] Duplicate data rows may lead to deviations in the analysis results, so these duplicate rows need to be removed to ensure the uniqueness of the data.
[0078] (6) Finding null values
[0079] Empty values (NaN or None) are common in data sets, but they may have an impact on the analysis results. Therefore, finding and handling these empty values is an important step in data cleaning.
[0080] It can be understood that by inputting the above-mentioned semi-structured data after data cleaning into the set information extraction model, and extracting the entities and relationships between entities in the semi-structured data through the model, it can lay the foundation for the subsequent fusion of information extracted from the structured data and the construction of the test identification supervision data set.
[0081] Step S102: input the semi-structured test identification data into the input layer of the setting information extraction model for encoding conversion operation, determine the corresponding conversion code, input the conversion code into the bidirectional encoder layer of the setting information extraction model for semantic conversion operation, determine the corresponding semantic features, input the semantic features into the bidirectional long short-term memory network layer of the setting information extraction model for dependency capture operation, determine the corresponding semantic dependency features, input the semantic dependency features into the conditional random field layer of the setting information extraction model, determine the corresponding semi-structured test identification data label sequence, and determine the corresponding test identification supervision data set according to the semi-structured test identification data label sequence and the structured test identification data;
[0082] Optionally, in this embodiment, the purpose of this step is to perform named entity recognition and relationship extraction on semi-structured data by setting an information extraction model, extract entities and relationships in the semi-structured data, and integrate them with the information extracted from the structured data to construct a test identification supervision data set.
[0083] Optionally, in this embodiment, in this step, the information extraction model is set to be a BERT-BiLSTM-CRF model based on a deep neural network, which performs named entity recognition and relationship extraction on the test identification supervision data text. The model is mainly divided into four parts, input layer, BERT layer (bidirectional encoder layer), bidirectional LSTM layer (bidirectional long short-term memory network layer), CRF layer (conditional random field layer).
[0084] It is worth noting that model training is a process of continuous updating and iterative tuning. Model tuning is performed through subsequent changes and retraining of model annotation information, and the model parameters generated by the tuning are updated at each functional layer in a back-propagation manner to make the model more accurate.
[0085] Specifically, in this embodiment, each layer is described as follows:
[0086] 1) Input layer: The {B, M, E, S, O} annotation system standard is used to perform encoding conversion on the previously established standard tagged corpus.
[0087] 2) BERT layer: The Bidirectional Encoder Representations from Transformers (BERT) layer mainly completes the distributed representation of the input and the conversion of the corpus to the word-level feature vector. The BERT pre-trained model is a pre-trained language model proposed by Google AI Research in October 2018, which has outstanding performance in the top-level test of machine reading comprehension. The pre-training process predicts randomly masked words based on contextual semantic information, which can better learn contextual content features; the sentence relationship prediction inserts [CLS] and [SEP] tags at the beginning and end of each sentence, respectively, and predicts whether the positions of two sentences are adjacent by learning the relationship features between sentences. The BERT model adopts a 12-layer or 24-layer bidirectional Transformer encoding structure, in which the input text is converted into a word vector, and the start and end marks are added to each sentence, thereby obtaining the input vector E1, E2, ..., EN, and the output vector T1, T2, ..., TN after passing through multiple layers of Transformer encoders.
[0088] 3) Bidirectional LSTM layer: The bidirectional long short-term memory (LSTM) layer can further combine context information to form a feature matrix, which can effectively solve the long-distance dependency problem in the recognition of longer entities in the experimental identification supervision corpus.
[0089] LSTM (Long Short-Term Memory) is a special recurrent neural network (RNN) structure that is used to solve the gradient vanishing or gradient exploding problems that exist in traditional RNNs when dealing with long-term dependency problems. LSTM captures and preserves long-term dependency information by introducing gating mechanisms and memory units.
[0090] The basic unit of LSTM consists of a memory cell (cell state) and three gating mechanisms: input gate, forget gate, and output gate. These gating mechanisms achieve the preservation and transmission of long-term dependent information by controlling the flow of information.
[0091] The following is the principle formula of LSTM:
[0092] 1. Forget Gate:
[0093] Among them, represents the output of the forget gate at time t, represents the sigmoid activation function, and represent the weight and bias of the forget gate respectively, represents the output of the previous moment, and represents the input of the current moment.
[0094] 2. Input Gate: The input gate receives the long-term memory output from the forget gate and the short-term memory output from the learn gate and directly merges the two together.
[0095] Among them, represents the output of the input gate at time t, represents the candidate memory unit of the input gate at time t, and represents the weight and bias of the input gate respectively.
[0096] 3. Memory Cell (Cell State):
[0097] Memory cells are used to store and update information. A memory cell can be thought of as a "cell" that contains a state vector (usually called c_t, where t represents the time step) that is passed between different time steps of the sequence, allowing long-term information to be stored.
[0098] 4. Output Gate:
[0099] The output gate controls how the information in the memory cell affects the output of the LSTM cell. It determines the output of the LSTM cell based on the state of the memory cell and the input of the current time step.
[0100] From the above formula, we can see that LSTM captures and preserves long-term dependency information through gating mechanisms and memory units. During the training process, LSTM updates weights and biases through back-propagation algorithms and gradient descent algorithms to optimize the performance of the model.
[0101] 4) CRF layer: The conditional random field (CRF) layer takes the feature matrix H generated by the bidirectional LSTM layer as input, and obtains the global optimal tag sequence based on the dependency relationship between adjacent tags, thereby completing the named entity recognition process.
[0102] Optionally, in this embodiment, a specific embodiment is given to illustrate:
[0103] Suppose we have the following structured data and semi-structured text:
[0104] Structured data:
[0105] Experiment ID: 001
[0106] Experiment Date:2024-07-26
[0107] Experimental location: Launch Site A
[0108] Experimental equipment: Missile X model
[0109] Semi-structured text:
[0110] "On July 26, 2024, a missile of a certain type was launched at location A."
[0111] Input the above semi-structured data into the set information extraction model for information extraction:
[0112] (1) Word segmentation: "July 26, 2024", "a certain model", "missile", "at", "location A", "conducted", "launch test".
[0113] (2) Encoding conversion: Convert the word segmentation result into an index sequence in the BERT vocabulary.
[0114] (3) BERT layer (bidirectional encoder layer): Input the index sequence to the BERT model to obtain the context vector representation of each token.
[0115] (4) BiLSTM layer (bidirectional long short-term memory network layer): BiLSTM processes the output of BERT and generates a feature matrix containing long-term dependency information.
[0116] (5) CRF layer (conditional random field layer): defines a set of labels (such as time, place, event, etc.) and uses the CRF layer to determine the optimal label sequence.
[0117] (6) Output tag sequence:
[0118] "July 26, 2024" (time)
[0119] "A certain model" (entity)
[0120] "Missile" (Entity)
[0121] "In" (O)
[0122] "A Location" (Location)
[0123] "Done" (O)
[0124] "Launch test" (event)
[0125] Construct a test identification supervision dataset based on the extracted semi-structured data and existing structured data:
[0126] (1) Entity linking operation: Confirm that "a certain type of missile" in the semi-structured text and "missile X model" in the structured data are the same entity. Confirm that "A location" in the semi-structured text and "launch site A" in the structured data are the same location.
[0127] (2) Relationship standardization operation: define the standard relationship as the "participation" or "use" relationship, and map the extracted "product-event" relationship to the "participation" or "use" relationship.
[0128] (3) Data fusion operation: Combine the extracted entities and relationships with structured data to form the following information:
[0129] Experiment ID: 001
[0130] Experiment Date:2024-07-26
[0131] Experimental location: Launch Site A
[0132] Experimental equipment: Missile X model
[0133] Experimental event: Launch test
[0134] Experimental relationship: Missile X model is used in the launch test at launch site A
[0135] It can be understood that a test identification supervision data set is established through the above steps. By similarity annotation of the test identification supervision data set and using the annotated test identification supervision data set to train the text semantic model, the text semantic model can learn the similarity relationship between the data, so that the trained text semantic model can receive real-time test identification data and generate corresponding similarity predictions based on the parameters in the training process, and finally obtain the corresponding matching score for evaluating the test identification results.
[0136] Step S103: Acquire real-time test identification data, determine a corresponding matching score based on the real-time test identification data, the test identification supervision data set and a set text semantic matching model, perform a result evaluation operation on the test identification task based on the matching score, and determine a corresponding test identification task evaluation result.
[0137] Optionally, in this embodiment, the purpose of this embodiment is to perform similarity annotation on the test identification supervision data set and use the annotated test identification supervision data set to train the text semantic model so that the text semantic model can generate corresponding similarity predictions for the real-time test identification data for evaluating the test identification results.
[0138] Optionally, in this embodiment, the technical means of this embodiment is to extract text features such as word vectors and sentence vectors from the test identification supervision data set, select a text semantic model suitable for the test identification task, and use the data set to train the text semantic model. The trained model can be used to input real data in actual test identification, match the standard data to obtain the test score.
[0139] Optionally, in this embodiment, the text semantic model is selected according to the specific requirements of the test identification task, and a suitable semantic matching model is selected. A Siamese network, a twin network or a Transformer model can be used.
[0140] Optionally, in this embodiment, the experimental identification supervision data set is divided into a training set, a test set and a validation set, wherein the number of training sets is much larger than that of test sets and validation sets, and the purpose is that the model can discover the text semantic features therein through the training set.
[0141] First, the datasets are annotated in pairs to determine their similarity labels. Second, the labeled training sets are input into the text semantic model. In this step, the model will learn how to predict the similarity between them.
[0142] In the model learning stage, the TF-IDF algorithm is used in the recognition stage of the data set for text similarity matching. This embodiment is optimized on the basis of the TF-IDF algorithm and adopts a semantic-based text similarity matching method. Specifically, by introducing word vectors and semantic similarity calculations, the text vector representation obtained by the TF-IDF algorithm is converted into a semantic vector representation, thereby improving the accuracy and efficiency of text similarity matching.
[0143] In terms of word vectors, a pre-trained word vector model is used to map each word into a high-dimensional vector space. In this way, each word in the text can be represented as a vector, and the text itself can be represented as a collection of these vectors. By calculating the similarity between vectors, the semantic similarity between texts can be obtained.
[0144] In terms of semantic similarity calculation, the cosine similarity calculation method is used. Specifically, for two text vectors A and B, their cosine similarity can be obtained by calculating the cosine value of the angle between their corresponding vectors. The value range of cosine similarity is [-1,1]. The larger the value, the more similar the two texts are.
[0145] During the model training phase, the model learns to predict the similarity between question pairs based on their semantic vector representations, and then continuously adjusts its parameters to minimize the difference between the predicted similarity and the true similarity labels.
[0146] It can be understood that by combining the TF-IDF algorithm and the semantic-based text similarity matching method, the accuracy and efficiency of text similarity matching can be effectively improved.
[0147] Through the above-mentioned trained model, after receiving the real-time test supervision data, the real-time test supervision data is first converted into a semantic vector representation, and then the test supervision data most similar to the real-time test supervision data is found, and the similarity score between them is calculated for result evaluation.
[0148] It can be understood that the test identification supervision model of this embodiment not only integrates the relationship extraction capability of the BERT-BiLSTM-CRF model, but also combines the advantages of the TF-IDF algorithm in text semantic matching. The BERT-BiLSTM-CRF model can accurately capture the complex relationships in the test identification process, further process the context representation output by the BERT model through the bidirectional long short-term memory network (BiLSTM), and then combine the conditional random field (CRF) for sequence labeling, effectively solving the problem of relationship extraction in the field of test identification.
[0149] The TF-IDF algorithm is used to semantically match the text related to the test identification. The algorithm measures the importance of a word in a specific text by calculating the word frequency (TF) and inverse document frequency (IDF) of each word in the text. By comparing the TF-IDF values of key words in different texts, the semantic similarity between them can be determined, thereby achieving accurate matching of the test identification text.
[0150] The application of the test identification supervision model in the field of test identification has significant advantages. On the one hand, by extracting the relationships in the test identification process, we can have a deeper understanding of the test process and results, providing strong support for decision makers. On the other hand, through text semantic matching technology, we can quickly and accurately find documents and information related to specific test identification, improving work efficiency.
[0151] This example demonstrates how this embodiment extracts information based on historical test supervision data to construct a supervision data set and trains a text semantic matching model so that the model can achieve test identification and evaluation.
[0152] From the above description, it can be seen that the test identification task evaluation method provided in the embodiment of the present application can obtain historical test identification data files, determine the corresponding structured test identification data and semi-structured test identification data according to preset data extraction rules, input the semi-structured test identification data into the conditional random field layer of the set information extraction model, determine the corresponding semi-structured test identification data tag sequence, determine the corresponding test identification supervision data set according to the semi-structured test identification data tag sequence and the structured test identification data, obtain real-time test identification data, determine the corresponding matching score according to the real-time test identification data, the test identification supervision data set and the set text semantic matching model, perform result evaluation operation on the test identification task according to the matching score, and determine the corresponding test identification task evaluation result, thereby being able to improve the efficiency and accuracy of test identification evaluation based on automated information extraction and semantic matching.
[0153] In one embodiment of the test identification task evaluation method of the present application, see Figure 2 , and can also include the following:
[0154] Step S201: performing a data extraction operation on the historical test identification data file according to a preset key node extraction rule to determine corresponding structured test identification data;
[0155] Step S202: performing a data extraction operation on the historical test identification data file according to a preset data cleaning rule to determine corresponding semi-structured test identification data.
[0156] Optionally, in this embodiment, for structured data, the format is usually fixed, and corresponding node extraction rules can be formulated for extraction according to business scenarios and data requirements.
[0157] The key node extraction rules are preset to define the key information points that need to be identified and extracted in the data file. Key nodes may include the name, date, time, location, participants, equipment used, test results, etc. of the test.
[0158] For example, in an Excel file containing multiple test records, each row records the detailed information of a test. Using the preset extraction rules, key information such as the test date, test location, and test results can be extracted from each row, annotated, and saved in a structured database table.
[0159] Optionally, in this embodiment, semi-structured data generally refers to data that has a certain structure but a less fixed format, such as XML or JSON files. During data extraction, it may be necessary to convert these data into a format that is easier to process, such as converting JSON to CSV.
[0160] Preset data cleaning rules define how to identify and handle inconsistencies, errors, or missing values in raw data. Rules may include removing duplicate records, correcting format errors, filling missing values, standardizing text, etc.
[0161] The data cleaning steps are as follows:
[0162] (1) Delete unnecessary columns and rows in DataFrame
[0163] DataFrame is a main data structure in the pandas library, used to store and manipulate structured data. It is similar to a table in Excel or a SQL table, but it has more powerful data processing and analysis capabilities. The data in a DataFrame can be of types such as numeric values, strings, Boolean values, and each column can be of a different data type.
[0164] (2) Rename columns to a set of more recognizable labels
[0165] ColumnsIn a DataFrame, columns are the vertical dimensions of data that define the fields or features of the data. Each column has a unique name (column name) that can be specified during the creation or modification of a DataFrame. Column names help users identify and access specific columns of data in a DataFrame.
[0166] (3) Change the data format
[0167] The data format may need to be adjusted according to the analysis requirements, such as converting a date string to a date type, or converting a text type number to a numeric type.
[0168] (4) Remove spaces or specific characters between data
[0169] When processing text data, it is often necessary to remove spaces, line breaks, special characters, etc. in the data to ensure the consistency and accuracy of the data.
[0170] (5) Remove duplicate rows
[0171] Duplicate data rows may lead to deviations in the analysis results, so these duplicate rows need to be removed to ensure the uniqueness of the data.
[0172] (6) Finding null values
[0173] Empty values (NaN or None) are common in data sets, but they may have an impact on the analysis results. Therefore, finding and handling these empty values is an important step in data cleaning.
[0174] Finally, the semi-structured data after data cleaning is input into the set information extraction model, and the entities and relationships between entities in the semi-structured data are extracted through the model, laying a solid data processing foundation for the subsequent fusion of the entities and relationships of the semi-structured data with the structured data.
[0175] Through step S202, this embodiment extracts structured data and semi-structured data respectively, laying a foundation for subsequently establishing a connection between the data.
[0176] In one embodiment of the test identification task evaluation method of the present application, see Figure 3 , and can also include the following:
[0177] Step S301: performing a word segmentation operation on the semi-structured test identification data according to a preset word segmentation rule to determine a corresponding minimum word unit;
[0178] Step S302: input the minimum word unit into the input layer of the setting information extraction model to perform a coding conversion operation, and determine the corresponding conversion code.
[0179] Optionally, in this embodiment, in the input layer of the BERT-BiLSTM-CRF model, it is first necessary to segment the input clean semi-structured data and perform encoding conversion on the segmented data to facilitate model recognition and learning.
[0180] Optionally, in this embodiment, a word segmentation rule is preset to decompose a sentence into words or smaller units (subwords, tokens). BERT uses WordPiece as its word segmentation method.
[0181] For example, suppose there is a semi-structured test identification text: "On July 26, 2024, a certain type of missile was launched at location A."
[0182] Word segmentation processing: "July 26, 2024", "a certain model", "missile", "at", "location A", "conducted", "launch test".
[0183] Optionally, in this embodiment, the encoding conversion operation is to convert the text into a format that can be understood by the model, which is an important step in natural language processing.
[0184] Each of the above minimal word units will be converted into unique indices that correspond to the BERT model vocabulary.
[0185] Through step S302, this embodiment obtains the semi-structured data conversion code, so that the model can recognize the semi-structured data and perform subsequent named entity recognition and relationship extraction operations.
[0186] In one embodiment of the test identification task evaluation method of the present application, see Figure 4 , and can also include the following:
[0187] Step S401: input the conversion code into the bidirectional encoder layer of the setting information extraction model, perform a vector conversion operation on the conversion code according to a preset word embedding algorithm, and determine a corresponding encoding vector;
[0188] Step S402: performing a coding stacking operation according to the coding vector and a preset number of coding structure layers to determine corresponding semantic features.
[0189] Optionally, in this embodiment, the transformation code is input into the next layer of the information extraction model, that is, the BERT layer (bidirectional encoder layer) of the BERT-BiLSTM-CRF model. The BERT layer uses the Transformer architecture to process the input tokenID sequence and output the context-dependent embedding of each token. These embeddings capture the semantic and syntactic relationships between words.
[0190] Optionally, in this embodiment, a word embedding algorithm is preset, and word embedding is converted into a vector, and position embedding and sentence segment embedding are added to the vector representation of each token.
[0191] Optionally, in this embodiment, the encoding stacking operation is performed, and after multiple layers of Transformer encoders, the vector representation of each token is continuously updated to reflect its context information in the sentence.
[0192] Optionally, in this embodiment, the BERT model adopts a 12-layer or 24-layer bidirectional Transformer encoding structure, in which the input text is converted into word vectors, and start and end marks are added to each sentence, thereby obtaining input vectors E1, E2,…, EN, and output vectors T1, T2,…, TN after passing through multiple layers of Transformer encoders.
[0193] Ultimately, each token has a context-dependent vector representation that can be used for downstream tasks such as text classification, named entity recognition, etc.
[0194] Through step S402, this embodiment successfully obtains the semantic representation relationship between semi-structured data through the BERT layer (bidirectional encoder layer) of the BERT-BiLSTM-CRF model, laying the foundation for subsequent accurate named entity recognition and relationship extraction.
[0195] In one embodiment of the test identification task evaluation method of the present application, see Figure 5 , and can also include the following:
[0196] Step S501: inputting the semantic features into the bidirectional long short-term memory network layer of the setting information extraction model, performing dependency capture operation according to a preset gating mechanism, and determining a corresponding semantic feature matrix;
[0197] Step S502: Determine corresponding semantic dependency features according to the semantic feature matrix and the preset memory unit.
[0198] Optionally, in this embodiment, the semantic features output by the BERT layer are input into the bidirectional LSTM layer (bidirectional long short-term memory network layer) of the BERT-BiLSTM-CRF model. LSTM (Long Short-Term Memory) is a special recurrent neural network (RNN) structure used to solve the gradient vanishing or gradient exploding problems of traditional RNN when dealing with long-term dependency problems. LSTM captures and saves long-term dependency information by introducing gating mechanisms and memory units.
[0199] Optionally, in this embodiment, a gating mechanism and a memory unit are preset, and the basic unit of LSTM includes a memory unit (cell state) and three gating mechanisms: input gate, forget gate and output gate. These gating mechanisms realize the preservation and transmission of long-term dependent information by controlling the flow of information.
[0200] 1. Forget Gate:
[0201] Among them, represents the output of the forget gate at time t, represents the sigmoid activation function, and represent the weight and bias of the forget gate respectively, represents the output of the previous moment, and represents the input of the current moment.
[0202] 2. Input Gate: The input gate receives the long-term memory output from the forget gate and the short-term memory output from the learn gate and directly merges the two together.
[0203] Among them, represents the output of the input gate at time t, represents the candidate memory unit of the input gate at time t, and represents the weight and bias of the input gate respectively.
[0204] 3. Memory Cell (Cell State):
[0205] Memory cells are used to store and update information. A memory cell can be thought of as a "cell" that contains a state vector (usually called c_t, where t represents the time step) that is passed between different time steps of the sequence, allowing long-term information to be stored.
[0206] 4. Output Gate:
[0207] The output gate controls how the information in the memory cell affects the output of the LSTM cell. It determines the output of the LSTM cell based on the state of the memory cell and the input of the current time step.
[0208] From the above formula, we can see that LSTM captures and preserves long-term dependency information through gating mechanisms and memory units. During the training process, LSTM updates weights and biases through back-propagation algorithms and gradient descent algorithms to optimize the performance of the model.
[0209] Through step S502, this embodiment successfully obtains the semantic dependency relationship between semi-structured data through the bidirectional LSTM layer (bidirectional long short-term memory network layer) of the BERT-BiLSTM-CRF model, laying the foundation for subsequent accurate named entity recognition and relationship extraction.
[0210] In one embodiment of the test identification task evaluation method of the present application, see Figure 6 , and can also include the following:
[0211] Step S601: inputting the test identification supervision data set into a preset text semantic matching model to perform a model training operation, thereby obtaining a preset text semantic matching model after the model training operation;
[0212] Step S602: input the real-time test identification data into the set text semantic matching model to determine the matching score output by the set text semantic matching model.
[0213] Optionally, in this embodiment, the experimental identification supervision data set is divided into a training set, a validation set and a test set. The training set is used to train the model, the validation set is used to adjust the model parameters, and the test set is used to evaluate the performance of the model.
[0214] The training set is input into the model for model training. The model can effectively learn the features contained in the training set and output the model's predicted features for the training set. The predicted features are compared with the actual features to obtain the loss function. The model obtains the minimized loss function through iterative training to obtain the optimal parameters.
[0215] It can be understood that the semantic matching model is trained and adjusted using the training set and the validation set to optimize the model's performance in semantic similarity matching. The trained model is evaluated using the test set to check its accuracy and efficiency in text similarity matching.
[0216] The optimal loss function is used to determine the set text semantic matching model whose predicted loss is the smallest. At this time, the model can be used to receive real-time test identification data and perform model prediction to output its predicted matching score for obtaining the test identification evaluation result.
[0217] Through step S602, this embodiment successfully obtains a trained text semantic matching model, so that the model can be used to receive real-time test identification data and perform model prediction to output its predicted matching score for obtaining the test identification evaluation result.
[0218] In one embodiment of the test identification task evaluation method of the present application, see Figure 7 , and can also include the following:
[0219] Step S701: input the test identification supervision data set into a preset text semantic matching model, and determine the corresponding matching loss according to the pre-trained word embedding model and the preset cosine similarity algorithm;
[0220] Step S702: performing a model parameter update operation according to the minimum matching loss and the back propagation algorithm to determine a corresponding set text semantic matching model.
[0221] Optionally, in this embodiment, the test identification supervision data set is input into a preset text semantic matching model, and the corresponding matching loss is determined according to the pre-trained word embedding model and the preset cosine similarity algorithm, including two steps:
[0222] (1) Input the test identification supervision dataset into the preset text semantic matching model to obtain the prediction of the model output. This process uses the pre-trained word embedding model and the preset cosine similarity algorithm, specifically including:
[0223] First, the TF-IDF algorithm is applied to the text data of the training set in the test identification supervision dataset to calculate the TF-IDF value of each word. According to the TF-IDF value, the text is represented as a vector. These vectors will be used as input for model training. At the same time, in order to overcome the limitations of the TF-IDF algorithm in semantic understanding, a pre-trained word embedding model is introduced. Each word is mapped to a high-dimensional vector space to capture its semantic information. Using word vectors, each word in the text is represented as a vector, and the entire text is represented as a collection of these vectors by averaging.
[0224] Secondly, the cosine similarity calculation method is used to calculate the similarity between the two text vectors. The larger the cosine similarity value, the more similar the two texts are.
[0225] Finally, the model aims to find a mapping relationship through learning, so that the model can predict the similarity between them based on the above text vectors and obtain the predicted similarity output by the model.
[0226] (2) Calculate the loss based on the prediction and true label output by the model, and iteratively update to obtain the minimum matching loss, including:
[0227] After the model obtains the predicted similarity, it needs to make the match between this prediction and the data label in the training set (i.e. the true similarity) as high as possible. The quantitative indicator of this matching degree is the matching loss, which measures the difference between the model prediction and the true label.
[0228] During the training process, the model will continuously calculate the loss value based on the current prediction results and the true label, and optimize the loss value through the backpropagation algorithm. The backpropagation algorithm calculates the gradient of the loss function with respect to the model parameters, and then updates the model parameters based on this gradient information. Through this iterative process, the model will continuously adjust its parameters to reduce the loss value, thereby improving the accuracy of the prediction.
[0229] When the loss value is reduced to a certain extent, that is, the difference between the similarity predicted by the model and the real label is small enough, it can be considered that the model has learned a good text vector similarity matching relationship. At this time, the model parameters have reached a better state, and the model parameters in this state can be used to build a text semantic matching model for text similarity prediction in practical applications.
[0230] Through step S702, this embodiment successfully obtains a trained text semantic matching model, so that the model can be used to receive real-time test identification data and perform model prediction to output its predicted matching score for obtaining the test identification evaluation result.
[0231] In order to improve the efficiency and accuracy of test identification and evaluation based on automated information extraction and semantic matching, the present application provides an embodiment of a test identification task evaluation device for implementing all or part of the contents of the test identification task evaluation method, see Figure 8 The test and identification task evaluation device specifically includes the following contents:
[0232] A data extraction module 10 is used to obtain a historical test identification data file, perform a data extraction operation on the historical test identification data file according to a preset data extraction rule, and determine the corresponding structured test identification data and semi-structured test identification data;
[0233] A supervision data set construction module 20 is used to input the semi-structured test identification data into the input layer of the setting information extraction model for encoding conversion operation, determine the corresponding conversion code, input the conversion code into the bidirectional encoder layer of the setting information extraction model for semantic conversion operation, determine the corresponding semantic features, input the semantic features into the bidirectional long short-term memory network layer of the setting information extraction model for dependency capture operation, determine the corresponding semantic dependency features, input the semantic dependency features into the conditional random field layer of the setting information extraction model, determine the corresponding semi-structured test identification data label sequence, and determine the corresponding test identification supervision data set according to the semi-structured test identification data label sequence and the structured test identification data;
[0234] The test identification evaluation module 30 is used to obtain real-time test identification data, determine the corresponding matching score according to the real-time test identification data, the test identification supervision data set and the set text semantic matching model, perform result evaluation operation on the test identification task according to the matching score, and determine the corresponding test identification task evaluation result.
[0235] From the above description, it can be seen that the test identification task evaluation device provided in the embodiment of the present application can obtain historical test identification data files, determine the corresponding structured test identification data and semi-structured test identification data according to preset data extraction rules, input the semi-structured test identification data into the conditional random field layer of the set information extraction model, determine the corresponding semi-structured test identification data tag sequence, determine the corresponding test identification supervision data set according to the semi-structured test identification data tag sequence and the structured test identification data, obtain real-time test identification data, determine the corresponding matching score according to the real-time test identification data, the test identification supervision data set and the set text semantic matching model, perform result evaluation operation on the test identification task according to the matching score, and determine the corresponding test identification task evaluation result, thereby being able to improve the efficiency and accuracy of test identification evaluation based on automated information extraction and semantic matching.
[0236] From the hardware level, in order to improve the efficiency and accuracy of test identification and evaluation based on automated information extraction and semantic matching, the present application provides an embodiment of an electronic device for implementing all or part of the contents of the test identification task evaluation method, and the electronic device specifically includes the following contents:
[0237] Processor, memory, communication interface and bus; wherein the processor, memory and communication interface communicate with each other through the bus; the communication interface is used to realize information transmission between the test identification task evaluation method and related devices such as the core business system, user terminal and related database; the logic controller can be a desktop computer, a tablet computer and a mobile terminal, etc., but the present embodiment is not limited thereto. In the present embodiment, the logic controller can be implemented with reference to the embodiment of the test identification task evaluation method in the embodiment, and the embodiment of the test identification task evaluation method, and the contents thereof are incorporated herein, and the repeated parts are not repeated.
[0238] It is understandable that the user terminal may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a personal digital assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Among them, the smart wearable device may include smart glasses, a smart watch, a smart bracelet, etc.
[0239] In practical applications, part of the test identification task evaluation method can be executed on the electronic device side as described above, or all operations can be completed in the client device. The specific selection can be based on the processing capability of the client device and the limitations of the user's usage scenario. This application does not limit this. If all operations are completed in the client device, the client device may also include a processor.
[0240] The client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and other implementation scenarios may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or a server cluster consisting of multiple servers, or a server structure of a distributed device.
[0241] Fig. 9 FIG. 9 is a schematic block diagram of the system structure of the electronic device 9600 according to an embodiment of the present application. Fig. 9 As shown, the electronic device 9600 may include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. It is worth noting that Fig. 9 is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunication functions or other functions.
[0242] In one embodiment, the test identification task evaluation method function may be integrated into the central processing unit 9100. The central processing unit 9100 may be configured to perform the following control:
[0243] Step S101: obtaining a historical test identification data file, performing a data extraction operation on the historical test identification data file according to a preset data extraction rule, and determining corresponding structured test identification data and semi-structured test identification data;
[0244] Step S102: input the semi-structured test identification data into the input layer of the setting information extraction model for encoding conversion operation, determine the corresponding conversion code, input the conversion code into the bidirectional encoder layer of the setting information extraction model for semantic conversion operation, determine the corresponding semantic features, input the semantic features into the bidirectional long short-term memory network layer of the setting information extraction model for dependency capture operation, determine the corresponding semantic dependency features, input the semantic dependency features into the conditional random field layer of the setting information extraction model, determine the corresponding semi-structured test identification data label sequence, and determine the corresponding test identification supervision data set according to the semi-structured test identification data label sequence and the structured test identification data;
[0245] Step S103: Acquire real-time test identification data, determine a corresponding matching score based on the real-time test identification data, the test identification supervision data set and a set text semantic matching model, perform a result evaluation operation on the test identification task based on the matching score, and determine a corresponding test identification task evaluation result.
[0246] From the above description, it can be seen that the electronic device provided in the embodiment of the present application obtains historical test identification data files, determines the corresponding structured test identification data and semi-structured test identification data according to preset data extraction rules, inputs the semi-structured test identification data into the conditional random field layer of the set information extraction model, determines the corresponding semi-structured test identification data tag sequence, determines the corresponding test identification supervision data set according to the semi-structured test identification data tag sequence and the structured test identification data, obtains real-time test identification data, determines the corresponding matching score according to the real-time test identification data, the test identification supervision data set and the set text semantic matching model, performs result evaluation operation on the test identification task according to the matching score, and determines the corresponding test identification task evaluation result, thereby being able to improve the efficiency and accuracy of test identification evaluation based on automated information extraction and semantic matching.
[0247] In another embodiment, the test identification task evaluation method can be configured separately from the central processing unit 9100. For example, the test identification task evaluation method can be configured as a chip connected to the central processing unit 9100, and the function of the test identification task evaluation method is realized through the control of the central processing unit.
[0248] like Fig. 9 As shown, the electronic device 9600 may also include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily have to include Fig. 9 In addition, the electronic device 9600 may also include Fig. 9 For components not shown, reference may be made to the prior art.
[0249] like Fig. 9 As shown, the central processing unit 9100 is sometimes also referred to as a controller or an operation control, and may include a microprocessor or other processor device and / or logic device. The central processing unit 9100 receives input and controls the operation of various components of the electronic device 9600.
[0250] The memory 9140 may be, for example, one or more of a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices. The above-mentioned information related to the failure may be stored, and a program for executing the relevant information may also be stored. The CPU 9100 may execute the program stored in the memory 9140 to implement information storage or processing, etc.
[0251] The input unit 9120 provides input to the central processing unit 9100. The input unit 9120 is, for example, a key or a touch input device. The power supply 9170 is used to provide power to the electronic device 9600. The display 9160 is used to display display objects such as images and texts. The display may be, for example, an LCD display, but is not limited thereto.
[0252] The memory 9140 may be a solid-state memory, such as a read-only memory (ROM), a random access memory (RAM), a SIM card, etc. It may also be a memory that saves information even when the power is off, can be selectively erased, and is provided with more data, examples of which are sometimes referred to as EPROMs, etc. The memory 9140 may also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142, which is used to store application programs and function programs or processes for executing the operation of the electronic device 9600 through the central processor 9100.
[0253] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for communication functions of the electronic device and / or for executing other functions of the electronic device (such as messaging applications, address book applications, etc.).
[0254] The communication module 9110 is a transmitter / receiver that sends and receives signals via the antenna 9111. The communication module 9110 (transmitter / receiver) is coupled to the central processor 9100 to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal.
[0255] Based on different communication technologies, multiple communication modules 9110 may be provided in the same electronic device, such as a cellular network module, a Bluetooth module and / or a wireless LAN module, etc. The communication module (transmitter / receiver) 9110 is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide an audio output via the speaker 9131 and receive an audio input from the microphone 9132, thereby realizing a common telecommunication function. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor 9130 is also coupled to the central processor 9100, so that recording can be performed on the local machine through the microphone 9132, and the sound stored on the local machine can be played through the speaker 9131.
[0256] The embodiments of the present application also provide a computer-readable storage medium capable of implementing all the steps of the test identification task evaluation method in the above embodiment, where the execution subject is a server or a client. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, all the steps of the test identification task evaluation method in the above embodiment, where the execution subject is a server or a client, are implemented. For example, when the processor executes the computer program, the following steps are implemented:
[0257] Step S101: obtaining a historical test identification data file, performing a data extraction operation on the historical test identification data file according to a preset data extraction rule, and determining corresponding structured test identification data and semi-structured test identification data;
[0258] Step S102: input the semi-structured test identification data into the input layer of the setting information extraction model for encoding conversion operation, determine the corresponding conversion code, input the conversion code into the bidirectional encoder layer of the setting information extraction model for semantic conversion operation, determine the corresponding semantic features, input the semantic features into the bidirectional long short-term memory network layer of the setting information extraction model for dependency capture operation, determine the corresponding semantic dependency features, input the semantic dependency features into the conditional random field layer of the setting information extraction model, determine the corresponding semi-structured test identification data label sequence, and determine the corresponding test identification supervision data set according to the semi-structured test identification data label sequence and the structured test identification data;
[0259] Step S103: Acquire real-time test identification data, determine a corresponding matching score based on the real-time test identification data, the test identification supervision data set and a set text semantic matching model, perform a result evaluation operation on the test identification task based on the matching score, and determine a corresponding test identification task evaluation result.
[0260] From the above description, it can be seen that the computer-readable storage medium provided in the embodiment of the present application obtains historical test identification data files, determines the corresponding structured test identification data and semi-structured test identification data according to preset data extraction rules, inputs the semi-structured test identification data into the conditional random field layer of the set information extraction model, determines the corresponding semi-structured test identification data tag sequence, determines the corresponding test identification supervision data set according to the semi-structured test identification data tag sequence and the structured test identification data, obtains real-time test identification data, determines the corresponding matching score according to the real-time test identification data, the test identification supervision data set and the set text semantic matching model, performs result evaluation operation on the test identification task according to the matching score, and determines the corresponding test identification task evaluation result, thereby being able to improve the efficiency and accuracy of test identification evaluation based on automated information extraction and semantic matching.
[0261] The embodiments of the present application also provide a computer program product capable of implementing all the steps of the test identification task evaluation method in the above embodiment, where the execution subject is a server or a client. When the computer program / instruction is executed by a processor, the steps of the test identification task evaluation method are implemented. For example, the computer program / instruction implements the following steps:
[0262] Step S101: obtaining a historical test identification data file, performing a data extraction operation on the historical test identification data file according to a preset data extraction rule, and determining corresponding structured test identification data and semi-structured test identification data;
[0263] Step S102: input the semi-structured test identification data into the input layer of the setting information extraction model for encoding conversion operation, determine the corresponding conversion code, input the conversion code into the bidirectional encoder layer of the setting information extraction model for semantic conversion operation, determine the corresponding semantic features, input the semantic features into the bidirectional long short-term memory network layer of the setting information extraction model for dependency capture operation, determine the corresponding semantic dependency features, input the semantic dependency features into the conditional random field layer of the setting information extraction model, determine the corresponding semi-structured test identification data label sequence, and determine the corresponding test identification supervision data set according to the semi-structured test identification data label sequence and the structured test identification data;
[0264] Step S103: Acquire real-time test identification data, determine a corresponding matching score based on the real-time test identification data, the test identification supervision data set and a set text semantic matching model, perform a result evaluation operation on the test identification task based on the matching score, and determine a corresponding test identification task evaluation result.
[0265] From the above description, it can be seen that the computer program product provided in the embodiment of the present application obtains historical test identification data files, determines the corresponding structured test identification data and semi-structured test identification data according to preset data extraction rules, inputs the semi-structured test identification data into the conditional random field layer of the set information extraction model, determines the corresponding semi-structured test identification data tag sequence, determines the corresponding test identification supervision data set according to the semi-structured test identification data tag sequence and the structured test identification data, obtains real-time test identification data, determines the corresponding matching score according to the real-time test identification data, the test identification supervision data set and the set text semantic matching model, performs result evaluation operation on the test identification task according to the matching score, and determines the corresponding test identification task evaluation result, thereby being able to improve the efficiency and accuracy of test identification evaluation based on automated information extraction and semantic matching.
[0266] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0267] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (apparatus), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable test identification task evaluation device to produce a machine, so that the instructions executed by the processor of the computer or other programmable test identification task evaluation device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0268] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable test qualification task evaluation device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0269] These computer program instructions may also be loaded onto a computer or other programmable test qualification task evaluation device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0270] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. A test identification task evaluation method, characterized in that: The method comprises: Acquire a historical test identification data file, perform a data extraction operation on the historical test identification data file according to a preset data extraction rule, and determine corresponding structured test identification data and semi-structured test identification data; Input the semi-structured test identification data into the input layer of the setting information extraction model for encoding conversion operation, determine the corresponding conversion code, input the conversion code into the bidirectional encoder layer of the setting information extraction model for semantic conversion operation, determine the corresponding semantic features, input the semantic features into the bidirectional long short-term memory network layer of the setting information extraction model for dependency capture operation, determine the corresponding semantic dependency features, input the semantic dependency features into the conditional random field layer of the setting information extraction model, determine the corresponding semi-structured test identification data label sequence, and determine the corresponding test identification supervision data set according to the semi-structured test identification data label sequence and the structured test identification data; Acquire real-time test identification data, determine a corresponding matching score based on the real-time test identification data, the test identification supervision data set and a set text semantic matching model, perform a result evaluation operation on the test identification task based on the matching score, and determine a corresponding test identification task evaluation result.
2. The test identification task evaluation method according to claim 1, characterized in that: The performing of data extraction operation on the historical test identification data file according to the preset data extraction rule to determine the corresponding structured test identification data and semi-structured test identification data includes: Performing data extraction operations on the historical test identification data file according to preset key node extraction rules to determine corresponding structured test identification data; A data extraction operation is performed on the historical test identification data file according to preset data cleaning rules to determine corresponding semi-structured test identification data.
3. The test identification task evaluation method according to claim 1, characterized in that: The step of inputting the semi-structured test identification data into the input layer of the setting information extraction model to perform a coding conversion operation and determining a corresponding conversion code comprises: Performing a word segmentation operation on the semi-structured test identification data according to a preset word segmentation rule to determine a corresponding minimum word unit; The minimum word unit is input into the input layer of the setting information extraction model to perform a coding conversion operation, and a corresponding conversion code is determined.
4. The test identification task evaluation method according to claim 1, characterized in that: The step of inputting the conversion code into the bidirectional encoder layer of the setting information extraction model to perform a semantic conversion operation and determine the corresponding semantic features includes: Input the conversion code into the bidirectional encoder layer of the setting information extraction model, perform a vector conversion operation on the conversion code according to a preset word embedding algorithm, and determine a corresponding encoding vector; A coding stacking operation is performed according to the coding vector and a preset number of coding structure layers to determine corresponding semantic features.
5. The test identification task evaluation method according to claim 1, characterized in that: The step of inputting the semantic features into the bidirectional long short-term memory network layer of the setting information extraction model to perform a dependency capture operation and determining corresponding semantic dependency features includes: Inputting the semantic features into the bidirectional long short-term memory network layer of the setting information extraction model, performing dependency capture operations according to a preset gating mechanism, and determining a corresponding semantic feature matrix; The corresponding semantic dependency features are determined according to the semantic feature matrix and the preset memory unit.
6. The test identification task evaluation method according to claim 1, characterized in that: The determining of a corresponding matching score according to the real-time test identification data, the test identification supervision data set and a set text semantic matching model includes: Inputting the test identification supervision data set into a preset text semantic matching model to perform a model training operation, thereby obtaining a set text semantic matching model after the model training operation; The real-time test identification data is input into the set text semantic matching model to determine the matching score output by the set text semantic matching model.
7. The test identification task evaluation method according to claim 6, characterized in that: The step of inputting the test identification supervision data set into a preset text semantic matching model for a model training operation to obtain a set text semantic matching model after the model training operation comprises: Inputting the test identification supervision data set into a preset text semantic matching model, and determining the corresponding matching loss according to a pre-trained word embedding model and a preset cosine similarity algorithm; The model parameter updating operation is performed according to the minimum matching loss and the back propagation algorithm to determine the corresponding set text semantic matching model.
8. A test identification task evaluation device, characterized in that: The device comprises: A data extraction module is used to obtain a historical test identification data file, perform a data extraction operation on the historical test identification data file according to a preset data extraction rule, and determine the corresponding structured test identification data and semi-structured test identification data; A supervised data set construction module, used for inputting the semi-structured test identification data into the input layer of the setting information extraction model for encoding conversion operation, determining the corresponding conversion code, inputting the conversion code into the bidirectional encoder layer of the setting information extraction model for semantic conversion operation, determining the corresponding semantic features, inputting the semantic features into the bidirectional long short-term memory network layer of the setting information extraction model for dependency capture operation, determining the corresponding semantic dependency features, inputting the semantic dependency features into the conditional random field layer of the setting information extraction model, determining the corresponding semi-structured test identification data label sequence, and determining the corresponding test identification supervision data set according to the semi-structured test identification data label sequence and the structured test identification data; The test identification evaluation module is used to obtain real-time test identification data, determine the corresponding matching score based on the real-time test identification data, the test identification supervision data set and the set text semantic matching model, perform result evaluation operation on the test identification task based on the matching score, and determine the corresponding test identification task evaluation result.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the test identification task evaluation method described in any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the test identification task evaluation method described in any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Method for constructing and storing knowledge graph in network space security field
CN111428054A
Construction method based on novel research and development institution scientific and technological innovation service knowledge graph system
CN116992042A
Evaluation task-oriented evaluation index screening method and device
CN118410126A
Cited By
Test verification evaluation method based on AI intelligent agent and related device
CN120873522A
