Unstructured sampling data processing method, device and equipment and readable storage medium

Abstract

The application relates to a non-structured sampling data processing method, device and equipment and a readable storage medium. The method comprises the following steps: extracting a basic keyword from a detection item name; generating a synonym expansion keyword based on a plurality of description texts; inputting the detection item name, the basic keyword and the synonym expansion keyword into a language large model to obtain a supplementary keyword; integrating the basic keyword, the synonym expansion keyword and the supplementary keyword to obtain a keyword knowledge base having a mapping relationship with the detection item; obtaining non-structured sampling data to be processed, performing word segmentation processing to obtain a word sequence, matching the word sequence with the keyword knowledge base; judging whether the detection items mapped by the matched keywords are consistent; and if all the keywords point to the same detection item, completing the structured conversion. The method can realize automatic processing of non-structured sampling data and improve processing efficiency and accuracy.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, apparatus, computer equipment, computer-readable storage medium, and computer program product for processing unstructured sampling inspection data. Background Technology

[0002] With the development of data processing technology, text classification and keyword matching technologies have emerged. This technology achieves automated text classification by extracting and comparing keywords from text content.

[0003] Traditional techniques typically employ rule-based or statistical keyword matching methods. This involves pre-establishing a keyword database, comparing the text to be processed against entries in the database, and determining the text's category based on the matching results.

[0004] However, current traditional methods mainly rely on static, manually maintained keyword databases for simple keyword matching, which cannot effectively deal with the rich variations of the same detection item or result in natural language descriptions (such as synonyms, near-synonyms, industry terms, colloquial expressions, etc.), resulting in low matching coverage and poor accuracy. Summary of the Invention

[0005] Therefore, it is necessary to provide a method, apparatus, computer equipment, computer-readable storage medium, and computer program product for processing unstructured sampling data that can effectively improve matching coverage and accuracy, in order to address the aforementioned technical problems.

[0006] Firstly, this application provides a method for processing unstructured sampling data, including:

[0007] Extract basic keywords from the names of the testing items;

[0008] Multiple descriptive texts belonging to the same testing item are identified from the structured and labeled historical sampling records, and synonymous extended keywords are generated based on these multiple descriptive texts;

[0009] Input the detection project name, basic keywords, and synonymous extended keywords into the language model to obtain supplementary keywords;

[0010] The basic keywords, synonymous extended keywords, and supplementary keywords are integrated to obtain a keyword knowledge base that has a mapping relationship with the detection items;

[0011] Obtain the unstructured sampling data to be processed, perform word segmentation on the unstructured sampling data to obtain word sequences, and match the word sequences with the keyword knowledge base;

[0012] Determine whether the detection items mapped to the matched keywords are consistent. If all keywords point to the same detection item, the structured conversion is completed. If all keywords point to multiple different detection items, the unstructured sampling data is marked as requiring manual review. If no keywords are matched, the unstructured sampling data is marked as unidentified.

[0013] In one embodiment, synonymous extended keywords are generated based on multiple descriptive texts, including:

[0014] Extract key descriptive words from multiple descriptive texts;

[0015] Calculate the semantic similarity between key descriptive words and preset candidate words, and determine synonymous extended keywords based on the similarity.

[0016] In one embodiment, the semantic similarity between key descriptive words and candidate words is calculated, and synonymous expanded keywords are determined based on the similarity, including:

[0017] A pre-trained Chinese word vector model is used to convert key descriptive words into descriptive word vectors;

[0018] Obtain candidate words from the vocabulary of the pre-trained Chinese word vector model, and convert each candidate word into a candidate word vector;

[0019] Calculate the cosine similarity between the descriptive word vector and the candidate word vector;

[0020] Candidate words with a cosine similarity exceeding a first preset threshold are used as synonymous extended keywords for key descriptors.

[0021] In one embodiment, the detected project name, basic keywords, and synonymous extended keywords are input into the language model to obtain supplementary keywords, including:

[0022] The detection item name, basic keywords and synonymous extended keywords are combined into prompt information. The prompt information also includes the item explanation content of the detection item and control instructions for generating diverse natural language descriptions.

[0023] The prompt information is input into the language model to obtain the output candidate descriptive words;

[0024] Candidate descriptors are filtered, and multiple candidate descriptors pointing to the same meaning are merged by synonyms to obtain supplementary keywords.

[0025] In one embodiment, matching the word sequence with a keyword knowledge base includes:

[0026] Compare each word in the word sequence with the keywords in the keyword knowledge base to see if they match.

[0027] If they match, the match is confirmed to be successful, and the successfully matched words and their corresponding mapping detection items are recorded.

[0028] If there is a discrepancy, a pre-trained Chinese word vector model is used to convert the unmatched words in the word sequence into word vectors;

[0029] Calculate the cosine similarity between the word vector and each keyword vector in the keyword knowledge base; if the cosine similarity exceeds the second preset threshold, confirm that the corresponding word and keyword are successfully matched.

[0030] In one embodiment, the method further includes:

[0031] Unstructured sampling data that will be marked as unidentified will be included as part of the historical sampling records to update the keyword knowledge base.

[0032] Secondly, this application also provides an unstructured sampling inspection data processing apparatus, comprising:

[0033] The acquisition module is used to extract basic keywords from the names of the detected items;

[0034] The acquisition module is also used to determine multiple descriptive texts belonging to the same detection item from the structured and labeled historical sampling records, and to generate synonymous extended keywords based on the multiple descriptive texts;

[0035] The acquisition module is also used to input the detection project name, basic keywords, and synonymous extended keywords into the language model to obtain supplementary keywords;

[0036] The module is used to integrate basic keywords, synonymous extended keywords, and supplementary keywords to obtain a keyword knowledge base that has a mapping relationship with the detection project;

[0037] The matching module is used to acquire unstructured sampling data to be processed, perform word segmentation on the unstructured sampling data to obtain word sequences, and match the word sequences with the keyword knowledge base;

[0038] The conversion module is used to determine whether the detection items mapped to the matched keywords are consistent. If all keywords point to the same detection item, the structured conversion is completed; if all keywords point to multiple different detection items, the unstructured sampling data is marked as requiring manual review; if no keywords are matched, the unstructured sampling data is marked as unrecognized.

[0039] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0040] Extract basic keywords from the names of the testing items;

[0041] Multiple descriptive texts belonging to the same testing item are identified from the structured and labeled historical sampling records, and synonymous extended keywords are generated based on these multiple descriptive texts;

[0042] Input the detection project name, basic keywords, and synonymous extended keywords into the language model to obtain supplementary keywords;

[0043] The basic keywords, synonymous extended keywords, and supplementary keywords are integrated to obtain a keyword knowledge base that has a mapping relationship with the detection items;

[0044] Obtain the unstructured sampling data to be processed, perform word segmentation on the unstructured sampling data to obtain word sequences, and match the word sequences with the keyword knowledge base;

[0045] Determine whether the detection items mapped to the matched keywords are consistent. If all keywords point to the same detection item, the structured conversion is completed. If all keywords point to multiple different detection items, the unstructured sampling data is marked as requiring manual review. If no keywords are matched, the unstructured sampling data is marked as unidentified.

[0046] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0047] Extract basic keywords from the names of the testing items;

[0048] Multiple descriptive texts belonging to the same testing item are identified from the structured and labeled historical sampling records, and synonymous extended keywords are generated based on these multiple descriptive texts;

[0049] Input the detection project name, basic keywords, and synonymous extended keywords into the language model to obtain supplementary keywords;

[0050] The basic keywords, synonymous extended keywords, and supplementary keywords are integrated to obtain a keyword knowledge base that has a mapping relationship with the detection items;

[0051] Obtain the unstructured sampling data to be processed, perform word segmentation on the unstructured sampling data to obtain word sequences, and match the word sequences with the keyword knowledge base;

[0052] Determine whether the detection items mapped to the matched keywords are consistent. If all keywords point to the same detection item, the structured conversion is completed. If all keywords point to multiple different detection items, the unstructured sampling data is marked as requiring manual review. If no keywords are matched, the unstructured sampling data is marked as unidentified.

[0053] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0054] Extract basic keywords from the names of the testing items;

[0055] Multiple descriptive texts belonging to the same testing item are identified from the structured and labeled historical sampling records, and synonymous extended keywords are generated based on these multiple descriptive texts;

[0056] Input the detection project name, basic keywords, and synonymous extended keywords into the language model to obtain supplementary keywords;

[0057] The basic keywords, synonymous extended keywords, and supplementary keywords are integrated to obtain a keyword knowledge base that has a mapping relationship with the detection items;

[0058] Obtain the unstructured sampling data to be processed, perform word segmentation on the unstructured sampling data to obtain word sequences, and match the word sequences with the keyword knowledge base;

[0059] Determine whether the detection items mapped to the matched keywords are consistent. If all keywords point to the same detection item, the structured conversion is completed. If all keywords point to multiple different detection items, the unstructured sampling data is marked as requiring manual review. If no keywords are matched, the unstructured sampling data is marked as unidentified.

[0060] The aforementioned method, apparatus, computer equipment, computer-readable storage medium, and computer program product for processing unstructured sampling inspection data include: extracting basic keywords from the name of the inspection item; identifying multiple descriptive texts belonging to the same inspection item from historically annotated sampling inspection records, and generating synonymous extended keywords based on these texts; inputting the inspection item name, basic keywords, and synonymous extended keywords into a language model to obtain supplementary keywords; integrating the basic keywords, synonymous extended keywords, and supplementary keywords to obtain a keyword knowledge base that maps to the inspection item; acquiring the unstructured sampling inspection data to be processed, performing word segmentation on the unstructured sampling inspection data to obtain a word sequence, and matching the word sequence with the keyword knowledge base; determining whether the inspection items mapped to each matched keyword are consistent; if all keywords point to the same inspection item, the structured conversion is completed; if all keywords point to multiple different inspection items, the unstructured sampling inspection data is marked as requiring manual review; if no keywords are matched, the unstructured sampling inspection data is marked as unidentified. By constructing a keyword knowledge base through a three-source fusion approach—extracting basic keywords from the names of detection items, mining synonymous extended keywords from historical sampling records, and generating supplementary keywords using a large language model—this keyword knowledge base covers standard terminology, historical authentic expressions, and potentially emerging expressions, overcoming the problems of incomplete coverage and inability to identify synonym variants in traditional static keyword databases. By segmenting unstructured sampling data and matching it with the keyword knowledge base, key information in the text can be accurately identified. By determining whether the detection items mapped to the matched keywords are consistent, the system automatically performs structured conversion when consistent, marks inconsistent data as requiring manual review when inconsistent, and marks data as unidentified when not matched. This achieves automated processing of unstructured sampling data, reduces manual intervention, and improves processing efficiency and accuracy. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 This is a diagram illustrating the application environment of an unstructured sampling data processing method in one embodiment.

[0063] Figure 2 This is a flowchart illustrating an unstructured sampling data processing method in one embodiment;

[0064] Figure 3 This is a flowchart illustrating the steps of an unstructured sampling data processing method in one embodiment.

[0065] Figure 4 This is a schematic diagram of the keyword knowledge base construction process in another embodiment;

[0066] Figure 5 Here is a flowchart of data matching and structuring in another embodiment;

[0067] Figure 6 This is a structural block diagram of an unstructured sampling data processing device in one embodiment;

[0068] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0070] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0071] The unstructured sampling data processing method provided in this application embodiment can be applied to, for example... Figure 1The application environment shown can involve only terminal 102, only server 104, or both terminal 102 and server 104. Terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located in the cloud or on other network servers. Specifically, terminal 102 or server 104 completes an unstructured sampling data processing method, which includes: extracting basic keywords from the name of the detection item; identifying multiple descriptive texts belonging to the same detection item from the structured and labeled historical sampling records, and generating synonymous extended keywords based on the multiple descriptive texts; inputting the detection item name, basic keywords, and synonymous extended keywords into a language model to obtain supplementary keywords; integrating the basic keywords, synonymous extended keywords, and supplementary keywords to obtain a keyword knowledge base that has a mapping relationship with the detection item; acquiring the unstructured sampling data to be processed, performing word segmentation on the unstructured sampling data to obtain a word sequence, and matching the word sequence with the keyword knowledge base; determining whether the detection items mapped by each matched keyword are consistent; if all keywords point to the same detection item, the structured conversion is completed; if all keywords point to multiple different detection items, the unstructured sampling data is marked as requiring manual review; if no keywords are matched, the unstructured sampling data is marked as unrecognized. By constructing a keyword knowledge base through a three-source fusion approach—extracting basic keywords from the names of detection items, mining synonymous extended keywords from historical sampling records, and generating supplementary keywords using a large language model—this keyword knowledge base covers standard terminology, historical authentic expressions, and potentially emerging expressions, overcoming the problems of incomplete coverage and inability to identify synonym variants in traditional static keyword databases. By segmenting unstructured sampling data and matching it with the keyword knowledge base, key information in the text can be accurately identified. By determining whether the detection items mapped to the matched keywords are consistent, the system automatically performs structured conversion when consistent, marks inconsistent data as requiring manual review when inconsistent, and marks data as unidentified when not matched. This achieves automated processing of unstructured sampling data, reduces manual intervention, and improves processing efficiency and accuracy.

[0072] The terminal 102 can be, but is not limited to, various personal computers, laptops, tablets, and industrial handheld terminals. The server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0073] In one exemplary embodiment, such as Figure 2 As shown, a method for processing unstructured sampling data is provided, which can be applied to... Figure 1Taking server 104 as an example, the explanation includes the following steps 202 to 212. Wherein:

[0074] Step 202: Extract basic keywords from the names of the detection items;

[0075] Among them, the testing items refer to the standard testing categories used in the power industry for random quality inspections of power equipment in power supply stations, substations, and transmission and distribution lines. For example, testing items may include "transformer temperature rise test", "circuit breaker opening and closing test", "insulator withstand voltage test", and "cable partial discharge test", etc.

[0076] For example, the name of the detected item is broken down into one or more lexical units that retain their original meaning. Based on the lexical units obtained from the breakdown, the lexical units are expanded to obtain basic keywords.

[0077] Step 204: Identify multiple descriptive texts belonging to the same testing item from the structured and labeled historical sampling records, and generate synonymous extended keywords based on the multiple descriptive texts;

[0078] Historical inspection records refer to structured and annotated historical inspection reports or items generated during the quality inspection of power materials. Each record contains original descriptive text belonging to a specific inspection item. Inspection items refer to the standard inspection categories used to evaluate the quality of power equipment, such as "transformer temperature rise test" or "circuit breaker opening and closing test." Synonymous extended keywords refer to words semantically similar to key descriptive terms in the descriptive text, used to expand the coverage of the keyword knowledge base to identify different expressions of the same meaning.

[0079] For example, key descriptive words are extracted from multiple descriptive texts belonging to the same detection item, the semantic similarity between the key descriptive words and candidate words is calculated, and synonymous extended keywords are determined based on the similarity.

[0080] Step 206: Input the detection project name, basic keywords, and synonymous extended keywords into the language model to obtain supplementary keywords;

[0081] Among them, the large-scale language model refers to a deep learning model pre-trained on a large-scale corpus, possessing natural language understanding and generation capabilities, and able to generate semantically relevant and expressively diverse natural language text based on input prompts. Large-scale language models can include ChatGLM, GPT series models, etc. Supplementary keywords refer to descriptive keywords generated by the large-scale language model based on the detection item name and existing keywords, which are likely to appear in real sampling reports. These are used to further expand the coverage of the keyword knowledge base, especially to cover new expressions that have not yet appeared in historical data.

[0082] For example, the detection project name, basic keywords, and synonymous extended keywords are combined into a prompt message and input into the language model; candidate descriptors output by the language model are obtained; the candidate descriptors are manually screened to remove those that do not correspond to the detection object and detection result of the detection project, and multiple candidate descriptors pointing to the same meaning are merged by synonyms to obtain supplementary keywords.

[0083] Step 208: Integrate the basic keywords, synonymous extended keywords, and supplementary keywords to obtain a keyword knowledge base that has a mapping relationship with the detection items;

[0084] The keyword knowledge base refers to a structured data set indexed by detection items and containing keywords. It serves as the basis for identifying the detection items to which unstructured sampling data belongs in subsequent matching steps. Each keyword is mapped to a detection item in the knowledge base, meaning that when the keyword appears in the sampling description text, it can be classified into the corresponding detection item.

[0085] For example, the basic keywords, synonymous extended keywords, and supplementary keywords are merged to obtain a set of keywords to be added to the database; the set of keywords to be added to the database is deduplicated to remove duplicate keyword entries; and a mapping relationship is established between each deduplicated keyword and its corresponding detection item to obtain a keyword knowledge base.

[0086] Step 210: Obtain the unstructured sampling data to be processed, perform word segmentation on the unstructured sampling data to obtain a word sequence, and match the word sequence with the keyword knowledge base;

[0087] Unstructured sampling data refers to inspection reports or records generated during the sampling inspection of power materials quality, existing in the form of natural language text. Their content is not constrained by predefined data formats or field structures, such as "the temperature was too high during the transformer temperature rise test" or "the circuit breaker opening and closing test was normal." Word segmentation refers to the process of dividing continuous natural language text into a sequence of words with independent semantic units. A word sequence is the set of words obtained after segmenting the original unstructured text according to semantic units. For example, for the text "the temperature was too high during the transformer temperature rise test," the segmentation yields the word sequence "transformer," "temperature rise test," "medium," "temperature," and "too high."

[0088] For example, unstructured sampling data is obtained, and the data is segmented to obtain a word sequence; the word sequence is matched with a keyword knowledge base, and the successfully matched keywords and their mapped detection items are recorded; for words that fail to match, semantic similarity matching is performed, and the successfully matched keywords and their mapped detection items are recorded.

[0089] Step 212: Determine whether the detection items mapped to the matched keywords are consistent. If all keywords point to the same detection item, the structured conversion is completed. If all keywords point to multiple different detection items, the unstructured sampling data is marked as requiring manual review. If no keywords are matched, the unstructured sampling data is marked as unidentified.

[0090] Structured transformation refers to binding the original descriptive text in unstructured sampling data with the matched detection items, forming a structured data record containing the original description, a list of matching keywords, and the corresponding detection items. Marking data as requiring manual review means that for data with conflicting matching results, reviewers will make the final determination of the detection item attribution. Marking data as unidentified means that for data without any matching keywords, the system or reviewers will further analyze its corresponding detection item.

[0091] For example, all matched keywords and their corresponding detection items are obtained; it is determined whether the detection items corresponding to each keyword are consistent; if all keywords point to the same detection item, the unstructured sampling data and its original description, the list of matched keywords and the corresponding detection items are bound together, and structured data is output; if each keyword points to multiple different detection items, the unstructured sampling data is marked as requiring manual review; if no keywords are matched, the unstructured sampling data is marked as unrecognized.

[0092] In the above-mentioned method for processing unstructured sampling data, basic keywords are extracted from the names of the detection items; multiple descriptive texts belonging to the same detection item are identified from the historical sampling records that have been structured and labeled, and synonymous extended keywords are generated based on these multiple descriptive texts; the names of the detection items, basic keywords, and synonymous extended keywords are input into a language model to obtain supplementary keywords; the basic keywords, synonymous extended keywords, and supplementary keywords are integrated to obtain a keyword knowledge base that has a mapping relationship with the detection items; the unstructured sampling data to be processed is obtained, and word segmentation is performed on the unstructured sampling data to obtain a word sequence, which is then matched with the keyword knowledge base; it is determined whether the detection items mapped by each matched keyword are consistent. If all keywords point to the same detection item, the structured conversion is completed; if all keywords point to multiple different detection items, the unstructured sampling data is marked as requiring manual review; if no keywords are matched, the unstructured sampling data is marked as unidentified. By constructing a keyword knowledge base through a three-source fusion approach—extracting basic keywords from the names of detection items, mining synonymous extended keywords from historical sampling records, and generating supplementary keywords using a large language model—this keyword knowledge base covers standard terminology, historical authentic expressions, and potentially emerging expressions, overcoming the problems of incomplete coverage and inability to identify synonym variants in traditional static keyword databases. By segmenting unstructured sampling data and matching it with the keyword knowledge base, key information in the text can be accurately identified. By determining whether the detection items mapped to the matched keywords are consistent, the system automatically performs structured conversion when consistent, marks inconsistent data as requiring manual review when inconsistent, and marks data as unidentified when not matched. This achieves automated processing of unstructured sampling data, reduces manual intervention, and improves processing efficiency and accuracy.

[0093] In one embodiment, synonymous extended keywords are generated based on multiple descriptive texts, including:

[0094] Extract key descriptive words from multiple descriptive texts;

[0095] Calculate the semantic similarity between key descriptive words and preset candidate words, and determine synonymous extended keywords based on the similarity.

[0096] Specifically, for the "transformer temperature rise test" inspection item, historical sampling records include multiple descriptive texts: "Temperature too high during transformer temperature rise test," "Winding temperature too high, exceeding standard requirements," "Severe heat generation during operation," and "High temperature anomaly detected." Key descriptive words "too high," "severe heat generation," and "high temperature anomaly" are extracted from these texts. Taking the key descriptive word "too high" as an example, a pre-trained Chinese word vector model is used to convert it into a descriptive word vector. Candidate words such as "too high," "overheating," "excessive temperature," and "too large" from the vocabulary are obtained and converted into candidate word vectors respectively. The cosine similarity between the descriptive word vector of "too high" and each candidate word vector is calculated. A preset threshold of 0.8 is set, and candidate words "too high," "overheating," and "excessive temperature" with a similarity exceeding 0.8 are used as synonymous extended keywords for "too high."

[0097] In the above embodiments, by extracting key descriptive words from multiple descriptive texts and calculating the semantic similarity between the key descriptive words and preset candidate words to determine synonymous extended keywords, it is possible to automatically mine multiple expression variations that are semantically similar to the key descriptive words from real historical data, overcome the problem of incomplete and inaccurate manual summarization of synonym variations, and effectively improve the keyword knowledge base's coverage of diverse natural language expressions.

[0098] In one embodiment, the semantic similarity between key descriptive words and candidate words is calculated, and synonymous expanded keywords are determined based on the similarity, including:

[0099] A pre-trained Chinese word vector model is used to convert key descriptive words into descriptive word vectors;

[0100] Obtain candidate words from the vocabulary of the pre-trained Chinese word vector model, and convert each candidate word into a candidate word vector;

[0101] Calculate the cosine similarity between the descriptive word vector and the candidate word vector;

[0102] Candidate words with a cosine similarity exceeding a first preset threshold are used as synonymous extended keywords for key descriptors.

[0103] Specifically, a pre-trained Chinese word vector model is used to convert key descriptive words into descriptive word vectors; candidate words in the vocabulary of the pre-trained Chinese word vector model are obtained, and each candidate word is converted into a candidate word vector; the cosine similarity between the descriptive word vector and the candidate word vectors is calculated; and candidate words with a cosine similarity exceeding a first preset threshold are used as synonymous expansion keywords for the key descriptive words. For example, for the key descriptive word "too high", the pre-trained Chinese word vector model is used to convert it into a vector representation, and candidate words such as "too high", "overheating", "excessive temperature", and "too big" in the vocabulary are obtained and converted into candidate word vectors respectively. The cosine similarity between the descriptive word vector of "too high" and each candidate word vector is calculated, and the first preset threshold is set to 0.8. Candidate words with a similarity exceeding 0.8, such as "too high", "overheating", and "excessive temperature", are used as synonymous expansion keywords for "too high", thus completing the synonym expansion of the key descriptive word.

[0104] In the above embodiments, a pre-trained Chinese word vector model is used to convert key descriptive words and candidate words into vectors, and the cosine similarity between them is calculated. Candidate words with similarity exceeding a preset threshold are used as synonymous expansion keywords, thus achieving automated and accurate mining of semantically similar words for key descriptive words. This method calculates similarity based on the semantic space of a large-scale pre-trained corpus, which can discover semantically similar expressions that are difficult to enumerate manually, effectively improving the keyword knowledge base's coverage and expansion efficiency for diverse natural language variants.

[0105] In one embodiment, the detected project name, basic keywords, and synonymous extended keywords are input into the language model to obtain supplementary keywords, including:

[0106] The detection item name, basic keywords and synonymous extended keywords are combined into prompt information. The prompt information also includes the item explanation content of the detection item and control instructions for generating diverse natural language descriptions.

[0107] The prompt information is input into the language model to obtain the output candidate descriptive words;

[0108] Candidate descriptors are filtered, and multiple candidate descriptors pointing to the same meaning are merged by synonyms to obtain supplementary keywords.

[0109] Specifically, for the test item "Transformer Temperature Rise Test", the item name "Transformer Temperature Rise Test", the acquired basic keywords and synonymous extended keywords (such as "transformer", "temperature rise", "temperature increase", "TR", "heating test", "excessive", "overheating", "exceeding temperature", etc.), the corresponding item explanation content, and the control instructions used to generate diverse natural language descriptions are combined into prompt information and input into the language model. The model outputs candidate descriptive words including "temperature exceeds standard", "temperature rise exceeds limit", "excessive heat generation", "temperature rises too fast", "winding heating", "local overheating", "hot spot temperature is too high", "poor heat dissipation leads to temperature rise", etc. After screening, candidate words that do not correspond to the meaning of transformer temperature rise test are removed, and multiple candidate descriptive words pointing to the same meaning are merged by synonym (such as merging "temperature exceeds standard" and "temperature rise exceeds limit" into the same meaning) to obtain supplementary keywords.

[0110] In the above embodiments, by combining the detection item name, existing basic keywords, and synonymous extended keywords into prompt information, and utilizing the generation capabilities of the language big data model, diverse candidate descriptive words are automatically obtained. This proactively discovers new expressions that have not yet appeared in historical data, effectively improving the forward-looking nature and coverage of the keyword knowledge base. By filtering and merging synonyms in the model output, the accuracy and refinement of the keywords in the database are ensured, and redundant entries are avoided from interfering with subsequent matching, thereby further improving the quality of the keyword knowledge base.

[0111] In one embodiment, matching the word sequence with a keyword knowledge base includes:

[0112] Compare each word in the word sequence with the keywords in the keyword knowledge base to see if they match.

[0113] If they match, the match is confirmed to be successful, and the successfully matched words and their corresponding mapping detection items are recorded.

[0114] If there is a discrepancy, a pre-trained Chinese word vector model is used to convert the unmatched words in the word sequence into word vectors;

[0115] Calculate the cosine similarity between the word vector and each keyword vector in the keyword knowledge base; if the cosine similarity exceeds the second preset threshold, confirm that the corresponding word and keyword are successfully matched.

[0116] Specifically, each word in the word sequence is compared sequentially with keywords in the keyword knowledge base. If the current word is exactly the same as a keyword in the keyword knowledge base, the match is confirmed directly, and the word and its mapped detection item are recorded. If the current word is different from all keywords in the keyword knowledge base, the word is input into a pre-trained Chinese word vector model and mapped to a word vector in a high-dimensional semantic space. At the same time, the vector representations of each keyword in the keyword knowledge base in the same semantic space are obtained, and the cosine similarity between the word vector and each keyword vector is calculated one by one. When the cosine similarity exceeds a preset threshold (e.g., 0.8), the semantic match between the word and the keyword is confirmed to be successful, and the word and its mapped detection item are recorded. Taking the transformer temperature rise test as an example, the word sequence "temperature rise test" is completely identical to the word sequence "temperature rise test" in the keyword knowledge base, and is directly matched and mapped to the "transformer temperature rise test" detection item. The word sequence "temperature" is different from any keyword in the keyword knowledge base. After converting it into a vector, the cosine similarity with each keyword vector is calculated. When the cosine similarity with "temperature rise" exceeds the preset threshold, it is confirmed that "temperature" and "temperature rise" are semantically matched successfully, and is also mapped to the "transformer temperature rise test" detection item.

[0117] In the above embodiments, the words in the word sequence are first compared with the keywords in the keyword knowledge base. When the comparison is inconsistent, a pre-trained Chinese word vector model is used to convert the unmatched words into word vectors, and the cosine similarity between the word vectors and the keyword vectors in the keyword knowledge base is calculated. When the similarity exceeds a preset threshold, the match is confirmed as successful, thus achieving an organic combination of precise matching and semantic similarity matching. Compared with a single precise matching method, this embodiment can effectively identify word expressions that are different in word form from the keywords in the knowledge base but similar in meaning, overcoming the problem of missed matches caused by synonyms and near-synonyms, and significantly improving the coverage and accuracy of matching.

[0118] In one embodiment, the method further includes:

[0119] Unstructured sampling data that will be marked as unidentified will be included as part of the historical sampling records to update the keyword knowledge base.

[0120] Specifically, unstructured sampling data that does not match any keywords is marked as unidentified. This unidentified data is then added to the structured, labeled historical sampling records as a new historical sampling entry. When the accumulated amount of unidentified data reaches a preset condition (e.g., a certain quantity or time period), the step of identifying multiple descriptive texts belonging to the same detection item from the historical sampling records and generating synonymous extended keywords is repeated. These newly generated synonymous extended keywords are then added to the keyword knowledge base, achieving iterative updates. Optionally, unidentified data can also be directly used as input to the language model, and the step of generating supplementary keywords can be repeated to update the keyword knowledge base.

[0121] In the above embodiments, by feeding back unidentified data to historical sampling records and using it to update the keyword knowledge base, the keyword knowledge base can continuously improve and dynamically evolve based on the unidentified expressions encountered in the actual matching process. As unidentified data accumulates, the coverage of the knowledge base gradually expands, and the ability to identify diverse expressions is continuously enhanced, thereby reducing the occurrence rate of unidentification in subsequent processing, realizing continuous optimization of the knowledge base and cyclical improvement of matching performance, fundamentally solving the defect of the traditional static keyword database that is "built once and remains unchanged".

[0122] In one embodiment, such as Figure 3 The image shows a specific embodiment of an unstructured sampling data processing method, including:

[0123] like Figure 4 As shown, basic keywords are extracted from the name of the test item. This involves breaking down the test item name into one or more lexical units that retain their original meaning, and then expanding these units with at least one of the following: synonyms, near-synonyms, industry abbreviations, or colloquial expressions. For the test item name "temperature rise test," the resulting lexical units include "temperature rise" and "test." "Temperature rise" is expanded to the synonym "temperature increase," and "test" is expanded to the synonyms "test" and "detection." The industry abbreviation "TR" is expanded, and the colloquial expression "heat test" is expanded, resulting in the basic keyword set: {temperature rise, temperature increase, temperature rise test, temperature rise test, heat test, TR}.

[0124] Multiple descriptive texts belonging to the same detection item are identified from the structured and labeled historical sampling records. Key descriptive words are extracted from these texts, and the semantic similarity between the key descriptive words and candidate words is calculated. Based on the similarity, synonymous extended keywords are determined. Optionally, a pre-trained Chinese word vector model (such as Tencent word vectors, which contains more than 8 million Chinese words, each word represented as a 200-dimensional real vector) is used to convert the key descriptive words into descriptive word vectors. Candidate words in the vocabulary of the pre-trained Chinese word vector model are obtained, and each candidate word is converted into a candidate word vector. The cosine similarity between the descriptive word vector and the candidate word vector is calculated, and candidate words with a cosine similarity exceeding a preset threshold (e.g., 0.8) are used as synonymous extended keywords for the key descriptive words. For the "temperature rise test" inspection item, the historical sampling records included multiple descriptive texts: "Temperature too high during transformer temperature rise test", "Winding temperature too high, exceeding standard requirements", "Severe heat generation during operation", "High temperature anomaly detected". Key descriptive words "too high", "severe heat generation", and "high temperature anomaly" were extracted from the above descriptive texts. Taking the key descriptive word "too high" as an example, the cosine similarity between it and the candidate words was calculated, and a threshold of 0.8 was set. "Too high" (0.92), "overheated" (0.88), "high temperature" (0.85), and "exceeding temperature" (0.82) were selected, while "too high" (0.43), "too slow" (0.21), and "too low" (0.08) were not selected. The candidate words "too high", "overheated", and "exceeding temperature" with a similarity of more than 0.8 were used as synonymous extended keywords for "too high".

[0125] Input the detection project name, basic keywords, and synonymous extended keywords into the language model to obtain supplementary keywords; optionally, combine the detection project name, basic keywords, and synonymous extended keywords into prompt information. The prompt information also includes the project definition of the detection project and control instructions for generating diverse natural language descriptions. Input the prompt information into the language model (e.g., ChatGLM) to obtain candidate descriptors output by the language model. Manually screen the candidate descriptors, remove inaccurate items, and merge multiple candidate descriptors pointing to the same meaning by synonyms to obtain supplementary keywords. For the "Temperature Rise Test" project, the following prompts were generated: "Project Name: Temperature Rise Test. Project Definition: A test to assess the temperature rise of the equipment during operation. Existing keywords include: too high, overheat, and exceeding temperature. Please generate 20 natural language phrases that may appear in the test report and describe abnormal temperature rise test results." The candidate keywords generated by the large model included: "temperature exceeds standard," "temperature rise exceeds limit," "excessive heat generation," "temperature rise too fast," "abnormal temperature rise rate," "severe heat accumulation," "temperature rise due to poor heat dissipation," "winding overheating," "localized overheating," and "high hotspot temperature," etc. After manual calibration, inaccurate items were removed, synonyms were merged, and valid keywords were retained for inclusion in the database.

[0126] The basic keywords, synonymous extended keywords, and supplementary keywords are integrated to obtain a keyword knowledge base that maps to the testing items. Optionally, the basic keywords, synonymous extended keywords, and supplementary keywords are merged, deduplicated, and a mapping relationship between each keyword and the corresponding testing item is established to obtain the keyword knowledge base. The keyword knowledge base records for the "temperature rise test" item are shown in Table 1.

[0127] Table 1. Keyword Knowledge Base Records for the "Temperature Rise Test" Project

[0128]

[0129] Where S1 represents the basic keyword extraction, S2 represents the semantic expansion, and S3 represents the large model expansion.

[0130] like Figure 5 As shown, unstructured sampling data to be processed is obtained, and word segmentation is performed on the unstructured sampling data (e.g., using the jieba word segmentation tool) to obtain a word sequence. The word sequence is then matched with a keyword knowledge base. Optionally, the words in the word sequence are compared with the keywords in the keyword knowledge base. If the comparison matches, the matching is confirmed as successful. If the comparison does not match, a pre-trained Chinese word vector model is used to convert the unmatched words into word vectors. The cosine similarity between the word vectors and the keyword vectors in the keyword knowledge base is calculated. When the cosine similarity exceeds a preset threshold (e.g., 0.8), the matching is confirmed as successful. For the input text "Transformer temperature rise test found the temperature was too high", after preprocessing, the word sequence is obtained as: {transformer, temperature rise test, found, temperature, too high}. After removing the stop word "found", the words in the word sequence are compared with the keywords in the keyword knowledge base. "Temperature rise test" matches the keyword "temperature rise test" in the keyword knowledge base (source S1), and the successfully matched words and their mapped detection items are recorded. For the words "temperature" and "too high" that cannot be matched, a pre-trained Chinese word vector model is used to convert them into word vectors, and the cosine similarity between the word vectors and the keyword vectors in the keyword knowledge base is calculated. If the similarity between "temperature" and "temperature rise" exceeds the preset threshold of 0.8 (source S1), and the similarity between "too high" and "excessively high" exceeds the preset threshold of 0.8 (source S2), both are considered semantically matched successfully and mapped to the "temperature rise test" detection item. After the matching is completed, a matching result list is obtained. For example, for the input text "Transformer temperature rise test found the temperature was too high", the matching results are shown in Table 2.

[0131] Table 2 Matching Results

[0132]

[0133] The system determines whether the detection items mapped to the matched keywords are consistent. If all keywords point to the same detection item, the structured conversion is completed, and the structured processing result is output. The structured processing result includes: the original description (the input unstructured text), the list of matched keywords (the matched keywords and their sources and matching methods), and the corresponding detection item. If all keywords point to multiple different detection items, the unstructured sampled data is marked as requiring manual review. If no keywords are matched, the unstructured sampled data is marked as unrecognized. For the input description "Transformer temperature rise test found the temperature to be too high", the matching results are shown in Table 2. The matched keywords "temperature rise test" (source S1), "too high" (source S2), and "temperature" (source S1) all point to the "temperature rise test" detection item. Therefore, this data is mapped to the "temperature rise test" item, completing the structured conversion.

[0134] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0135] Based on the same inventive concept, this application also provides an unstructured sampling data processing apparatus for implementing the unstructured sampling data processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method. Therefore, the specific limitations in one or more embodiments of the unstructured sampling data processing apparatus provided below can be found in the limitations of the unstructured sampling data processing method described above, and will not be repeated here.

[0136] In one exemplary embodiment, such as Figure 6 As shown, an unstructured sampling data processing device is provided, including: an acquisition module 602, a construction module 604, a matching module 606, and a conversion module 608, wherein:

[0137] Module 602 is used to extract basic keywords from the name of the detection item;

[0138] The acquisition module 602 is also used to determine multiple descriptive texts belonging to the same detection item from the structured and labeled historical sampling records, and generate synonymous extended keywords based on the multiple descriptive texts;

[0139] The acquisition module 602 is also used to input the detection project name, basic keywords and synonymous extended keywords into the language model to obtain supplementary keywords;

[0140] Module 604 is used to integrate basic keywords, synonymous extended keywords, and supplementary keywords to obtain a keyword knowledge base that has a mapping relationship with the detection project;

[0141] The matching module 606 is used to acquire the unstructured sampling data to be processed, perform word segmentation on the unstructured sampling data to obtain a word sequence, and match the word sequence with the keyword knowledge base;

[0142] The conversion module 608 is used to determine whether the detection items mapped to the matched keywords are consistent. If all keywords point to the same detection item, the structured conversion is completed; if all keywords point to multiple different detection items, the unstructured sampling data is marked as requiring manual review; if no keywords are matched, the unstructured sampling data is marked as unrecognized.

[0143] In one embodiment, the acquisition module 602 is further configured to generate synonymous extended keywords based on multiple descriptive texts, including:

[0144] Extract key descriptive words from multiple descriptive texts;

[0145] Calculate the semantic similarity between key descriptive words and preset candidate words, and determine synonymous extended keywords based on the similarity.

[0146] In one embodiment, the acquisition module 602 is further configured to calculate the semantic similarity between the key descriptive words and the candidate words, and determine synonymous extended keywords based on the similarity, including:

[0147] A pre-trained Chinese word vector model is used to convert key descriptive words into descriptive word vectors;

[0148] Obtain candidate words from the vocabulary of the pre-trained Chinese word vector model, and convert each candidate word into a candidate word vector;

[0149] Calculate the cosine similarity between the descriptive word vector and the candidate word vector;

[0150] Candidate words with a cosine similarity exceeding a first preset threshold are used as synonymous extended keywords for key descriptors.

[0151] In one embodiment, the acquisition module 602 is further configured to input the detected project name, basic keywords, and synonymous extended keywords into the language model to obtain supplementary keywords, including:

[0152] The detection item name, basic keywords and synonymous extended keywords are combined into prompt information. The prompt information also includes the item explanation content of the detection item and control instructions for generating diverse natural language descriptions.

[0153] The prompt information is input into the language model to obtain the output candidate descriptive words;

[0154] Candidate descriptors are filtered, and multiple candidate descriptors pointing to the same meaning are merged by synonyms to obtain supplementary keywords.

[0155] In one embodiment, the matching module 606 is used to match the word sequence with a keyword knowledge base, including:

[0156] Compare each word in the word sequence with the keywords in the keyword knowledge base to see if they match.

[0157] If they match, the match is confirmed to be successful, and the successfully matched words and their corresponding mapping detection items are recorded.

[0158] If there is a discrepancy, a pre-trained Chinese word vector model is used to convert the unmatched words in the word sequence into word vectors;

[0159] Calculate the cosine similarity between the word vector and each keyword vector in the keyword knowledge base; if the cosine similarity exceeds the second preset threshold, confirm that the corresponding word and keyword are successfully matched.

[0160] In one embodiment, the method further includes:

[0161] Unstructured sampling data that will be marked as unidentified will be included as part of the historical sampling records to update the keyword knowledge base.

[0162] Each module in the aforementioned unstructured sampling data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0163] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 7As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When executed by the processor, the computer program implements an unstructured sampling data processing method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0164] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0165] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0166] Extract basic keywords from the names of the testing items;

[0167] Multiple descriptive texts belonging to the same testing item are identified from the structured and labeled historical sampling records, and synonymous extended keywords are generated based on these multiple descriptive texts;

[0168] Input the detection project name, basic keywords, and synonymous extended keywords into the language model to obtain supplementary keywords;

[0169] The basic keywords, synonymous extended keywords, and supplementary keywords are integrated to obtain a keyword knowledge base that has a mapping relationship with the detection items;

[0170] Obtain the unstructured sampling data to be processed, perform word segmentation on the unstructured sampling data to obtain word sequences, and match the word sequences with the keyword knowledge base;

[0171] Determine whether the detection items mapped to the matched keywords are consistent. If all keywords point to the same detection item, the structured conversion is completed. If all keywords point to multiple different detection items, the unstructured sampling data is marked as requiring manual review. If no keywords are matched, the unstructured sampling data is marked as unidentified.

[0172] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0173] Extract key descriptive words from multiple descriptive texts; calculate the semantic similarity between key descriptive words and preset candidate words; and determine synonymous extended keywords based on the similarity.

[0174] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0175] A pre-trained Chinese word vector model is used to convert key descriptive words into descriptive word vectors; candidate words are obtained from the vocabulary of the pre-trained Chinese word vector model and each candidate word is converted into a candidate word vector; the cosine similarity between the descriptive word vector and the candidate word vector is calculated; and candidate words with a cosine similarity exceeding a first preset threshold are used as synonymous extended keywords of the key descriptive words.

[0176] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0177] The detection project name, basic keywords, and synonymous extended keywords are combined into prompt information. The prompt information also includes the project explanation and control instructions for generating diverse natural language descriptions. The prompt information is input into the language model to obtain the output candidate descriptors. The candidate descriptors are filtered, and multiple candidate descriptors pointing to the same meaning are merged by synonyms to obtain supplementary keywords.

[0178] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0179] Compare each word in the word sequence with the keywords in the keyword knowledge base. If they match, the match is confirmed as successful, and the matched words and their corresponding mapping detection items are recorded. If they do not match, a pre-trained Chinese word vector model is used to convert the unmatched words in the word sequence into word vectors. The cosine similarity between the word vectors and the keyword vectors in the keyword knowledge base is calculated. If the cosine similarity exceeds the second preset threshold, the corresponding word is confirmed as a successful match with the keyword.

[0180] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0181] Unstructured sampling data that will be marked as unidentified will be included as part of the historical sampling records to update the keyword knowledge base.

[0182] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0183] Extract basic keywords from the names of the testing items;

[0184] Multiple descriptive texts belonging to the same testing item are identified from the structured and labeled historical sampling records, and synonymous extended keywords are generated based on these multiple descriptive texts;

[0185] Input the detection project name, basic keywords, and synonymous extended keywords into the language model to obtain supplementary keywords;

[0186] The basic keywords, synonymous extended keywords, and supplementary keywords are integrated to obtain a keyword knowledge base that has a mapping relationship with the detection items;

[0187] Obtain the unstructured sampling data to be processed, perform word segmentation on the unstructured sampling data to obtain word sequences, and match the word sequences with the keyword knowledge base;

[0188] Determine whether the detection items mapped to the matched keywords are consistent. If all keywords point to the same detection item, the structured conversion is completed. If all keywords point to multiple different detection items, the unstructured sampling data is marked as requiring manual review. If no keywords are matched, the unstructured sampling data is marked as unidentified.

[0189] In one embodiment, when the computer program is executed by a processor, it further performs the following steps:

[0190] Extract key descriptive words from multiple descriptive texts; calculate the semantic similarity between key descriptive words and preset candidate words; and determine synonymous extended keywords based on the similarity.

[0191] In one embodiment, when the computer program is executed by a processor, it further performs the following steps:

[0192] A pre-trained Chinese word vector model is used to convert key descriptive words into descriptive word vectors; candidate words are obtained from the vocabulary of the pre-trained Chinese word vector model and each candidate word is converted into a candidate word vector; the cosine similarity between the descriptive word vector and the candidate word vector is calculated; and candidate words with a cosine similarity exceeding a first preset threshold are used as synonymous extended keywords of the key descriptive words.

[0193] In one embodiment, when the computer program is executed by a processor, it further performs the following steps:

[0194] The detection project name, basic keywords, and synonymous extended keywords are combined into prompt information. The prompt information also includes the project explanation and control instructions for generating diverse natural language descriptions. The prompt information is input into the language model to obtain the output candidate descriptors. The candidate descriptors are filtered, and multiple candidate descriptors pointing to the same meaning are merged by synonyms to obtain supplementary keywords.

[0195] In one embodiment, when the computer program is executed by a processor, it further performs the following steps:

[0196] Compare each word in the word sequence with the keywords in the keyword knowledge base. If they match, the match is confirmed as successful, and the matched words and their corresponding mapping detection items are recorded. If they do not match, a pre-trained Chinese word vector model is used to convert the unmatched words in the word sequence into word vectors. The cosine similarity between the word vectors and the keyword vectors in the keyword knowledge base is calculated. If the cosine similarity exceeds the second preset threshold, the corresponding word is confirmed as a successful match with the keyword.

[0197] In one embodiment, when the computer program is executed by a processor, it further performs the following steps:

[0198] Unstructured sampling data that will be marked as unidentified will be included as part of the historical sampling records to update the keyword knowledge base.

[0199] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0200] Extract basic keywords from the names of the testing items;

[0201] Multiple descriptive texts belonging to the same testing item are identified from the structured and labeled historical sampling records, and synonymous extended keywords are generated based on these multiple descriptive texts;

[0202] Input the detection project name, basic keywords, and synonymous extended keywords into the language model to obtain supplementary keywords;

[0203] The basic keywords, synonymous extended keywords, and supplementary keywords are integrated to obtain a keyword knowledge base that has a mapping relationship with the detection items;

[0204] Obtain the unstructured sampling data to be processed, perform word segmentation on the unstructured sampling data to obtain word sequences, and match the word sequences with the keyword knowledge base;

[0205] Determine whether the detection items mapped to the matched keywords are consistent. If all keywords point to the same detection item, the structured conversion is completed. If all keywords point to multiple different detection items, the unstructured sampling data is marked as requiring manual review. If no keywords are matched, the unstructured sampling data is marked as unidentified.

[0206] In one embodiment, when the computer program is executed by a processor, it further performs the following steps:

[0207] Extract key descriptive words from multiple descriptive texts; calculate the semantic similarity between key descriptive words and preset candidate words; and determine synonymous extended keywords based on the similarity.

[0208] In one embodiment, when the computer program is executed by a processor, it further performs the following steps:

[0209] A pre-trained Chinese word vector model is used to convert key descriptive words into descriptive word vectors; candidate words are obtained from the vocabulary of the pre-trained Chinese word vector model and each candidate word is converted into a candidate word vector; the cosine similarity between the descriptive word vector and the candidate word vector is calculated; and candidate words with a cosine similarity exceeding a first preset threshold are used as synonymous extended keywords of the key descriptive words.

[0210] In one embodiment, when the computer program is executed by a processor, it further performs the following steps:

[0211] The detection project name, basic keywords, and synonymous extended keywords are combined into prompt information. The prompt information also includes the project explanation and control instructions for generating diverse natural language descriptions. The prompt information is input into the language model to obtain the output candidate descriptors. The candidate descriptors are filtered, and multiple candidate descriptors pointing to the same meaning are merged by synonyms to obtain supplementary keywords.

[0212] In one embodiment, when the computer program is executed by a processor, it further performs the following steps:

[0213] Compare each word in the word sequence with the keywords in the keyword knowledge base. If they match, the match is confirmed as successful, and the matched words and their corresponding mapping detection items are recorded. If they do not match, a pre-trained Chinese word vector model is used to convert the unmatched words in the word sequence into word vectors. The cosine similarity between the word vectors and the keyword vectors in the keyword knowledge base is calculated. If the cosine similarity exceeds the second preset threshold, the corresponding word is confirmed as a successful match with the keyword.

[0214] In one embodiment, when the computer program is executed by a processor, it further performs the following steps:

[0215] Unstructured sampling data that will be marked as unidentified will be included as part of the historical sampling records to update the keyword knowledge base.

[0216] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0217] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0218] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0219] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for processing unstructured sampling inspection data, characterized in that, The method includes: Extract basic keywords from the names of the testing items; Multiple descriptive texts belonging to the same testing item are identified from the structured and labeled historical sampling records, and synonymous extended keywords are generated based on the multiple descriptive texts; Input the detection item name, the basic keywords, and the synonymous extended keywords into the language model to obtain supplementary keywords; The basic keywords, the synonymous extended keywords, and the supplementary keywords are integrated to obtain a keyword knowledge base that has a mapping relationship with the detection items; Obtain unstructured sampling data to be processed, perform word segmentation on the unstructured sampling data to obtain a word sequence, and match the word sequence with the keyword knowledge base; Determine whether the detection items mapped to each matched keyword are consistent. If all keywords point to the same detection item, the structured conversion is completed. If all keywords point to multiple different detection items, the unstructured sampling data is marked as requiring manual review. If no keywords are matched, the unstructured sampling data is marked as unidentified.

2. The method according to claim 1, characterized in that, The generation of synonymous expanded keywords based on multiple descriptive texts includes: Extract key descriptive words from multiple descriptive texts; Calculate the semantic similarity between the key descriptive words and preset candidate words, and determine synonymous extended keywords based on the similarity.

3. The method according to claim 2, characterized in that, The calculation of semantic similarity between the key descriptor and candidate words, and the determination of synonymous extended keywords based on the similarity, includes: The key descriptive words are converted into descriptive word vectors using a pre-trained Chinese word vector model; Obtain candidate words from the vocabulary of the pre-trained Chinese word vector model, and convert each candidate word into a candidate word vector; Calculate the cosine similarity between the descriptive word vector and the candidate word vector; Candidate words with a cosine similarity exceeding a first preset threshold are used as synonymous extended keywords for the key descriptor.

4. The method according to claim 1, characterized in that, The process involves inputting the detection item name, the basic keywords, and the synonymous extended keywords into the language model to obtain supplementary keywords, including: The detection item name, the basic keywords, and the synonymous extended keywords are combined into a prompt message. The prompt message also includes an explanation of the detection item and control instructions for generating diverse natural language descriptions. The prompt information is input into the language model to obtain the output candidate descriptive words; The candidate descriptors are filtered, and multiple candidate descriptors pointing to the same meaning are merged by synonyms to obtain supplementary keywords.

5. The method according to claim 1, characterized in that, The step of matching the word sequence with the keyword knowledge base includes: Compare each word in the word sequence with the keywords in the keyword knowledge base to see if they match. If they match, the match is confirmed to be successful, and the successfully matched words and their corresponding mapping detection items are recorded. If there is a discrepancy, a pre-trained Chinese word vector model is used to convert the unmatched words in the word sequence into word vectors; Calculate the cosine similarity between the word vector and each keyword vector in the keyword knowledge base; if the cosine similarity exceeds a second preset threshold, confirm that the corresponding word and keyword are successfully matched.

6. The method according to claim 1, characterized in that, The method further includes: Unstructured sampling data that will be marked as unidentified will be included as part of the historical sampling records to update the keyword knowledge base.

7. A device for processing unstructured sampling inspection data, characterized in that, The device includes: The acquisition module is used to extract basic keywords from the names of the detected items; The acquisition module is also used to determine multiple descriptive texts belonging to the same detection item from the structured and labeled historical sampling records, and to generate synonymous extended keywords based on the multiple descriptive texts; The acquisition module is also used to input the detection project name, the basic keywords, and the synonymous extended keywords into the language model to obtain supplementary keywords; A construction module is used to integrate the basic keywords, the synonymous extended keywords, and the supplementary keywords to obtain a keyword knowledge base that has a mapping relationship with the detection item; The matching module is used to acquire unstructured sampling data to be processed, perform word segmentation on the unstructured sampling data to obtain a word sequence, and match the word sequence with the keyword knowledge base; The conversion module is used to determine whether the detection items mapped to each matched keyword are consistent. If all keywords point to the same detection item, the structured conversion is completed. If all keywords point to multiple different detection items, the unstructured sampling data is marked as requiring manual review. If no keywords are matched, the unstructured sampling data is marked as unrecognized.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images