Data sample processing method and device based on similarity clustering and similarity retrieval

Abstract

The invention discloses a data sample processing method and device based on similarity clustering and similarity retrieval, and relates to the field of artificial intelligence. The method comprises the following steps: performing similarity clustering on N data samples to obtain X clusters; searching at least one target data sample related to the to-be-analyzed message from the X clusters; adjusting an initial cue word corresponding to the to-be-analyzed message according to the target data sample to obtain a target cue word; and analyzing the to-be-analyzed message through the target prompt word and the large language model. According to the method and the device, the technical problem that the analysis accuracy is relatively low due to the fact that complicated semantics implied in the announcement cannot be effectively processed when the analysis of the navigation announcement message is realized by depending on rule matching or simple pattern recognition in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and more specifically, to a method and device for processing data samples based on similarity clustering and similarity retrieval. Background Art

[0002] As a civil aviation message data sample, NOTAMs play a vital role in shipping, aviation, and other fields. They contain important information related to navigation safety, weather warnings, route changes, and more. NOTAMs are rich in content, diverse in format, and require a certain level of professionalism. To parse NOTAMs, existing technologies typically rely on rule matching or simple pattern recognition. However, this approach cannot effectively handle the complex semantics implicit in NOTAMs, resulting in low parsing accuracy.

[0003] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0004] The embodiments of the present application provide a data sample processing method and device based on similarity clustering and similarity retrieval, so as to at least solve the technical problem in the prior art of relying on rule matching or simple pattern recognition to implement the parsing of navigation notice messages, which is unable to effectively process the complex semantics implicit in the notices, resulting in low parsing accuracy.

[0005] According to one aspect of an embodiment of the present application, a data sample processing method based on similarity clustering and similarity retrieval is provided, comprising: performing similarity clustering on N data samples to obtain X clusters, where N is an integer greater than 1 and X is a positive integer less than N, and each cluster is used to represent a class of data samples having a similarity greater than a preset threshold; retrieving at least one target data sample related to a message to be parsed from the X clusters, where the similarity between the target data sample and the message to be parsed is greater than the similarity between the non-target data sample and the message to be parsed; adjusting an initial prompt word corresponding to the message to be parsed based on the target data sample to obtain a target prompt word; and parsing the message to be parsed using the target prompt word and a large language model.

[0006] Optionally, similarity clustering is performed on N data samples to obtain X clusters, including: parsing each field information of each data sample; preprocessing target field information of each data sample, wherein the target field information is used to describe operation step information, and the preprocessing includes: converting the target field information into plain text, converting abbreviations in the target field information into full names, and deleting preset characters in the target field information; converting the preprocessed target field information for each data sample into a numerical vector corresponding to the data sample; calculating the similarity of the numerical vectors between different data samples, and performing similarity clustering on the N data samples based on the similarity of the numerical vectors to obtain X clusters.

[0007] Optionally, retrieving at least one target data sample related to the message to be parsed from the X cluster groups includes: determining a representative sample from the data samples included in each cluster group, wherein the representative sample of each cluster group is used to represent the feature information of all data samples of the cluster group; determining a target cluster group from the X cluster groups based on the feature similarity between the representative sample of each cluster group and the message to be parsed; and retrieving at least one target data sample related to the message to be parsed from the target cluster group.

[0008] Optionally, at least one target data sample related to the message to be parsed is retrieved from the target cluster group, including: sorting all data samples in the target cluster group according to the feature similarity between each data sample in the target cluster group and the message to be parsed to obtain a sorting result; and determining at least one target data sample from the target cluster group based on the sorting result.

[0009] Optionally, after determining a representative sample from the data samples included in each cluster, the representative samples of the X clusters are copied to a sample library, wherein the sample library includes multiple samples that have completed feature annotation; and based on the feature similarity between each sample in the sample library and the message to be parsed, a target data sample is determined from the sample library.

[0010] Optionally, the initial prompt word corresponding to the message to be parsed is adjusted according to the target data sample to obtain the target prompt word, including: parsing multiple context information in the initial prompt word; determining the target data sample most similar to each context information from at least one target data sample based on the semantic similarity between each target data sample and each context information of the initial prompt word; based on the position of each context information in the initial prompt word, embedding the target data sample most similar to each context information into the initial prompt word to obtain the target prompt word.

[0011] Optionally, the initial prompt word corresponding to the message to be parsed is adjusted according to the target data sample to obtain the target prompt word, including: arranging at least one target data sample in descending order of similarity according to the semantic similarity between each target data sample and the initial prompt word to obtain an arrangement result; inserting each target data sample into the initial prompt word according to the arrangement result to obtain the target prompt word, wherein the target data sample with higher similarity to the initial prompt word is inserted closer to the front of the initial prompt word.

[0012] Optionally, the initial prompt word corresponding to the message to be parsed is adjusted according to the target data sample to obtain the target prompt word, including: splitting the initial prompt word and each target data sample based on the information fusion template to obtain Y sub-prompt words corresponding to the initial prompt word and Y sub-samples corresponding to each target data sample, where Y is an integer greater than 1, and the information fusion template is used to split the text into Y sub-texts according to the preset Y information types, and each sub-text corresponds to one information type; determining the sub-sample most similar to each sub-prompt word from all sub-samples corresponding to all target data samples and splicing them; generating the target prompt word according to the splicing result of the Y sub-prompt words.

[0013] According to another aspect of an embodiment of the present application, a data sample processing device based on similarity clustering and similarity retrieval is further provided, including: a first processing unit, used to perform similarity clustering on N data samples to obtain X clusters, wherein N is an integer greater than 1, and X is a positive integer less than N, and each cluster is used to represent a class of data samples whose similarity is greater than a preset threshold; a retrieval unit, used to retrieve at least one target data sample related to the message to be parsed from the X clusters, wherein the similarity between the target data sample and the message to be parsed is greater than the similarity between the non-target data sample and the message to be parsed; a second processing unit, used to adjust the initial prompt word corresponding to the message to be parsed according to the target data sample to obtain a target prompt word; and a parsing unit, used to parse the message to be parsed using the target prompt word and a large language model.

[0014] According to another aspect of the present application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer-readable storage medium. When the computer program is executed, the device where the computer-readable storage medium is located executes the above-mentioned data sample processing method based on similarity clustering and similarity retrieval.

[0015] According to another aspect of the present application, an electronic device is also provided, wherein the electronic device includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors execute the above-mentioned data sample processing method based on similarity clustering and similarity retrieval.

[0016] As can be seen from the above content, the present application performs similarity clustering on N data samples to obtain X clusters, where N is an integer greater than 1 and X is a positive integer less than N, and each cluster is used to represent a class of data samples whose similarity is greater than a preset threshold; at least one target data sample related to the message to be parsed is retrieved from the X clusters, where the similarity between the target data sample and the message to be parsed is greater than the similarity between the non-target data sample and the message to be parsed; the initial prompt word corresponding to the message to be parsed is adjusted according to the target data sample to obtain a target prompt word; and the message to be parsed is parsed using the target prompt word and a large language model.

[0017] In the embodiment of the present application, similarity clustering and similarity retrieval are adopted to construct a high-quality sample library through cluster analysis, and similarity retrieval is used to select examples, thereby achieving the purpose of optimizing sample selection and model guidance, thereby achieving the technical effect of significantly improving the accuracy and efficiency of navigation notice parsing, and further solving the technical problem of low parsing accuracy caused by relying on rule matching or simple pattern recognition to parse navigation notice messages in the existing technology due to the inability to effectively process the complex semantics implicit in the notices. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 is a flowchart of a data sample processing method based on similarity clustering and similarity retrieval according to an embodiment of the present application;

[0020] Figure 2 is a schematic diagram of an optional similarity clustering algorithm according to an embodiment of the present application;

[0021] Figure 3 This is a schematic diagram of an optional sample learning based on similarity retrieval according to an embodiment of the present application.

[0022] Figure 4 is an architectural diagram of an optional NOTAM message sample selection method according to an embodiment of the present application;

[0023] Figure 5 This is a schematic diagram of an optional data sample processing device based on similarity clustering and similarity retrieval according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] It should also be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) collected by this application are information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data comply with the relevant laws, regulations and standards of the relevant regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse. For example, an interface is set up between this system and relevant users or institutions. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or institution through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or institution.

[0027] According to an embodiment of the present application, a method embodiment of a data sample processing method based on similarity clustering and similarity retrieval is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0028] Optionally, according to an embodiment of the present application, a system is provided as the execution subject of the data sample processing method based on similarity clustering and similarity retrieval in the embodiment of the present application, wherein the system can be a software system or an embedded system combining software and hardware. Of course, the method execution subject in the embodiment of the present application can also be other forms of execution subjects, such as devices, equipment, etc. Those skilled in the art should know that this application does not specifically limit the specific form of expression of the method execution subject.

[0029] Figure 1 FIG. 1 is a flow chart of a data sample processing method based on similarity clustering and similarity retrieval according to an embodiment of the present application. Figure 1 As shown, the method includes the following steps:

[0030] Step S101 : performing similarity clustering on N data samples to obtain X clusters.

[0031] In step S101 , N is an integer greater than 1, X is a positive integer less than N, and each cluster is used to represent a class of data samples whose similarity is greater than a preset threshold.

[0032] Optionally, the system uses a deep learning model to vectorize the message body from a large amount of NOTAM data (i.e., N data samples), converting the text into a numerical vector representation. Subsequently, algorithms such as similarity clustering and spectral clustering are used to perform cluster analysis based on the similarity between vectors, dividing the data samples into X clusters, each containing data samples with similar characteristics and content.

[0033] Optionally, the above process is essentially a deep optimization of the existing sample library. By discovering and summarizing the intrinsic structure of the data set, representative samples are automatically identified and extracted, thereby providing a richer and more diverse data foundation for subsequent sample learning.

[0034] Step S102: Retrieve at least one target data sample related to the message to be parsed from the X clusters.

[0035] In step S102 , the similarity between the target data sample and the message to be parsed is greater than the similarity between the non-target data sample and the message to be parsed.

[0036] Optionally, for a new message to be parsed (e.g., a specific NOTAM that currently needs to be parsed), after the system completes clustering and forms X clusters, it selects one or more data samples that are most relevant to the message from these X clusters as target data samples. This is achieved by calculating the similarity between the message to be parsed and all samples in the cluster, for example, using cosine similarity as a metric, and selecting the samples with the highest similarity as target samples.

[0037] Optionally, the above process can enable the large language model to learn and predict based on the most relevant known samples when parsing a new message, thereby improving the accuracy and efficiency of the parsing.

[0038] Step S103: adjusting the initial prompt word corresponding to the message to be parsed according to the target data sample to obtain the target prompt word.

[0039] Optionally, the system can leverage information from the target data sample to adjust the initial prompts received by the large language model. These initial prompts contain basic instructions and context for the large model. By incorporating the specific content of the target data sample, the initial prompts become richer and more specific, helping the model better understand the unique attributes and context of the message being parsed.

[0040] For example, if the target data sample reveals a specific type of announcement or contains key professional terms, the above information will be integrated into the initial prompt word to form the target prompt word.

[0041] Optionally, this adjustment process is a key step designed to enhance the large model's parsing capabilities for specific tasks, based on the acquisition of target data samples. By finely customizing prompts, the system can guide the large model to accurately capture and understand the core points of NOTAMs, resulting in more efficient and accurate automated parsing.

[0042] Step S104: Parse the message to be parsed using the target prompt word and the large language model.

[0043] Optionally, the system feeds the optimized prompts into a large language model. Rich sample data guides and standardizes the large model's learning, further improving the accuracy of NOTAM parsing. Sample learning for the large model involves learning parsing strategies from provided samples and sample parsing results, conducting supervised learning using a diverse set of NOTAM samples, and optimizing model parameters through backpropagation.

[0044] From the above content, it can be seen that the present application adopts similarity clustering and similarity retrieval methods, constructs a high-quality sample library through cluster analysis, and uses similarity retrieval to select examples, thereby achieving the purpose of optimizing sample selection and model guidance, thereby achieving the technical effect of significantly improving the accuracy and efficiency of navigation notice parsing, and thus solving the technical problem of low parsing accuracy caused by relying on rule matching or simple pattern recognition to realize the parsing of navigation notice messages in the existing technology due to the inability to effectively process the complex semantics implicit in the notice.

[0045] In an optional embodiment, similarity clustering is performed on N data samples to obtain X clusters, including: parsing each field information of each data sample; preprocessing target field information of each data sample, wherein the target field information is used to describe operation step information, and the preprocessing includes: converting the target field information into plain text, converting abbreviations in the target field information into full names, and deleting preset characters in the target field information; converting the preprocessed target field information for each data sample into a numerical vector corresponding to the data sample; calculating the similarity of the numerical vectors between different data samples, and performing similarity clustering on the N data samples based on the numerical vector similarity to obtain X clusters.

[0046] Optionally, Figure 2 is a schematic diagram of an optional similarity clustering algorithm according to an embodiment of the present application, such as Figure 2 As shown in the figure, the system first cleans the NOTAM data, extracting the target field information of each sample from the NOTAM dataset, such as Item E (operation details) in the NOTAM. Next, data preprocessing is performed, including noise removal, format standardization, conversion of telegraph codes to plain text, conversion of abbreviations to full names, and removal of numerical information to ensure readability and consistency of the text.

[0047] The system then vectorizes the preprocessed text using the BGE-m3 model, converting the natural language text into a numerical vector representation. All vectorization results are stored securely in the database. This step maps the text information into a high-dimensional vector space, laying the foundation for subsequent similarity calculations and cluster analysis.

[0048] After obtaining the vector representations of all samples, the system constructs a similarity matrix. This matrix quantifies the similarity between different NOTAM samples by calculating the cosine similarity between any two vectors. Cosine similarity measures text similarity by comparing the angular difference between vectors rather than their lengths. It is a method for evaluating text similarity in high-dimensional space.

[0049] Specifically, for any two message vectors v i and v j , its similarity S ij It can be calculated by the following formula:

[0050]

[0051] Subsequently, clustering algorithms such as spectral clustering were used to perform cluster analysis on the similarity matrix. These algorithms can automatically identify the internal structure of the dataset and form several clusters, each representing a class of highly similar NOTAM samples, thereby effectively expanding the sample library. The system selects the item in each cluster that is most similar to the other vectors as the representative sample.

[0052] Finally, the system will manually annotate the representative samples identified by clustering, and through expert review or the use of an annotation platform, the samples will be annotated in detail, including analysis results, key information, etc., to further improve the sample library, ensure its high quality and professionalism, and provide more accurate guidance information for subsequent large-scale model learning.

[0053] Optionally, Figure 3 is a schematic diagram of an optional sample learning based on similarity retrieval according to an embodiment of the present application, such as Figure 3 As shown in the figure, first, the system selects a NOTAM message to be parsed from the constructed and optimized sample library. This message may be a brand new, unseen notice, or a complex notice that requires further processing and understanding.

[0054] Next, the system can calculate the similarity between the message to be parsed and all historical messages in the sample library based on cosine similarity. By comparing the angular difference between each historical message vector and the message vector to be parsed, the system can obtain a series of similarity values.

[0055] After determining the similarity between all historical messages and the message to be parsed, the system selects the top three most similar samples as target data samples. Selecting the most relevant samples provides the most direct and relevant contextual information to the large model, thereby better guiding the model to understand and parse the message to be processed.

[0056] Based on these top three target data samples, the system then adjusted and reconstructed the initial prompts to more accurately reflect the message context and requirements. These adjusted prompts, which now include more detailed background information and examples, effectively assist the large-scale model in identifying and parsing the unique patterns and details of NOTAMs, ensuring targeted and effective model learning.

[0057] Finally, the system combines the adjusted target prompts with the large language model. The large model then learns from samples, performs a deep analysis of the message being parsed, and generates a notification parsing result. By receiving rich and relevant contextual information, the large model can more accurately and quickly understand the message content, thereby improving parsing efficiency and accuracy, meeting the high-precision requirements of practical applications.

[0058] Optionally, the above process emphasizes the use of similar samples in historical data, and the system optimizes the ability of large models to parse unknown and complex navigation notices through precise similarity retrieval and context learning.

[0059] In an optional embodiment, retrieving at least one target data sample related to the message to be parsed from X cluster groups includes: determining a representative sample from the data samples included in each cluster group, wherein the representative sample of each cluster group is used to represent the feature information of all data samples of the cluster group; determining a target cluster group from the X cluster groups based on the feature similarity between the representative sample of each cluster group and the message to be parsed; and retrieving at least one target data sample related to the message to be parsed from the target cluster group.

[0060] Optionally, within the X clusters derived using similarity clustering, the system first selects a representative sample from each cluster. This representative sample is a concentrated representation of the characteristics of the data samples within the cluster and represents the common characteristics of all samples within the cluster. Selecting a representative sample is a natural extension of cluster analysis, ensuring that each cluster has a single, representative sample, reducing the number of samples required for subsequent processing and improving efficiency.

[0061] Next, the system calculates the feature similarity between each cluster's representative sample and the message to be parsed. This calculation quantifies the correlation between the representative sample and the message to be parsed. Based on the calculated feature similarity, the system identifies the target cluster from the X clusters that is most relevant to the message to be parsed. This step compares the similarity between the representative samples of different clusters and the message to be parsed, identifying the cluster with the highest match and thus determining the source of samples for subsequent learning.

[0062] Finally, at least one target data sample related to the message to be parsed is retrieved from the identified target cluster. This step, based on retrieval technology, is not limited to representative samples but may also select more data samples from the target cluster that have similar characteristics to the message to be parsed, providing a richer set of learning examples for the large model.

[0063] In an optional embodiment, at least one target data sample related to the message to be parsed is retrieved from the target cluster group, including: sorting all data samples in the target cluster group according to the feature similarity between each data sample in the target cluster group and the message to be parsed to obtain a sorting result; and determining at least one target data sample from the target cluster group based on the sorting result.

[0064] Optionally, after determining the cluster most relevant to the message being parsed, the system can calculate the feature similarity between each data sample within the target cluster and the message being parsed. Feature similarity is measured by calculating the cosine similarity between the vector representations of different messages. This measure reflects the angular difference between two vectors in high-dimensional space. The smaller the angle, i.e., the closer the cosine value is to 1, the more similar the two samples are.

[0065] The system then sorts all the data samples in the target cluster based on the calculated feature similarity, generating a ranking result. The ranking process aims to identify the samples in the cluster that are most similar to the message to be parsed, providing the most relevant guidance for subsequent large-scale model learning.

[0066] Finally, based on the sorting results, the system identifies at least one data sample from the target cluster that is highly relevant to the message being parsed as the target data sample. Typically, the top few samples in the sorting results are selected to ensure their characteristics are most similar to the message being parsed. These samples are then fed into the larger model as examples, effectively guiding the model's parsing and learning process.

[0067] In an optional embodiment, after determining a representative sample from the data samples included in each cluster, the method further includes: copying the representative samples of the X clusters into a sample library, wherein the sample library includes a plurality of samples for which feature annotation has been completed; and determining a target data sample from the sample library based on a feature similarity between each sample in the sample library and the message to be parsed.

[0068] Optionally, after similarity clustering is completed and representative samples are selected from each cluster, they are copied and added to a dedicated sample library. This sample library already contains a large number of feature-labeled samples. These samples have been carefully selected and processed through expert manual annotation or other methods, and have high-quality and diverse feature information.

[0069] Copying representative samples to the sample library is essentially a dynamic update and optimization of the sample library. This not only expands the coverage of the sample library, making the sample types in the sample library more diverse and comprehensive, but also ensures the high quality of the sample library.

[0070] Next, the system again uses feature similarity calculations to identify at least one target data sample from the sample library that is relevant to the message being parsed. This time, the search targets the entire sample library, rather than a specific cluster, including samples that have been feature-labeled, as well as newly added representative samples. By calculating the feature similarity between each sample in the sample library and the message being parsed, the system can identify the most relevant samples, which serve as input for subsequent large-scale model learning, further improving the model's parsing capabilities and accuracy for specific tasks.

[0071] Optionally, the above process integrates representative samples into the sample library and retrieves them from the entire sample library, which not only fully utilizes the typical characteristics of the representative samples, but also fully taps the potential of all samples in the sample library, providing richer and more comprehensive learning examples for the large model, thereby improving the parsing efficiency while ensuring the high quality and high credibility of the parsing results.

[0072] In an optional embodiment, the initial prompt word corresponding to the message to be parsed is adjusted according to the target data sample to obtain the target prompt word, including: parsing multiple context information in the initial prompt word; determining the target data sample most similar to each context information from at least one target data sample based on the semantic similarity between each target data sample and each context information of the initial prompt word; based on the position of each context information in the initial prompt word, embedding the target data sample most similar to each context information into the initial prompt word to obtain the target prompt word.

[0073] Optionally, after determining at least one target data sample related to the message to be parsed through similarity retrieval, the system will first parse multiple contextual information in the initial prompt words. These contextual information may include the definition of the role of the large model, the type of message, the key information to be parsed, etc., which constitute the context environment for the large model to be parsed.

[0074] Next, the system can calculate the semantic similarity between each target data sample and each contextual information in the initial prompt word based on the semantic similarity of text understanding. It usually uses a pre-trained deep learning model, such as BERT or its variants, to evaluate the degree of semantic relevance between the target data sample and the contextual information. Through this calculation, the system can identify the target data sample that is most similar to a specific contextual information, ensuring that each contextual information is reinforced by the most relevant and representative samples.

[0075] Then, based on the position of each piece of contextual information within the initial prompt, the system accurately embeds the most similar target data sample into the corresponding position within the initial prompt. This not only maintains the structural and logical integrity of the prompt, but also, through the inclusion of the target data sample, provides targeted contextual reinforcement for each part of the prompt, ensuring that the large model accurately understands the meaning and requirements of each piece of contextual information during the learning and parsing process.

[0076] Ultimately, through the above adjustments, the system can obtain a target prompt word, which contains optimized contextual information and specific examples, and can more effectively guide the large model for deep learning and parsing, thereby improving the accuracy and reliability of the parsing results.

[0077] In an optional embodiment, the initial prompt word corresponding to the message to be parsed is adjusted according to the target data sample to obtain the target prompt word, including: arranging at least one target data sample in descending order of similarity according to the semantic similarity between each target data sample and the initial prompt word to obtain an arrangement result; inserting each target data sample into the initial prompt word according to the arrangement result to obtain the target prompt word, wherein the target data sample with higher similarity to the initial prompt word is inserted closer to the front of the initial prompt word.

[0078] Optionally, based on the calculated semantic similarity, the system can sort all target data samples in descending order, with samples with higher similarity placed first and samples with lower similarity placed last. This sorting process generates a sorting result that reflects the degree of semantic fit of each sample with the initial prompt word, providing a basis for subsequent sample embedding.

[0079] The system then inserts each target data sample into the initial prompt word based on the ranking results, in descending order of similarity, to form the target prompt word. The insertion logic is that data samples with higher similarity to the initial prompt word are inserted more preferentially and at an earlier position. This ensures that the contextual information received by the large model during parsing is the most instructive and representative set of samples, thereby improving the model's learning efficiency and parsing accuracy.

[0080] In an optional embodiment, the initial prompt word corresponding to the message to be parsed is adjusted according to the target data sample to obtain the target prompt word, including: splitting the initial prompt word and each target data sample based on an information fusion template to obtain Y sub-prompt words corresponding to the initial prompt word and Y sub-samples corresponding to each target data sample, wherein Y is an integer greater than 1, and the information fusion template is used to split the text into Y sub-texts according to Y preset information types, and each sub-text corresponds to one information type; determining the sub-sample most similar to each sub-prompt word from all sub-samples corresponding to all target data samples and splicing them; and generating the target prompt word according to the splicing result of the Y sub-prompt words.

[0081] Optionally, the system can first split the initial prompt word and each target data sample based on a preset information fusion template. This information fusion template defines Y information types. Based on the specific requirements of the NOTAM parsing task, it splits the text into Y subtexts according to different information dimensions (such as time information, location information, operational details, etc.), each corresponding to a specific information type. This aims to decompose complex information structures into smaller, more manageable components, facilitating subsequent similarity calculation and information fusion.

[0082] Next, the system splits the initial prompt into Y sub-prompts, each of which corresponds to a specific information type in the initial prompt. Simultaneously, each target data sample is also split into Y sub-samples, each focusing on a specific information type in the sample.

[0083] The system can then identify the subsample that is most similar to each subsample of the initial prompt word by calculating the semantic similarity between each subsample of the initial prompt word and the corresponding subsample of all target data samples, that is, the subsample that best matches the subsample in terms of specific information type.

[0084] Optionally, after determining the most similar subsamples, the system concatenates them with the subprompt words one by one in the order of the subprompt words to generate a new text sequence. This process essentially precisely matches the most representative information fragments in the target data sample with the information requirements of the initial prompt word, and through information fusion, constructs a more comprehensive and targeted target prompt word.

[0085] Ultimately, the system concatenates all sub-cues with the concatenation of the most similar sub-samples to generate a complete target cue. This target cue not only retains the basic framework and task description of the initial cue, but also incorporates the most instructive information from the target data sample, providing a richer and more refined context for deep learning in large models.

[0086] Optionally, Figure 4 is an architectural diagram of an optional NOTAM message sample selection method according to an embodiment of the present application, such as Figure 4 As shown, the system first collects aeronautical notice data and performs preliminary cleaning and annotation to build a preliminary sample library. This includes noise removal, format standardization, conversion of telegraph codes to plain text, replacement of all abbreviations with their full names, and removal of numeric information from target fields. The goal is to reduce interference factors in the data and improve the accuracy and efficiency of the parsing process.

[0087] The system vectorizes the processed message body (target field information) using the BGE-m3 model. It then uses a similarity clustering algorithm to cluster the data in the existing sample library and identify representative samples. By calculating the similarity between samples, the clustering method can automatically discover the underlying structure in the data and generate a diverse sample set.

[0088] Subsequently, the system manually labels the representative samples obtained through clustering to further improve the data in the sample library, ensuring that each sample has accurate feature information and context, thereby enhancing the training effect of the model.

[0089] In the expanded sample library, the system can use similarity retrieval technology to quickly locate the samples most relevant to the message being parsed. The retrieval results are then used for prompt word optimization and model learning. The system combines the retrieved target data samples with the initial prompt words and optimizes them using specific algorithms (such as information fusion templates and contextual information matching) to more accurately guide the large model's learning and parsing.

[0090] Next, the optimized prompt words are input into the large language model, which performs deep learning based on this information to improve its understanding and execution capabilities of the navigation notice parsing task.

[0091] After deep learning and analysis of the large model, the system generates analysis results, which means that the new navigation notice has been processed automatically and efficiently and accurately, completing the closed loop of the entire process.

[0092] According to another aspect of the present application, a data sample processing device based on similarity clustering and similarity retrieval is also provided, wherein: Figure 5 is a schematic diagram of an optional data sample processing device based on similarity clustering and similarity retrieval according to an embodiment of the present application, such as Figure 5 As shown, the data sample processing device includes: a first processing unit 501 , a retrieval unit 502 , a second processing unit 503 , and a parsing unit 504 .

[0093] Optionally, the first processing unit 501 is used to perform similarity clustering on N data samples to obtain X clusters, where N is an integer greater than 1 and X is a positive integer less than N, and each cluster is used to represent a class of data samples whose similarity is greater than a preset threshold; the retrieval unit 502 is used to retrieve at least one target data sample related to the message to be parsed from the X clusters, where the similarity between the target data sample and the message to be parsed is greater than the similarity between the non-target data sample and the message to be parsed; the second processing unit 503 is used to adjust the initial prompt word corresponding to the message to be parsed based on the target data sample to obtain a target prompt word; the parsing unit 504 is used to parse the message to be parsed using the target prompt word and the large language model.

[0094] Optionally, the first processing unit includes: a first parsing subunit, a first processing subunit, a conversion subunit, and a second processing subunit. The first parsing subunit is used to parse each field information of each data sample; the first processing subunit is used to preprocess the target field information of each data sample, wherein the target field information is used to describe operation step information, and the preprocessing includes: converting the target field information into plain text, converting abbreviations in the target field information into full names, and deleting preset characters in the target field information; the conversion subunit is used to convert the preprocessed target field information for each data sample into a numerical vector corresponding to the data sample; and the second processing subunit is used to calculate the numerical vector similarity between different data samples, and perform similarity clustering on N data samples based on the numerical vector similarity to obtain X clusters.

[0095] Optionally, the retrieval unit includes: a first determining subunit, a second determining subunit, and a retrieval subunit. The first determining subunit is configured to determine a representative sample from the data samples included in each cluster, wherein the representative sample of each cluster is used to represent the feature information of all data samples in the cluster; the second determining subunit is configured to determine a target cluster from the X clusters based on the feature similarity between the representative sample of each cluster and the message to be parsed; and the retrieval subunit is configured to retrieve at least one target data sample related to the message to be parsed from the target cluster.

[0096] Optionally, the retrieval subunit includes: a processing module and a determination module. The processing module is configured to sort all data samples in the target cluster according to the feature similarity between each data sample in the target cluster and the message to be parsed, thereby obtaining a sorting result; and the determination module is configured to determine at least one target data sample from the target cluster based on the sorting result.

[0097] Optionally, the retrieval unit further includes: a third processing subunit and a third determination subunit. The third processing subunit is configured to copy representative samples of the X clusters into a sample library, wherein the sample library includes a plurality of samples that have been feature-labeled; and the third determination subunit is configured to determine a target data sample from the sample library based on a feature similarity between each sample in the sample library and the message to be parsed.

[0098] Optionally, the second processing unit includes: a second parsing subunit, a fourth determination subunit, and a fourth processing subunit. The second parsing subunit is configured to parse multiple contextual information in the initial prompt word; the fourth determination subunit determines, based on the semantic similarity between each target data sample and each contextual information of the initial prompt word, a target data sample that is most similar to each contextual information from at least one target data sample; and the fourth processing subunit is configured to embed the target data sample that is most similar to each contextual information into the initial prompt word based on the position of each contextual information in the initial prompt word, thereby obtaining the target prompt word.

[0099] Optionally, the second processing unit further includes: a fifth processing subunit and a sixth processing subunit. The fifth processing subunit is configured to sort at least one target data sample in descending order of similarity based on the semantic similarity between each target data sample and the initial prompt word to obtain a sorting result; and the sixth processing subunit is configured to insert each target data sample into the initial prompt word based on the sorting result to obtain a target prompt word, wherein the target data sample with higher similarity to the initial prompt word is inserted closer to the front of the initial prompt word.

[0100] Optionally, the second processing unit further includes: a seventh processing subunit, a fifth determination subunit, and a generation subunit. The seventh processing subunit is configured to split the initial prompt word and each target data sample based on the information fusion template to obtain Y sub-prompt words corresponding to the initial prompt word and Y subsamples corresponding to each target data sample, where Y is an integer greater than 1, and the information fusion template is configured to split the text into Y subtexts according to Y preset information types, each subtext corresponding to one information type; the fifth determination subunit is configured to determine the subsample most similar to each sub-prompt word from all subsamples corresponding to all target data samples and to concatenate the subsamples; and the generation subunit is configured to generate the target prompt word based on the concatenation result of the Y sub-prompt words.

[0101] According to another aspect of the present application, a computer-readable storage medium is provided, including: a computer program stored in the computer-readable storage medium, wherein when the computer program runs, the device where the computer-readable storage medium is located executes the above-mentioned data sample processing method.

[0102] According to another aspect of the present application, an electronic device is also provided, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors execute the above-mentioned data sample processing method.

[0103] The above-mentioned embodiments or examples disclosed in this application are not exhaustive, but are only illustrations of some embodiments or examples, and are not intended to be specific limitations on the scope of protection disclosed in this application. In the absence of contradiction, each step in a certain embodiment or example in this application can be implemented as an independent example, and the steps can be arbitrarily combined. For example, the solution after removing some steps in a certain embodiment or example can also be implemented as an independent example, and the order of the steps in a certain embodiment or example can be arbitrarily exchanged. In addition, the optional methods or optional examples in a certain embodiment or example can be arbitrarily combined; in addition, the various embodiments or examples can be arbitrarily combined. For example, some or all of the steps in different embodiments or examples can be arbitrarily combined, and a certain embodiment or example can be arbitrarily combined with the optional methods or optional examples of other embodiments or examples.

[0104] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0105] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0106] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0107] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected to achieve the purpose of the present embodiment according to actual needs.

[0108] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0109] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program code.

[0110] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data sample processing method based on similarity clustering and similarity retrieval, characterized in that: include: Perform similarity clustering on N data samples to obtain X clusters, where N is an integer greater than 1 and X is a positive integer less than N. Each cluster is used to represent a class of data samples whose similarity is greater than a preset threshold; Retrieving at least one target data sample related to the message to be parsed from the X clusters, wherein the similarity between the target data sample and the message to be parsed is greater than the similarity between the non-target data samples and the message to be parsed; Adjusting the initial prompt word corresponding to the message to be parsed according to the target data sample to obtain a target prompt word; The message to be parsed is parsed using the target prompt word and the large language model.

2. The method according to claim 1, characterized in that Perform similarity clustering on N data samples and obtain X clusters, including: Parse each field information of each data sample; Preprocessing the target field information of each data sample, wherein the target field information is used to describe operation step information, the preprocessing comprising: converting the target field information into plain text, converting abbreviations in the target field information into full names, and deleting preset characters in the target field information; Converting the preprocessed target field information of each data sample into a numerical vector corresponding to the data sample; Calculate the numerical vector similarities between different data samples, and perform similarity clustering on the N data samples according to the numerical vector similarities to obtain X clusters.

3. The method according to claim 1, characterized in that Retrieving at least one target data sample related to the message to be parsed from the X clusters, including: Determining a representative sample from the data samples included in each cluster, wherein the representative sample of each cluster is used to represent feature information of all data samples of the cluster; Determining a target cluster from the X clusters based on feature similarity between a representative sample of each cluster and the message to be parsed; At least one target data sample related to the message to be parsed is retrieved from the target cluster.

4. The method according to claim 3, characterized in that Retrieving at least one target data sample related to the message to be parsed from the target cluster, comprising: Sorting all data samples in the target cluster according to the feature similarity between each data sample in the target cluster and the message to be parsed to obtain a sorting result; At least one target data sample is determined from the target cluster according to the sorting result.

5. The method according to claim 3, characterized in that After determining a representative sample from the data samples included in each cluster, the method further includes: Copying representative samples of the X clusters into a sample library, wherein the sample library includes a plurality of samples that have completed feature annotation; The target data sample is determined from the sample library according to the feature similarity between each sample in the sample library and the message to be parsed.

6. The method according to claim 1, characterized in that Adjusting the initial prompt word corresponding to the message to be parsed according to the target data sample to obtain the target prompt word includes: parsing multiple context information in the initial prompt word; Determining, from the at least one target data sample, a target data sample that is most similar to each context information according to the semantic similarity between each target data sample and each context information of the initial prompt word; Based on the position of each piece of context information in the initial prompt word, a target data sample that is most similar to each piece of context information is embedded into the initial prompt word to obtain the target prompt word.

7. The method according to claim 1, characterized in that Adjusting the initial prompt word corresponding to the message to be parsed according to the target data sample to obtain the target prompt word includes: Arrange the at least one target data sample in descending order of similarity according to the semantic similarity between each target data sample and the initial prompt word to obtain an arrangement result; Each target data sample is inserted into the initial prompt word according to the arrangement result to obtain the target prompt word, wherein the target data sample with a higher similarity to the initial prompt word is inserted closer to the front of the initial prompt word.

8. The method according to claim 1, characterized in that Adjusting the initial prompt word corresponding to the message to be parsed according to the target data sample to obtain the target prompt word includes: Splitting the initial prompt word and each target data sample based on an information fusion template to obtain Y sub-prompt words corresponding to the initial prompt word and Y sub-samples corresponding to each target data sample, where Y is an integer greater than 1, and the information fusion template is used to split the text into Y sub-texts according to Y preset information types, each sub-text corresponding to one information type; Determine the subsample that is most similar to each sub-prompt word from all subsamples corresponding to all target data samples and concatenate them; The target prompt word is generated according to the splicing result of the Y sub-prompt words.

9. A data sample processing device based on similarity clustering and similarity retrieval, characterized in that: include: A first processing unit is configured to perform similarity clustering on N data samples to obtain X clusters, where N is an integer greater than 1 and X is a positive integer less than N, and each cluster is used to represent a class of data samples whose similarity is greater than a preset threshold; a retrieval unit, configured to retrieve at least one target data sample related to the message to be parsed from the X clusters, wherein a similarity between the target data sample and the message to be parsed is greater than a similarity between a non-target data sample and the message to be parsed; A second processing unit is configured to adjust the initial prompt word corresponding to the message to be parsed according to the target data sample to obtain a target prompt word; The parsing unit is used to parse the message to be parsed using the target prompt word and the large language model.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located executes the data sample processing method based on similarity clustering and similarity retrieval according to any one of claims 1 to 8.

11. An electronic device, characterized in that: The method comprises one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the data sample processing method based on similarity clustering and similarity retrieval as described in any one of claims 1 to 8.

Images