Method and System for Eliminating Redundancy in Training Data of Large Models Based on Semantic Recognition

Abstract

The present invention relates to the technical field of natural language processing. The present invention discloses a method and system for removing redundant training data of a large model based on semantic recognition, including accurately identifying and removing redundant parts in the data by constructing a domain knowledge graph, and combining advanced technologies such as graph neural networks to improve the quality and efficiency of data preprocessing. Compared with the prior art, by adopting a multi-level data encoding and feature propagation mechanism, the present invention not only effectively eliminates redundant information in the data, but also promotes the high-dimensional representation of the data, enabling the data to have higher interpretability and accuracy before entering the model training stage.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and particularly to a method and system for removing redundant training data of a large model based on semantic recognition. Background Art

[0002] In the context of the rapid development of information technology today, artificial intelligence (AI) and machine learning (ML) have become important forces driving innovation in various industries. Especially in the field of natural language processing (NLP), large models based on semantic recognition have gradually emerged and become powerful tools for understanding human language, extracting semantic information, and generating natural language text. In recent years, with the progress of deep learning technology, researchers have made significant progress in the construction and application of knowledge graphs. By representing entities and relationships, knowledge graphs can effectively capture and organize domain knowledge, providing strong semantic support for downstream tasks. At the same time, during the training process of large models, the quality and diversity of data have a more obvious impact on model performance. Therefore, how to effectively remove redundant data and improve data quality has become a research hotspot.

[0003] Although current technologies have achieved certain achievements in semantic recognition and data cleaning, there are still many deficiencies. Existing data preprocessing methods mostly focus on simple duplicate data deletion or cleaning based on specific rules, lacking a deep understanding of the internal semantic structure of data. This results in the possible omission of some redundant information caused by semantic similarity during the process of removing redundant data, thereby affecting the generalization ability of the model. In addition, when faced with large-scale and diverse data sets, traditional manual annotation and review mechanisms are inefficient and prone to introducing human errors, reducing data quality. Therefore, there is an urgent need for a more efficient method for removing redundant data that can comprehensively analyze and process large-scale data by combining advanced semantic recognition and knowledge graph technologies. Summary of the Invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a method for removing redundant training data of a large model based on semantic recognition, which has significant advantages in aspects such as enriching semantic understanding, improving data quality, and optimizing model training efficiency, effectively solving the problems of data redundancy and unqualified quality existing in traditional methods.

[0006] To solve the above technical problems, the present invention provides the following technical solutions. A method for eliminating redundant training data of a large model based on semantic recognition includes: constructing a domain knowledge graph, performing semantic association mining based on the domain knowledge graph, and identifying redundant parts of the data in the domain knowledge graph; eliminating irrelevant data in the data preprocessing stage of constructing the domain knowledge graph by combining semantic recognition technology; after identifying the redundant parts of the data, removing the redundant data through a data cleaning algorithm; and optimizing the data governance process in combination with data quality assessment.

[0007] As a preferred solution of the method for eliminating redundant training data of a large model based on semantic recognition according to the present invention, wherein: the construction of the domain knowledge graph includes collecting domain professional literature and large-scale data sets; performing vector representation on the content of the domain professional literature in combination with a domain vocabulary; extracting the relationships between the vectorized representation contents by using a graph attention network model on the basis of content recognition;

[0008] Running a GNN-based entity alignment algorithm, through the review and adjustment of the alignment results, using a meta-path-based similarity measurement method to verify and optimize the accuracy of entity alignment, implementing a multi-modal information fusion technology, integrating and analyzing relationship information from different sources, integrating the aligned entities and the fused relationships into a fine-grained knowledge graph, and performing periodic updates and maintenance.

[0009] As a preferred solution of the method for eliminating redundant training data of a large model based on semantic recognition according to the present invention, wherein: the semantic association mining includes using a graph neural network to analyze multi-hop relationships between entities; extracting entities and relationships from the data of the domain knowledge graph to generate the structure of graph G, including nodes V and edges E,

[0010] Selecting a graph neural network architecture and initializing the feature vector of each node , obtaining the initial feature matrix of all nodes , and updating the node feature vectors through multiple rounds of propagation:

[0011] ;

[0012] wherein, is the feature vector of node i after the k-th round of transmission, is the feature vector of node i after the (k - 1)-th round of transmission, is the set of neighbor nodes of node i, is an aggregation function that integrates the features of neighbor node j of node i together, is a function for updating the feature of node i, is the feature vector of neighbor node j of node i after the (k - 1)-th round of transmission;

[0013] Through multiple rounds of propagation updates, the high-dimensional feature vectors of the nodes are finally obtained. The high-dimensional feature vector of each node contains information of multi-hop neighbors.

[0014] As a preferred solution of the training data redundancy elimination method for the large model based on semantic recognition according to the present invention, wherein: the semantic association mining further includes identifying redundant instances in the data by calculating semantic similarity. For any two entities and , calculate their similarity, and the similarity is defined as:

[0015] ;

[0016] Wherein, is the similarity score between entities and , is the feature vector of neighbor node j of node i after the k-th round of transmission. Set the similarity threshold . When , it is considered that and are redundant.

[0017] As a preferred solution of the training data redundancy elimination method for the large model based on semantic recognition according to the present invention, wherein: the elimination of irrelevant data includes unifying the formats of the original data obtained from different data sources, removing obvious errors and missing values, and storing them in a database;

[0018] Adopt a multi-level coding mechanism to encode the original data into three levels:

[0019] Extract the basic features of the original data to generate basic vectors; use a pre-trained deep learning model to perform context semantic encoding on the text or image in the original data and convert it into high-dimensional semantic vectors; according to the knowledge graph in the target domain, calculate the similarity between high-dimensional semantic vectors to generate weighted semantic feature vectors, and map each data point in the original data to a feature space with higher interpretability;

[0020] When the original data enters the encoding stage, it will be matched with the knowledge graph to generate high-level semantic features;

[0021] Perform neighborhood search on each data point. If the number of points in the neighborhood of the current data point including itself is greater than the density threshold, it is regarded as a core point; otherwise, it is regarded as an edge point; classify the core points and their neighbor points into the same cluster. If one core point can reach another core point, the points adjacent to these two core points will also be merged into the same cluster;

[0022] Mark all edge points and unclassified points as outliers, that is, points that cannot be classified into any cluster formed by core points, and mark data points outside the specified density and size groups;

[0023] Randomly sample the automatically marked data points proportionally or audit them according to the number of outliers. After the audit is completed, enter the modification results and update the marking database.

[0024] As a preferred solution of the training data redundancy elimination method of the large model based on semantic recognition described in the present invention, wherein: the removal of redundant data includes generating an identifier for each data point, marking the data point as "normal" or "redundant", and creating a marking set M,

[0025] ;

[0026] Wherein, is the marked data point, represents the redundant data set. According to the marking set M, an operation for removing redundant data is defined:

[0027] ;

[0028] Here, is the data point marked as redundant, is the data set after removing redundant data, represents the difference operation of sets, indicating removing all data points marked as redundant from D.

[0029] As a preferred solution of the training data redundancy elimination method of the large model based on semantic recognition described in the present invention, wherein: the combination of data quality assessment includes performing a quality assessment on the data set after removing redundant data and defining a quality assessment function Q to measure the quality of the data retained in the data set:

[0030] ;

[0031] Wherein, is the weight of the marked data point and is dynamically adjusted by selecting relevance according to features:

[0032] ;

[0033] Wherein, is the mean value of the features corresponding to the current data set, used to reflect the overall distribution of all data points, so that the feature weights are adjusted in real time according to the actual data distribution, is the feature vector.

[0034] As a preferred solution of the training data redundancy elimination system for the large model based on semantic recognition of the present invention, it includes: a knowledge graph construction module, a semantic association mining module, a data preprocessing module, a redundant data identification and elimination module, and a data quality evaluation and optimization module;

[0035] The knowledge graph construction module is responsible for constructing a domain knowledge graph to provide basic data and semantic references for subsequent semantic analysis and association mining;

[0036] The semantic association mining module uses the constructed knowledge graph to analyze the relationships between entities in the data and mine semantic information;

[0037] The data preprocessing module cleans and formats the original data to ensure the quality and consistency of the data and provide reliable input for subsequent analysis;

[0038] The redundant data identification and elimination module identifies and eliminates redundant data to improve the conciseness and effectiveness of the data set;

[0039] The data quality evaluation and optimization module, after eliminating redundant data, evaluates the quality of the data set to ensure the integrity and accuracy of the data, and at the same time optimizes the data governance process.

[0040] Advantages of the present invention: By constructing a domain knowledge graph, combining semantic association mining, implementing data cleaning and quality evaluation, etc., this method effectively optimizes the data governance process. This technical solution can improve the quality and representativeness of the data set while identifying and eliminating redundant data, thereby improving the training efficiency and accuracy of the deep learning model. In addition, the present invention effectively integrates the information of multiple data sources, uses graph neural network technology to strengthen the mining of semantic relationships, and ensures that the model exerts its maximum potential when processing complex data. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings. Among them:

[0042] Figure 1 It is a schematic flowchart of the training data redundancy elimination method for the large model based on semantic recognition provided by an embodiment of the present invention.

[0043] Figure 2 It is a schematic diagram of the working modules of the training data redundancy elimination system for the large model based on semantic recognition provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0044] To make the above objects, features, and advantages of the present invention more obvious and understandable, the following will describe in detail the specific embodiments of the present invention in conjunction with the accompanying drawings of the specification. Obviously, the described examples are part of the embodiments of the present invention, rather than all embodiments. Based on the embodiments in the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0045] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can also be implemented in other ways different from those described herein. Those skilled in the art can make similar generalizations without departing from the connotation of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below.

[0046] Secondly, the so-called "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that can be included in at least one implementation manner of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it an embodiment that is separate or selectively mutually exclusive with other embodiments.

[0047] The present invention is described in detail in conjunction with schematic diagrams. When detailing the embodiments of the present invention, for the convenience of explanation, the cross-sectional views showing the device structure will be enlarged locally in a non-general proportion, and the schematic diagrams are only examples and should not limit the protection scope of the present invention herein. In addition, in actual production, three-dimensional spatial dimensions including length, width, and depth should be included.

[0048] At the same time, in the description of the present invention, it should be noted that the orientation or positional relationships indicated by terms such as "upper, lower, inner, and outer" are based on the orientation or positional relationships shown in the drawings. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation to the present invention. In addition, the terms "first, second, or third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0049] Unless otherwise clearly defined and limited in the present invention, the terms "mounted, connected, and coupled" shall be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can also be a mechanical connection, an electrical connection, or a direct connection, or can be indirectly connected through an intermediate medium, or can be the communication inside two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0050] Example 1, referring to Figure 1, which is the first embodiment of the present invention. This embodiment provides a method for eliminating redundant training data of a large model based on semantic recognition, including:

[0051] S1: Construct a domain knowledge graph, perform semantic association mining based on the domain knowledge graph, and identify redundant parts of the data in the domain knowledge graph.

[0052] Furthermore, the construction of the domain knowledge graph includes collecting domain professional literature and large-scale data sets; vectorizing the content of the domain professional literature in combination with the domain vocabulary; on the basis of content recognition, using a graph attention network model to extract the relationships between the vectorized content;

[0053] Run the entity alignment algorithm based on GNN. After the review and adjustment of the alignment results, use the similarity measurement method based on meta-path to verify and optimize the accuracy of entity alignment. Implement the multi-modal information fusion technology, integrate and analyze the relationship information from different sources, integrate the aligned entities and the fused relationships into a fine-grained knowledge graph, and perform periodic updates and maintenance.

[0054] Furthermore, use a graph neural network to analyze the multi-hop relationships between entities; extract entities and relationships from the data of the domain knowledge graph to generate the structure of graph G, including nodes V and edges E,

[0055] Select a graph neural network architecture and initialize the feature vectors of each node , and obtain the initial feature matrix of all nodes , and the node feature vectors are updated through multiple rounds of propagation:

[0056] ;

[0057] Among them, is the feature vector of node i after the k-th round of transmission, is the feature vector of node i after the (k - 1)-th round of transmission, is the set of neighbor nodes of node i, is an aggregation function that integrates the features of neighbor node j of node i together, is a function for updating the features of node i, is the feature vector of neighbor node j of node i after the (k - 1)-th round of transmission;

[0058] Through multiple rounds of propagation updates, the high-dimensional feature vectors of the nodes are finally obtained, and the high-dimensional feature vectors of each node contain information about multi-hop neighbors.

[0059] Identify redundant instances in the data through semantic similarity calculation. For any two entities and , calculate their similarity, where the similarity is defined as:

[0060] ;

[0061] where, is the similarity score between entities and , is the feature vector of neighbor node j of node i after the k - th round of message passing. Set a similarity threshold . When , it is considered that and are redundant.

[0062] It should be noted that the similarity score here is the cosine similarity. When = 1, it means that the directions of the two vectors are the same, indicating that they are very similar.

[0063] When = 0, it means that the angle between the two vectors is 90°, indicating that there is no similarity between them.

[0064] When = - 1, it means that the directions of the two vectors are exactly opposite. [[ID=XXX]] [[ID=XXX]]

[0065] It should be noted that by constructing a knowledge graph, a clear domain knowledge framework is formed, which helps to understand the complex relationships within the domain. With the help of the Graph Attention Network (GAT), the relationships between entities can be more accurately identified and extracted, improving the data integration ability. By periodically updating and maintaining the knowledge graph, the identification of redundant data is ensured to be real - time and accurate.

[0066] S2: Combine semantic recognition technology to eliminate irrelevant data in the data pre - processing stage of constructing the domain knowledge graph.

[0067] Furthermore, unify the formats of the original data obtained from different data sources, remove obvious errors and missing values, and store them in the database;

[0068] Adopt a multi - level coding mechanism to encode the original data at three levels:

[0069] Extract the basic features of the original data to generate basic vectors; Use a pre - trained deep - learning model to perform context - semantic encoding on the text or images in the original data and transform them into high - dimensional semantic vectors; According to the knowledge graph in the target domain, calculate the similarity between high - dimensional semantic vectors to generate weighted semantic feature vectors, and map each data point in the original data to a feature space with higher interpretability.

[0070] When the original data enters the encoding stage, it will be matched with the knowledge graph to generate advanced semantic features;

[0071] Perform a neighborhood search for each data point. If the number of points within the neighborhood of the current data point, including itself, is greater than the density threshold, it is regarded as a core point; otherwise, it is regarded as an edge point. Group the core points and their neighbor points into the same cluster. If one core point can reach another core point, the points adjacent to these two core points will also be merged into the same cluster;

[0072] Mark all edge points and unclassified points as outliers, that is, points that cannot be assigned to any cluster formed by core points, and mark data points outside the specified density and size groups;

[0073] Randomly sample the automatically marked data points proportionally or audit them according to the number of outliers. After the audit is completed, enter the modification results and update the marking database.

[0074] It should be noted that by unifying the format and removing errors, the quality of the data is significantly improved, facilitating subsequent analysis. Adopting a multi-level encoding mechanism transforms the original data into higher-dimensional semantic features, improving the performance of subsequent models. Through the density clustering method, core points and outliers are accurately identified, facilitating further data processing and cleaning.

[0075] S3: After identifying the redundant parts in the data, remove the redundant data through a data cleaning algorithm.

[0076] Furthermore, generate an identifier for each data point, mark the data point as "normal" or "redundant", and create a marking set M,

[0077] ;

[0078] where, is the marked data point, represents the redundant data set. According to the marking set M, define an operation to remove redundant data:

[0079] ;

[0080] Here, is the data point marked as redundant, is the data set after removing the redundant data, represents the set difference operation, indicating removing all data points marked as redundant from D.

[0081] It should be noted that generating an identifier for each data point facilitates the management and tracking of the data status, ensuring the effectiveness and reliability of the data. Using set operations to quickly remove redundant data reduces the scale of the data set and improves the subsequent processing efficiency.

[0082] S4: Optimize the data governance process in combination with data quality assessment.

[0083] Furthermore, the combination of data quality assessment includes performing quality assessment on the data set after removing redundant data, defining a quality assessment function Q to measure the quality of the remaining data in the data set: ;

[0084] ;

[0085] where is the weight of the marked data point and is dynamically adjusted by selecting relevance according to features:

[0086] ;

[0087] where is the mean value of the features corresponding to the current data set, used to reflect the overall distribution of all data points, so that the feature weights are adjusted in real time according to the actual data distribution, is the feature vector.

[0088] Furthermore, establish a user feedback channel, collect application feedback on the used data, and adjust the data governance strategy in a timely manner. Classify and mark the user feedback for subsequent analysis. The feedback content can be divided into "data accuracy problems", "data missing feedback", "usage experience suggestions", etc., which is convenient for the team to identify the problems to be processed first. Regularly generate reports on the data governance effect and quality assessment for the reference of management decision-making to promote the continuous improvement of the data governance process.

[0089] It should be noted that through quality assessment, the integrity and effectiveness of the final data set are ensured, providing a guarantee for the use of data. By updating the feature weights in real time, the data governance process is made flexible and adaptable to respond to environmental and data changes in a timely manner.

[0090] Example 2, the second example of the present invention, which is different from the previous example in that:

[0091] When the above-mentioned functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs.

[0092] The logic and / or steps represented in the flowchart or described in other ways herein, for example, can be considered as a definite sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or in combination with these instruction execution systems, apparatus, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

[0093] More specific examples (non-exhaustive list) of computer-readable media include the following: electrical connection parts with one or more wirings (electronic devices), portable computer disk cartridges (magnetic devices), random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memories), fiber optic devices, and portable compact disc read-only memories (CDROM). Additionally, the computer-readable medium can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, then editing, interpreting, or otherwise processing it as appropriate, and then storing it in a computer memory.

[0094] It should be understood that each part of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0095] Example 3, referring to Figure 2 , which is an embodiment of the present invention, provides a training data redundancy elimination system for a large model based on semantic recognition, including a knowledge graph construction module, a semantic association mining module, a data preprocessing module, a redundant data recognition and elimination module, and a data quality evaluation and optimization module;

[0096] The knowledge graph construction module is responsible for constructing a domain knowledge graph to provide basic data and semantic references for subsequent semantic analysis and association mining;

[0097] The semantic association mining module analyzes the relationships between entities in the data using the constructed knowledge graph and mines semantic information;

[0098] The data preprocessing module cleans and formats the original data to ensure the quality and consistency of the data and provides reliable input for subsequent analysis;

[0099] The redundant data recognition and elimination module identifies and eliminates redundant data to improve the conciseness and effectiveness of the data set;

[0100] The data quality evaluation and optimization module evaluates the quality of the data set after redundant data elimination to ensure the integrity and accuracy of the data, and at the same time optimizes the data governance process.

[0101] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention, and they should all be covered by the scope of the claims of the present invention.

Claims

1. A method for eliminating redundant training data of a large model based on semantic recognition, characterized in that: include, Construct a domain knowledge graph, perform semantic association mining based on the domain knowledge graph, and identify redundant parts of the data in the domain knowledge graph; In the data preprocessing stage of constructing the domain knowledge graph, irrelevant data in the original data is eliminated by combining semantic recognition technology, wherein the original data is text or image data; After identifying the redundant parts in the data, the redundant data is removed through data cleaning algorithms; Optimize data governance processes by combining data quality assessments; The semantic association mining includes analyzing the multi-hop relationship between entities using graph neural network; extracting entities and relationships from the data of the domain knowledge graph to generate the structure of graph G, which includes nodes V and edges E. Select a graph neural network architecture and initialize the feature vectors of each node , and obtain the initial feature matrix of all nodes , and the feature vectors of the nodes are updated through multiple rounds of propagation; The semantic association mining also includes identifying redundant instances in the data through semantic similarity calculation.

2. The method for eliminating redundant training data of a large model based on semantic recognition according to claim 1, wherein: The construction of the domain knowledge graph includes collecting raw data consisting of domain professional literature and large-scale data sets, combining the domain vocabulary, and vectorizing the content contained in the raw data; Based on content recognition, the graph attention network model is used to extract and quantify the relationship between the contents. Run the GNN-based entity alignment algorithm, review and adjust the alignment results, use the meta-path-based similarity measurement method to verify and optimize the accuracy of entity alignment, implement multimodal information fusion technology, integrate and analyze relationship information from different sources, integrate the aligned entities and fused relationships into a fine-grained knowledge graph, and perform periodic updates and maintenance.

3. The method for eliminating redundant training data of a large model based on semantic recognition according to claim 2, wherein: In the semantic association mining: ； Among them, is the feature vector of node i after the k-th round of message passing, is the feature vector of node i after the (k - 1)-th round of message passing, is the set of neighbor nodes of node i, is an aggregation function that integrates the features of neighbor node j of node i, is a feature function used to update the feature of node i, is the feature vector of neighbor node j of node i after the (k - 1)-th round of message passing; Through multiple rounds of propagation updates, the high-dimensional feature vector of the node is finally obtained. The high-dimensional feature vector of each node contains the information of multi-hop neighbors.

4. The method for eliminating redundant training data of a large model based on semantic recognition according to claim 3, wherein: For any two entities and , calculate their similarity, where the similarity is defined as: ； Among them, is the similarity score between and is the feature vector of neighbor node j of node i after the k-th round of message passing. Set the similarity threshold When it is considered that and are redundant.

5. The method for eliminating redundant training data of a large model based on semantic recognition according to claim 3, characterized in that: Eliminating irrelevant data includes unifying the format of raw data obtained from different data sources, removing obvious errors and missing values, and storing them in the database; A multi-level coding mechanism is used to divide the original data into three levels for coding: Extract the basic features of the original data to generate basic vectors; use a pre-trained deep learning model to perform contextual semantic encoding on the text or images in the original data and convert them into high-dimensional semantic vectors; calculate the similarity between high-dimensional semantic vectors to generate weighted semantic feature vectors, mapping each data point in the original data to a feature space with higher explanatory power; When the raw data enters the encoding stage, high-level semantic features are generated; Perform a neighborhood search for each data point. If the number of points in the neighborhood of the current data point, including itself, is greater than the density threshold, it is considered a core point; otherwise, it is considered an edge point. The core point and its neighboring points are grouped into the same cluster. If a core point reaches another core point, the points adjacent to the two core points will also be merged into the same cluster. Mark all edge points and unclassified points as outliers, that is, data points that cannot be classified into any cluster formed by core points, and mark data points outside the specified density and size group; The automatically marked data points are randomly sampled in proportion or reviewed according to the number of outliers. After the review is completed, the modification results are entered and the marking database is updated.

6. The method for eliminating redundant training data of a large model based on semantic recognition according to claim 4, characterized in that: The redundant data removal includes generating an identifier for each data point, marking the data point as "normal" or "redundant", and creating a tag set M. ； Among them, is a marked data point, represents a redundant data set. According to the marker set M, an operation for removing redundant data is defined as follows: ； Here, is to mark redundant data points, is the data set after removing redundant data, represents the set difference operation, which means removing all data points marked as redundant from D.

7. The method for eliminating redundant training data of a large model based on semantic recognition according to claim 5, characterized in that: The combined data quality assessment includes performing quality assessment on the data set after removing redundant data and defining a quality assessment function Q to measure the quality of the remaining data in the data set: ； Among them, is the weight of the marked data point and is dynamically adjusted by selecting relevance according to features: ； Among them, is the mean value of the features corresponding to the current data set, which is used to reflect the overall distribution of all data points, so that the feature weights can be adjusted in real time according to the actual data distribution. is the feature vector.

8. A training data redundancy elimination system for a large model based on semantic recognition, applied to the training data redundancy elimination method for a large model based on semantic recognition described in claim 6, characterized in that: It includes a knowledge graph construction module, a semantic association mining module, a data preprocessing module, a redundant data identification and elimination module, and a data quality assessment and optimization module; The knowledge graph construction module is responsible for constructing a domain knowledge graph, providing basic data and semantic references for subsequent semantic analysis and association mining; The semantic association mining module uses the constructed knowledge graph to analyze the relationships between entities in the data and mine semantic information; The data preprocessing module cleans and formats the original data to ensure the quality and consistency of the data, providing reliable input for subsequent analysis; The redundant data identification and elimination module identifies and eliminates redundant data to improve the conciseness and effectiveness of the data set; The data quality assessment and optimization module conducts a quality assessment of the data set after eliminating redundant data to ensure the integrity and accuracy of the data, and at the same time optimizes the data governance process.

Images