A data processing method and system for interpretable AI model training

Abstract

This invention discloses a data processing method and system for interpretable AI model training, relating to the field of model training technology. The method includes performing unified vector encoding, concept assignment, concept evidence generation, and consistency verification on each piece of raw data in a concept candidate data pool to form concept bottleneck data. Based on the concept bottleneck data, confirmed concept clusters are established. Then, a concept combination sparse region is determined using the confirmed concept clusters and a density-adaptive clustering algorithm. Data located in the concept combination sparse region and meeting the conditions of concept evidence completeness and source credibility are selected for supplementary labeling to generate a concept-complete training set. This invention, by combining concept evidence completeness and source credibility to screen and supplement samples, achieves data completion for concept combination coverage gaps, improving the interpretability and completeness of the interpretation chain in subsequent model training.

Description

Technical Field

[0001] This invention relates to the field of model training technology, and in particular to a data processing method and system for interpretable AI model training. Background Technology

[0002] With the rapid development of big data and deep learning technologies, artificial intelligence models have been widely applied in fields such as industrial monitoring, intelligent manufacturing, financial risk control, and medical auxiliary diagnosis. Traditional artificial intelligence model training methods mainly rely on multi-source heterogeneous data and achieve task prediction through feature engineering or end-to-end neural networks. Common methods include data encoding based on vector representation learning, sample training based on supervised or semi-supervised learning, and complex model structures based on deep neural networks.

[0003] However, existing training methods still have some shortcomings in terms of interpretability. First, they lack the prior organization and verification of conceptual structure and conceptual evidence in the training data, which leads to the model training relying on implicit features and the interpretation path being unstable. Second, they lack sample selection and supplementation mechanisms oriented towards the coverage of concept combinations, the completeness of conceptual evidence, and the credibility of sources, which results in gaps in the training set at the conceptual level, affecting the interpretability of subsequent model training and the completeness of the interpretation chain. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a data processing method for interpretable AI model training to solve the problems of unstable interpretation paths and gaps at the conceptual level.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides a data processing method for interpretable AI model training, comprising, Obtain the raw data and merge the raw data. At the same time, combine the concept dictionary to perform preprocessing and cross-source consistency comparison on the merged raw data to generate a concept candidate data pool. Perform unified vector encoding, concept assignment, concept evidence generation, and consistency verification on each piece of raw data in the concept candidate data pool to form concept bottleneck data; Based on the conceptual bottleneck data, confirmed concept clusters are established, and the concept combination sparse regions are determined by the confirmed concept clusters and density adaptive clustering algorithm. Data located in the concept combination sparse regions and meeting the conditions of concept evidence completeness and source credibility are selected for supplementary labeling to generate a concept complete training set. The concept prediction model and the task prediction model are trained in two stages using a concept-complete training set. The concept prediction model outputs concept prediction results and compares them with concept evidence for consistency. The task prediction model outputs the final task label based on the concept prediction results. At the same time, the concept prediction results with abnormal consistency comparison results are corrected to generate interpretable prediction results.

[0007] As a preferred embodiment of the data processing method for interpretable AI model training described in this invention, the merging of raw data refers to accessing raw data from business databases, log streams, sensor streams, text records, and image / video streams, extracting source identifiers, collection times, and business entity identifiers during the access process, and performing time window merging on the raw data according to the collection time and business entity identifiers to form a merged data set.

[0008] As a preferred embodiment of the data processing method for interpretable AI model training described in this invention, the generation of the concept candidate data pool specifically includes: Based on the pre-defined concept field mapping relationship in the concept dictionary, the merged data set is subjected to field alignment, missing data filling, anomaly pruning, and format unification to form a preprocessed data set. Perform cross-source consistency comparison on the preprocessed data set, identify data conflict records corresponding to business entity identifiers and complete conflict resolution, retain data records that pass the consistency verification, and summarize them to form a conceptual candidate data pool.

[0009] As a preferred embodiment of the data processing method for interpretable AI model training described in this invention, the formation of conceptual bottleneck data specifically includes: Extract each piece of raw data from the concept candidate data pool, and perform unified vector encoding on each piece of raw data to form the corresponding basic feature vector; Based on the pre-defined concept field mapping relationship in the concept dictionary, the concept value is determined according to the basic feature vector and the concept field mapping relationship in the concept assignment process, thus forming a concept vector; Extract field segments, time segments, and region segments corresponding to concept vectors from the raw data, establish the correspondence between concept vectors and field segments, time segments, and region segments, and form concept evidence; Consistency checks are performed on concept vectors and concept evidence. Inconsistent records between concept vectors and concept evidence are identified and removed. Basic feature vectors, concept vectors, and concept evidence that pass the consistency check are retained and aggregated to form concept bottleneck data.

[0010] As a preferred embodiment of the data processing method for interpretable AI model training described in this invention, the determination of the scarce region of concept combination specifically includes: Extract basic representation vectors and concept vectors from the concept bottleneck data. Calculate the Euclidean distance between the fundamental representation vectors and the concept vectors to form a distance matrix; Calculate the neighborhood density value for each basic representation vector based on the distance matrix, and determine the basic representation vectors with neighborhood density values ​​greater than a preset density threshold as the initial cluster centers; The remaining basic representation vectors are assigned to the initial cluster centers with the closest Euclidean distance in descending order of neighborhood density values, and the corresponding cluster boundaries are dynamically adjusted according to the neighborhood density values ​​during the assignment process. After the basic representation vectors are assigned, clusters whose Euclidean distance between cluster centers is less than a preset distance threshold and whose neighborhood density difference is less than a preset density difference threshold are merged to form multiple confirmed concept clusters. The number of data corresponding to each concept vector is counted from multiple confirmed concept clusters. Concept combinations with a number of data below a preset threshold are identified as scarce concept combinations, and the distribution range of the basic representation vectors corresponding to scarce concept combinations is identified as the scarce region of concept combinations.

[0011] As a preferred embodiment of the data processing method for interpretable AI model training described in this invention, the generation of a conceptually complete training set specifically includes: Extract data located in the scarce region of concept combinations, and read the conceptual evidence and source credibility corresponding to each data point; Perform conceptual evidence integrity and source credibility assessments on data located in areas of scarce conceptual combinations, and select data with complete conceptual evidence and source credibility reaching a preset credibility threshold to form a set of data to be supplemented; Extract concept vectors and task label placeholder states from the dataset to be supplemented; perform supplementation labeling on the unconfirmed concept values ​​in the concept vectors and the task labels corresponding to the task label placeholder states to form a supplemented dataset. The supplementary data set is merged with the data in the concept bottleneck data that have complete concept values ​​and complete task labels to form a concept-complete training set.

[0012] As a preferred embodiment of the data processing method for interpretable AI model training described in this invention, the output concept prediction result specifically includes: Extract training records that have been merged, preprocessed and labeled with concepts from the concept-complete training set, and input the raw data into the front-end concept prediction model; The front-end concept prediction model performs concept reasoning on the original data, outputs the concept value results corresponding to the original data, and uses them as the concept prediction results.

[0013] As a preferred embodiment of the data processing method for interpretable AI model training described in this invention, the consistency comparison specifically includes: Extract concept value results from the concept prediction results, and extract concept evidence corresponding to the concept value results from the concept completeness training set; Based on the concept field mapping relationship, perform item-by-item matching and verification of concept value results and concept evidence, identify and mark inconsistent records between concept value results and concept evidence, and output consistency comparison results.

[0014] As a preferred embodiment of the data processing method for interpretable AI model training described in this invention, the output of the final task label specifically includes: Input the concept value results from the concept prediction results into the subsequent task prediction model; The control back-end task prediction model performs task inference on the concept value results, outputs the task label results corresponding to the concept value results, and forms the final task label.

[0015] Secondly, this invention provides a data processing system for interpretable AI model training, comprising, The preprocessing module acquires the raw data and merges it. At the same time, it combines the concept dictionary to perform preprocessing and cross-source consistency comparison on the merged raw data to generate a concept candidate data pool. The encoding module performs unified vector encoding, concept assignment, concept evidence generation, and consistency verification on each piece of raw data in the concept candidate data pool to form concept bottleneck data. The supplementation module establishes confirmed concept clusters based on the concept bottleneck data, and determines the scarce regions of concept combinations through the confirmed concept clusters and density adaptive clustering algorithm. It then selects data located in the scarce regions of concept combinations that meet the conditions of concept evidence completeness and source credibility for supplementation, and generates a concept-complete training set. The training module uses a concept-complete training set to perform two-stage training on the front-end concept prediction model and the back-end task prediction model. This allows the front-end concept prediction model to output concept prediction results and compare them with concept evidence for consistency. The back-end task prediction model then outputs the final task label based on the concept prediction results. Simultaneously, it corrects concept prediction results with abnormal consistency comparison results and generates interpretable prediction results.

[0016] The beneficial effects of this invention are as follows: By performing unified vector encoding, concept assignment, concept evidence generation, and consistency verification on the original data in the concept candidate data pool, the training data possesses explicit concept structure and concept evidence constraints before entering model training. By eliminating data records where concepts and evidence are inconsistent, the model training process avoids over-reliance on implicit features, enhancing the correspondence and traceability between concept values ​​and original data, thereby improving the stability of the interpretation path. Simultaneously, by establishing confirmed concept clusters, identifying scarce regions of concept combinations, and filtering and labeling samples based on the completeness of concept evidence and the credibility of the source, data filling for gaps in concept combination coverage is achieved, enhancing the training set in terms of concept coverage, evidence completeness, and source reliability, thereby improving the interpretability and interpretation chain completeness of subsequent model training. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating a data processing method for interpretable AI model training.

[0019] Figure 2 A flowchart for generating a pool of candidate concept data.

[0020] Figure 3 A flowchart for generating a conceptually complete training set.

[0021] Figure 4 A flowchart for generating interpretable prediction results after training is completed. Detailed Implementation

[0022] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0023] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0024] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0025] Reference Figures 1-4 As one embodiment of the present invention, this embodiment provides a data processing method for interpretable AI model training, including the following steps: S1: Obtain the original data and merge the original data. At the same time, combine the concept dictionary to perform preprocessing and cross-source consistency comparison on the merged original data to generate a concept candidate data pool.

[0026] S1.1: When accessing raw data from the business database, log stream, sensor stream, text records, and image / video streams, read the business database records, log stream events, sensor stream sampling points, text record content fragments, and image / video stream frame fragments in a unified access order, and append an access sequence number to each piece of raw data. Simultaneously, extract the source information, time information, and object identification information from each type of raw data, and write them uniformly into the source identifier field, acquisition time field, and business entity identifier field, respectively. After extracting the source identifier, acquisition time, and business entity identifier, perform a unified format conversion on the acquisition time to ensure that all business database records, log stream events, sensor stream sampling points, text record content fragments, and image / video stream frame fragments correspond to the same time base. Then, classify the data according to the business entity identifier, grouping raw data with the same business entity identifier and acquisition times falling within the same time window into the same merged data. The time window is divided into continuous time intervals, such as a 1-minute time window, a 5-minute time window, or a 10-minute time window. During the merging process, raw data with empty business entity identifiers are directly discarded. Raw data with missing collection times but recoverable from context fields, sampling order, metadata, or frame sequence numbers are supplemented with collection times before participating in the merging. Raw data with missing source identifiers are supplemented with source identifiers according to the access channel name before participating in the merging. Merged data formed within the same business entity identifier and the same time window retains the business database record set, log stream event set, sensor stream sampling point set, text record content fragment set, and image / video stream frame fragment set, and records the corresponding time window start time, time window end time, and business entity identifier. All merged data are aggregated to form a merged data set.

[0027] S1.2: Read the merged data one by one from the merged data set, and synchronously call the pre-defined concept field mapping relationship in the concept dictionary corresponding to the current business task. The pre-defined concept field mapping relationship in the concept dictionary is used to clarify the correspondence between concept names and original fields, so that the content from different sources in the merged data set can be organized around the same set of concepts, and avoid the original fields in the merged data set entering the subsequent processing in a scattered form.

[0028] It should also be noted that the pre-defined concept field mapping relationships in the concept dictionary are pre-organized based on the business task interpretation objectives. During the organization process, for each concept name that needs to be clearly explained in the business task, the field source, field name, field type, field format requirements, and field value boundaries corresponding to the concept name are determined, and the organization results are written into the concept dictionary. The concept names in the concept dictionary remain fixed. The field source indicates which type of content the concept name comes from: business database record, log stream event, sensor stream sampling point, text record content fragment, or image / video stream frame fragment. The field name indicates the specific field corresponding to the concept name. The field type is used to distinguish between numerical fields, status fields, text fields, and region fields. The field format requirements are used to unify the subsequent writing format, and the field value boundaries are used to support anomaly pruning. For example, "temperature rise anomaly" corresponds to the temperature sequence field in the sensor stream sampling point, "load fluctuation" corresponds to the load curve field in the sensor stream sampling point, "rule conflict" corresponds to the rule trigger field in the log stream event, and "vibration anomaly" corresponds to the vibration sequence field in the sensor stream sampling point.

[0029] The business database records, log stream events, sensor stream sampling points, text record content fragments, and image / video stream frame fragments in the merged dataset are aligned, missing data is filled, anomaly is pruned, and formatted according to the predefined concept field mapping relationships in the concept dictionary. Field alignment unifies fields with the same meaning but different names from different sources into the same field position; missing data filling completes missing fields with the same business entity identifier or within the same time window; anomaly pruning deletes or truncates content that exceeds field value boundaries, is obviously misaligned, or cannot correspond to the concept name; formatting unification organizes time, status, category, numerical unit, text content, and region content into a consistent writing style. After completing field alignment, missing data filling, anomaly pruning, and formatting unification, a preprocessed dataset is obtained.

[0030] For the preprocessed dataset, records with different source identifiers but the same field names or corresponding semantics are placed in the same comparison range according to business entity identifiers and time windows, and the consistency of field content is checked item by item. The checks include whether status fields correspond to the same business status, whether numerical fields fall within the same field value boundary, whether time fields are within the same time window, whether the event description in text record content fragments is consistent with log stream events or business database records, and whether the object content in image / video stream frame fragments corresponds to sensor stream sampling points or log stream events. Records with consistent field content or semantics are considered to have passed the consistency check; records with contradictory field content or semantic mismatch are considered to be conflicting data records.

[0031] Based on the identified data conflict records, conflict resolution is performed according to the source credibility and field type corresponding to the source identifier: when a business database record conflicts with a log stream event, the structured field content of the business database record is retained; when a sensor stream sampling point conflicts with a log stream event, the content of the sensor stream sampling point with the same acquisition time and time window position is retained; when a text record content fragment conflicts with a business database record, the structured field content of the business database record is retained; when an image / video stream frame fragment conflicts with a sensor stream sampling point, the content of the image / video stream frame fragment with the same business entity identifier and acquisition time is retained. After conflict resolution, only data records that pass the consistency check are retained in the preprocessed data set.

[0032] Data records that pass consistency verification are written to the concept candidate data pool one by one. The written content includes at least the original features, source credibility, concept candidate values, and original task label placeholders. The original features come from the field content retained in the preprocessed dataset; the source credibility comes from the credibility record corresponding to the source identifier; the concept candidate values ​​come from the results corresponding to the pre-defined concept field mapping relationships in the concept dictionary; and the original task label placeholders are used to mark that the current record has not yet completed subsequent task label processing. All data records that pass consistency verification are aggregated to form the concept candidate data pool.

[0033] S2: Perform unified vector encoding, concept assignment, concept evidence generation, and consistency verification on each piece of raw data in the concept candidate data pool to form concept bottleneck data.

[0034] S2.1: Extract each merged and preprocessed training record from the concept candidate data pool in record order, retaining the original features, source credibility, concept candidate values, and task label placeholder status. Input the training records into a unified vector encoding process, where structured fields, time series, text content, and image / video frame fragments are organized into vector form according to the concept dictionary order of structured fields, the sampling time order of time series, the appearance order of text keywords, and the frame number and region order of image / video frames. These vectors are then concatenated in a unified arrangement order to generate a basic feature vector of uniform length and format, used to represent the basic distribution information of training records in the overall sample space. After input processing, according to the pre-set field order, time series extraction order, text keyword order, and image / video stream frame fragment statistical order, extract a fixed number (determined by the input length requirement of the unified vector encoding process, e.g., 20) of structured field values, time series statistical values, text statistical values, and image / video stream frame fragment statistical values, respectively, and concatenate them in a unified arrangement order to form a basic feature vector of uniform length and format. The basic feature vector is only used to represent the basic distribution position of the original data in the overall sample space, and does not directly undertake the final task of label determination.

[0035] To further explain, the field order is set according to the preset concept field mapping relationship in the concept dictionary. Fields that directly correspond to the concept name are arranged first, and fields used for supplementary explanations are arranged later. Business database record fields and log stream event fields are arranged according to the predefined field order. The field order is determined by the fields corresponding to each concept in the concept dictionary, and the field order remains unchanged in all the original data after it is determined.

[0036] The time series sampling order is set according to the chronological order of acquisition time. Sensor stream sampling points are arranged sequentially from earliest to latest, with the start time of the time window as the starting point and the end time of the time window as the ending point. When the number of sampling points exceeds the preset number, the sampling content at fixed positions is retained; when the number of sampling points is less than the preset number, gaps are filled at the end. The preset number is determined in advance according to the number of fixed positions that need to be retained for various types of raw data in the unified vector encoding process, to ensure that the length of the basic feature vectors formed by different raw data is consistent. For example, it can be set to 20 sampling positions.

[0037] The order of text keywords is set according to the order of concept names in the concept dictionary and the order of appearance in the text record content fragments. In practice, a keyword list is preset for each concept name in the concept dictionary, and then words matching the keyword list are extracted sentence by sentence from the text record content fragments as text keywords. Text keywords corresponding to preceding concept names are arranged first, and those corresponding to subsequent concept names are arranged last. When the same concept name corresponds to multiple text keywords, they are arranged according to their actual appearance order in the text record content fragments. For example, the concept name "abnormal temperature rise" corresponds to the keywords "temperature rise, heat generation, overheating, and rapid temperature rise." When the text record content fragment shows "equipment overheating accompanied by temperature rise," "overheating" and "temperature rise" are extracted as text keywords and written in the order of appearance.

[0038] It should also be noted that the corresponding keyword table is determined by pre-organizing a set of words that are semantically consistent with the concept names in the concept dictionary. This is used to ensure that the text keyword extraction results in the text record content fragments correspond to the concept names. For example, the keyword table corresponding to the concept name 'rule conflict' can be set as 'mutually exclusive, conflict, simultaneous triggering, inconsistent conditions'.

[0039] The statistical order of image and video stream frame segments is set according to frame number order and region position order. First, the frames are arranged from smallest to largest by frame number. Then, within a single frame, statistical content is extracted according to the preset region position order. The region position order uses a fixed direction, such as from left to right or from top to bottom. After the statistical order of image and video stream frame segments is fixed, the positions of the statistical values ​​formed by different image and video stream frame segments remain consistent.

[0040] The system reads the original field content from the raw data corresponding to the basic feature vector and determines the validity of each concept name according to the concept field mapping relationship in the concept dictionary, forming concept values. Concept values ​​are explicit judgment results corresponding to each concept name, with the corresponding fields, judgment criteria, and result writing format predefined in the concept dictionary. When a concept name corresponds to a single state judgment, the concept value is written as "valid" or "invalid"; when a concept name corresponds to a level judgment, the concept value is written as a preset level, determined by the predefined level categories in the concept dictionary, with each level corresponding to a specific value range or judgment rule. For example, temperature rise anomalies can be divided into 'low', 'medium', and 'high'; when a concept name corresponds to a category judgment, the concept value is written as a preset category name. For example, 'temperature rise anomaly' is determined as 'valid' or 'invalid' based on the temperature sequence field and the anomaly judgment boundary; 'rule conflict' is determined as 'valid' or 'invalid' based on whether mutually exclusive rules are triggered simultaneously in the rule trigger field. After each concept name is judged, the concept values ​​are written sequentially according to the preset concept order in the concept dictionary, forming a concept vector. Each element in the concept vector corresponds to a fixed concept name, and the content of each element is the judgment result of the corresponding concept name. Therefore, the concept vector can directly reflect the establishment status, level status, or category status of each concept name in the current original data.

[0041] It should also be noted that the concept order comes from the concept arrangement order in the concept dictionary. When the concept dictionary pre-organizes the concept names, concept field mapping relationships, judgment criteria, and result writing formats, it assigns a fixed position to each concept name. When forming the concept vector, the values ​​of each concept are written sequentially according to this fixed position. For example, if "vibration anomaly" is placed in position 1, "temperature rise anomaly" in position 2, "load fluctuation" in position 3, and "rule conflict" in position 4, then the concept vector will be formed by writing the corresponding concept values ​​in the order of "vibration anomaly—temperature rise anomaly—load fluctuation—rule conflict." The purpose of this processing is to ensure that the concept vectors formed from different original data have consistent positions, facilitating subsequent consistency checks and clustering processing.

[0042] S2.2: After the concept vector is formed, the corresponding original data is retrieved based on the concept name and concept value in the concept vector, and content fragments that can directly prove the validity of the concept value are extracted as concept evidence. Specifically: when the concept name corresponds to a structured field, the corresponding field content is extracted from business database records, log stream events, or sensor stream sampling points as field fragments; when the concept name corresponds to a time series, the continuous sampling interval corresponding to the concept value is extracted from sensor stream sampling points as time fragments; when the concept name corresponds to text semantics, the sentence content corresponding to the concept value is extracted from text record content fragments as field fragments; when the concept name corresponds to image / video content, the target region corresponding to the concept value is extracted from image / video stream frame fragments as region fragments. After the field fragments, time fragments, and region fragments are extracted, the concept name, concept value, field fragment, time fragment, and region fragment are established according to the same record number to form concept evidence. The concept evidence retains the field name, field location, time start and end range, frame number, and region coordinates, so that each concept value in the concept vector can correspond to specific original data content.

[0043] The concept vector and concept evidence are checked item by item according to the same record number. It is verified that the concept name and concept value in the concept vector can be found in the corresponding field segment, time segment, or region segment in the concept evidence. Simultaneously, it is checked whether the field segment, time segment, or region segment is consistent with the concept field mapping relationship in the concept dictionary. If a concept value exists in the concept vector but the corresponding field segment, time segment, or region segment does not exist in the concept evidence, it is identified as a record of inconsistency due to lack of evidence. If a content segment exists in the concept evidence but the source field of the content segment is inconsistent with the preset concept field mapping relationship in the concept dictionary, it is identified as a record of inconsistency due to incorrect mapping. If the time position or region position in the concept evidence is inconsistent with the position of the original data record, it is identified as a record of inconsistency due to position. Records of inconsistency due to lack of evidence, inconsistency due to incorrect mapping, and inconsistency due to position are removed from the current processing results and do not proceed to subsequent steps.

[0044] Records that pass consistency verification retain basic feature vectors, concept vectors, and conceptual evidence, and are written in association with source credibility and original task label placeholder status. Each written record adopts a fixed organizational format, which includes original data, basic feature vectors, concept vectors, conceptual evidence, source credibility, and original task label placeholder status. All records that pass consistency verification are aggregated to form the concept bottleneck data. The smallest record center in the concept bottleneck data already possesses unified representation results, explicit concept results, and traceable evidence results simultaneously; therefore, the concept bottleneck data can be directly used as input for subsequent establishment of confirmed concept clusters and selection of supplementary labeling objects.

[0045] Preferably, before training an interpretable AI model, the present invention organizes the multi-source raw data into basic feature vectors of uniform length and format, and further forms concept vectors and concept evidence that correspond to the original field content, so that the concept values ​​have clear sources and backtracking basis; at the same time, by eliminating inconsistent records with missing evidence, inconsistent records with incorrect mapping, and inconsistent records with inconsistent positions through consistency verification, it can reduce the entry of erroneous concepts into the training process and improve the interpretability, consistency, and subsequent labeling efficiency of the concept bottleneck data.

[0046] S3: Establish confirmed concept clusters based on the concept bottleneck data, and determine the scarce regions of concept combinations through the confirmed concept clusters and density adaptive clustering algorithm. Select data located in the scarce regions of concept combinations that meet the conditions of concept evidence completeness and source credibility for supplementary labeling, and generate a concept complete training set.

[0047] S3.1: Read the basic representation vectors and concept vectors one by one from the concept bottleneck data, maintaining the correspondence between the basic representation vectors and concept vectors. Calculate the Euclidean distance between samples for the basic representation vectors and concept vectors, forming a distance matrix. The expression for the Euclidean distance is: ; in, Representation of the basic representation vector With the basic representation vector Distance between them Representation of the basic representation vector In the Feature values ​​at each position, Representation of the basic representation vector In the Feature values ​​at each position, This represents the total dimension of the basic representation vector, i.e., the number of features.

[0048] After obtaining the distance matrix, the number of base representation vectors whose Euclidean distance to the current base representation vector is less than a preset neighborhood radius is counted, and the statistical result is determined as the neighborhood density value of the current base representation vector. Base representation vectors with neighborhood density values ​​greater than a preset density threshold are determined as initial cluster centers. The neighborhood density value is used to represent the degree of concentration of base representation vectors within their corresponding neighborhoods.

[0049] It should also be noted that the neighborhood radius is set to 20% of the mean of all Euclidean distances based on the statistical distribution of the Euclidean distances between the basic representation vectors. For example, a value of 0.2 is used to limit the statistical range of neighborhood density values, and can be customized according to needs. The density threshold is set to the average neighborhood density value based on the distribution of neighborhood density values ​​of all basic representation vectors in the distance matrix. For example, a value of 5 is used to select initial cluster centers that meet the clustering initiation conditions.

[0050] After determining the initial cluster centers, the unassigned basic representation vectors are sequentially compared with each initial cluster center according to their neighborhood density values ​​from highest to lowest using Euclidean distance. The vector is first assigned to the initial cluster center with the smallest Euclidean distance. After each assignment, the average Euclidean distance between the newly added basic representation vector and all basic representation vectors in the current cluster is calculated, and this average Euclidean distance is determined as the radius of the current cluster boundary. If the Euclidean distance between a subsequent unassigned basic representation vector and the current cluster center is less than or equal to the current cluster boundary radius, and the neighborhood density value of the subsequent unassigned basic representation vector is less than or equal to the neighborhood density value of the current cluster center... When the difference is less than or equal to a preset density difference threshold, the subsequent basic representation vectors to be assigned are allocated to the current cluster. When the Euclidean distance between the subsequent basic representation vectors to be assigned and the center of the current cluster is greater than the radius of the current cluster boundary, or when the difference between the neighborhood density value of the subsequent basic representation vectors to be assigned and the neighborhood density value of the center of the current cluster is greater than the preset density difference threshold, the allocation to the current cluster is stopped, and the current cluster boundary remains unchanged. This completes the dynamic adjustment of the cluster boundary based on the neighborhood density value, so that the basic representation vectors in the same confirmed concept cluster are kept close in data distribution, and the division of the scarce region of the corresponding concept combination is more accurate.

[0051] It should also be noted that the density difference threshold is set based on the statistical distribution of the neighborhood density values ​​of all basic representation vectors, preferably 0.1. This can distinguish basic representation vectors with large differences in neighborhood density values ​​during the basic representation vector allocation process, while retaining basic representation vectors with similar neighborhood density values ​​to enter the same cluster.

[0052] After the basic representation vectors are assigned, the Euclidean distance and neighborhood density difference between the cluster centers are compared. Clusters whose Euclidean distance between their centers is lower than a preset distance threshold and whose neighborhood density difference is lower than a preset density difference threshold are merged to form multiple confirmed concept clusters. After clustering, each confirmed concept cluster retains its corresponding concept vector, basic representation vector set, and identification information.

[0053] To further explain, the distance threshold is set based on the statistical distribution of the Euclidean distance between the cluster centers, for example, 0.1, which can ensure that the basic representation vectors within the same confirmed concept cluster are closely distributed and that different confirmed concept clusters are effectively distinguished.

[0054] S3.2: After multiple confirmed concept clusters are formed, the number of data corresponding to the concept vector combinations formed by the fixed order of the concept values ​​in the concept vectors is counted: confirmed concept clusters with the same concept vector combination are merged and counted to obtain the number of data corresponding to each concept combination, and then the number of data corresponding to each concept combination is compared with a preset number threshold. Concept vector combinations with a number of data below the number threshold are determined as scarce concept combinations, while concept vector combinations with a number of data at or above the number threshold are kept as non-scarce concept combinations.

[0055] To further explain, the quantity threshold is preset according to the minimum requirement for the number of concept coverages in the concept-complete training set. It is used to limit the minimum number of concept combinations in the concept bottleneck data. For example, a value of 30 can ensure that each concept combination has at least a fixed number of concept bottleneck data, which is convenient for supplementing the insufficient number of concept combinations in subsequent labeling.

[0056] After identifying scarce concept combinations, the distribution range of basic representation vectors is extracted from the corresponding confirmed concept clusters, and this distribution range is defined as the scarce region of concept combinations. The scarce region of concept combinations is bounded by the distribution boundary of the basic representation vectors in the corresponding confirmed concept clusters. Through this process, the scarce region of concept combinations not only clarifies which concept combinations have insufficient data, but also identifies the specific distribution location of these combinations in the basic representation vector space, thus providing a direct basis for subsequent filtering of data located in the scarce region of concept combinations.

[0057] S3.3: Based on the identified concept combination scarcity region, examine the concept evidence and source credibility of each data point located in the concept combination scarcity region. Concept evidence completeness determination: Read the basic representation vector, concept vector, concept evidence, and source credibility one by one in the record order, and check the correspondence between the concept vector and concept evidence in the same record. The concept vector records the names of each concept and their corresponding values, while the concept evidence records field fragments, time fragments, and region fragments. Therefore, to determine whether the concept evidence is complete, it is necessary to check whether the concept names and concept values ​​in the concept vector have corresponding supporting content in the concept evidence. For example, if the concept name corresponds to a structured field, check whether there is a corresponding field fragment in the concept evidence; if the concept name corresponds to a time series, check whether there is a corresponding time fragment in the concept evidence. During the verification process, if any concept name in the concept vector cannot find a corresponding field fragment, time fragment, or region fragment, or if the supporting content in the concept evidence does not correspond to the concept name, the current record is determined to have incomplete concept evidence; only when all concept names in the concept vector can find corresponding supporting content in the concept evidence is the current record determined to have complete concept evidence.

[0058] After determining the completeness of conceptual evidence, the system continues to read the source credibility of the current record and compares it with a preset credibility threshold. If the source credibility is lower than the preset credibility threshold, the current record is not included in the supplementary data set; if the source credibility reaches or exceeds the preset credibility threshold, the current record passes the source credibility determination. After both the conceptual evidence completeness determination and the source credibility determination are completed, records that simultaneously meet both conditions of "conceptual evidence completeness" and "source credibility reaching the preset credibility threshold" are retained and summarized according to the original record order in the conceptual candidate data pool to form the supplementary data set.

[0059] To further explain, the credibility threshold is preset according to the source credibility screening requirements. For example, a value of 0.8 can retain records with stable source quality and clear evidence support in the supplementary bidding process, and avoid low credibility records from affecting the subsequent supplementary bidding results.

[0060] S3.4: Read the concept vectors and task label placeholder statuses one by one from the dataset to be supplemented, in record order. First, check whether the concept values ​​corresponding to each concept name in the concept vector have been confirmed. If a concept name in the concept vector has a clear judgment result and the corresponding field segment, time segment, or region segment exists in the concept evidence, keep the corresponding concept value unchanged. If a concept name in the concept vector lacks a clear judgment result, or if a concept name in the concept vector has a concept value but the concept evidence cannot provide effective support, determine the corresponding concept name as an unconfirmed concept value and mark it as a supplementary labeling object. After completing the concept vector check, read the task label placeholder status. The task label placeholder status is used to indicate whether the current record has a clear task label. If the task label placeholder status is blank, missing, or only a placeholder mark is retained, determine the corresponding task label as a task label to be supplemented.

[0061] Once the unconfirmed concept values ​​and task labels to be supplemented are determined, supplementation is performed based on the original data content, conceptual evidence, and source credibility of the same record. During the supplementation process, the field segment, time segment, or regional segment corresponding to each unconfirmed concept name is used as the basis. According to the preset concept field mapping relationship, judgment criteria, and result writing format in the concept dictionary, the concept value is supplemented, and the supplemented result is written to the corresponding position in the original concept vector. The supplementation of task labels to be supplemented is based on the original data content, confirmed concept values, and conceptual evidence of the current record. The task label placeholder status is updated to a clear task label, and it is written to the corresponding position.

[0062] After all unconfirmed concept values ​​and task labels in the same record have been filled in, the original data, basic representation vector, updated concept vector, concept evidence, source credibility, and filled task labels are retained. The records that have been filled in are then summarized in sequence to form a filled data set.

[0063] From the conceptual bottleneck data, records where all concept names in the concept vectors have explicit concept values ​​and whose task label placeholder status has been updated to explicit task labels through supplementation are selected, forming a set of records with complete concept values ​​and task labels. The records with supplemented concept values ​​and task labels during the supplementation process are then merged with the set of records with complete concept values ​​and task labels, following a unified record structure. During the merging process, the order of the original data, basic representation vector, concept vector, concept evidence, source credibility, and task label fields remains consistent, and the values ​​of each field are not changed. After merging all records, a conceptually complete training set is formed.

[0064] Preferably, the present invention performs density-adaptive clustering using concept vectors and basic representation vectors to form multiple confirmed concept clusters. Based on these confirmed concept clusters, it identifies scarce regions of concept combination and then filters and supplements data located in these scarce regions of concept combination by combining the completeness of concept evidence and the credibility of the source, thus forming a concept-complete training set. This concept-complete training set simultaneously possesses the characteristics of complete concept coverage, clear concept evidence, and satisfactory source credibility, and provides a data foundation for the subsequent two-stage training of the front-end concept prediction model and the back-end task prediction model.

[0065] S4: Utilize a concept-complete training set to perform two-stage training on the front-end concept prediction model and the back-end task prediction model. This allows the front-end concept prediction model to output concept prediction results and perform consistency comparison with concept evidence. The back-end task prediction model then outputs the final task label based on the concept prediction results. Simultaneously, it corrects concept prediction results with abnormal consistency comparison results and generates interpretable prediction results.

[0066] S4.1: The structures of the front-end concept prediction model and the back-end task prediction model are as follows: The front-end concept prediction model receives raw data from a concept-complete training set as input, including structured fields, time series data, text content, and image / video frames. Structured fields are encoded using a fully connected neural network, which consists of an input layer, two hidden layers (each with 128 or 256 neurons), and an output layer, generating structured field feature vectors. Time series data are encoded using a Long Short-Term Memory (LSTM) network with 1 to 2 stacked recurrent hidden layers, each with a hidden state vector dimension of 128 or 256, generating time series feature vectors. Text content is processed using a recurrent neural network or word vector encoding structure, with a sequence of word vectors as input, passing through 1 to 2 stacked recurrent hidden layers (hidden state vectors of 128 or 256) to generate text semantic feature vectors. Image and video frames are encoded using a convolutional neural network with 3 to 5 convolutional layers, each with 32, 64, or 128 kernels, passing through pooling layers and fully connected layers to generate image / video feature vectors.

[0067] Structured field feature vectors, time series feature vectors, text semantic feature vectors, and image / video feature vectors are concatenated or weighted in a fixed order (i.e., the pre-defined concept order in the concept dictionary) at the fusion layer to form a unified representation vector with dimensions of 256 or 512. This unified representation vector is input to the concept output layer. Output nodes are set according to the concept dictionary order, with each node corresponding to a concept name. Output status concepts are "true" or "false," level concepts are preset levels, and category concepts are preset category names, thus generating a concept value vector corresponding to the original data.

[0068] The back-end task prediction model consists of a fully connected neural network, receiving the concept value vectors output from the front-end concept prediction model as input. The concept value vectors are input into the fully connected neural network, which has 1 to 2 hidden layers, each containing 64 or 128 neurons, used to combine the features of the concept value vectors. The output of the hidden layers is input into the output layer of the task prediction model; the number of output nodes equals the number of task label categories. Normalization is performed to obtain the probability distribution of each task label, and the label with the highest probability is selected as the final task label. The output task labels correspond one-to-one with the input record numbers, ensuring a clear input-output correspondence between the concept value vectors and task labels.

[0069] S4.2: Read the raw data, concept value vectors, and task labels from the concept-complete training set one by one in the order of records, while maintaining the correspondence between the raw data and the concept value vectors, concept evidence, source credibility, and task labels. Organize the raw data into a unified input format for the front-end concept prediction model, where structured fields are arranged in a fixed order, time series are arranged in the order of collection time, text content is arranged in text order, and image and video frames are arranged in frame number order.

[0070] The processed raw data is input into the front-end concept prediction model to obtain concept prediction results. Concept value vectors are extracted from the concept prediction results, and corresponding concept evidence is extracted from the concept completeness training set. Based on the concept field mapping relationship, the concept value vectors and concept evidence are matched item by item, and inconsistencies between the concept value vectors and concept evidence are identified and marked to generate consistency comparison results.

[0071] During training, a loss function is calculated based on the difference between the concept prediction result and the concept value vector. Simultaneously, the consistency comparison result is incorporated as a weight into the loss function, giving inconsistent records a higher priority for correction during parameter updates. For example, when calculating the loss function, the loss weight for consistent records is set to 1, and the loss weight for inconsistent records is set to 2. Specific settings can be customized according to requirements. This makes the inconsistent record have a greater impact on the total loss, and the front-end concept prediction model will prioritize correcting inconsistent concept prediction results during backpropagation parameter adjustments. By adjusting the parameters of the front-end concept prediction model through the backpropagation algorithm, the model's output concept prediction results gradually approach the concept value vector and remain consistent with the concept evidence, thereby ensuring that the front-end concept prediction model can accurately learn concept expressions supported by evidence.

[0072] The concept value vector output by the front-end concept prediction model is input into the back-end task prediction model to obtain the task label prediction result. The task label prediction result is compared with the corresponding task labels in the training set, the loss function is calculated, and the parameters of the back-end task prediction model are adjusted according to the difference to ensure that the task label prediction result accurately matches the task labels in the training set.

[0073] The criteria for completing the two-stage training are as follows: the consistency between the output of the front-end concept prediction model and the concept value vector and concept evidence reaches a preset consistency threshold; and the consistency between the output task label of the back-end task prediction model and the task label of the training set reaches a preset accuracy threshold. Once these criteria are met, the training of the front-end concept prediction model and the back-end task prediction model is complete, and they can be used for subsequent concept prediction, consistency comparison, and task label output, forming interpretable prediction results.

[0074] It should also be noted that the consistency threshold is set based on statistical analysis of the consistency between concept prediction results and concept evidence. A level that ensures most concept prediction results match the concept evidence is chosen as the consistency threshold; for example, 90% consistency means that at least 90% of the concept prediction results on the training set are consistent with the concept evidence. This ensures that the concept values ​​output by the front-end concept prediction model are reliable and supported by clear evidence, thereby enhancing interpretability and stability. The accuracy threshold is set based on the statistical accuracy of task labels on the concept-complete training set; for example, 95% accuracy is chosen to ensure that the back-end task prediction model can accurately output task labels, reliably perform task judgments after completing concept prediction, and maintain the controllability and stability of the training process.

[0075] S4.3: After training, the trained front-end concept prediction model and back-end task prediction model are used to predict the concept-complete training set or newly input raw data: The raw data is organized into a unified input format for the front-end concept prediction model and input into the trained front-end concept prediction model to obtain the concept prediction results (concept value vectors). Based on the concept field mapping relationship, the concept prediction results are compared with the concept evidence for consistency, inconsistent records are marked, and abnormal concept prediction results are corrected as needed to ensure that the concept values ​​are consistent with the concept evidence.

[0076] After calibration, the concept value vectors output by the front-end concept prediction model are input into the back-end task prediction model to generate corresponding task label results. Finally, the original data, calibrated concept prediction results, concept evidence, task labels, and consistency markers are combined to form interpretable prediction results. This ensures that each prediction record contains a task label and can be traced back to the concept value and corresponding evidence supporting that label, thereby achieving interpretable output after training.

[0077] This embodiment also provides a data processing system that can interpret AI model training, including: The preprocessing module acquires the raw data and merges it. At the same time, it combines the concept dictionary to perform preprocessing and cross-source consistency comparison on the merged raw data to generate a concept candidate data pool. The encoding module performs unified vector encoding, concept assignment, concept evidence generation, and consistency verification on each piece of raw data in the concept candidate data pool to form concept bottleneck data. The supplementation module establishes confirmed concept clusters based on the concept bottleneck data, and determines the scarce regions of concept combinations through the confirmed concept clusters and density adaptive clustering algorithm. It then selects data located in the scarce regions of concept combinations that meet the conditions of concept evidence completeness and source credibility for supplementation, and generates a concept-complete training set. The training module uses a concept-complete training set to perform two-stage training on the front-end concept prediction model and the back-end task prediction model. This allows the front-end concept prediction model to output concept prediction results and compare them with concept evidence for consistency. The back-end task prediction model then outputs the final task label based on the concept prediction results. Simultaneously, it corrects concept prediction results with abnormal consistency comparison results and generates interpretable prediction results.

[0078] This embodiment also provides a computer device applicable to the data processing method for interpretable AI model training, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the data processing method for interpretable AI model training as proposed in the above embodiment.

[0079] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0080] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the data processing method for training an interpretable AI model as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0081] In summary, this invention achieves the following: By performing unified vector encoding, concept assignment, concept evidence generation, and consistency verification on the original data in the concept candidate data pool, the training data possesses explicit concept structure and concept evidence constraints before entering model training. Furthermore, by eliminating data records where concepts and evidence are inconsistent, the model training process avoids over-reliance on implicit features, enhancing the correspondence and traceability between concept values ​​and original data, thereby improving the stability of the interpretation path. Simultaneously, by establishing confirmed concept clusters, identifying scarce regions of concept combinations, and filtering and labeling samples based on the completeness of concept evidence and the reliability of the source, data filling for gaps in concept combination coverage is achieved, enhancing the training set in terms of concept coverage, evidence completeness, and source reliability, thereby improving the interpretability and completeness of the interpretation chain in subsequent model training.

[0082] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A data processing method for interpretable AI model training, characterized in that: include, Obtain the raw data and merge the raw data. At the same time, combine the concept dictionary to perform preprocessing and cross-source consistency comparison on the merged raw data to generate a concept candidate data pool. Each piece of raw data in the concept candidate data pool is subjected to unified vector encoding, concept assignment, concept evidence generation, and consistency verification to form concept bottleneck data. Based on the conceptual bottleneck data, confirmed concept clusters are established, and the concept combination sparse regions are determined by the confirmed concept clusters and density adaptive clustering algorithm. Data located in the concept combination sparse regions and meeting the conditions of concept evidence completeness and source credibility are selected for supplementary labeling to generate a concept complete training set. The concept prediction model and the task prediction model are trained in two stages using a concept-complete training set. The concept prediction model outputs concept prediction results and compares them with concept evidence for consistency. The task prediction model outputs the final task label based on the concept prediction results. At the same time, the concept prediction results with abnormal consistency comparison results are corrected to generate interpretable prediction results.

2. The data processing method for interpretable AI model training as described in claim 1, characterized in that: The process of merging raw data refers to accessing raw data from business databases, log streams, sensor streams, text records, and image / video streams, extracting source identifiers, collection times, and business entity identifiers during the access process, and then performing time-window merging on the raw data according to the collection time and business entity identifiers to form a merged data set.

3. The data processing method for interpretable AI model training as described in claim 1, characterized in that: The generation of the candidate data pool for concepts specifically includes: Based on the pre-defined concept field mapping relationship in the concept dictionary, the merged data set is subjected to field alignment, missing data filling, anomaly pruning, and format unification to form a preprocessed data set. Perform cross-source consistency comparison on the preprocessed data set, identify data conflict records corresponding to business entity identifiers and complete conflict resolution, retain data records that pass the consistency verification, and summarize them to form a conceptual candidate data pool.

4. The data processing method for interpretable AI model training as described in claim 1, characterized in that: The bottleneck data that forms the concept specifically includes: Extract each piece of raw data from the concept candidate data pool, and perform unified vector encoding on each piece of raw data to form the corresponding basic feature vector; Based on the pre-defined concept field mapping relationship in the concept dictionary, the concept value is determined according to the basic feature vector and the concept field mapping relationship in the concept assignment process, thus forming a concept vector; Extract field segments, time segments, and region segments corresponding to concept vectors from the raw data, establish the correspondence between concept vectors and field segments, time segments, and region segments, and form concept evidence; Consistency checks are performed on concept vectors and concept evidence. Inconsistent records between concept vectors and concept evidence are identified and removed. Basic feature vectors, concept vectors, and concept evidence that pass the consistency check are retained and aggregated to form concept bottleneck data.

5. The data processing method for interpretable AI model training as described in claim 1, characterized in that: The determination of the scarce region of concept combination specifically refers to: Extract basic representation vectors and concept vectors from the conceptual bottleneck data, and calculate the Euclidean distance between the basic representation vectors and concept vectors to form a distance matrix; Calculate the neighborhood density value for each basic representation vector based on the distance matrix, and determine the basic representation vectors with neighborhood density values ​​greater than a preset density threshold as the initial cluster centers; The remaining basic representation vectors are assigned to the initial cluster centers with the closest Euclidean distance in descending order of neighborhood density values, and the corresponding cluster boundaries are dynamically adjusted according to the neighborhood density values ​​during the assignment process. After the basic representation vectors are assigned, clusters whose Euclidean distance between cluster centers is less than a preset distance threshold and whose neighborhood density difference is less than a preset density difference threshold are merged to form multiple confirmed concept clusters. The number of data corresponding to each concept vector is counted from multiple confirmed concept clusters. Concept combinations with a number of data below a preset threshold are identified as scarce concept combinations, and the distribution range of the basic representation vectors corresponding to scarce concept combinations is identified as the scarce region of concept combinations.

6. The data processing method for interpretable AI model training as described in claim 1, characterized in that: The generated conceptually complete training set is specifically as follows: Extract data located in the scarce region of concept combinations, and read the conceptual evidence and source credibility corresponding to each data point; Perform conceptual evidence integrity and source credibility assessments on data located in areas of scarce conceptual combinations, and select data with complete conceptual evidence and source credibility reaching a preset credibility threshold to form a set of data to be supplemented; Extract concept vectors and task label placeholder states from the dataset to be supplemented; perform supplementation labeling on the unconfirmed concept values ​​in the concept vectors and the task labels corresponding to the task label placeholder states to form a supplemented dataset. The supplementary data set is merged with the data in the concept bottleneck data that have complete concept values ​​and complete task labels to form a concept-complete training set.

7. The data processing method for interpretable AI model training as described in claim 1, characterized in that: The output concept prediction result is specifically as follows: Extract the raw data that has been merged, preprocessed and labeled with concepts from the concept-complete training set and input it into the front-end concept prediction model; The front-end concept prediction model performs concept reasoning on the original data, outputs the concept value results corresponding to the original data, and uses them as the concept prediction results.

8. The data processing method for interpretable AI model training as described in claim 1, characterized in that: The consistency comparison is performed as follows: Extract concept value results from the concept prediction results, and extract concept evidence corresponding to the concept value results from the concept completeness training set; Based on the concept field mapping relationship, perform item-by-item matching and verification of concept value results and concept evidence, identify and mark inconsistent records between concept value results and concept evidence, and output consistency comparison results.

9. The data processing method for interpretable AI model training as described in claim 1, characterized in that: The final output task label is specifically as follows: Input the concept value results from the concept prediction results into the subsequent task prediction model; The control back-end task prediction model performs task inference on the concept value results, outputs the task label results corresponding to the concept value results, and forms the final task label.

10. A data processing system for interpretable AI model training, based on the data processing method for interpretable AI model training according to any one of claims 1 to 9, characterized in that: include, The preprocessing module acquires the raw data and merges it. At the same time, it combines the concept dictionary to perform preprocessing and cross-source consistency comparison on the merged raw data to generate a concept candidate data pool. The encoding module performs unified vector encoding, concept assignment, concept evidence generation, and consistency verification on each piece of raw data in the concept candidate data pool to form concept bottleneck data. The supplementation module establishes confirmed concept clusters based on the concept bottleneck data, and determines the scarce regions of concept combinations through the confirmed concept clusters and density adaptive clustering algorithm. It then selects data located in the scarce regions of concept combinations that meet the conditions of concept evidence completeness and source credibility for supplementation, and generates a concept-complete training set. The training module uses a concept-complete training set to perform two-stage training on the front-end concept prediction model and the back-end task prediction model. This allows the front-end concept prediction model to output concept prediction results and compare them with concept evidence for consistency. The back-end task prediction model then outputs the final task label based on the concept prediction results. Simultaneously, it corrects concept prediction results with abnormal consistency comparison results and generates interpretable prediction results.

Images