Data processing method and device, equipment, medium and program product

Abstract

The invention provides a data processing method which can be applied to the technical field of artificial intelligence. The data processing method comprises the following steps: acquiring original training data, wherein the types of the original training data comprise a numeric type, a text type and an image type; constructing a topological space corresponding to the original training data by using an adaptive topology measurement algorithm; recognizing topological features in the topological space by using a dynamic threshold recognition algorithm, wherein the topological features comprise holes and rings; and based on the data type and the topological features of the original training data, repairing the topological features to obtain repaired data. The invention further provides a data processing device and equipment, a storage medium and a program product.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more specifically to a data processing method, apparatus, device, medium, and program product. Background Technology

[0002] With the rapid advancement of artificial intelligence (AI) technology, its application in the financial sector is becoming increasingly widespread and in-depth, especially in the critical business scenario of anti-money laundering (AML). AI technology is gradually becoming an important tool for improving regulatory effectiveness and preventing financial risks. However, the effectiveness of AI models is highly dependent on high-quality data input, making the preprocessing of model training data crucial for improving algorithm performance.

[0003] In anti-money laundering (AML) operations, the complexity and diversity of raw data constitute the primary challenge for data preprocessing. This data encompasses not only numerical information, such as transaction amounts and account balances—indicators directly reflecting economic activity—but also rich textual data, such as customer identification information and transaction notes, which often contain crucial clues about the transaction's background and purpose. Furthermore, with the acceleration of digitalization, image-based data, such as scanned copies of ID cards and images of receipts, is gradually becoming an indispensable part of AML analysis. These diverse data types collectively constitute the complex data landscape of AML analysis.

[0004] However, traditional technologies often struggle to accurately construct a topological space that truly reflects the original data structure and inherent relationships when processing such complex data. This prevents the full uncovering of potential connections and patterns within the data. Traditional technologies are particularly limited in revealing the intricate relationships between transaction behavior, customer characteristics, and risk factors. This limitation directly impacts the identification capabilities of anti-money laundering models, making it difficult for them to make accurate judgments when faced with potentially suspicious transactions, thus increasing the risk of false positives and false negatives.

[0005] More seriously, the shortcomings of traditional technologies severely hinder users' in-depth understanding of data correlations. In anti-money laundering work, a deep understanding of data is fundamental to developing effective monitoring strategies and identifying complex money laundering patterns. However, because traditional technologies cannot effectively capture the hidden correlations between data, business personnel often only see superficial, isolated information when analyzing data, making it difficult to discern the underlying logic and potential risks. This can not only lead to biased business judgments but also potentially expose financial institutions to compliance risks unknowingly, and even legal sanctions.

[0006] The limitations of traditional technologies become particularly apparent when dealing with complex transaction networks. Special structures such as "hole structures" and "ring structures" within these networks are common tools used by money launderers to conceal fund flows and evade regulation. However, traditional technologies often struggle to effectively identify these hidden connections, making it difficult for anti-money laundering systems to comprehensively and accurately identify complex money laundering patterns.

[0007] Therefore, exploring and applying more advanced data preprocessing technologies and algorithm models to address the complex data challenges in anti-money laundering operations has become an urgent issue for financial institutions. Summary of the Invention

[0008] In view of the above problems, this application provides data processing methods, apparatus, devices, media and program products that improve the accuracy and applicability of data preprocessing.

[0009] According to a first aspect of this application, a data processing method is provided, comprising: acquiring raw training data, the types of which include numerical, textual, and image data; constructing a topological space corresponding to the raw training data using an adaptive topological metric algorithm; identifying topological features in the topological space using a dynamic threshold recognition algorithm, the topological features including holes and loops; and repairing the topological features based on the data type and topological features of the raw training data to obtain repaired data.

[0010] According to an embodiment of this application, an adaptive topology metric algorithm is used to construct a topology space corresponding to the original training data, including: calculating the information entropy of each feature attribute based on the probability distribution of all possible values ​​of the feature attribute, wherein the original training data includes n data points, each data point has m feature attributes, and n and m are positive integers; constructing a feature difference matrix based on the information entropy and the feature similarity between each data point; and using a nonlinear mapping function to transform the feature difference matrix into a topology metric matrix to obtain the topology space, which can reflect the neighborhood relationship and distance metric between data points.

[0011] According to an embodiment of this application, if a feature attribute has missing values, the information entropy of each feature attribute is calculated, including: calculating the information entropy using an interpolation estimation method, wherein calculating the information entropy using the interpolation estimation method includes: dividing the data points with complete feature attributes in the original training data into q subgroups based on the probability distribution of all possible values ​​of the feature attribute, wherein the values ​​of the data points in each subgroup on the feature attribute with missing values ​​are within a preset deviation range, and q is a positive integer greater than or equal to 2; calculating the average value of the data points in the subgroup on the feature attribute with missing values ​​as the representative value of the subgroup; assigning the data points with missing values ​​to the corresponding subgroups according to the correlation of feature attributes, and using the representative value of the assigned subgroup as the substitute value for the missing value; and calculating the information entropy of the feature attribute with missing values ​​based on the substitute value.

[0012] According to an embodiment of this application, if the value of the feature difference matrix is ​​greater than a preset value, a nonlinear mapping function is used to convert the feature difference matrix into a topological metric matrix, including: converting the nonlinear mapping function into a logic function, wherein the output value of the logic function is in the range of (0, 1).

[0013] According to an embodiment of this application, identifying topological features in a topological space using a dynamic threshold recognition algorithm includes: using a dynamic threshold recognition algorithm to calculate topological invariants of the topological space; setting a basic threshold mean and a threshold standard deviation based on the size of the original training data and the expected frequency of occurrence of the topological features, and calculating a dynamic threshold; comparing the topological invariants with the dynamic threshold to obtain the topological features in the topological space, including the position, size, and shape of the topological features.

[0014] According to an embodiment of this application, a dynamic threshold recognition algorithm is used to calculate the topological invariants of the topological space, including: dividing the topological space into p levels according to the geometric structure or data distribution characteristics of the topological space, where p is a positive integer greater than or equal to 2; calculating sub-topological invariants for each level to obtain the local topological features of each level; and obtaining the topological invariants of the topological space based on the relationship between each level and the sub-topological invariants of each level.

[0015] According to an embodiment of this application, the distribution of the original training data exhibits seasonal or periodic changes. Setting a base threshold mean and a threshold standard deviation includes: setting different base threshold mean and threshold standard deviations according to different stages of the season or cycle. Specifically, during the peak period of the original training data distribution, the base threshold mean is increased and the threshold standard deviation is decreased; during the trough period of the original training data distribution, the base threshold mean is decreased and the threshold standard deviation is increased.

[0016] According to embodiments of this application, if the number of holes in the topological features exceeds a preset number, it is determined that the original training data has missing values ​​or is discontinuous. The topological features are then repaired, including: using a topology-guided hybrid method to repair the topological features. For numerical data, the filling direction is determined based on the topological relationship of the data points around the holes, and a weighted interpolation formula is used to fill along the determined filling direction to repair the topological features. For textual data, the text is supplemented or connected based on the contextual semantic relationship and the position of the text in the topological space to repair the topological features. In this case, if a text fragment has a topological connection relationship with other fragments in the topological space and there are missing parts in the contextual semantics, the supplementary content is inferred based on the connection relationship and contextual semantic logic.

[0017] According to an embodiment of this application, the topological feature is a ring structure. The original training data is determined to have a specific hierarchical structure or clustering characteristics. The topological feature is repaired by: based on the topological distance between the data point and other data points within the ring structure, using a topological weighted feature extraction algorithm to calculate the topological centrality of each data point in the ring structure; based on the topological centrality, weighting each feature attribute of the data point to obtain weighted feature attributes; and extracting features reflecting the characteristics of the ring structure from the weighted feature attributes as new feature extraction objects.

[0018] According to an embodiment of this application, determining the filling direction based on the topological relationship of data points around the hole includes: decomposing the topological structure around the hole to obtain multiple basic topological units; determining the contribution weight of each basic topological unit to the filling of data points inside the hole based on the relative positional relationship between each basic topological unit and the hole; and determining the filling direction based on the contribution weight of each basic topological unit to the filling of data points inside the hole.

[0019] According to an embodiment of this application, before constructing the topological space corresponding to the original training data using the adaptive topological metric algorithm, the method further includes: preprocessing the original training data, wherein preprocessing the original training data includes: removing format errors, illegal characters, and irrelevant data fragments from the original training data to obtain cleaned data; normalizing the numerical data points in the cleaned data to map the numerical data points to the range of 0-1; and converting the characters in the text data into numerical values ​​before normalizing them.

[0020] A second aspect of this application provides a data processing apparatus, comprising: a data acquisition module for acquiring raw training data, the types of which include numerical, textual, and image data; a space construction module for constructing a topological space corresponding to the raw training data using an adaptive topological metric algorithm; a feature recognition module for recognizing topological features in the topological space using a dynamic threshold recognition algorithm, the topological features including holes and loops; and a data repair module for repairing the topological features based on the data type and topological features of the raw training data to obtain repaired data.

[0021] A third aspect of this application provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0022] A fourth aspect of this application also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0023] The fifth aspect of this application also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method. Attached Figure Description

[0024] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0025] Figure 1 The illustrations depict application scenarios of data processing methods, apparatuses, devices, media, and program products according to embodiments of this application.

[0026] Figure 2 A flowchart illustrating a data processing method according to an embodiment of this application is shown schematically.

[0027] Figure 3 A flowchart illustrating another data processing method according to an embodiment of this application is shown schematically;

[0028] Figure 4 This illustration shows a flowchart of constructing the topology space corresponding to the original training data using an adaptive topology metric algorithm according to an embodiment of this application.

[0029] Figure 5 A flowchart illustrating the identification of topological features in a topological space using a dynamic threshold recognition algorithm according to an embodiment of this application is shown.

[0030] Figure 6A schematic block diagram of a data processing apparatus according to an embodiment of this application is shown; and

[0031] Figure 7 A block diagram schematically illustrates an electronic device suitable for implementing a data processing method according to an embodiment of this application. Detailed Implementation

[0032] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0033] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0034] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0035] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0036] In the technical solution of this application, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0037] In scenarios involving automated decision-making using personal information, the methods, devices, and systems provided in this application all offer users corresponding entry points for choosing to agree to or reject the automated decision-making results. If the user chooses to reject, the process proceeds to the expert decision-making stage. Here, "automated decision-making" refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests, or economic, health, and credit status through computer programs, and then making a decision. Here, "expert decision-making" refers to the activity of making decisions by personnel who specialize in a particular field, possess specialized experience, knowledge, and skills, and have reached a certain level of professional expertise.

[0038] Embodiments of this application provide a data processing method, including: acquiring raw training data, the types of which include numerical, text, and image data; constructing a topological space corresponding to the raw training data using an adaptive topological metric algorithm; identifying topological features in the topological space using a dynamic threshold recognition algorithm, the topological features including holes and loops; and repairing the topological features based on the data type and topological features of the raw training data to obtain repaired data.

[0039] Figure 1 The diagram illustrates an application scenario of the data processing method according to an embodiment of this application.

[0040] like Figure 1 As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0041] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0042] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0043] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0044] It should be noted that the data processing method provided in the embodiments of this application can generally be executed by server 105. Correspondingly, the data processing device provided in the embodiments of this application can generally be located in server 105. The data processing method provided in the embodiments of this application can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the data processing device provided in the embodiments of this application can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0045] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0046] The following will be based on Figure 1 The described scene, through Figures 2-5 The data processing method according to the embodiments of this application will be described in detail.

[0047] Figure 2 A flowchart illustrating a data processing method according to an embodiment of this application is shown.

[0048] like Figure 2 As shown, the data processing in this embodiment includes operations S210 to S240.

[0049] In operation S210, the raw training data is obtained. The types of raw training data include numerical, text, and image data.

[0050] In the embodiments of this application, numerical data includes transaction amounts and account balances. These data are presented as specific numbers, intuitively reflecting the scale of financial transactions and the status of account funds. Textual data includes customer information and transaction remarks. Customer information includes textual descriptions such as name and contact information, while transaction remarks record supplementary explanations related to the transaction. Both are stored in text form and carry a large amount of unstructured key information. Image data includes images of ID cards and receipts. ID cards contain image information such as portraits and text, while receipt images record the graphical content of transaction receipts, serving as important evidence for verifying the authenticity of transactions and customer identities. When processing customer transaction records, numerical data such as transaction amounts and account balances can be directly used for statistical analysis. Age, gender, and other components of customer information can be converted into numerical codes for analysis, and keywords in transaction remarks can also be extracted and quantified. ID card images can have key information extracted and converted into numerical features using image recognition technology, and receipt images can similarly extract and quantify key information such as amount and date for in-depth analysis.

[0051] In embodiments of this application, after obtaining the original training data, operation S250 may also be included, see [link to relevant documentation]. Figure 3 .

[0052] The S250 is used to preprocess the raw training data.

[0053] In embodiments of this application, preprocessing the original training data may include removing format errors (such as missing fields or abnormal symbols in transaction records), illegal characters (such as garbled text or invalid input), and irrelevant data fragments (such as irrelevant notes or invalid images) from the original training data to obtain cleaned data; normalizing the numerical data points in the cleaned data to map the numerical data points to the range of 0-1; and converting the characters in the text data into numerical values ​​before normalizing them.

[0054] For example, in customer information, besides information that is difficult to quantify directly, such as name and contact information, age and gender can be converted into numerical codes for analysis. For instance, males can be coded as 1 and females as 0; age can be directly represented by a specific numerical value. Suppose that when analyzing a certain type of high-risk transaction, it is found that the proportion of female customers participating in a specific age group (such as 30-40 years old) is abnormally high, which is significantly different from the normal proportion. This may indicate that female customers in this age group pose a higher risk of suspicious transactions.

[0055] For example, natural language processing (NLP) technology can be used to extract keywords from transaction notes, such as "large amount," "frequent," and "out-of-town," and each keyword can be assigned a weight value to indicate its relevance to suspicious transactions. For instance, the keyword "large amount" might have a weight of 0.8, "frequent" 0.7, and "out-of-town" 0.6. When multiple keywords appear in a transaction note, the sum of these keyword weight values ​​is calculated as a numerical feature of the transaction note. If the sum of keyword weights in a transaction note is high, it indicates that the transaction may have a higher risk of being suspicious.

[0056] By quantifying textual data, unstructured information can be transformed into quantifiable features, which can then be incorporated into a model for comprehensive analysis. This allows for a more thorough uncovering of potential risk factors behind transactions and improves the ability to identify suspicious transactions.

[0057] In the embodiments of this application, the normalization formula is as follows:

[0058] (1)

[0059] Among them, X i For the i-th data point in the dataset, which is the original value that needs to be normalized, the normalization (X) is... i (x) represents data point X i The normalized value of X min X is the minimum value in the dataset. max This represents the maximum value in the dataset.

[0060] For example, taking the dataset [80,90,70,85,95] as an example, X min =70, X max =95, for X i =80, according to the formula, normalized(80)= .

[0061] By converting character data to numerical values ​​and normalizing it, these data can be used together with numerical data for unified analysis and processing within the model.

[0062] Differentiated preliminary format checks and basic cleanup operation standards were adopted for different types of data involved in anti-money laundering operations to ensure the accuracy and applicability of data preprocessing.

[0063] In operation S220, the adaptive topology metric algorithm is used to construct the topology space corresponding to the original training data.

[0064] Figure 4The flowchart illustrating the process of constructing the topology space corresponding to the original training data using an adaptive topology metric algorithm according to an embodiment of this application is shown.

[0065] like Figure 4 As shown, the topology space corresponding to the original training data constructed using the adaptive topology metric algorithm in this embodiment includes operations S221 to S223.

[0066] In operation S221, the information entropy of each feature attribute is calculated based on the probability distribution of all possible values ​​of the feature attribute. The original training data includes n data points, each data point has m feature attributes, and n and m are positive integers.

[0067] Information entropy is a metric used to measure the uncertainty in a dataset. By calculating the information entropy of each feature attribute, we can understand the degree to which that feature contributes to the classification or prediction task of the dataset. The higher the information entropy, the greater the uncertainty of that feature, and the greater its potential contribution to the classification or prediction task. For a dataset containing n data points, each with m feature attributes, the dataset can be represented as an n×m matrix, where each row represents a data point and each column represents a feature attribute. Information entropy H(X) measures the uncertainty of a feature attribute X, and is defined as:

[0068] (2)

[0069] Where p(x) i ) is the feature attribute X, which takes the value x. i The probability is given by k, where k is the number of all possible values ​​for the feature attribute X.

[0070] When missing values ​​exist in a feature attribute, directly calculating the information entropy may lead to inaccurate results. Therefore, an interpolation estimation method is used to handle missing values ​​and calculate the information entropy of each feature attribute. Specifically, this involves: based on the probability distribution of all possible values ​​of the feature attribute, dividing the data points with complete feature attributes in the original training data into q subgroups, where the values ​​of the data points in each subgroup for the feature attribute with missing values ​​are within a preset deviation range, and q is a positive integer greater than or equal to 2; calculating the average value of the data points in the subgroup for the feature attribute with missing values, as the representative value of the subgroup; assigning the data points with missing values ​​to the corresponding subgroups based on the correlation of the feature attributes, and using the representative value of the assigned subgroup as the substitute value for the missing value; and calculating the information entropy of the feature attribute with missing values ​​based on the substitute value.

[0071] By calculating the information entropy of features, we can quantify the uncertainty contribution of each feature to data classification or prediction tasks. Features with high information entropy tend to contain more discriminative information, and their similarity is given higher weight when constructing the difference matrix, thus highlighting key patterns in the data. For example, in customer segmentation tasks, if the information entropy of the "purchase frequency" feature is higher than that of "registration time," the difference matrix will strengthen the representation of differences in purchase frequency, helping to identify customer groups with different consumption behaviors.

[0072] In operation S222, a feature difference matrix is ​​constructed based on information entropy and feature similarity between various data points.

[0073] In the embodiments of this application, the original training data set is assumed to be... ,in Let d be the number of data points, and for each data point d i It has m characteristic attributes, which can be represented as d i =(f i1 ,f i2 ,…,f im The core of the adaptive topology metric algorithm lies in dynamically determining the topology metric based on the feature similarity between data points, and calculating the feature difference matrix M between data points, where elements M ij Represents data point d i and d j The degree of difference in characteristics between them is calculated using the following formula:

[0074] (3)

[0075] in, For data points Corresponding feature attributes The value, To adjust the parameter, the value ranges from 1 to 3. The specific value is determined based on the data distribution characteristics and subsequent processing requirements. When the data distribution is relatively uniform, a value of 1 can be used, W. k The weight of the k-th feature attribute is calculated as follows:

[0076] (4)

[0077] Where P is the number of values ​​for the feature attribute, P kp This represents the probability that the k-th feature attribute in the dataset takes the p-th value.

[0078] Feature similarity matrices (such as cosine similarity and Euclidean distance) can capture the local relationships between data points in the feature space. Combined with information entropy weighting, the difference matrix reflects both the importance of global features and preserves local similarity information, revealing complex clustering structures in the data. For example, in image recognition, even if two images have similar overall color distributions, the difference matrix can still accurately distinguish them if certain high-information-entropy features (such as texture) differ significantly.

[0079] In operation S223, a nonlinear mapping function is used to transform the feature difference matrix into a topological metric matrix, resulting in a topological space. The topological space can reflect the neighborhood relationship and distance metric between data points.

[0080] In embodiments of this application, a topological metric matrix is ​​used to quantify neighborhood relationships and distance metrics between data points in a topological space. Elements in the matrix represent the topological similarity or proximity between data points, typically ranging from (0,1). The topological space is a space constructed using the topological metric matrix, where the relationships between each data point are represented by the values ​​of the topological metric matrix.

[0081] By transforming the feature difference matrix into a topological metric matrix, the neighborhood relationships and distance metrics between data points can be better reflected in the topological space. This not only facilitates data visualization and analysis but also provides more intuitive input for subsequent tasks such as clustering and classification. Using a nonlinear mapping function, difference information can be effectively converted into metrics suitable for topological analysis, helping to reveal the underlying structures and patterns in the data.

[0082] In the embodiments of this application, when the value of the feature difference matrix is ​​greater than a preset value, a nonlinear mapping function is used to convert the feature difference matrix into a topological metric matrix, including: converting the nonlinear mapping function into a logic function, wherein the output value of the logic function is in the range of (0, 1).

[0083] For example, in the topology space construction step, when the nonlinear mapping function F transforms the feature difference matrix M into the topology metric matrix T, if the feature difference of the data is large, i.e., M... ij When the value of is large, to avoid the topological metric being too sensitive and causing excessive subdivision or distortion in the construction of the topological space, the function F is locally adjusted. Specifically, the adjustment method is as follows: when M... ij When >γ, modify function F to F' (M ij )=1 / (1+e^(-β×(M ij / θ))), where θ is an adjustment parameter greater than 1, used to slow down the rate of change of the function when the difference is large. Through this local adjustment, a reasonable topological space can still be constructed even when the data features are very different. This ensures that the topological space can reflect the real structure and relationship of the data, and will not have an unreasonable topological structure due to oversensitivity, thus ensuring the smooth progress of subsequent processing.

[0084] return Figure 2 In operation S230, a dynamic threshold identification algorithm is used to identify topological features in the topological space, including holes and loops. In the topological space, the geometric structural features of data are typically related to the neighborhood relationships and connectivity of data points. Loop structures can reveal money laundering patterns such as cyclical transactions or fund repatriation, while hole structures can reflect abnormal behavior such as concealed fund flow paths or missing key nodes. This method can more comprehensively consider the complex relationships and network structures between transaction data, thereby significantly improving the accuracy and efficiency of anti-money laundering data preprocessing.

[0085] Figure 5 The flowchart illustrating the use of a dynamic threshold recognition algorithm to identify topological features in a topological space according to an embodiment of this application is shown.

[0086] like Figure 5 As shown, the method of using a dynamic threshold recognition algorithm to identify topological features in the topological space in this embodiment includes operations S231 to S233.

[0087] In operation S231, a dynamic threshold recognition algorithm is used to calculate the topological invariants of the topological space.

[0088] Topological invariants are properties of a topological space that do not change with continuous deformation, such as the number of connected components, Eulerian characteristic, and homology groups. Calculating topological invariants of a topological space using a dynamic thresholding algorithm can adaptively adjust the threshold to suit the characteristics of different data and the expected frequency of topological features. This method improves the flexibility and accuracy of topological analysis and contributes to a more comprehensive understanding of the data's topological structure.

[0089] In the embodiments of this application, the dynamic threshold recognition algorithm employs a hierarchical calculation method to address the geometric structure and data distribution characteristics of complex topological spaces. Operation S231 specifically includes: dividing the topological space into p levels based on its geometric structure or data distribution characteristics, where p is a positive integer greater than or equal to 2. For example, in a trading network, the network can be divided into a high-frequency trading layer, a low-frequency trading layer, or a specific account group layer based on features such as transaction amount, transaction frequency, or account type. Sub-topological invariants (such as the Betty number βk) are calculated for each level to obtain the local topological features of each level. For example, β0 represents the number of connected components in each level, reflecting the distribution of isolated accounts or isolated trading groups in the trading network; β1 represents the number of loop structures in each level, revealing patterns of cyclical trading or fund reflux; β2 represents the number of hole structures in each level, identifying abnormal behaviors such as hidden fund flow paths or missing key nodes. Based on the relationships between the levels (such as fund flows between accounts at different levels in the trading network) and the sub-topological invariants of each level, the topological invariants of the topological space are obtained. For example, by using a weighted approach, topological invariants at each level can be integrated into global topological invariants for the entire topological space. The weighting formula is as follows:

[0090] (5)

[0091] in, Let L be the topological invariant of the topological space; L is the number of levels in the topological space. Let be the weight of the i-th level. The weight can be dynamically adjusted according to the importance of the level or the characteristics of the data distribution. Let be the sub-topological invariant of the i-th level.

[0092] By dividing the topological space into multiple levels and calculating sub-topological invariants separately, the local characteristics of the topological space can be analyzed in greater detail, thereby improving the accuracy of topological analysis. The calculation of sub-topological invariants (such as the Betti number) can reveal complex structures in the topological space, such as loop structures and hole structures. These structures may correspond to abnormal behaviors in trading networks, such as circular transactions, fund reflux, or concealed fund flow paths. The weights in the weighted summation formula can be dynamically adjusted according to the importance of the levels or the characteristics of data distribution. This allows global topological invariants to more flexibly reflect the overall characteristics of the topological space, improving the adaptability and flexibility of the analysis. Through hierarchical partitioning and sub-topological invariant calculation, multi-level analysis can be supported, allowing for both a holistic grasp of the characteristics of the topological space and in-depth analysis at each level, providing more comprehensive support for decision-making.

[0093] In operation S232, based on the size of the original training data and the expected frequency of occurrence of topological features, the mean of the basic threshold and the standard deviation of the threshold are set, and the dynamic threshold is calculated.

[0094] In the embodiments of this application, the data size is assumed to be... That is, the total number of data points in the dataset is The expected frequency of occurrence of a certain topological feature is The dynamic threshold represents the proportion of the expected occurrence of this topological feature in the dataset. The calculation formula is:

[0095] (6)

[0096] in, It is the mean of the base threshold, determined based on historical data or prior knowledge, reflecting the typical threshold under conditions without specific topological feature expectations. The standard deviation of the threshold reflects the range of fluctuation of the threshold under different data or conditions.

[0097] Dynamic threshold TH k Taking into account data size, expected frequency of topological features, and historical data or prior knowledge, the dynamic threshold TH can adaptively adjust based on data characteristics. When the data size N increases or the expected frequency q of the topological features decreases, the dynamic threshold TH... k It will increase accordingly to adapt to more complex topologies or sparser topological features.

[0098] When constructing the topology space, the dynamic threshold TH k Used to determine the connectivity between data points. Only when the distance between data points is less than or equal to TH... k Only when the threshold is set is the data considered connected. By dynamically adjusting the threshold, topological features at different scales can be identified, thus enabling a more comprehensive calculation of topological invariants.

[0099] For example, suppose we have a dataset of size N=1000, and the expected frequency of a certain topological feature (such as a ring) is q=0.1. Based on historical data or prior knowledge, we determine the baseline thresholds as mean μ=0.5 and standard deviation σ=0.1. We then calculate the dynamic threshold: TH k =0.5+0.1×0.11000=0.5+0.1×10000=0.5+0.1×100=10.5.

[0100] When the distribution of the original training data exhibits seasonal or periodic variations, setting a base threshold mean and standard deviation can include: setting different base threshold mean and standard deviation based on different stages of the season or cycle. Specifically, during peak periods of the original training data distribution, the base threshold mean is increased and the standard deviation is decreased; during trough periods of the original training data distribution, the base threshold mean is decreased and the standard deviation is increased. By dynamically adjusting the threshold according to the seasonal or periodic changes in data distribution, the topological features in the data can be identified more accurately, making it more consistent with the actual situation of the data at different stages, and providing a more accurate basis for subsequent processing.

[0101] For example, a financial institution's transaction data shows frequent trading activity in the fourth quarter (peak season) and less activity in the first quarter (off-season). In the dynamic threshold adjustment targeting the seasonal characteristics of this financial institution's transaction data, to address the frequent trading activity during the peak season (fourth quarter), the average base threshold is increased to 1 million yuan (from 800,000 yuan), while the standard deviation of the threshold is decreased to 100,000 yuan (from 150,000 yuan) to more accurately identify money laundering patterns such as high-value transactions and recurring transactions. Conversely, during the off-season (first quarter), when trading activity decreases, the average base threshold is decreased to 600,000 yuan (from 800,000 yuan), and the standard deviation of the threshold is increased to 200,000 yuan (from 150,000 yuan), thereby enhancing sensitivity to unusually small transactions and concealed fund flow paths, and more effectively capturing potential money laundering activities.

[0102] In operation S233, the topological invariants are compared with the dynamic threshold to obtain the topological features in the topological space, including the location, size and shape of the topological features.

[0103] In the embodiments of this application, the location of a topological feature is determined by recording the nodes (transaction accounts) contained within the topological feature, thus establishing its position within the transaction network; its size is inferred based on the threshold range for the appearance and disappearance of the topological feature. For example, if a topological feature appears at a threshold of 1 million yuan and disappears at 800,000 yuan, its size is related to this threshold range; its shape is inferred by analyzing topological invariants such as homology groups. For example, a one-dimensional "hole" may represent a ring structure, while a higher-dimensional "hole" may represent a more complex topological structure.

[0104] For example, in the analysis of transaction data during the peak season (fourth quarter), key topological features were identified through dynamic threshold adjustment and topological data analysis techniques: Among them, the ring structure involves three transaction accounts A, B, and C, forming a circular transaction chain. This feature exists stably within the threshold range of 900,000 to 1.1 million yuan, presenting a one-dimensional "hole" shape, which may reveal a money laundering pattern of circular transactions or fund repatriation; the hole structure is related to two transaction accounts D and E, whose transaction relationships with other accounts are weak. Within the threshold range of 800,000 to 1 million yuan, it appears as an isolated connected component (zero-dimensional "hole"), which, combined with the transaction background, may point to more complex hidden fund flow paths or abnormal behavior of missing key nodes.

[0105] By dynamically adjusting thresholds based on seasonal variations, topological features within transaction networks can be identified more accurately. Topological features such as loop structures and hole structures can reveal money laundering patterns like circular transactions and fund repatriation, as well as abnormal behaviors such as concealing fund flow paths. This provides strong technical support to financial institutions, helping them to more effectively identify and prevent money laundering activities.

[0106] return Figure 2 In operation S240, based on the data type and topological features of the original training data, the topological features are repaired to obtain the repaired data.

[0107] In the embodiments of this application, if the number of holes in the topological features exceeds a preset number, it is determined that the original training data contains missing values ​​or is discontinuous. The topological features are then repaired, including: using a topology-guided hybrid method to repair the topological features. For numerical data, the filling direction is determined based on the topological relationship of the data points surrounding the holes. Along the determined filling direction, a weighted interpolation formula is used for filling, thereby repairing the topological features. By using the topology-guided hybrid method to determine the filling direction based on the topological relationship of the data points surrounding the holes and using a weighted interpolation formula for filling, missing values ​​or discontinuities in numerical data can be effectively repaired, improving data integrity. For textual data, supplementation or connection is performed based on contextual semantic relationships and the position of the text in the topological space to repair the topological features. Specifically, text fragments have topological connections with other fragments in the topological space, and there are missing parts in the contextual semantics. Supplementary content is inferred based on the connection relationships and contextual semantic logic. For textual data, supplementation or connection based on contextual semantic relationships and the position of the text in the topological space can fill in the missing parts of the text, making the text data more complete and coherent. Topology-guided hybrid methods fully consider the topological structure of the data, ensuring that the original topological relationships are not destroyed during the repair process. Topology-guided hybrid methods are not only applicable to simple linear or planar topologies, but can also handle complex topologies (such as rings and networks). By repairing holes in topological features, complex topological information in the data can be preserved or recovered, providing strong support for subsequent advanced data analysis (such as network analysis and social network analysis).

[0108] When using the topology-guided hybrid method to fill numerical data, for data points around holes forming complex topological structures, the method for determining the filling direction is further refined. Determining the filling direction based on the topological relationships of the data points around the hole includes: decomposing the topological structure around the hole into multiple basic topological units; determining the contribution weight of each basic topological unit to the filling of data points within the hole based on the relative positional relationship between each basic topological unit and the hole; and determining the filling direction based on the contribution weights of each basic topological unit to the filling of data points within the hole. This refined method for determining the filling direction enables more accurate determination of the filling direction when data points around holes form complex topological structures, improving the accuracy and rationality of the filling process, and resulting in more complete and realistically consistent repaired numerical data.

[0109] For example, suppose the set of data points within the hole region is... ,in To determine the number of data points inside the hole, the filling direction is determined based on the topological relationship of the data points surrounding the hole. If the data points around the hole form a ring structure, the tangent direction of the ring is used as the filling direction. If other topological structures are formed, the filling direction is determined based on their geometric characteristics. Along the determined filling direction, a weighted interpolation formula is used for filling. Let the adjacent data points in the filling direction be... and For data points inside the cave Its fill value The calculation formula is:

[0110] (7)

[0111] in, and Adjacent data points and The known value, and Assign the corresponding weights to the data points. and The topological distance between them is data points and The topological distance between them is The weight calculation formula is as follows:

[0112] (8)

[0113] (9)

[0114] For text-based data, supplementation or connection is made based on contextual semantic relationships and the position of the text in the topological space. If a text fragment has a topological connection relationship with other fragments in the topological space and there are missing parts in the contextual semantics, the supplementary content is inferred based on the connection relationship and contextual semantic logic.

[0115] In the embodiments of this application, the topological feature is a ring structure. The original training data is determined to have a specific hierarchical structure or clustering characteristics. The topological feature is repaired by: calculating the topological centrality of each data point in the ring structure using a topological weighted feature extraction algorithm based on the topological distance between the data point and other data points within the ring structure; weighting each feature attribute of the data point based on the topological centrality to obtain weighted feature attributes; and extracting features reflecting the ring structure characteristics from the weighted feature attributes as new feature extraction objects. By weighting each feature attribute of the data point, the feature extraction process becomes more targeted, highlighting those feature attributes that contribute significantly to the ring structure characteristics while suppressing those that contribute less. This targeted feature extraction helps improve the accuracy and efficiency of subsequent analyses (such as classification, clustering, etc.). By repairing the ring structure in the topological feature and extracting features reflecting the ring structure characteristics, this method provides strong support for complex data analysis and mining tasks. For example, in social network analysis, a ring structure may represent close connections or community structures between users; in bioinformatics, a ring structure may represent an interaction network between genes or proteins. By extracting these features that reflect the characteristics of the ring structure, we can gain a deeper understanding of the inherent patterns and mechanisms of the data.

[0116] For example, let the set of data points within the ring structure be... ,in The number of data points within the ring structure, for each data point It has Each feature attribute can be represented as Calculate each data point Topological centrality in a ring structure The calculation formula is based on the topological distance between the data point and other data points within the ring structure, and the specific formula is as follows:

[0117] (10)

[0118] in, For data points and The topological distance between them.

[0119] Each feature attribute is weighted according to its topological centrality. Let the weighted feature attributes be . The calculation formula is as follows:

[0120] (11)

[0121] Extract features that reflect the characteristics of the ring structure from the weighted feature attributes, and for the weighted feature mean... The calculation formula is as follows:

[0122] (12)

[0123] For weighted characteristic variance The calculation formula is as follows:

[0124] (13)

[0125] These features extracted based on topological structure features will serve as new feature extraction objects for training subsequent artificial intelligence algorithm models.

[0126] Complete and continuous data is the foundation of data analysis. By repairing holes in topological features, noise and anomalies in the data can be eliminated, improving the accuracy of data analysis. For numerical data, repaired data more closely resembles the actual distribution, helping models to more accurately capture patterns and trends. For textual data, repaired textual data is more coherent and semantically complete, contributing to improved accuracy in natural language processing tasks such as text classification and sentiment analysis. Using repaired data for model training can enhance the model's generalization ability. Because repaired data is closer to the real data distribution, the model can learn more robust feature representations during training. For numerical data, repaired data reduces the impact of missing values ​​and discontinuities on the model, enabling the model to better adapt to unseen data. For textual data, repaired textual data provides richer contextual information, helping the model better understand textual semantics and improving generalization performance.

[0127] Based on the above data processing method, this application also provides a data processing apparatus. The following will be combined with... Figure 6 The device is described in detail.

[0128] Figure 6 A schematic block diagram of a data processing apparatus according to an embodiment of this application is shown.

[0129] like Figure 6 As shown, the data processing device 800 of this embodiment includes a data acquisition module 810, a space construction module 820, a feature recognition module 830, and a data repair module 840.

[0130] The data acquisition module 810 is used to acquire raw training data, which may be numerical, textual, or image-based. In one embodiment, the data acquisition module 810 may be used to perform the operation S210 described above, which will not be repeated here.

[0131] The space construction module 820 is used to construct the topological space corresponding to the original training data using an adaptive topology metric algorithm. In one embodiment, the space construction module 820 can be used to perform the operation S220 described above, which will not be repeated here.

[0132] The feature recognition module 830 is used to identify topological features in the topological space using a dynamic threshold recognition algorithm. These topological features include holes and loops. In one embodiment, the feature recognition module 830 can be used to perform the operation S230 described above, which will not be repeated here.

[0133] The data repair module 840 is used to repair the topological features based on the data type and topological features of the original training data to obtain repaired data. In one embodiment, the data repair module 840 can be used to perform the operation S240 described above, which will not be repeated here.

[0134] According to an embodiment of this application, the space construction module 820 may include a first calculation submodule 821, a first construction submodule 822, and a first transformation submodule 823.

[0135] The first calculation submodule 821 is used to calculate the information entropy of each feature attribute based on the probability distribution of all possible values ​​of the feature attribute. The original training data includes n data points, each data point has m feature attributes, and n and m are positive integers. In one embodiment, the first calculation submodule 821 can be used to perform the operation S221 described above, which will not be repeated here.

[0136] The first construction submodule 822 is used to construct a feature difference matrix based on information entropy and feature similarity between various data points. In one embodiment, the first construction submodule 822 can be used to perform the operation S222 described above, which will not be repeated here.

[0137] The first transformation submodule 823 is used to transform the feature difference matrix into a topological metric matrix using a nonlinear mapping function, thereby obtaining a topological space. The topological space can reflect the neighborhood relationships and distance metrics between data points. In one embodiment, the first transformation submodule 823 can be used to perform the operation S223 described above, which will not be repeated here.

[0138] According to an embodiment of this application, the feature recognition module 830 may include a second calculation submodule 831, a third calculation submodule 832, and an acquisition submodule 833.

[0139] The second calculation submodule 831 is used to calculate the topological invariants of the topological space using a dynamic threshold recognition algorithm. In one embodiment, the second calculation submodule 831 can be used to perform the operation S231 described above, which will not be repeated here.

[0140] The third calculation submodule 832 is used to calculate the dynamic threshold by setting the mean of the base threshold and the standard deviation of the threshold based on the size of the original training data and the expected occurrence frequency of the topological features. In one embodiment, the third calculation submodule 832 can be used to perform the operation S232 described above, which will not be repeated here.

[0141] The acquisition submodule 833 is used to compare topological invariants with dynamic thresholds to obtain topological features in the topological space, including the position, size, and shape of the topological features. In one embodiment, the acquisition submodule 833 can be used to perform the operation S233 described above, which will not be repeated here.

[0142] According to embodiments of this application, the data processing device 800 in Figure 6 In addition, a preprocessing module 850 can be added.

[0143] The preprocessing module 850 is used to preprocess the original training data. In one embodiment, the preprocessing module 850 can be used to perform the operation S250 described above, which will not be repeated here.

[0144] According to embodiments of this application, any multiple modules among the data acquisition module 810, space construction module 820, feature recognition module 830, and data repair module 840 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this application, at least one of the data acquisition module 810, space construction module 820, feature recognition module 830, and data repair module 840 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in a suitable combination of any of these. Alternatively, at least one of the data acquisition module 810, space construction module 820, feature recognition module 830, and data repair module 840 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0145] Figure 7 A block diagram schematically illustrates an electronic device suitable for implementing a data processing method according to an embodiment of this application.

[0146] like Figure 7As shown, an electronic device 900 according to an embodiment of this application includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage portion 908 into a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0147] RAM 903 stores various programs and data required for the operation of electronic device 900. Processor 901, ROM 902, and RAM 903 are interconnected via bus 904. Processor 901 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 902 and / or RAM 903. It should be noted that programs may also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in one or more memories.

[0148] According to embodiments of this application, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to a bus 904. The electronic device 900 may also include one or more of the following components connected to the input / output (I / O) interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the input / output (I / O) interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 910 as needed so that computer programs read from it can be installed into the storage section 908 as needed.

[0149] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0150] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 902 and / or RAM 903 and / or one or more memories other than ROM 902 and RAM 903 described above.

[0151] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to enable the computer system to implement the data processing methods provided in the embodiments of this application.

[0152] When the computer program is executed by the processor 901, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0153] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 909, and / or installed from a removable medium 911. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0154] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 909, and / or installed from the removable medium 911. When the computer program is executed by the processor 901, it performs the functions defined in the system of this application embodiment. According to the embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0155] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0156] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0157] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

Claims

1. A data processing method, characterized in that, The method includes: Obtain raw training data, the types of which include numerical, text, and image data; The topological space corresponding to the original training data is constructed using an adaptive topology metric algorithm; The topological features in the topological space, including holes and loops, are identified using a dynamic threshold recognition algorithm; and Based on the data type and topological features of the original training data, the topological features are repaired to obtain repaired data.

2. The method according to claim 1, characterized in that, The step of constructing the topology space corresponding to the original training data using an adaptive topology metric algorithm includes: Based on the probability distribution of all possible values ​​of the feature attribute, the information entropy of each feature attribute is calculated, wherein the original training data includes n data points, each data point has m feature attributes, and n and m are positive integers; Based on the information entropy and the feature similarity between each data point, a feature difference matrix is ​​constructed; The feature difference matrix is ​​transformed into a topological metric matrix using a nonlinear mapping function to obtain a topological space, which reflects the neighborhood relationship and distance metric between data points.

3. The method according to claim 2, characterized in that, If the feature attribute has missing values, calculating the information entropy of each feature attribute includes: calculating the information entropy using an interpolation estimation method. The calculation of the information entropy using interpolation estimation includes: Based on the probability distribution of all possible values ​​of the feature attribute, the data points with complete feature attributes in the original training data are divided into q subgroups, wherein the data points in each subgroup have values ​​of the feature attribute with missing values ​​within a preset deviation range, and q is a positive integer greater than or equal to 2. Calculate the average value of the data points in the subgroup on the feature attribute with missing values, and use it as the representative value of the subgroup; The data points containing the missing values ​​are assigned to the corresponding subgroups based on the correlation of the feature attributes, and the representative value of the assigned subgroup is used as the replacement value for the missing value. Based on the substitute value, the information entropy of the feature attribute with missing values ​​is calculated.

4. The method according to claim 2, characterized in that, If the value of the feature difference matrix is ​​greater than a preset value, the step of converting the feature difference matrix into a topological metric matrix using a nonlinear mapping function includes: The nonlinear mapping function is transformed into a logic function, the output value of which is in the range of (0, 1).

5. The method according to claim 1, characterized in that, The method of identifying topological features in the topological space using a dynamic threshold recognition algorithm includes: A dynamic threshold identification algorithm is used to calculate the topological invariants of the topological space; Based on the scale of the original training data and the expected frequency of occurrence of topological features, the mean of the basic threshold and the standard deviation of the threshold are set, and the dynamic threshold is calculated. The topological invariant is compared with the dynamic threshold to obtain the topological features in the topological space, including the position, size and shape of the topological features.

6. The method according to claim 5, characterized in that, The method of using a dynamic threshold recognition algorithm to calculate the topological invariants of the topological space includes: Based on the geometric structure or data distribution characteristics of the topological space, the topological space is divided into p levels, where p is a positive integer greater than or equal to 2; Calculate sub-topological invariants for each level to obtain the local topological features of each level; Based on the relationships between different levels and the sub-topological invariants of each level, the topological invariants of the topological space are obtained.

7. The method according to claim 5, characterized in that, The distribution of the original training data exhibits seasonal or periodic variations, and the set base threshold mean and standard deviation include: Different baseline thresholds and standard deviations are set for different seasons or different stages of a cycle. Specifically, during the peak period of the original training data distribution, the mean of the base threshold is increased and the standard deviation of the threshold is decreased. During the low point of the original training data distribution, the mean of the base threshold is reduced and the standard deviation of the threshold is increased.

8. The method according to claim 1, characterized in that, If the number of holes in the topological feature exceeds a preset limit, it is determined that the original training data contains missing values ​​or is discontinuous. The step of repairing the topological feature includes: using a topology-guided hybrid method to repair the topological feature. For numerical data, the filling direction is determined based on the topological relationship of the data points around the hole. Along the determined filling direction, a weighted interpolation formula is used to fill the hole, thereby restoring the topological features. For text-based data, supplementation or connection is performed based on contextual semantic relationships and the position of the text in the topological space to repair the topological features. Specifically, text fragments have topological connections with other fragments in the topological space, and there are missing parts in the contextual semantics. Supplementary content is inferred based on the connection relationships and contextual semantic logic.

9. The method according to claim 1, characterized in that, The topological feature is a ring structure, and it is determined that the original training data has a specific hierarchical structure or clustering characteristics. The repair of the topological feature includes: Based on the topological distance between the data point and other data points within the ring structure, a topological weighted feature extraction algorithm is used to calculate the topological centrality of each data point in the ring structure. Based on the topological centrality, each feature attribute of the data point is weighted to obtain the weighted feature attribute. Features reflecting the characteristics of the ring structure are extracted from the weighted feature attributes and used as new feature extraction objects.

10. The method according to claim 8, characterized in that, The step of determining the filling direction based on the topological relationship of data points around the hole includes: The topological structure around the hole is decomposed to obtain multiple basic topological units; Based on the relative positional relationship between each basic topological unit and the hole, determine the contribution weight of each basic topological unit to the filling of data points within the hole; The filling direction is determined based on the contribution weight of each basic topological unit to the filling of data points within the hole.

11. The method according to claim 1, characterized in that, Before constructing the topological space corresponding to the original training data using the adaptive topology metric algorithm, the method further includes: preprocessing the original training data. The preprocessing of the original training data includes: Remove format errors, illegal characters, and irrelevant data fragments from the original training data to obtain cleaned data; The numerical data points in the cleaned data are normalized, mapping them to the range 0-1; and The characters in the text data are converted into numerical values ​​and then normalized.

12. A data processing apparatus, characterized in that, The device includes: The data acquisition module is used to acquire raw training data, the types of which include numerical, text, and image data; The space construction module is used to construct the topological space corresponding to the original training data using an adaptive topology metric algorithm. A feature recognition module is used to identify topological features in the topological space using a dynamic threshold recognition algorithm, the topological features including holes and loops; and The data repair module is used to repair the topological features based on the data type and topological features of the original training data to obtain repaired data.

13. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 11.

14. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 11.

15. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 11.

Images