Data processing method, related method, device, equipment and storage medium

Abstract

The invention discloses a data processing method, a related method, a related device, equipment and a storage medium. Comprises: obtaining a target sample set; performing set decomposition on the target sample set to obtain a plurality of target sample decomposition sets, and taking each target sample decomposition set as a to-be-merged set; for the plurality of to-be-merged sets, performing set merging on every two to-be-merged sets to obtain a middle merged set after set merging, and screening the middle merged set based on a merging standard threshold to obtain a corresponding sample merged set; and when the set number of the sample merging sets is not one, taking each sample merging set as a to-be-merged set, and returning to the step of performing set merging on every two to-be-merged sets until the set number of the sample merging sets obtained by set merging is one, thereby obtaining a target merging set. According to the method, the time complexity of multi-source sample fusion can be reduced, the quality of multi-source sample fusion is improved, and the method can be applied to scenes such as sample clustering.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and more particularly, to a data processing method and related method, device, equipment and storage medium. BACKGROUND

[0002] In the data-driven era, the performance of machine learning / deep learning models largely depends on the quality and richness of data. However, with the continuous growth and diversification of data, especially in application fields with high confidentiality, such as finance, medicine and intelligent manufacturing, the demand for data isolation, heterogeneity and confidentiality is increasingly prominent. To solve the data silos and take into account the demand for data confidentiality, multi-source sample fusion technology has emerged.

[0003] The core of multi-source sample fusion technology is how to effectively fuse data from different data sources while ensuring data confidentiality, and then the model can capture more features and patterns from the fused sample set, thereby improving the accuracy and robustness of prediction. For example, in the financial security scenario, the object data of multiple banks is fused for credit risk assessment, and in the Internet communication scenario, the communication data of multiple communication platforms is fused for information recommendation.

[0004] Currently, related technologies mainly perform multi-source sample fusion based on the exhaustive idea and heuristic rules. For example, new samples are fused one by one into a number of seed sets obtained according to historical experience, and whether to fuse is determined based on the modeling effect of the observed fused sample set, or different samples are clustered into multiple sample clusters by referring to specific business rules or using clustering algorithms. However, these multi-source sample fusion methods have high time complexity and poor sample fusion quality. SUMMARY

[0005] Embodiments of the present application provide a data processing method and related method, device, equipment and storage medium. The time complexity of multi-source sample fusion can be reduced, and the quality of multi-source sample fusion can be improved.

[0006] According to an aspect of the present application, a data processing method is provided, which includes: obtaining a target sample set; performing set decomposition on the target sample set to obtain a plurality of target sample decomposition sets, and taking each target sample decomposition set as a to-be-merged set; for a plurality of to-be-merged sets, performing set merging on each two to-be-merged sets to obtain an intermediate merged set after set merging, and performing filtering on the intermediate merged set based on a merging standard threshold to obtain a corresponding sample merged set; when the number of sets of the sample merged set is not one, taking each sample merged set as a to-be-merged set, and returning to the step of performing set merging on each two to-be-merged sets until the number of sets of the sample merged set obtained by set merging is one, to obtain a target merged set.

[0007] According to another aspect of the present application, a sample clustering method is provided, which comprises: initializing a recombination number; performing sequential recombination on a plurality of sample initial subsets in a target sample set to obtain a recombined sample set; performing set decomposition and set recombination on the recombined sample set according to the data processing method, storing the obtained recombined sample recombination set, and performing numerical updating on the recombination number; when it is detected that the recombination number does not satisfy a preset recombination condition, returning to iteratively perform the step of performing sequential recombination on a plurality of sample initial subsets in a target sample set to obtain a recombined sample set until the recombination number satisfies the preset recombination condition, and terminating the iteration; determining a recombination co-occurrence matrix corresponding to the target sample set based on the stored plurality of recombined sample recombination sets; and clustering the plurality of sample initial subsets according to each matrix element in the recombination co-occurrence matrix, and determining a target sample clustering set corresponding to the plurality of sample initial subsets through the clustering result.

[0008] According to another aspect of the present application, a data processing apparatus is provided, which comprises: a set obtaining module configured to obtain a target sample set; a set decomposition module configured to perform set decomposition on the target sample set to obtain a plurality of target sample decomposition sets, and take each target sample decomposition set as a to-be-recombined set; a first recombination module configured to, for a plurality of to-be-recombined sets, perform set recombination on each two to-be-recombined sets to obtain an intermediate recombination set after set recombination, and perform screening on the intermediate recombination set based on a recombination standard threshold to obtain a sample recombination set corresponding to each two to-be-recombined sets; and a second recombination module configured to, when the number of recombination sets of the sample recombination set is not one, take each sample recombination set as a to-be-recombined set, and return to the step of performing set recombination on each two to-be-recombined sets until the number of recombination sets of the sample recombination set obtained by set recombination is one to obtain a target recombination set.

[0009] Optionally, the first recombination module can comprise an intermediate set determination unit, a recombination gain calculation unit, and a combined subset screening unit. The intermediate set determination unit is configured to sequentially combine two to-be-recombined sets to obtain a corresponding intermediate recombination set, the intermediate recombination set comprising a plurality of combined subset pairs. The recombination gain calculation unit is configured to, for the plurality of combined subset pairs in each intermediate recombination set, sequentially perform recombination on one combined subset pair, and perform gain calculation based on the recombined subset to obtain a recombination gain corresponding to each combined subset pair. The combined subset screening unit is configured to screen the plurality of combined subset pairs of each intermediate recombination set according to the recombination standard threshold and the recombination gain corresponding to each combined subset pair to obtain a sample recombination set after screening of each intermediate recombination set.

[0010] Optionally, the merging gain calculation unit can include a first subunit, a second subunit, a third subunit, a fourth subunit, and a fifth subunit. The first subunit is configured to merge the first sample subset and the second sample subset to obtain a sample merging subset; the second subunit is configured to train a first model according to the first sample subset and calculate a first statistical index corresponding to the first model on the first sample subset; the third subunit is configured to train a second model according to the second sample subset and calculate a second statistical index corresponding to the second model on the second sample subset; the fourth subunit is configured to train a merging model according to the sample merging subset and calculate a merging statistical index corresponding to the merging model on the sample merging subset; and the fifth subunit is configured to calculate a merging gain corresponding to the one combination subset pair by using the first statistical index, the second statistical index, and the merging statistical index.

[0011] Optionally, the first sample subset includes a first training set, a first test set, and a first out-of-time set, the second sample subset includes a second training set, a second test set, and a second out-of-time set, and the first subunit can be specifically configured to: merge the first training set and the second training set to obtain a merging training subset; merge the first test set and the second test set to obtain a merging test subset; merge the first out-of-time set and the second training set to obtain a merging out-of-time subset; obtain a first blind test set corresponding to the first sample subset and a second blind test set corresponding to the second sample subset; merge the first blind test set and the second blind test set to obtain a merging blind test subset; and determine a sample merging subset based on the merging training subset, the merging test subset, the merging out-of-time subset, and the merging blind test subset.

[0012] Optionally, the fourth subunit can be specifically configured to: train a preset model according to the merging training subset to obtain a trained merging model; perform test discrimination statistics on the merging model on the merging test subset to obtain a test statistical index; perform out-of-time discrimination statistics on the merging model on the merging out-of-time subset to obtain an out-of-time statistical index; perform blind test discrimination statistics on the merging model on the merging blind test subset to obtain a blind test statistical index; and determine a merging statistical index corresponding to the merging model on the sample merging subset based on the test statistical index, the out-of-time statistical index, and the blind test statistical index.

[0013] Optionally, the fifth subunit can be specifically configured to: calculate a first index difference between the merged statistical indicator and the first statistical indicator, and take a quotient of the first index difference and the first statistical indicator as a first gain; calculate a second index difference between the merged statistical indicator and the second statistical indicator, and take a quotient of the second index difference and the second statistical indicator as a second gain; and determine the merged gain corresponding to each combination subset pair according to the first gain and the second gain.

[0014] Optionally, the combination subset screening unit can include a first sub-subunit, a second sub-subunit, a third sub-subunit, a fourth sub-subunit, and a fifth sub-subunit. The first sub-subunit is configured to, for each intermediate merged set, respectively compare the merged gain corresponding to each combination subset pair of the intermediate merged set with a merged standard threshold; the second sub-subunit is configured to screen out, according to the comparison result, the combination subset pair with a merged gain not less than the merged standard threshold as a target subset pair corresponding to each intermediate merged set; the third sub-subunit is configured to, for each intermediate merged set, obtain a gain descending order determined by each target subset pair, and sequentially select, from a head of the gain descending order, a target subset pair with a shared sample quantity greater than a shared sample threshold as an available subset pair; the fourth sub-subunit is configured to determine, from two to-be-merged sets corresponding to the available subset pair, a non-positive gain subset pair corresponding to the available subset pair, the non-positive gain subset pair having no intersection with the available subset pair; and the fifth sub-subunit is configured to construct, according to each available subset pair and the corresponding non-positive gain subset pair, a sample merged set corresponding to each intermediate merged set.

[0015] Optionally, the fourth sub-subunit can be specifically configured to: based on an available sample subset in the available subset pair, obtain, from two to-be-merged sets corresponding to the available subset pair, a non-positive gain subset that does not match the available sample subset; and generate a non-positive gain subset pair according to the non-positive gain subset.

[0016] Optionally, the set decomposition module can be specifically configured to: take the target sample set as a to-be-decomposed set; perform set decomposition on the to-be-decomposed set to obtain two sample decomposition sets after set decomposition; when a decomposition set quantity of the sample decomposition set is not equal to a subset quantity corresponding to the target sample set, take each sample decomposition set as a to-be-decomposed set, and return to the step of performing set decomposition on the to-be-decomposed set, until the decomposition set quantity of the sample decomposition set after set decomposition is equal to the subset quantity corresponding to the target sample set, and stop set decomposition; and take each target sample decomposition set obtained after stopping set decomposition as a to-be-merged set.

[0017] According to another aspect of the present application, a sample clustering apparatus is provided, which comprises: a number of times determining module, configured to initialize a recombination number of times; a sequential recombination module, configured to sequentially recombine a plurality of sample initial subsets in a target sample set to obtain a recombined sample set; a set storage module, configured to perform multi-layer set decomposition and multi-layer set recombination on the recombined sample set according to the data processing method as described above, to obtain a recombined sample recombination set and store the same, and to perform a numerical update operation on the recombination number of times; an iterative execution module, configured to return to the step of sequentially recombining the plurality of sample initial subsets in the target sample set to obtain the recombined sample set when it is detected that the recombination number of times does not satisfy a preset recombination condition, until the recombination number of times satisfies the preset recombination condition, and terminate the iteration; a matrix determining module, configured to determine a recombination co-occurrence matrix corresponding to the target sample set based on the stored plurality of recombined sample recombination sets; and a sample clustering module, configured to cluster the plurality of sample initial subsets according to each matrix element in the recombination co-occurrence matrix, and determine a target sample clustering set corresponding to the plurality of sample initial subsets through the clustering result.

[0018] Optionally, the matrix determining module can be specifically configured to: determine a plurality of co-occurrence subset pairs for the plurality of sample initial subsets; determine a same-species probability that each co-occurrence subset pair belongs to a same recombined sample recombination set based on the stored plurality of recombined sample recombination sets; and determine a recombination co-occurrence matrix corresponding to the plurality of sample initial subsets according to the same-species probability corresponding to each co-occurrence subset pair.

[0019] According to another aspect of the present application, a computer readable storage medium is provided, which stores a computer program, wherein the computer program, when executed by a processor, performs the data processing method and the sample clustering method as described above.

[0020] According to another aspect of the present application, a computer device is provided, which comprises a processor and a memory, and the memory stores a computer program, which, when invoked by the processor, performs the data processing method and the sample clustering method as described above.

[0021] According to another aspect of the present application, a computer program product is provided, which comprises a computer program or computer instructions stored in a storage medium; a processor of a computer device reads the computer program or computer instructions from the storage medium, and the processor executes the computer program or computer instructions, so that the computer device performs the data processing method and the sample clustering method as described above.

[0022] The application can obtain a target sample set, perform set decomposition on the target sample set, obtain a plurality of target sample decomposition sets, and then take each target sample decomposition set as a to-be-merged set, so as to decompose the complex sample set division problem of the target sample set into simple sample set division sub-problems. Further, for the plurality of to-be-merged sets, set merging is performed on each two to-be-merged sets to obtain a set-merged intermediate merging set, and the intermediate merging set is filtered based on a merging standard threshold to obtain a corresponding sample merging set. When the number of sets of the sample merging set is not one, each sample merging set is taken as a to-be-merged set, and the step of performing set merging on each two to-be-merged sets is returned until the number of sets of the sample merging set obtained by set merging is one, and a target merging set is obtained.

[0023] Through set decomposition, the complex sample set division problem of the target sample set is finally decomposed into sample set division sub-problems that can be simply and directly solved. Further, after set merging, the solutions of the sample set division sub-problems are merged into the solution of the sample set division problem, that is, the target merging set is obtained. Compared with the related art with a time complexity of O(n 2 ) based on the exhaustive strategy, the application can quickly complete multi-source sample fusion of the target sample set in a linear time complexity of only O(n log2n), thereby improving the efficiency of multi-source sample fusion. In addition, by using the merging standard threshold, the local greedy strategy is used to solve the optimal solution of the sample set division sub-problem in the set merging, that is, the intermediate merging set after set merging is filtered to obtain a high-quality sample merging set that better meets the requirements of the application scenario, so that the optimal solutions of the sample set division sub-problems are merged into the optimal solution of the sample set division problem, and a high-quality target merging set is obtained. Compared with the related art, the quality of multi-source sample fusion can be improved.

[0024] Other features and advantages of the application will be set forth in the following description, and in part will become apparent to those skilled in the art from the description, or can be learned by practice of the application. The objects and other advantages of the application will be realized and attained by the structure particularly pointed out in the description and claims. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0026] Figure 1 A system architecture diagram provided by an embodiment of the application is shown.

[0027] Figure 2 A system deployment diagram provided by an embodiment of the application is shown.

[0028] Figure 3 An application scenario diagram of a data processing method provided by an embodiment of the application is shown.

[0029] Figure 4 An application scenario diagram of another data processing method provided by an embodiment of the application is shown.

[0030] Figure 5 A flow diagram of a data processing method provided by an embodiment of the application is shown.

[0031] Figure 6 An acquisition diagram of a target sample set provided by an embodiment of the application is shown.

[0032] Figure 7 A division diagram of a sample initial subset provided by an embodiment of the application is shown.

[0033] Figure 8 A flow architecture diagram of multi-source sample fusion provided by an embodiment of the application is shown.

[0034] Figure 9 A flow diagram of problem decomposition and merging solution provided by an embodiment of the application is shown.

[0035] Figure 10 A generation flow diagram of a sample merging set provided by an embodiment of the application is shown.

[0036] Figure 11 A statistical index calculation diagram provided by an embodiment of the application is shown.

[0037] Figure 12 A diagram of a non-positive gain subset pair provided by an embodiment of the application is shown.

[0038] Figure 13 A generation diagram of a sample merging set provided by an embodiment of the application is shown.

[0039] Figure 14 A flow diagram of a sample clustering method provided by an embodiment of the application is shown.

[0040] Figure 15 A flow diagram of a sample clustering method provided by an embodiment of the application is shown.

[0041] Figure 16A multi-source sample fusion effect schematic diagram provided by an embodiment of the present application is shown.

[0042] Figure 17 A module block diagram of a data processing apparatus provided by an embodiment of the present application is shown.

[0043] Figure 18 A module block diagram of a sample clustering apparatus provided by an embodiment of the present application is shown.

[0044] Figure 19 A module block diagram of a computer device provided by an embodiment of the present application is shown.

[0045] Figure 20 A module block diagram of a computer readable storage medium provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0046] In order to enable persons skilled in the art to better understand the schemes of the present application, the technical schemes in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. The described embodiments are exemplary and are only used to explain the present application, and cannot be understood as a limitation on the present application. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without making creative efforts fall within the scope of protection of the present application.

[0047] In some processes described in the specification, claims and the above drawings, a plurality of steps appearing in a specific order are included, but it should be clearly understood that these steps can be executed or executed in parallel without the order appearing in the text, and the step number is only used to distinguish different steps, and the number itself does not represent any execution order. In addition, the descriptions such as "first", "second" or "target" in this paper are used to distinguish similar objects, and do not necessarily describe a specific order, sequence or quantity.

[0048] It is worth noting that in the specific embodiments of the present application, related data such as target sample set is involved, when the above embodiments of the present application are applied to specific products or technologies, the permission or consent of the object needs to be obtained, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards. For example, when the embodiments of the present application need to obtain a target sample set, a pop-up window or a jump to a confirmation page can be used to obtain individual permission or individual consent of the target sample set, and after obtaining individual permission or individual consent, the necessary related data for the normal operation of the embodiments of the present application is obtained.

[0049] Multi-source sample fusion refers to the process of dividing, combining and fusing multiple sample data from different customers or different business systems to generate a new set of sample collections. "Multi-source" means that the data samples come from different customers, different business departments or different systems. Each data source may contain unique features and business backgrounds, and directly merging these data may face data privacy, data heterogeneity and data quality problems. Traditional machine learning or deep learning models usually rely on data from a single data source or a single business system for training. This limits the generalization ability and prediction accuracy of the model to some extent, especially when facing complex and variable actual scenarios, this limitation is more obvious.

[0050] For example, in the field of financial security, different banks have credit data of their own service objects, but due to data privacy and competition, these data are often closed in their own systems and cannot be shared and integrated. This results in a single bank's security model being able to train only with limited internal data, and being unable to fully capture potential risk characteristics, thereby affecting the accuracy of the prediction. Similar situations exist in the fields of medical health, e-commerce recommendation, intelligent manufacturing, etc. In order to solve the problem of data silos while taking into account the need for data privacy, researchers have proposed multi-source sample fusion technology, aiming to break down data barriers and effectively fuse data from different sources while ensuring data privacy. How to design an efficient and feasible multi-source sample fusion method that guarantees fusion quality is a problem of great business value.

[0051] Currently, the multi-source sample fusion methods commonly used by related technologies mainly include the exhaustive method and the heuristic rule method. Among them, the exhaustive method usually obtains several initial seed sets according to historical experience, then adds new samples one by one to each seed set, and observes the modeling effect of each sample set after fusion. If the modeling effect based on a certain seed set has gain, the new sample is added to the seed set, and if the modeling effect of all seed sets has no gain, it is divided into a new sample set. The heuristic rule method usually uses common business rules or clustering algorithms based on the feature distribution of different samples to cluster samples from different data sources into multiple sample sets, thereby achieving the purpose of fusing multi-source samples.

[0052] However, the related art has problems of high time complexity and poor sample fusion quality. For example, the above-mentioned exhaustive method can achieve the effect of partially reducing the solution size by setting the initial seed set cluster, but a large number of enumerations are still required for the remaining samples to be fused. In addition, the setting of the initial seed set itself is a difficult problem, and the past experience settings often cannot be applied. Because samples and models change with market and business changes, past experience settings still need to be adjusted when applied to new iteration cycles. The heuristic rule method also faces the problem of strong dependence on rule design. Since the business background information of the sample is often missing, the business rule cannot cover each sample to be fused.

[0053] If clustering is only based on the feature information of the sample, the label definition of the sample is ignored, the sample fusion process lacks a clear effect orientation, and the fusion result may not help the business effect, so the quality of sample fusion is poor. In summary, the technical solutions of the related art for multi-source sample fusion have problems of high time complexity and poor sample fusion quality. In order to solve the above problems, the inventors have studied and proposed the data processing method provided in the embodiments of the present application. The system implementation environment and related application scenarios of the above-mentioned method involved in the present application will be described below.

[0054] Please refer to Figure 1 , Figure 1 A system architecture diagram is shown. The above-mentioned method provided in the embodiments of the present application can be applied in the system 100. Among them, the sample acquisition device 110 is used to acquire multi-source sample data, for example, the sample acquisition device 110 can acquire service data of different service objects from a database storing service data of service objects, and pre-process the service data to obtain sample initial subsets corresponding to different service objects. The sample acquisition device 110 can store the sample initial subsets corresponding to different service objects in the database 120, and the sample fusion device 140 can perform multi-source sample fusion based on the plurality of sample initial subsets stored in the database 120 to obtain a fused target merged set.

[0055] The processing module 141 of the sample fusion device 140 is configured to perform relevant computing processing. The sample fusion device 140 can invoke data, programs, and the like in the data storage system 142 for the processing module 141 to perform relevant computing processing, and store data or instructions obtained by the computing processing into the data storage system 142. For example, the processing module 141 can perform multi-source sample fusion on a plurality of sample initial subsets, and in the middle process of the multi-source sample fusion, the training device 130 can be used to train a model based on the fused intermediate sample set to calculate a merging gain. After the processing module 141 completes the multi-source sample fusion, the target merging set can be sent to the training device 130 to train a corresponding target model 101 for different service scenarios.

[0056] The target network 101 can be a deep neural network or a network model composed of a plurality of neural networks. For example, a network framework composed of a fully connected layer, a Transformer model, or an XGBoost model, and the like, which are not limited herein. The training device 130 can deploy the corresponding target model 101 trained for different service scenarios in the execution device 150. When the execution device 150 receives service data sent by the client device 160, the execution device 150 can use the target model 101 conforming to the business requirements to perform corresponding service calculation on the service data to obtain a calculation result. The execution device 150 can return the calculation result to the client device 160. The execution device 150 and the client device 160 can perform data transmission through an input / output (I / O) interface.

[0057] The training device 130, the sample fusion device 140, the execution device 150, and the client device 160 described above can be a server or a terminal computer device. The server can be a standalone physical server, a server cluster or a distributed system composed of a plurality of physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (CDNs), blockchains, and big data and artificial intelligence platforms. The terminal can be a smart mobile device, a notebook computer, a vehicle-mounted computer, a desktop computer, and the like.

[0058] It should be noted that, Figure 1 The system architecture and application scenarios described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. For example, Figure 1The data storage system 160 is an external memory relative to the execution device 150. In other cases, the data storage system 160 can also be placed inside the execution device 150. For example, the client device 150 can also be directly used as an execution device to deploy the trained target network to perform computing tasks related to business services.

[0059] Please see Figure 2 , Figure 2 A schematic diagram of a system deployment is shown. Exemplarily, the data processing method provided in this application embodiment can also be deployed on, for example, Figure 2 The system shown is a C / S (Client / Server) architecture system. This system may include servers, gateways, the internet, and devices.

[0060] A server is a computer system that provides certain services to the server-side. Compared to ordinary computer equipment, servers have higher requirements in terms of stability, security, and performance. Figure 2 As shown, server 210 can be a single high-performance computer in a network platform, a cluster of multiple high-performance computers, a portion of a single high-performance computer (e.g., a virtual machine), or a combination of portions of multiple high-performance computers (e.g., virtual machines).

[0061] A gateway is also known as an internetwork connector or protocol converter. For example... Figure 2 As shown, gateway 220 achieves network interconnection at the transport layer via Internet 230, and is a computer system or device that acts as a translator. Between two systems using different communication protocols, data formats, or languages, or even completely different architectures, a gateway is a translator. At the same time, a gateway can also provide filtering and security functions.

[0062] Devices can send requests to servers, requesting access to or use resources, data, or services on the server. Devices are typically the interface through which users interact with the system; users perform operations and generate requests on the client side. For example... Figure 2 As shown, the device can include various forms such as a smartphone 240a, a laptop 240b, a desktop computer 240c, a personal digital assistant (PDA), and a server. Furthermore, it can be a single device or a collection of multiple devices.

[0063] The device can communicate with the internet and exchange data via wired or wireless means. For example... Figure 2As shown, smartphone 240a can communicate with the Internet 230 via wireless router 250. Similarly, messages sent by laptop 240b to server 210 can be transmitted to the corresponding server 210 via gateway 220. Messages sent by server 210 to device 240 can also be transmitted to the corresponding desktop computer 240c via gateway 220.

[0064] In one implementation method, the data processing method and related methods of this application can be implemented partly on a server and partly on a device. For example, the server acquires sample data from different devices to construct a target sample set, and performs multi-source sample fusion on multiple initial subsets of samples in the target sample set, including set decomposition and set merging, or performs clustering on multiple initial subsets of samples based on the merge co-occurrence matrix after multiple executions of set decomposition and set merging, to obtain a target merged set, which includes multiple merged sample sets. The server can then distribute each sample set in the target merged set to the device required for the specific modeling scenario.

[0065] It should be noted that, Figure 2 This is merely a schematic diagram of a system deployment provided in this application embodiment. The system deployment scheme described in this application embodiment is only for more clearly illustrating the technical solutions of this application embodiment and does not constitute a limitation on the technical solutions provided in this application embodiment. For example, "device" can refer to one of multiple devices, and this embodiment only uses one as an example. Figure 3 The three devices shown (smartphone 240a, laptop 240b, and desktop computer 240c) are used as examples. Those skilled in the art will recognize that, as system deployment schemes evolve, the technical solutions provided in these embodiments are equally applicable to similar technical problems.

[0066] The embodiments of this application can be applied in various scenarios, such as Figure 4 The financial data fusion scenario shown and Figure 3 The intelligent transportation scenarios shown are examples of this.

[0067] (1) Financial data integration scenario.

[0068] Financial security control refers to the comprehensive activities of financial institutions to identify, assess, detect and manage various potential risks in their business processes. With the deepening of the complexity and globalization of financial markets, traditional security control methods are increasingly difficult to cope with new and diversified risk challenges. For example, in credit risk management, traditional risk assessment methods based on static credit scoring often fail to reflect the dynamic changes in the behavior of service objects in a timely manner, resulting in insufficient and inaccurate risk warning. The data processing method proposed in this application can integrate and fuse diversified financial sample data from different data sources to generate a more comprehensive and rich data set, thereby providing more accurate and comprehensive input for security control models.

[0069] The security service platform provided by the financial security service provider can use deep learning technology to train security control models based on financial data, and package the security control models to third-party financial companies, so that the financial companies have risk control capabilities in specific businesses. At the same time, the financial security service provider can accumulate a large amount of financial data from the special businesses of different financial companies in the process of business expansion. As the base of security control models, in order to improve the competitiveness of the security control models trained by the security service platform, the financial security service provider will perform multi-source sample fusion on these multi-source financial data. Exemplarily, the security service platform can obtain multiple financial data from the servers of multiple financial companies, and perform sample fusion on the multiple financial data based on the data processing method proposed in this application to obtain multiple fusion sample sets, and then train corresponding security control models according to each fusion sample set.

[0070] For example, Figure 4 The service background 320 of the security service platform shown can obtain corresponding financial data sets from the first server 311, the second server 312 and the third server 313 corresponding to different financial companies, respectively. The service background 320 can perform data preprocessing on the multiple financial data sets, respectively, to obtain corresponding financial sample sets. The data preprocessing can include data cleaning, data conversion, data labeling (such as assigning appropriate labels to each data item in the data set), and the like. Further, the service background 320 can perform multiple sample fusions on the target sample set composed of these financial sample sets, including set decomposition and set merging. Further, the service background 320 can cluster each financial sample set based on the co-occurrence matrix of each financial sample set obtained by multiple sample fusions, and then obtain multiple fusion financial sample sets.

[0071] (2) Education artificial intelligence scene.

[0072] Education-Artificial Intelligence (E-AI) improves and optimizes various aspects of the education system by applying artificial intelligence technology to enhance teaching quality, individualized learning experience, and the efficiency of education management. This concept covers the intelligent transformation of multiple dimensions such as classroom teaching, learning analysis, evaluation testing, and education management. For example, an intelligent learning platform can automatically recommend the next step of learning content based on the student's historical learning records and current progress, and provide immediate tutoring and practice when the student encounters difficulties. This intelligent and personalized learning experience can enhance the student's learning initiative. Typically, an education artificial intelligence platform can provide customized education-related artificial intelligence services for multiple third-party education companies, such as student academic performance prediction, personalized learning path recommendation, online tutoring intelligent matching, etc.

[0073] Each education company has a large amount of education data, and these data cover various aspects of information such as learning objects, course content, and learning behavior. However, each education company operates independently, and their data formats, content focuses, and data structures may differ greatly and cannot be directly shared. In order to better provide unified and high-quality artificial intelligence services, the education artificial intelligence platform can fuse multi-source education data from different education companies to generate more comprehensive sample sets for training more accurate AI education models. For example, the education artificial intelligence platform can obtain corresponding education data from different education companies, such as Figure 4 The service cluster 410 of the education artificial intelligence platform shown can obtain a large amount of education data from multiple education companies, and after data preprocessing, generate multiple education sample sets.

[0074] Further, the service cluster 410 can perform multiple sample fusions on the target sample set composed of these education sample sets, including set decomposition and set merging. Further, the service cluster 410 can cluster each education sample set based on the co-occurrence matrix of each education sample set obtained by multiple sample fusions, and further obtain multiple fused education sample sets. For example, the fused education sample set A mainly includes historical mathematics learning data, the fused education sample set B mainly includes professional skill evaluation results and learning behavior, and the fused education sample set C mainly includes the learning progress of language learners and ability test scores. Further, the service cluster 410 can train AI education models for different education scenarios according to the needs of different education companies.

[0075] For example, the service cluster 410 of the education artificial intelligence platform shown can obtain a large amount of education data from multiple education companies, and after data preprocessing, generate multiple education sample sets. Figure 3As shown, service cluster 410 can train a prediction model for primary and secondary school mathematics learning performance based on the integrated education sample set A, and privatize this learning performance prediction model on server 421 of education company A. This model helps mathematics teachers understand students' learning progress and provide personalized mathematics tutoring suggestions through the teaching aid system on laptop 430a. Service cluster 410 can train a prediction model for professional skills learning effectiveness based on the integrated education sample set B, and privatize this model on server 422 of education company B. Students can log in to the skills learning APP on smartphone 430b to obtain professional development suggestions and learn recommended skills courses. Service cluster 410 can train a language learning progress assessment model based on the integrated education sample set C, and privatize this model on server 423 of education company C. Language learners can use the language optimization learning strategy on the voice learning platform 430c on desktop computers to improve their language skills.

[0076] It should be noted that, Figure 4 and Figure 5 These are merely illustrative diagrams illustrating two application scenarios provided in this application's embodiments. The application scenarios described in this application's embodiments are only for more clearly illustrating the technical solutions of this application's embodiments and do not constitute a limitation on the technical solutions provided in this application's embodiments. The data processing method proposed in this application can also be used in application scenarios requiring multi-source sample fusion, such as joint modeling of medical samples between different medical institutions and joint modeling of product recommendation samples between different e-commerce platforms. As those skilled in the art will understand, with the evolution of application scenarios, the technical solutions provided in the embodiments of this invention are also applicable to similar technical problems.

[0077] According to one embodiment of this application, a data processing method is provided. This data processing method can be executed by a computer device (e.g., a server, edge computing device, or other terminal device with certain processing capabilities), which at least has storage, computing, and communication functions. Figure 5 A flowchart illustrating the data processing method provided in this embodiment is shown. Figure 6 As shown, the data processing method may specifically include:

[0078] S110: Obtain the target sample set.

[0079] In this embodiment, the target sample set may include multiple initial sample subsets. The initial sample subset refers to the sample set to be fused from multiple sources. This sample set may be obtained from different customer services, such as financial security services, medical services, and product recommendations. Each initial sample subset may include multiple samples, each containing sample features and sample labels. Sample features describe the attributes or input variables of the sample and are the input information used by the model to learn and make predictions. Sample labels are the target values ​​or output variables corresponding to the sample features and are the content predicted by the model during the learning process. For example, in the financial security service business, sample features may include the bank's deposit and withdrawal records, consumption patterns, and credit history, and the corresponding sample labels may be "with default behavior" and "without default behavior."

[0080] Please see Figure 6 , Figure 6 This diagram illustrates how a target sample set can be obtained. (For example...) Figure 6 As shown, raw data of e-commerce service objects on different e-commerce platforms can be obtained, such as information browsing history, purchase records, and query keywords. This raw data can then be preprocessed to obtain an initial subset of samples corresponding to each e-commerce platform, and this initial subset constitutes the target sample set S = {S1, S2, ..., S...}. i ,…,S n}, where 0 < n and n ∈ N * , where n represents the number of initial subsets of samples in the target sample set S, which is also the number of e-commerce platforms.

[0081] like Figure 7 As shown, each initial subset of samples can also include multiple samples, and each sample can include corresponding sample features and sample labels. The i-th initial subset of samples... Where 0 < m and m ∈ N * m represents the initial subset S of the samples i The number of samples in the middle, and the initial subset S of the i-th sample. i The sample feature corresponding to the j-th sample is and the corresponding sample labels are Optionally, sample features and sample labels can be set according to the specific business scenario requirements. Specifically, a feature list of samples can be determined first, and features can be backtracked for all samples. For example, in financial security services, since it is necessary to summarize the basic security service elements of different service objects accumulated by the business side: object identifier, application time (the time when the object applied for the business), and default label, the sample features can be set to include multi-dimensional information such as object identifier, application time, and location-based services (LBS). Correspondingly, the sample labels can be default labels, including: there is a default behavior (label=1) or there is no default behavior (label=0).

[0082] As one implementation method, to ensure that model training, validation, and evaluation accurately reflect model performance, each initial subset of samples can be divided into datasets with different functions. For example, in business scenarios with time characteristics, the initial subset of samples corresponding to a continuous time range can be divided into datasets with different functions according to different time periods. Please refer to [link / reference]. Figure 7 , Figure 7 This diagram illustrates the partitioning of an initial subset of samples. The initial subset can be divided into a training set, a test set, and an out-of-time (OOT) set. The OOT set can be an independent dataset partitioned based on the sample's arrival time (e.g., the start time of a service). For example, within the time range corresponding to the initial subset, a specific time point is selected, and samples exceeding that time point are considered the OOT set, primarily used to evaluate the model's stability over time.

[0083] Optionally, by determining the time node, 85% of the samples in the initial subset before the time node can be used as the training and testing sets, and 15% of the samples in the initial subset after the time node can be used as the out-of-time set. Optionally, such as... Figure 8 As shown, samples from the initial subset can also be used as a blind test set. The blind test set contains data that is not used for model training or preliminary testing, and is used to evaluate the model's performance on different blind test targets, such as the model's cross-customer generalization effect across different customer groups. It should be noted that the ratio of samples in the training set, test set, out-of-time set, and blind test set can be set according to business needs and the total number of samples in the initial subset; no limit is imposed here.

[0084] Existing techniques for multi-source sample fusion suffer from high time complexity. For example, exhaustive methods find the answer by traversing all possible solutions, adding new samples one by one to the seed set and observing the modeling effect of the fused sample set. This means it requires observation and computation on all possible fused sample sets, resulting in typically very high time complexity, often exponential. Even if setting the initial seed set can partially reduce the size of the solution search, the remaining samples to be fused still need to be enumerated, preventing a significant reduction in time complexity. To address this, this application, considering the common large-scale sample problem in multi-source sample fusion tasks, proposes decomposing the sample fusion problem into smaller subproblems and merging their solutions to reduce the problem size, avoid unnecessary computation, and thus lower the time complexity of multi-source sample fusion.

[0085] In this embodiment, the large problem of multi-source sample fusion for a target sample set can be decomposed layer by layer into smaller sub-problems of the same form until each sub-problem can be directly solved. At this point, the sub-problem has a size of one, and solving the sub-problem is equivalent to fusing an initial subset of individual samples in the target sample set. Based on this, the sub-problems of size one are merged sequentially until all sub-problems are merged, thus completing the solution to the large problem of multi-source sample fusion for the target sample set and obtaining the result of multi-source sample fusion. The specific implementation process is as follows.

[0086] S120: Perform set decomposition on the target sample set to obtain multiple target sample decomposition sets, and use each target sample decomposition set as a set to be merged.

[0087] Generally, multi-source sample fusion of multiple initial subsets of a target sample set can be understood as dividing these initial subsets into different sample sets. Since the number of initial subsets is large, multi-source sample fusion of a target sample set is actually a complex problem of dividing a large number of initial subsets. Considering that complex problems are often difficult to solve directly, if a large-scale complex problem is decomposed into multiple independent subproblems, where these subproblems are smaller in scale and easier to solve, then the complex problem can be simplified by solving these smaller and easier subproblems.

[0088] In some embodiments, the problem of multi-source sample fusion of the target sample set can be decomposed into a subproblem of multi-source sample fusion of multiple initial subsets of samples by performing set decomposition on the target sample set. Set decomposition refers to performing a layer-by-layer set decomposition based on the target sample set. In each layer of set decomposition, each set to be decomposed at that layer can be decomposed into two sample decomposition subsets, until each sample decomposition subset obtained at a certain layer corresponds to an initial subset of samples in the target sample set, at which point the set decomposition stops.

[0089] Thus, through multi-level set decomposition, the complex problem of partitioning multiple initial subsets of the target sample set is decomposed layer by layer into subproblems that can be solved simply and directly. Then, through set reduction, the solutions to these subproblems are merged into the solution to the main sample set partitioning problem, thereby quickly dividing the multiple initial subsets of the target sample set into different sample sets.

[0090] In one implementation, the target sample set can be used as the set to be decomposed, and set decomposition can be performed on the set to be decomposed to obtain two sample decomposed sets. When the number of decomposed sets of the sample decomposed sets is not equal to the number of subsets corresponding to the target sample set, each sample decomposed set is used as the set to be decomposed, and the step of performing set decomposition on the set to be decomposed is returned until the number of decomposed sets of the sample decomposed sets obtained by set decomposition is equal to the number of subsets corresponding to the target sample set. At this point, set decomposition stops, and each target sample decomposed set obtained by stopping set decomposition is used as a set to be merged.

[0091] The step of returning to the set to be decomposed and performing set decomposition can be implemented recursively or iteratively. When the number of decomposition sets of the sample decomposition set is not equal to the number of subsets corresponding to the target sample set, it means that the sample decomposition set obtained by the current decomposition cannot be directly used for simple multi-source sample fusion. Therefore, the set decomposition set obtained by the current decomposition can be continued until the number of decomposition sets of the sample decomposition set obtained by set decomposition is equal to the number of subsets corresponding to the target sample set. At this point, it means that each sample decomposition set obtained by the current decomposition is actually an initial sample set in the target sample set, and each initial sample set itself can be directly used as the result of the corresponding multi-source sample fusion. That is, through the set decomposition layer by layer, the sample set partitioning subproblem that can be directly solved is finally obtained.

[0092] Specifically, the initial subset number of the initial subsets included in the target sample set is obtained, and the target sample set is used as the set to be decomposed in the current layer. Further, set decomposition is performed on each set to be decomposed in the current layer to obtain two sample decomposition subsets after decomposition for each set, and the number of decomposition subsets after set decomposition is obtained. When it is detected that the number of decomposition subsets is not equal to the initial subset number, the sample decomposition subsets after set decomposition can be used as the set to be decomposed in the next layer, and the set to be decomposed in the next layer is updated to the set to be decomposed in the current layer. The process of performing set decomposition on each set to be decomposed in the current layer to obtain two sample decomposition subsets after decomposition for each set is repeated iteratively in the next layer until the number of decomposition subsets after set decomposition equals the initial subset number, at which point set decomposition stops.

[0093] Please see Figure 8 , Figure 8 A flowchart illustrating a multi-source sample fusion process is shown. Figure 9 As shown, multi-source sample fusion can include a multi-level set decomposition stage and a multi-level set merging stage. In the multi-level set decomposition stage, the problem of partitioning the target sample set is addressed. The size of this problem can be considered as n, where n represents the number of initial subsets of samples in the target sample set. The decomposition is recursively performed layer by layer. First, the sample set partitioning problem of size n is recursively decomposed into two sample set partitioning subproblems of size n / 2. That is, the target sample set {S1, S2, S3, S4, ..., S...} is used as the set to be partitioned. n-2 S n-1 S n The sample decomposition is divided into sample decomposition sets {S1, S2, S3, ...} and sample decomposition sets {..., S...}. n-2 S n-1 S n}, and then the sample decomposition sets {S1, S2, S3, ...} and {..., S...} can be respectively decomposed. n-2 S n-1 S n The sample sets are divided separately.

[0094] Furthermore, by recursively decomposing downwards layer by layer, the two sample sets of size n / 2 are each recursively decomposed into two sample set subproblems of size n / 4. That is, the set to be decomposed {S1, S2, S3, ...} is decomposed into two sample decomposition sets, and then the set to be decomposed {..., S...} is further decomposed into two sample decomposition sets. n-2 S n-1 S nThe sample set is decomposed into two sets, and then each of these four sets can be further partitioned. This process is repeated layer by layer until the size of the subproblem is 1, that is, when the target sample set is obtained: {S1}, {S2}, ..., {S...}. n The recursive decomposition ends, and at this point, each target sample decomposition set obtained corresponds to an initial subset of the target sample set.

[0095] S130: For multiple sets to be merged, merge every two sets to be merged to obtain an intermediate merged set. Then, filter the intermediate merged set based on the merging standard threshold to obtain the corresponding sample merged set.

[0096] In this embodiment, the sample merge set refers to a set in which different initial subsets of samples are grouped together. During the set decomposition stage of the target set, when the size of the sample set partitioning subproblem is 1, it indicates that the current sample set partitioning subproblem is a problem that is easy to solve directly. This is because each sample decomposition subset at this time corresponds to an initial subset of the target sample set, and an initial subset can itself be directly regarded as a merged sample set, i.e., a sample merge set. For example, when the set decomposition stops, i.e., when the set decomposition of the target sample set is completed, each sample decomposition subset obtained corresponds to an initial subset of the target sample set. At this time, each sample decomposition subset is a sample merge set, for example, the i-th sample merge set Set. i =S i S i Let i be the initial subset of the i-th sample in the target sample set.

[0097] By performing multi-level set decomposition on the target sample set, the problem of partitioning the sample set for multiple initial subsets of the target sample set is decomposed layer by layer until it is finally decomposed into a sample set partitioning subproblem that can be solved simply and directly, that is, a sample set partitioning subproblem of size 1. At this point, based on the directly solvable sample set partitioning subproblem of size 1, each sample set partitioning subproblem can be solved layer by layer, and the solutions of each sample set partitioning subproblem can be merged into the solution of the sample set partitioning problem, that is, multi-level set merging.

[0098] To better understand the solution process for the sample set partitioning subproblem, please refer to [link / reference]. Figure 9 , Figure 8This diagram illustrates a process for problem decomposition and merging solutions. For example, suppose the target sample set includes four initial subsets. These four initial subsets are then fused using multi-source samples, meaning they are divided into different sample sets. To this end, the sample set partitioning problem Q1 (size 4) can be decomposed into two sample set partitioning problems Q2 and Q3 (size 2). Further, sample set partitioning problem Q2 is decomposed into two sample set partitioning problems Q4 and Q5 (size 1), and sample set partitioning problem Q3 is further decomposed into two sample set partitioning problems Q6 and Q7 (size 1).

[0099] When the decomposition yields sample set partitioning problems Q4, Q5, Q6, and Q7 of size 1, these four sample set partitioning problems of size 1 can be solved directly. That is, the initial subsets of samples corresponding to Q4, Q5, Q6, and Q7 are partitioned into sample sets, and the resulting sample sets are the solutions to each problem. Further, the solutions to sample set partitioning problems Q4 and Q5 are merged and used to solve sample set partitioning problem Q2, and the solutions to sample set partitioning problems Q6 and Q7 are merged and used to solve sample set partitioning problem Q3. Further, the solutions to sample set partitioning problems Q2 and Q3 are merged and used to solve sample set partitioning problem Q1. Thus, by first decomposing the sample set partitioning problem layer by layer, and then merging and solving each decomposed problem layer by layer, the target merged set after partitioning the target sample set can be obtained.

[0100] In some embodiments, set merging is actually a solution process for the problem of partitioning sample sets. Since many merged sample sets are obtained during the solution process, in order to ensure the quality of the merged sample sets, operations such as filtering can be performed on the merged sample sets. The target merged set is constructed based on the high-quality merged sample sets, which may specifically include the following steps:

[0101] (1) The two sets to be merged are sequentially combined in an ordered manner to obtain the corresponding intermediate merge set. An ordered combination refers to combining a sample set from one set to be merged with a sample set from the other set to be merged into a combined subset pair. The intermediate merge set can include multiple combined subset pairs. For example, ... Figure 10 As shown, two sets to be merged are combined into a single pair for ordered merging. Considering the possibility that the number of sets to be merged may be odd, a pair can also be formed by combining one set to be merged with an empty set. For multiple sets to be merged, each pair of sets can be sequentially combined in an ordered manner to obtain the intermediate merge set corresponding to each pair of sets.

[0102] In one implementation, the sets to be merged may include a first set to be merged and a second set to be merged. Further, based on the ordered combination of sample subsets in the first set to be merged and sample subsets in the second set to be merged, multiple combined subset pairs are obtained. Here, ordered combination means that in the set to be merged, each sample subset in the first set to be merged is sequentially paired with each sample subset in the second set to be merged, resulting in multiple combined subset pairs. For example, each sample subset in the first set to be merged and each sample subset in the second set to be merged can be subjected to a Cartesian product to generate multiple combined subset pairs.

[0103] Please see Figure 10 , Figure 1 This diagram illustrates a process for generating a sample merge set. Figure 10 As shown in Figure 0, the sets to be merged include the first set to be merged. Where v∈[1,2,…,V], and V represents the first set to be merged, Set. left The number of middle sample subsets SET. The second set to be merged. Where u∈[1,2,…,U], and U represents the second set to be merged, Set. right The number of sample subsets SET in the middle. For the first set to be merged, Set... left Each sample subset in the set and the second set to be merged in the set. right Performing a Cartesian product on each sample subset yields multiple pairs of combined subsets, which together form the first set to be merged, Set. left Second set to be merged Set right The corresponding intermediate merge set:

[0104]

[0105] (2) For each pair of combined subsets in the intermediate merge set, merge one of the combined subsets in turn, and calculate the gain based on the merged subsets to obtain the merge gain corresponding to each pair of combined subsets.

[0106] Since the main purpose of multi-source sample fusion is to merge samples from different data sources into multiple sample sets so as to model based on these multiple sample sets, it is necessary to evaluate the merging effect of the merged subsets obtained by merging the combined subset pairs. This is so that the combined subset pairs with the best merging effect can be selected from the multiple combined subset pairs in each intermediate merged set for modeling.

[0107] Considering that the merged samples often differ significantly in dimensionality, distribution, etc., directly judging the merging effect of these combined subset pairs is not easy. Therefore, the merging effect can be indirectly reflected by the gain generated by modeling with these combined subset pairs. This gain can reflect the performance difference between a model trained on the unmerged samples and a model trained on the merged samples (combined subset pairs). For example, ... Figure 11 As shown, the combination subset pairs Includes the first set to be merged (Set) left The first sample subset and from the second set to be merged Set right The second sample subset The sample merge subset is formed by the first sample subset. Second sample subset The fused sample set SET lr Merge gain measures the extent to which modeling based on merged subsets of samples improves model performance, thus reflecting the merging effect of combined subset pairs.

[0108] For example, based on the first sample subset Modeling was performed to obtain model L. This was based on the second sample subset. Modeling was performed to obtain model R. This was based on the sample merge subset SET. lr Modeling is performed to obtain model LR. If the performance of model LR is improved relative to the performance of model L and model R, it proves that the merging effect of the combined subset pairs is positive, and the merging gain is positive. If the performance of model LR is not improved or even decreased relative to the performance of model L or model R, it proves that the merging effect of the combined subset pairs is not positive, and the merging gain is not positive (negative merging gain or no merging gain). Therefore, to select the combined subset pairs whose merging effect can meet the modeling requirements of specific application scenarios from multiple combined subset pairs obtained from ordered combinations, we can first calculate the merging gain corresponding to each combined subset pair.

[0109] In some embodiments, the first and second sample subsets in the combined subset pair can be merged, and the model can be trained based on the merged sample subset to obtain a model based on the merged sample subset. The merge gain of the model based on the merged sample subset is calculated relative to the model based on the first sample subset and the model based on the second sample subset, respectively. Specific steps may include:

[0110] (2.1) For a combined subset pair, merge the first sample subset and the second sample subset to obtain the merged sample subset.

[0111] Since each sample subset is divided into training set, test set, and time-out set according to different functions, for example, the first sample subset includes the first training set, the first test set, and the first time-out set, and the second sample subset includes the second training set, the second test set, and the second time-out set, the two sample subsets can be merged according to the type of dataset.

[0112] As one implementation method, the first training set and the second training set are merged to obtain a merged training subset; the first test set and the second test set are merged to obtain a merged test subset; and the first time-out set and the second training set are merged to obtain a merged time-out subset. To evaluate the model's performance on different blind test objects, blind test sets corresponding to the sample subsets are also obtained for blind test set merging. Further, the first blind test set corresponding to the first sample subset and the second blind test set corresponding to the second sample subset are obtained, and the first blind test set and the second blind test set are merged to obtain a merged blind test subset. Then, based on the merged training subset, the merged test subset, the merged time-out subset, and the merged blind test subset, the sample merged subset is determined.

[0113] For example, merging different types of sample sets can concatenate two sample subsets by row or column to create a larger sample set. For instance, when two sample subsets have the same features (columns) but different samples (rows), the rows of the two subsets can be directly concatenated, resulting in a new sample set that includes all samples from both subsets. Similarly, when two sample subsets have the same samples (rows) but different features (columns), the columns of the two subsets can be directly concatenated to form a larger sample set that includes all features from both subsets. Optionally, if two sample subsets contain duplicate samples or features, deduplication or other preprocessing can be performed first. During concatenation, it's also possible to choose whether to ignore the original index or regenerate the index. The method for merging sample sets can depend on the structure of the sample sets and the analysis objectives, and is not limited here.

[0114] To accurately calculate the merge gain, this application proposes using statistical indicators that reflect the model's discriminative ability. These statistical indicators refer to the model's ability to distinguish between positive and negative samples: a larger statistical indicator indicates a better ability to differentiate between positive and negative samples, while a smaller indicator suggests a weaker discriminative ability. Statistical indicators may include the Kolmogorov-Smirnov statistic, the Receiver Operating Characteristic Curve (ROC curve), and the Gini coefficient, among others, and are not limited to these specific indicators.

[0115] The reason for calculating the merge gain based on statistical metrics is that while traditional model evaluation metrics such as accuracy, precision, and recall can measure the overall performance of a model, in some application scenarios, such as credit scoring and fraud detection, a more detailed evaluation of the model's performance on samples from different classes is needed. For example, accuracy may lose its effectiveness in cases of class imbalance; even if the model can hardly distinguish between positive and negative samples, the accuracy may still remain high.

[0116] Therefore, using these traditional evaluation metrics to assess the improvement of model performance by sample merging may lead to bias and misleading results. Statistical metrics, however, are independent of class proportions and can directly measure the model's ability to distinguish between positive and negative samples. Thus, statistical metrics are more advantageous than traditional model performance evaluation metrics in assessing model performance. Therefore, this application can accurately calculate the merging gain based on statistical metrics to accurately assess the impact of sample merging on model performance.

[0117] (2.2) Train the first model based on the first sample subset and calculate the first statistical index corresponding to the first model on the first sample subset.

[0118] When setting the multi-factor attenuation coefficient vector corresponding to the merging criterion threshold, this application sets corresponding attenuation thresholds based on datasets with different functions divided from the sample set. train DC test DC oot DC general ], dc train Used to evaluate the modeling gain on the training set, dc test Used to evaluate the modeling gain on the test set, dc oot Used to evaluate the modeling gain on the time outer set, dc general This is used to evaluate the modeling gain on the blind test set. Therefore, when calculating statistical indicators, it is necessary to calculate the statistical indicators for different datasets representing different functions of the model, according to the set of the merging criterion thresholds.

[0119] As one implementation method, a pre-set model is trained on a first training set to obtain a trained first model. Further, the merged model can be subjected to test discrimination statistics on a first test set to obtain a first test index, and the merged model can be subjected to out-of-time discrimination statistics on a first out-of-time set to obtain a first out-of-time index. Further, the merged model can be subjected to blind test discrimination statistics on a first blind test set to obtain a first blind test index. Then, based on the first test index, the first out-of-time index, and the first blind test index, the first statistical index corresponding to the first model on the first sample subset is determined.

[0120] For example, such as Figure 10The diagram illustrates the calculation of a statistical indicator. (Example: XGB) oo The St model and the KS statistic are used as statistical indicators. Sample merge subset {S} train S test S oot S general}, S train S represents the merging of training subsets. test Represents the merge test subset, S oot S represents the sum of outer subsets during merging. general This indicates the merging of blind test subsets. The first model, XGBoost, is obtained by training a predefined model using the training set based on the first sample subset. l Furthermore, the first XGBoost model can be tested on the test set of the first sample subset. l The test discrimination statistics were performed to obtain the first test index.

[0121] Specifically, the sample features of the test samples in the test set are input into the first model XGBoost. l The first model, XGBoost l Model inference is performed to obtain predicted values. Further, based on the sample label and predicted value for each test sample, the True Positive Rate (TPR) and False Positive Rate (FPR) for each test sample are calculated. Further, the absolute value of the difference between the TPR and FPR for each test sample is calculated, and the maximum value among these absolute values ​​is taken as the first test indicator. The calculation formula is as follows:

[0122]

[0123] in, This represents the first test index. Based on the calculation principle of the first test index, the first time-out index can be calculated separately. and the first blind test indicator Furthermore, based on the first test metric First time-out indicator and the first blind test indicator It is possible to construct indicator tuples about the first statistical indicator. Index l This represents a tuple index.

[0124] (2.3) Train the second model based on the second sample subset, and calculate the second statistical index corresponding to the second model on the second sample subset.

[0125] As one implementation method, a preset model is trained using a second training set to obtain the trained second model, XGBoost.r The merge model was tested and its discrimination was statistically analyzed on the second test set to obtain the second test metric; the second model XGBoost was tested on the second time set. r Perform out-of-time discrimination statistics to obtain the second out-of-time index; test the second model XGBoost on the second blind test set. r Blind test discrimination statistics were performed to obtain the second blind test index; based on the second test index, the second out-of-time index, and the second blind test index, the second model XGBoost was determined. r The second statistical indicator corresponding to the second sample subset. For example, the indicator tuple for the second statistical indicator. Specifically, the calculation process of the second statistical indicator is the same as step (2.2), and will not be repeated here.

[0126] (2.4) Train a merge model based on the merge subset of samples and calculate the merge statistics corresponding to the merge model on the merge subset of samples.

[0127] As one implementation method, a preset model is trained based on a merged training subset to obtain the trained merged model XGBoost. lr On the merge test subset, the merge model XGBoost was tested. lr Perform test discrimination statistics to obtain test statistical indicators; evaluate the XGBoost merging model on the outer subset during merging. lr Out-of-time discrimination statistics were performed to obtain out-of-time statistical indicators; the XGBoost merging model was tested on a merged blind test subset. lr Blind test discrimination statistics are performed to obtain blind test statistical indicators. Based on the test statistical indicators, out-of-time statistical indicators, and blind test statistical indicators, the merge statistical indicators corresponding to the merge model on the sample merge subset are determined. For example, the indicator tuple for the second statistical indicator. Specifically, the calculation process of the second statistical indicator is the same as step (2.2), and will not be repeated here.

[0128] (2.5) Calculate the merge gain corresponding to a combination subset pair using the first statistical indicator, the second statistical indicator, and the merge statistical indicator.

[0129] In the embodiments of this application, merge gain refers to the degree to which modeling based on merged subsets of samples can improve model performance. In other words, merge gain refers to the gain of model performance based on merged subsets of samples relative to model performance based on a first subset of samples, i.e., the first gain, and the gain of model performance based on merged subsets of samples relative to model performance based on a second subset of samples, i.e., the second gain.

[0130] As one implementation method, the difference between the first indicator and the first statistical indicator can be calculated, and the quotient of the difference between the first indicator and the first statistical indicator can be used as the first gain. Further, the difference between the second indicator and the second statistical indicator can be calculated, and the quotient of the difference between the second indicator and the second statistical indicator can be used as the second gain. Then, the merge gain corresponding to the merged subset pair can be determined based on the first gain and the second gain.

[0131] Since the calculation principles of the first gain and the second gain are the same, the following explanation will only use the first gain as an example to explain the gain calculation process, based on the first statistical index. and merged statistical indicators To calculate the first gain corresponding to the merged subset pair, it is necessary to calculate the gain on the datasets with different functions corresponding to the first sample subset. The calculation process is as follows:

[0132]

[0133] in, XGBoost represents the merge model lr Compared to the first model XGBoost l Test gain on the first test set, XGBoost represents the merge model lr Compared to the first model XGBoost l Temporal gain on the first temporal outer set, XGBoost represents the merge model lr Compared to the first model XGBoost l The blind test gain on the first blind test set. Further, a gain tuple about the first gain can be constructed from the test gain, the time-out gain, and the blind test gain. Index lr-l This is the index of the tuple. Following the calculation principle of the first gain, the gain tuple for the second gain can be calculated.

[0134] (3) Based on the merging standard threshold and the merging gain corresponding to each combination subset pair, multiple combination subset pairs of each intermediate merge set are filtered to obtain the sample merge set after filtering each intermediate merge set.

[0135] When constructing neural network models based on multi-source sample fusion, the business characteristics of different application scenarios mean that the specific business requirements must be considered when fusing samples from different data sources. Since each business has its unique goals and characteristics, simply relying on general evaluation metrics may not be sufficient to evaluate the modeling performance of the fused sample set in different business scenarios. Therefore, this application proposes that evaluation metrics need to be customized according to the specific characteristics of the business in order to accurately select the optimal combination subset pairs.

[0136] In this embodiment, the merging criterion threshold is an evaluation metric used to evaluate the fused sample set (i.e., the combined subset pairs). For example, the merging criterion threshold can be represented in the form of a multi-factor decay coefficient vector. Each dimension of this multi-factor decay coefficient vector represents a minimum standard for evaluating the gain based on the combined subset pairs in a business scenario, i.e., the decay threshold. For example, the multi-factor decay coefficient vector Decay_coeff[dc1, dc2, ..., dc] corresponding to the merging criterion threshold... z ], where z represents the vector dimension of the multi-factor decay coefficient vector. For example, based on the different purposes of the dataset divided into training set, test set, out-of-time set, and blind test set, corresponding decay thresholds [dc] are set. train DC test DC oot DC general ], dc train Used to evaluate the modeling gain on the training set, dc test Used to evaluate the modeling gain on the test set, dc oot Used to evaluate the modeling gain on the time outer set, dc general Used to evaluate the modeling gain on the blind test set.

[0137] In some embodiments, the sample merge set after filtering each intermediate merge set can be constructed by comparing the merge gain corresponding to each combined subset pair, selecting combined subset pairs that meet the merge criterion threshold as target subset pairs, and determining the number of shared samples for the target subset pairs and obtaining non-positive gain subset pairs. Specifically, the steps may include:

[0138] (3.1) For each intermediate merge set, compare the merge gain corresponding to each combination subset of the intermediate merge set with the merge standard threshold.

[0139] (3.2) Based on the comparison results, select the combined subset pairs whose merging gain is not less than the merging standard threshold, and use them as the target subset pairs corresponding to each intermediate merge set.

[0140] When obtaining the merge gain corresponding to each combination subset pair, the merge gain corresponding to each combination subset pair can be compared with the merge standard threshold, and then, as follows: Figure 10 As shown, the subset pairs that are less than the merging criterion threshold are filtered out, that is, the subset pairs that meet the minimum gain criterion for modeling in the business scenario are obtained, that is, the target subset pairs.

[0141] As one implementation method, for each combined subset pair, the first gain and the second gain corresponding to the combined subset pair are compared with the merge standard threshold. If the first gain is not less than the merge standard threshold and the second gain is not less than the merge standard threshold, then the combined subset pair is taken as the target subset pair. Taking the comparison of the first gain with the merge standard threshold as an example, the gain values ​​of the first gain on different datasets are compared with the gain standard of the corresponding dataset at the merge standard threshold.

[0142] For example, for the first gain and the merging standard threshold [dc] test DC oot DC general ],when Greater than dc test , Greater than dc oot and Greater than dc general When the first gain is greater than the merging criterion threshold, then the first gain is greater than the merging criterion threshold; otherwise, the first gain is not greater than the merging criterion threshold. Regarding the second gain... and the merging standard threshold [dc] test ,dc oot ,dc general ],when Greater than dc test , Greater than dc oot and Greater than dc general If the first gain is greater than the merging criterion threshold, then the second gain is greater than the merging criterion threshold; otherwise, the second gain is not greater than the merging criterion threshold. Further, if both the first gain and the second gain are greater than the merging criterion threshold, then the combined subset pair corresponding to the first gain and the second gain is taken as the target subset pair.

[0143] (3.3) For each intermediate merge set, obtain the gain descending order sorting determined by each target subset pair, and select the target subset pairs with a shared sample number greater than the shared sample threshold as available subset pairs from the first position of the gain descending order sorting.

[0144] The shared sample threshold represents the threshold number of shared samples in a target subset pair. Samples that are identical or similar between two sample subsets in a target subset pair are considered shared samples. Multi-source sample fusion divides sample sets from different data sources into multiple sample sets, thus enabling the divided sample sets to better reflect the relationships between samples and helping the model to perform analysis and prediction more accurately.

[0145] Ideally, the relationships between samples in each merged sample set can be expressed as sample similarity, correlation, or some underlying pattern. To ensure the quality of multi-source sample fusion, only when the number of shared samples reaches a certain threshold can it be said that these sample sets from different data sources have sufficient common information or features, thus making merging reasonable. If the number of shared samples is too small, it may indicate that there are significant differences between these data sources, and direct merging may lead to inaccurate results or loss of important characteristics of the original data. If the number of shared samples between two sample sets is very small, direct merging may introduce noise or bias, leading to distorted merging results. Therefore, in the process of multi-source sample fusion, to ensure that the samples in the merged sample sets have sufficient similarity and correlation, a target subset pair with a shared sample number greater than the shared sample threshold can be selected for sample merging, thereby improving the quality of the merged sample set, reducing noise and bias, and enhancing the reliability and accuracy of the merging results.

[0146] As one implementation method, the total gain can be calculated based on the first and second gains corresponding to each target subset pair. Further, the target subset pairs are sorted in descending order of gain based on the total gain corresponding to each target subset pair, resulting in a descending gain sort. Further, as... Figure 12 As shown, in the descending gain sorting, the number of shared samples in each target subset pair is judged in descending order of total gain. If not, the target subset pair is deleted (i.e., pruned). If so, the target subset pair is used as a usable subset pair.

[0147] For example, the first gain corresponding to the target subset pair is The second gain is The calculation process for the total gain is as follows:

[0148]

[0149] Among them, Gain totalThe total gain corresponds to the target subset pair. w1 represents the weight of the test gain, w2 represents the weight of the out-of-time gain, and w3 represents the weight of the blind test gain. The sum of w1, w2, and w3 is 1. The values ​​of w1, w2, and w3 can be set according to the modeling requirements of specific application scenarios. The larger the weight value, the more important the corresponding type of gain is in evaluating the gain.

[0150] Furthermore, when obtaining the total gain corresponding to each target subset pair, the target subset pairs can be sorted in descending order based on the total gain to obtain a gain descending order sort. Then, starting from the first position of the gain descending order sort, target subset pairs with a shared sample number greater than the shared sample threshold are selected as usable subset pairs. In this way, while prioritizing the merge pairs with higher gains (i.e., target subset pairs), merge pairs that do not meet the shared sample threshold are deleted, resulting in merge pairs with both high gains and a shared sample number that meets the shared sample threshold, i.e., usable subset pairs.

[0151] During the set merging process, when solving the subproblem of each sample partitioning, target subset pairs are selected based on the fusion standard threshold. This ensures that the fused target subset pairs meet the modeling gain requirements of specific application scenarios, thereby improving the flexibility of multi-source sample fusion in different application scenarios. Then, usable subset pairs are selected by sorting the gains determined by each target subset pair in descending order, thus obtaining the merged pairs with the best quality.

[0152] Thus, by using this greedy strategy, the optimal solution is found for each sample partitioning subproblem. Then, by merging a series of local optimal solutions to the sample partitioning subproblems, the solution process for the sample partitioning subproblems of the target sample set can be simplified, and the corresponding global optimal solution can be obtained quickly, thereby improving the efficiency and quality of multi-source sample fusion.

[0153] (3.4) From the two sets to be merged corresponding to the available subset pairs, determine the non-positive gain subset pairs corresponding to the available subset pairs.

[0154] In this context, there is no intersection between non-positive gain subset pairs and available subset pairs. A non-positive gain subset pair corresponding to an available subset pair refers to a subset pair consisting of sample subsets belonging to the two sets to be merged corresponding to the available subset pair, but not belonging to the available subset pair itself. Considering that available subset pairs, as positive gain sample subset pairs, exhibit better model performance improvement, it may still be necessary to include some non-positive gain sample subset pairs to ensure the generalization ability and robustness of the final model. While non-positive gain sample subset pairs do not significantly improve model performance, they may contain certain specific patterns or features that contribute to the model's performance in practical applications, especially when handling anomalies or low-probability events.

[0155] Furthermore, in the process of multi-source sample fusion, the model's performance may not solely depend on improved performance metrics, i.e., simply producing positive merge gains. It may also be affected by certain special cases where it does not show significant gains, such as being driven by specific business needs. Even if it does not show significant gains, non-positive gain subsets may play an important role in special cases such as business logic. Even if their gains are not as significant as those of positive gain subsets, they can still be retained to meet the needs of sample diversity.

[0156] As one implementation method, based on the available sample subsets in the available subset pair, non-positive gain subsets that do not match the available sample subsets are obtained from the first and second sets to be merged, and non-positive gain subset pairs are generated based on the non-positive gain subsets. See also... Figure 12 , Figure 12 A schematic diagram of a non-positive gain subset pair is shown. For example... Figure 10 As shown, the first and second sets to be merged are merged in an ordered manner, resulting in four combined subset pairs S. α S λ S α S ψ S β S λ and S β S ψ After performing operations such as gain comparison and gain descending sort, a usable subset pair S with positive gain is obtained. α S λ This leads to the non-positive gain subset pair S. β S ψ Among them, the non-positive gain subset pairs S β S ψ The sample subset S in the pair β and S ψ It belongs to the first and second sets to be merged, but not to the available subset pair S. α S λ .

[0157] (3.5) Construct the sample merge set corresponding to each intermediate merge set based on each available subset pair and the corresponding non-positive gain subset pair.

[0158] As one implementation method, such as Figure 8 As shown, after obtaining the available subset pairs and non-positive gain subset pairs corresponding to each intermediate merge set, the set consisting of each available subset pair and each non-positive gain subset pair can be used as the sample merge set after set merging.

[0159] S140: When the number of sets in the sample merge set is not one, treat each sample merge set as a set to be merged, and return the step of merging every two sets to be merged until the number of sets in the sample merge set obtained by the set merging is one, and obtain the target merge set.

[0160] By performing multi-level set decomposition on the target sample set, the problem of partitioning the sample set for multiple initial subsets of the target sample set is decomposed layer by layer until it is finally decomposed into a sample set partitioning subproblem that can be solved simply and directly, that is, a sample set partitioning subproblem of size 1. Then, based on the directly solvable sample set partitioning subproblem of size 1, each sample set partitioning subproblem can be solved layer by layer, and the solutions of each sample set partitioning subproblem can be merged into the solution of the sample set partitioning problem, that is, multi-level set merging.

[0161] As one implementation method, when the number of sets in the sample merge set is not one, it means that the sample set partitioning problem for the target sample set has not yet been solved. Therefore, the set merging is not finished. So, the sample merge set after the current set merging can be used as the set to be merged, and the set merging step can be continued until the number of sets in the sample merge set obtained by set merging is one, and the target merge set is obtained.

[0162] like Figure 13 As shown, when the size of the sample set partitioning subproblem is 1, the set decomposition ends. At this point, each sample subset obtained by the set decomposition corresponds to an initial subset of the target sample set. Then, the multi-level set merging stage begins. Initially, for a sample set partitioning problem of size 1, the initial subset itself can be directly used as a sample merge set Set. i =S i That is, the solution to the sample set partitioning problem of size 1. At this point, since the number of sample merge sets (n) is not one, it means that the sub-problem of sample set partitioning of size n / 2 has not yet been merged. That is, the multi-level set merging is not yet finished. Therefore, the solutions of the two sample set partitioning problems of size 1 can be merged and used to solve the sample set partitioning problem of size 2. That is, the sample merge set of the current level is used as the set to be merged in the next level. This set merging continues until the number of sample merge sets after merging is one, indicating that the sub-problem of sample set partitioning of size n / 2 has been merged, and the multi-level set merging is finished. The sample merge set obtained after the set merging is the target merge set T = {T1, T2, ..., T}. k}, where 0 < k and k ∈ N * , k represents the number of sample sets T obtained by merging the target merge set T.

[0163] For example, such as Figure 14 The diagram illustrates the generation of a sample merge set. The target sample set S = {S1, S2, ..., S...} i ,…,S n}. Wherein, the i-th initial subset of samples Each sample includes its corresponding sample features and sample labels. After the set decomposition and set merging as described in steps S110 to S140 above, multiple sample sets {T1, T2, ..., T} can be obtained. k In this context, each sample set T can be used to train a corresponding neural network model for use in a specific service scenario. For example, multi-source financial samples obtained from multiple financial institutions can be fused using this application to obtain three sample sets. These three sample sets can then be used to train loan default risk prediction models for business default risk prediction scenarios, credit fraud detection models for credit card fraud detection scenarios, and customer churn prediction models for customer churn prediction scenarios.

[0164] This embodiment can obtain a target sample set and perform set decomposition on the target sample set to obtain multiple target sample decomposition sets. Each target sample decomposition set is then used as a set to be merged. Thus, the complex sample set partitioning problem for the target sample set is decomposed into a simpler sample set partitioning sub-problem. Further, for multiple sets to be merged, set merging is performed on every two sets to obtain intermediate merged sets. These intermediate merged sets are then filtered based on a merging criterion threshold to obtain the corresponding sample merged sets. When the number of sample merged sets is not one, each sample merged set is used as a set to be merged, and the step of merging every two sets to be merged is repeated until the number of sample merged sets obtained is one, thus obtaining the target merged set.

[0165] By using set decomposition, the complex problem of partitioning the target sample set is reduced to simpler, directly solvable subproblems. Then, through set merging, the solutions to these subproblems are merged into the solution to the main partitioning problem, yielding the target merge set. This merge-and-conquer strategy, based on decomposition followed by merging, has a time complexity of O(n^2) compared to an exhaustive search strategy. 2 Based on the relevant technologies, this application can quickly complete the fusion of multi-source samples of the target sample set with a linear time complexity of only O(nlog2n), thereby improving the efficiency of multi-source sample fusion.

[0166] By using a merging criterion threshold, the optimal solution to the sample set partitioning subproblem is obtained in set merging using a local greedy strategy. That is, the intermediate merged set after merging is filtered to obtain a high-quality sample merged set that better meets the application scenario requirements. Thus, the optimal solutions of each sample set partitioning subproblem are merged into the optimal solution of the sample set partitioning problem, resulting in a high-quality target merged set. Compared with related technologies, this can improve the quality of multi-source sample fusion.

[0167] In the process of multi-source sample fusion of a target sample set, the final fusion result is affected by the combination order of the initial subsets of samples in the target sample set. This application proposes a sample clustering method that performs multiple sample fusions on the initial subsets of the target sample set to improve the stability of multi-source sample fusion. This data processing method can be executed by computer equipment (e.g., servers, edge computing devices, or other terminal devices with certain processing capabilities), and the computer equipment must at least have storage, computing, and communication functions. Please refer to [link to relevant documentation]. Figure 14 , Figure 15 A flowchart illustrating a sample clustering method provided in this embodiment is shown.

[0168] In the embodiments of this application, multi-source sample fusion is the task of dividing multiple initial subsets of samples in the target sample set into different sample clusters. However, the division result is easily affected by the different combination order of the initial subsets of samples in the target sample set.

[0169] On the one hand, this impact may stem from the randomness and initialization mechanisms of computer devices when performing partitioning tasks. When a computer device relies on initial conditions to perform a partitioning task, the order of multiple initial subsets of samples in the target sample set may directly affect the selection of these initial conditions, leading to differences in results. For example, different initial conditions may cause the algorithm to run on different paths, ultimately converging to different solutions. Furthermore, during the incremental update or iteration process, if the combination order of the initial subsets of samples in the target sample set is different, it may affect the computation order of each step, thereby changing the final output result. This is particularly evident when dealing with the complex problem of multi-source sample fusion.

[0170] On the other hand, a key factor contributing to this effect may be the problem of local optima. When a computer device relies on heuristic search or local optimization, initial conditions (such as the order of combinations of initial subsets of samples) may guide the computer device to converge to different local optima. Thus, even when faced with the same dataset, i.e., the target sample set, the computer device may give different partitioning results due to different combinations of initial subsets of samples in the target sample set.

[0171] Therefore, the order of multiple initial subsets of samples in the target sample set plays a crucial role in multi-source sample fusion tasks. To address this uncertainty, this application proposes to reduce its impact by running set decomposition and set merging multiple times with different combinations of initial subsets and integrating multiple merging results, thereby obtaining more stable and reliable merging results. The following will combine... Figure 16 The flowchart shown illustrates the specific steps of this data processing method for clustering samples.

[0172] S210: Initialize the number of reorganization and merging operations.

[0173] In the embodiments of this application, the number of reorganization and merging operations refers to the number of times set decomposition and set merging are run multiple times. As one implementation, the number of reorganization and merging operations can be initialized before starting to run set decomposition and set merging multiple times, for example, by setting the number of reorganization and merging operations to zero or one.

[0174] S220: Reorganize the initial subsets of multiple samples in the target sample set into a subset order to obtain a reorganized sample set.

[0175] As one implementation method, subset order reorganization refers to the random sequential reorganization of multiple initial subsets of samples. By performing subset order reorganization on multiple initial subsets of samples in the target sample set, multiple initial subsets of samples with different arrangements can be obtained, forming a reorganized sample set.

[0176] S230: Perform set decomposition and set merging on the recombined sample set, store the resulting merged recombined sample set, and update the numerical value of the recombination and merging count.

[0177] As one implementation method, the recombined sample set after subset sequential recombination can be subjected to set decomposition and set merging, and the resulting merged recombined sample set can be stored. The specific processes of set decomposition and set merging can be found in steps S110 to S140 of the above embodiments, and will not be elaborated here. Furthermore, a numerical update operation is performed on the recombination and merging count, for example, by incrementing the recombination and merging count by one to obtain the updated recombination and merging count.

[0178] S240: When it is detected that the number of recombination and merging does not meet the preset recombination and merging condition, return to the iterative execution of the step of sequentially recombining multiple initial subsets of samples in the target sample set to obtain the recombined sample set, until the number of recombination and merging meets the preset recombination and merging condition, and then terminate the iteration.

[0179] As one implementation, after updating the number of recombinations and merges, it is possible to check whether the number of recombinations and merges meets the preset recombination and merge conditions. If the current number of recombinations and merges does not meet the preset recombination and merge conditions, steps S220 and S230 can be executed iteratively until the number of recombinations and merges meets the preset recombination and merge conditions, at which point the iteration is terminated.

[0180] S250: Based on the stored merged sets of multiple recombined samples, determine the merge co-occurrence matrix corresponding to the target sample set.

[0181] The merge co-occurrence matrix (MCO) is a matrix that describes the co-occurrence relationships among the initial subsets of samples in a merged set of multiple recombined samples. The MCO is a square matrix of size n×n for a target sample set with n initial subsets. The matrix elements represent the probability that two initial subsets (i.e., co-occurrence subset pairs) belong to the same sample type.

[0182] As one implementation method, after performing set decomposition and set merging on the target sample set through multiple iterations, the merge co-occurrence matrix corresponding to the target sample set can be calculated based on the stored multiple recombined sample merge sets. For example, based on the stored multiple recombined sample merge sets, the probability of each co-occurrence subset pair belonging to the same recombined sample merge set is determined, and the merge co-occurrence matrix corresponding to multiple initial subsets of samples is determined according to the probability of each co-occurrence subset pair.

[0183] For example, suppose there are five initial subsets of samples {A, B, C, D, E}. After three iterations of multi-level set decomposition and multi-level set merging, the merged sample sets corresponding to each iteration are obtained as follows:

[0184] The merged set of recombined samples corresponding to the first iteration is {{A,B},{C,D,E}};

[0185] The merged set of recombined samples corresponding to the second iteration is {{A,B,C},{D,E}};

[0186] The merged set of recombined samples corresponding to the third iteration is {{A,B,D},{C,E}};

[0187] Based on the above three iterations, we can statistically conclude that: A and B: In all three iterations, A and B were assigned to the same subset, therefore their co-occurrence probability is 1; A and C: In all three iterations, A and C were in the same subset only once, therefore their co-occurrence probability is 1 / 3; C and E: In all three iterations, C and E were in the same subset twice, therefore their co-occurrence probability is 2 / 3; A and E were never in the same subset in all three iterations, therefore their co-occurrence probability is 0. This leads to the corresponding merge co-occurrence matrix:

[0188]

[0189] In this matrix, the diagonal elements represent samples with a co-occurrence probability of 1, as they are always in the same subset. The merge co-occurrence matrix provides a more intuitive view of the probability that initial subsets of samples will be grouped together into the same sample type across multiple runs.

[0190] S260: Based on each element in the merge co-occurrence matrix, cluster multiple initial subsets of samples, and determine the target sample cluster set corresponding to the multiple initial subsets of samples based on the clustering results.

[0191] As one implementation method, after obtaining the merge co-occurrence matrix, multiple initial subsets of samples can be clustered based on each element of the merge co-occurrence matrix. The clustering method can include various clustering methods such as K-Means clustering and hierarchical clustering, which are limited in this step. For example, taking K-Means clustering, the matrix elements in the merge co-occurrence matrix can serve as the probability that the initial subsets of samples are commonly assigned to the same cluster. The merge co-occurrence matrix can then be transformed into a distance matrix, a common method being the calculation of the 1-same-category probability. Further, K-Means clustering is performed using this distance matrix, ultimately yielding multiple stable cluster sets of initial subsets of samples. These cluster sets can constitute the target sample cluster set.

[0192] For example, to verify the effectiveness of the data processing method proposed in this application on multi-source sample fusion tasks, a specific verification experiment can be designed. This verification experiment includes modeling the sample set after multi-source sample fusion and calculating the KS value of the model. The verification method is a control verification, that is, on the same verification dataset, the KS values ​​of existing methods and the method of this application are calculated respectively.

[0193] Experimental data: 31 modeling samples and 15 corresponding blind test samples;

[0194] Experimental metric: KS (Kolmogorov-Smirnov) value;

[0195] Experimental subjects: existing methods (Benchmark) and the method of this application;

[0196] Experiment Implementation:

[0197] (1) The business rule-based fusion method, as an existing technology, performs multi-source sample fusion on 31 modeling samples to obtain 3 sample sets, and generates 3 XGBoost models based on the 3 sample sets. Then, KS values ​​are calculated on 31 modeling samples and 15 blind test samples based on these 3 XGBoost models, and the optimal KS value among the 3 XGBoost models is taken as the comparison value.

[0198] (2) Using the method of this application, multi-source sample fusion is performed on 31 modeling samples to obtain 5 sample sets. Based on the 5 sample sets, 5 XGBoost models are generated respectively. Then, KS values ​​are calculated on the 31 modeling samples and 15 blind test samples according to these 5 XGBoost models. The optimal KS value among the 5 XGBoost models is taken as the comparison value.

[0199] Experimental analysis: Please refer to Figure 16 , Figure 16 A schematic diagram illustrating the effect of multi-source sample fusion is shown. For example... Figure 17 As shown, `benchmark_best` represents the best performance of the benchmark method on the corresponding customer group, and `spl_best` represents the best result of the model partitioned by the method in this application on the corresponding customer group. It can be seen that the method in this application outperforms the benchmark method on almost all customer groups. The test performance on the modeled samples improves by 2.67ks points, while the improvement on cross-customer group samples is 0.5ks points, which also demonstrates the effectiveness of the method in this application.

[0200] This embodiment initializes the number of recombination and merging iterations, and performs subset sequential recombination on multiple initial subsets of samples in the target sample set to obtain a recombined sample set. Further, it performs set decomposition and set merging on the recombined sample set, stores the resulting recombined sample merge set, and updates the recombination and merging iterations. When the number of recombination and merging iterations is found to be less than the preset recombination and merging condition, the iteration returns to iteratively performing subset sequential recombination on multiple initial subsets of samples in the target sample set to obtain a recombined sample set, until the number of recombination and merging iterations meets the preset recombination and merging condition, at which point the iteration terminates. Based on the stored multiple recombined sample merge sets, the merge co-occurrence matrix corresponding to the target sample set is determined. Further, based on each element in the merge co-occurrence matrix, multiple initial subsets of samples are clustered, and the target sample cluster set corresponding to the multiple initial subsets of samples is determined based on the clustering results.

[0201] By performing multiple multi-level set decompositions and multi-level set mergings through subset order recombination, each multi-level set decomposition and merge provides a different sample merging perspective as the number of decompositions and merges increases. After multiple multi-level set decompositions and merges, sufficient sample co-occurrence information can be accumulated, thereby reducing sample merging interference caused by different combination orders of initial sample subsets in the target sample set. At this point, multiple initial sample subsets are clustered based on a stable co-occurrence matrix to obtain stable sample clusters. Thus, based on the multi-source sample merging solution strategy of divide and conquer followed by merging, sample clustering based on the co-occurrence matrix ensures that the final merged sample set is both high-quality and stable.

[0202] Please see Figure 18 This diagram illustrates a structural block diagram of a data processing apparatus 300 provided in an embodiment of this application. The apparatus 300 may include:

[0203] The set acquisition module 310 is used to acquire the target sample set;

[0204] The set decomposition module 320 is used to perform set decomposition on the target sample set to obtain multiple target sample decomposition sets, and to use each target sample decomposition set as a set to be merged.

[0205] The first merging module 330 is used to perform set merging on every two sets to be merged for multiple sets to be merged, to obtain an intermediate merged set after set merging, and to filter the intermediate merged set based on the merging standard threshold to obtain a sample merged set corresponding to every two sets to be merged.

[0206] The second merging module 340 is used to treat each sample merging set as a set to be merged when the number of merging sets in the sample merging set is not one, and return the step of merging every two sets to be merged until the number of merging sets in the sample merging set obtained by the set merging is one, and obtain the target merging set.

[0207] In some embodiments, the first merging module 330 may include an intermediate set determination unit, a merging gain calculation unit, and a combined subset filtering unit.

[0208] An intermediate set determination unit is used to sequentially combine two sets to be merged in an ordered manner to obtain a corresponding intermediate merge set, wherein the intermediate merge set includes multiple pairs of combined subsets;

[0209] The merge gain calculation unit is used to merge one combined subset pair in turn for each intermediate merge set, and calculate the gain based on the merged subset to obtain the merge gain corresponding to each combined subset pair.

[0210] The combined subset filtering unit is used to filter multiple combined subset pairs of each intermediate merge set according to the merge criterion threshold and the merge gain corresponding to each combined subset pair, so as to obtain the sample merge set after filtering each intermediate merge set.

[0211] In some embodiments, the merge gain calculation unit may include a first subunit, a second subunit, a third subunit, a fourth subunit, and a fifth subunit.

[0212] The first subunit is used to merge the first sample subset and the second sample subset for a given combination subset pair to obtain a merged sample subset;

[0213] The second subunit is used to train a first model based on the first sample subset and calculate the first statistical index corresponding to the first model on the first sample subset.

[0214] The third subunit is used to train the second model based on the second sample subset and calculate the second statistical index corresponding to the second model on the second sample subset.

[0215] The fourth subunit is used to train a merging model based on the sample merging subset and to calculate the merging statistics corresponding to the merging model on the sample merging subset.

[0216] The fifth subunit is used to calculate the merge gain corresponding to the combined subset pair using the first statistical indicator, the second statistical indicator, and the merge statistical indicator.

[0217] In some embodiments, the first sample subset includes a first training set, a first test set, and a first time-out set; the second sample subset includes a second training set, a second test set, and a second time-out set; and the first subunit may be specifically used for:

[0218] Merge the first training set and the second training set to obtain a merged training subset.

[0219] Merge the first test set and the second test set to obtain a merged test subset;

[0220] Merge the first time-out set and the second training set to obtain a merged time-out subset;

[0221] Obtain the first blind test set corresponding to the first sample subset and the second blind test set corresponding to the second sample subset;

[0222] The first blind test set and the second blind test set are merged to obtain a merged blind test subset;

[0223] Based on the merged training subset, the merged test subset, the merged out-of-time subset, and the merged blind test subset, a sample merge subset is determined.

[0224] In some embodiments, the fourth subunit may be specifically used for:

[0225] The preset model is trained based on the merged training subset to obtain the trained merged model.

[0226] The test discrimination statistics of the merge model are performed on the merge test subset to obtain test statistical indicators;

[0227] The out-of-time discrimination index of the merging model is obtained by performing out-of-time discrimination statistics on the out-of-time subset of the merging process.

[0228] Blind test discrimination statistics are performed on the merged blind test subset to obtain blind test statistical indicators;

[0229] Based on the test statistics, the out-of-time statistics, and the blind test statistics, the merge statistics corresponding to the merge model on the sample merge subset are determined.

[0230] In some embodiments, the fifth subunit may be specifically used for:

[0231] Calculate the first indicator difference between the merged statistical index and the first statistical index, and use the quotient of the first indicator difference and the first statistical index as the first gain;

[0232] Calculate the second indicator difference between the merged statistical index and the second statistical index, and use the quotient of the second indicator difference and the second statistical index as the second gain;

[0233] The merge gain corresponding to the combined subset pair is determined based on the first gain and the second gain.

[0234] In some embodiments, the combined subset filtering unit may include a first sub-sub-unit, a second sub-sub-unit, a third sub-sub-unit, a fourth sub-sub-unit, and a fifth sub-sub-unit.

[0235] The first sub-unit is used to compare the merge gain corresponding to each combination subset of the intermediate merge set with the merge standard threshold for each intermediate merge set.

[0236] The second sub-unit is used to select a combination subset pair whose merging gain is not less than the merging standard threshold based on the comparison results, and use it as the target subset pair corresponding to each intermediate merge set.

[0237] The third sub-unit is used to obtain the gain descending order sort determined by each target subset pair for each intermediate merge set, and select target subset pairs with a shared sample number greater than the shared sample threshold as available subset pairs from the first position of the gain descending order sort.

[0238] The fourth sub-unit is used to determine, from the two sets to be merged corresponding to the available subset pair, the non-positive gain subset pair corresponding to the available subset pair, wherein there is no intersection between the non-positive gain subset pair and the available subset pair;

[0239] The fifth sub-unit is used to construct the sample merge set corresponding to each intermediate merge set based on each available subset pair and the corresponding non-positive gain subset pair.

[0240] In some embodiments, the fourth subunit may be specifically used for:

[0241] Based on the available sample subsets in the available subset pair, obtain the non-positive gain subsets that do not match the available sample subsets from the two sets to be merged corresponding to the available subset pair;

[0242] Based on the non-positive gain subset, generate non-positive gain subset pairs.

[0243] In some embodiments, the set decomposition module 320 may be specifically used for:

[0244] The target sample set is taken as the set to be decomposed;

[0245] Perform set decomposition on the set to be decomposed to obtain two sample decomposed sets;

[0246] When the number of decomposed sets of the sample decomposition set is not equal to the number of subsets corresponding to the target sample set, each sample decomposition set is taken as a set to be decomposed, and the step of performing set decomposition on the set to be decomposed is returned until the number of decomposed sets of the sample decomposition set obtained by set decomposition is equal to the number of subsets corresponding to the target sample set, and the set decomposition is stopped.

[0247] Each target sample decomposition set obtained from the stopping set decomposition is used as the set to be merged.

[0248] This embodiment can obtain a target sample set and perform set decomposition on it to obtain multiple target sample decomposition sets. Each target sample decomposition set is then used as a set to be merged. Thus, the complex sample set partitioning problem for the target sample set is decomposed into a simpler sample set partitioning sub-problem. Further, for multiple sets to be merged, set merging is performed on every two sets to obtain intermediate merge sets. These intermediate merge sets are then filtered based on a merging criterion threshold to obtain the corresponding sample merge set. When the number of sample merge sets is not one, each sample merge set is used as a set to be merged, and the step of merging every two sets to be merged is repeated until the number of sample merge sets obtained is one, resulting in the target merge set. Through set decomposition, the complex sample set partitioning problem for the target sample set is finally decomposed into sample set partitioning sub-problems that can be solved simply and directly. Furthermore, by combining the solutions to the sub-problems of partitioning each sample set into a single solution to the overall sample set partitioning problem, the target merged set is obtained. This merge-and-conquer strategy, based on decomposition followed by merging, achieves faster multi-source sample fusion of the target sample set with lower linear time complexity compared to exhaustive search strategies, thus improving the efficiency of multi-source sample fusion. In addition, by using a merging criterion threshold, a local greedy strategy is employed to find the optimal solution to the sub-problems of partitioning the sample set during set merging. This involves filtering the intermediate merged sets after set merging to obtain a high-quality merged set that better meets the application scenario requirements. This merges the optimal solutions to the sub-problems of partitioning each sample set into the optimal solution to the overall sample set partitioning problem, resulting in a high-quality target merged set. Compared to related technologies, this improves the quality of multi-source sample fusion.

[0249] Please see Figure 19 This diagram illustrates a structural block diagram of a sample clustering device 400 provided in an embodiment of this application. The device 400 may include...

[0250] The count determination module 410 is used to initialize the number of recombination and merging operations;

[0251] The sequential recombination module 420 is used to perform sequential recombination of multiple initial subsets of samples in the target sample set to obtain a recombined sample set.

[0252] The collection storage module 430 is used to perform multi-level collection decomposition and multi-level collection merging on the recombined sample collection according to the data processing method according to any one of claims 1 to 10, store the resulting recombined sample merged collection, and perform a numerical update operation on the number of recombination and merging.

[0253] The iterative execution module 440 is used to return to the step of iteratively executing the initial subset of multiple samples in the target sample set to perform subset sequential recombination to obtain a recombined sample set when the number of recombination and merging is not satisfied with the preset recombination and merging condition, until the iteration is terminated when the number of recombination and merging meets the preset recombination and merging condition;

[0254] The matrix determination module 450 is used to determine the merge co-occurrence matrix corresponding to the target sample set based on multiple stored merge sets of recombined samples;

[0255] The sample clustering module 460 is used to cluster the multiple initial subsets of samples according to each matrix element in the merge co-occurrence matrix, and determine the target sample cluster set corresponding to the multiple initial subsets of samples based on the clustering results.

[0256] In some embodiments, the matrix determination module may be specifically used to: determine multiple co-occurrence subset pairs for the multiple initial subsets of samples; determine the probability of each co-occurrence subset pair belonging to the same recombined sample merge set based on the stored multiple recombined sample merge sets; and determine the merge co-occurrence matrix corresponding to the multiple initial subsets of samples according to the probability of belonging to the same recombined sample merge set.

[0257] This embodiment can perform multiple multi-level set decompositions and multi-level set mergings based on subset order recombination. As the number of multi-level set decompositions and multi-level set mergings increases, each multi-level set decomposition and multi-level set merging will provide a different sample merging perspective. After multiple multi-level set decompositions and multi-level set mergings, sufficient sample co-occurrence information can be accumulated, thereby reducing sample merging interference caused by different combination orders of the initial sample subsets in the target sample set. At this time, multiple initial sample subsets are clustered based on a stable co-occurrence matrix to obtain stable sample clusters, thereby effectively enhancing the stability of the merging results.

[0258] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the described devices, modules, etc., can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0259] In the several embodiments provided in this application, the coupling between modules can be electrical, mechanical, or other forms of coupling.

[0260] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0261] likeFigure 20 As shown in the figure, this application embodiment also provides a computer device 500, which includes a processor 510, a memory 520, a power supply 530, and an input unit 540. The memory 520 stores a computer program, which, when called by the processor 510, can execute the various method steps provided in the above embodiments. Those skilled in the art will understand that the structure of the computer device shown in the figures does not constitute a limitation on the computer device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0262] Processor 510 may include one or more processing cores. Processor 510 connects to various parts of the entire battery management system using various interfaces and lines. It executes instructions, programs, instruction sets, or program sets stored in memory 520, calls data stored in memory 520, performs various functions and processes data within the battery management system, and performs various functions and processes data within the computer device, thereby providing overall control of the computer device. Optionally, processor 510 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 510 may integrate one or more of a Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 510 and may be implemented separately using a communication chip.

[0263] The memory 520 may include random access memory (RAM) or read-only memory (ROM). The memory 520 can be used to store instructions, programs, instruction sets, or program assemblies. The memory 520 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described above. The data storage area may also store data created during the use of the computer device (such as phonebook and audio / video data). Accordingly, the memory 520 may also include a memory controller to provide the processor 510 with access to the memory 520.

[0264] The power supply 530 can be logically connected to the processor 510 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 530 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0265] The input unit 540 can be used to receive input digital or character information, and to generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0266] Although not shown, the computer device 500 may also include an output unit, a display unit, etc., which will not be described in detail here. Specifically, in the embodiments of this application, the processor 510 of the computer device loads the executable files corresponding to the processes of one or more computer programs into the memory 520 according to the following instructions, and the processor 510 runs the data stored in the memory 520, such as telephone books and audio and video data, thereby implementing the various method steps provided in the foregoing embodiments.

[0267] like ​ As shown, this application embodiment also provides a computer-readable storage medium 600, which stores a computer program or computer instructions 610. The computer program or computer instructions 610 can be invoked by the processor of a computer device to execute various method steps provided in this application embodiment.

[0268] Computer-readable storage media can be electronic storage devices such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, computer-readable storage media include non-volatile computer-readable storage medium. Computer-readable storage medium 600 has storage space for computer programs or computer instructions that perform any of the method steps in the above embodiments. These computer programs or computer instructions can be read from or written to one or more computer program products. The computer programs or computer instructions can be compressed in an appropriate form.

[0269] According to one aspect of this application, a computer program product is provided, comprising a computer program or computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer program or computer instructions from the computer-readable storage medium and executes the computer program or computer instructions, causing the computer device to perform the various method steps provided in the above embodiments.

[0270] The terms "comprising" and "including," and any variations thereof, in the specification and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such process, method, product, or apparatus. The terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined objective, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) may be used to implement one or more modules or units. Furthermore, each module or unit may be part of a larger module or unit that includes the functionality of that module or unit.

[0271] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0272] It should be understood that in the description of the embodiments of this application, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.

[0273] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0274] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units.

[0275] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. It should also be understood that the various implementation methods provided in the embodiments of this application can be arbitrarily combined to achieve different technical effects.

[0276] The above are merely preferred embodiments of this application and are not intended to limit this application in any way. Although this application has disclosed preferred embodiments as above, it is not intended to limit this application. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the technical solution of this application. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of this application without departing from the scope of the technical solution of this application shall still fall within the scope of the technical solution of this application.

Claims

1. A data processing method, characterized by, The method comprises: acquiring a target sample set; performing set decomposition on the target sample set to obtain a plurality of target sample decomposition sets, and taking each target sample decomposition set as a to-be-merged set respectively; for a plurality of to-be-merged sets, performing set merging on each two to-be-merged sets to obtain an intermediate merged set after set merging, and performing filtering on the intermediate merged set based on a merging standard threshold to obtain a corresponding sample merged set; when the number of sets of the sample merged set is not one, taking each sample merged set as a to-be-merged set, and returning to the step of performing set merging on each two to-be-merged sets until the number of sets of the sample merged set obtained by set merging is one, thereby obtaining a target merged set.

2. The method of claim 1, wherein, The step of performing set merging on each two to-be-merged sets to obtain an intermediate merged set after set merging, and performing filtering on the intermediate merged set based on a merging standard threshold to obtain a corresponding sample merged set comprises: performing ordered combination on two to-be-merged sets in sequence to obtain a corresponding intermediate merged set, wherein the intermediate merged set comprises a plurality of combined subset pairs; for the plurality of combined subset pairs in each intermediate merged set, performing merging on one combined subset pair in sequence, and performing gain calculation based on a merged subset obtained by merging to obtain a merging gain corresponding to each combined subset pair; according to the merging standard threshold and the merging gain corresponding to each combined subset pair, filtering the plurality of combined subset pairs in each intermediate merged set to obtain a sample merged set after filtering of each intermediate merged set.

3. The method of claim 2, wherein, The combined subset pair comprises a first sample subset and a second sample subset, and the step of performing merging on one combined subset pair and performing gain calculation based on a merged subset obtained by merging comprises: for one combined subset pair, performing merging on the first sample subset and the second sample subset to obtain a sample merged subset; training a first model according to the first sample subset, and calculating a first statistical index corresponding to the first model on the first sample subset; training a second model according to the second sample subset, and calculating a second statistical index corresponding to the second model on the second sample subset; training a merging model according to the sample merged subset, and calculating a merging statistical index corresponding to the merging model on the sample merged subset; calculating the merging gain corresponding to the one combined subset pair through the first statistical index, the second statistical index and the merging statistical index.

4. The method of claim 3, wherein, The first sample subset comprises a first training set, a first test set and a first out-of-time set, the second sample subset comprises a second training set, a second test set and a second out-of-time set, and the step of performing merging on the first sample subset and the second sample subset to obtain a sample merged subset comprises: performing training set merging on the first training set and the second training set to obtain a merged training subset; performing test set merging on the first test set and the second test set to obtain a merged test subset; performing out-of-time set merging on the first out-of-time set and the second training set to obtain a merged out-of-time subset; obtain a first blind test set corresponding to the first sample subset and a second blind test set corresponding to the second sample subset; perform blind test set merging on the first blind test set and the second blind test set to obtain a merged blind test subset; determine a sample merged subset based on the merged training subset, the merged test subset, the merged out-of-time subset and the merged blind test subset.

5. The method of claim 4, wherein, training a merged model according to the sample merged subset, and calculating a merged statistical index corresponding to the merged model on the sample merged subset, comprising: perform model training on a preset model according to the merged training subset to obtain a trained merged model; perform test discriminant statistics on the merged model on the merged test subset to obtain a test statistical index; perform out-of-time discriminant statistics on the merged model on the merged out-of-time subset to obtain an out-of-time statistical index; perform blind test discriminant statistics on the merged model on the merged blind test subset to obtain a blind test statistical index; determine a merged statistical index corresponding to the merged model on the sample merged subset based on the test statistical index, the out-of-time statistical index and the blind test statistical index.

6. The method of claim 3, wherein, calculate a merged gain corresponding to the one combination subset pair through the first statistical index, the second statistical index and the merged statistical index, comprising: calculate a first index difference between the merged statistical index and the first statistical index, and take a quotient of the first index difference and the first statistical index as a first gain; calculate a second index difference between the merged statistical index and the second statistical index, and take a quotient of the second index difference and the second statistical index as a second gain; determine a merged gain corresponding to the combination subset pair according to the first gain and the second gain.

7. The method of claim 2, wherein, screen a plurality of combination subset pairs of each intermediate merged set according to a merged standard threshold and a merged gain corresponding to each combination subset pair to obtain a sample merged set after screening of each intermediate merged set, comprising: for each intermediate merged set, respectively compare the merged gain corresponding to each combination subset pair of the intermediate merged set with the merged standard threshold; screen out, as a target subset pair corresponding to each intermediate merged set, a combination subset pair whose merged gain is not less than the merged standard threshold according to a comparison result; for each intermediate merged set, obtain a gain descending order determined by each target subset pair, and sequentially select, from a first position of the gain descending order, a target subset pair whose shared sample quantity is greater than a shared sample threshold as an available subset pair; from two to-be-merged sets corresponding to the available subset pair, determine a non-positive gain subset pair corresponding to the available subset pair, and there is no intersection between the non-positive gain subset pair and the available subset pair; construct a sample merged set corresponding to each intermediate merged set according to each available subset pair and a corresponding non-positive gain subset pair.

8. The method of claim 7, wherein, the method for determining the non-positive gain subset pair corresponding to the available subset pair from the two to-be-merged sets corresponding to the available subset pair, comprising: acquiring, from two to-be-merged sets corresponding to the available subset pair, a non-positive gain subset that does not match the available sample subset; generating a non-positive gain subset pair according to the non-positive gain subset.

9. The method of claim 1, wherein, The set decomposition on the target sample set to obtain a plurality of target sample decomposition sets, and each target sample decomposition set is taken as a to-be-merged set, comprising: taking the target sample set as a to-be-decomposed set; performing set decomposition on the to-be-decomposed set to obtain two sample decomposition sets after set decomposition; when the decomposition set number of the sample decomposition set is not equal to the subset number corresponding to the target sample set, taking each sample decomposition set as a to-be-decomposed set, and returning to the step of performing set decomposition on the to-be-decomposed set until the decomposition set number of the sample decomposition set obtained by set decomposition is equal to the subset number corresponding to the target sample set, and stopping set decomposition; taking each target sample decomposition set obtained by stopping set decomposition as a to-be-merged set.

10. A sample clustering method, characterized by, The method comprises: initializing a recombination merging number; performing subset order recombination on a plurality of sample initial subsets in a target sample set to obtain a recombined sample set; performing set decomposition and set merging on the recombined sample set according to the data processing method of any one of claims 1 to 9, storing the obtained recombined sample merging set, and performing a numerical update operation on the recombination merging number; when it is detected that the recombination merging number does not satisfy a preset recombination merging condition, returning to iteratively perform the step of performing subset order recombination on a plurality of sample initial subsets in a target sample set to obtain a recombined sample set until the recombination merging number satisfies the preset recombination merging condition, and terminating iteration; determining a merging co-occurrence matrix corresponding to the target sample set based on the stored plurality of recombined sample merging sets; performing clustering on the plurality of sample initial subsets according to each matrix element in the merging co-occurrence matrix, and determining a target sample clustering set corresponding to the plurality of sample initial subsets through the clustering result.

11. The method of claim 10, wherein, The determination of the merging co-occurrence matrix corresponding to the target sample set based on the stored plurality of recombined sample merging sets comprises: determining a plurality of co-occurrence subset pairs for the plurality of sample initial subsets; determining a same-species probability of each co-occurrence subset pair belonging to the same recombined sample merging set based on the stored plurality of recombined sample merging sets; determining a merging co-occurrence matrix corresponding to the plurality of sample initial subsets according to the same-species probability corresponding to each co-occurrence subset pair.

12. A data processing apparatus, characterized by The device comprises: a set acquisition module configured to acquire a target sample set; a set decomposition module configured to perform set decomposition on the target sample set to obtain a plurality of target sample decomposition sets, and take each target sample decomposition set as a to-be-merged set; a first merging module configured to perform set merging on each two to-be-merged sets to obtain an intermediate merging set after set merging, and perform screening on the intermediate merging set based on a merging standard threshold to obtain a sample merging set corresponding to each two to-be-merged sets; The second merging module is configured to, when the number of merging sets of the sample merging set is not one, take each sample merging set as a merging set to be merged, and return to the step of performing set merging on each two merging sets to be merged until the number of merging sets of the sample merging set obtained by set merging is one, and a target merging set is obtained.

13. A sample clustering apparatus, characterized by, The device comprises: The number of times determination module is configured to initialize a reorganization merging number of times; The sequential reorganization module is configured to perform sequential reorganization on the multiple sample initial subsets in the target sample set to obtain a reorganized sample set. The set storage module is configured to perform multi-layer set decomposition and multi-layer set merging on the reorganized sample set according to the data processing method of any one of claims 1 to 9, to obtain a reorganized sample merging set for storage, and to perform numerical updating on the reorganization merging number of times. The iteration execution module is configured to, when it is detected that the reorganization merging number of times does not satisfy a preset reorganization merging condition, return to the step of performing sequential reorganization on the multiple sample initial subsets in the target sample set to obtain a reorganized sample set until the reorganization merging number of times satisfies the preset reorganization merging condition, and terminate the iteration. The matrix determination module is configured to determine a merging co-occurrence matrix corresponding to the target sample set based on the stored multiple reorganized sample merging sets. The sample clustering module is configured to cluster the multiple sample initial subsets according to each matrix element in the merging co-occurrence matrix, and to determine a target sample clustering set corresponding to the multiple sample initial subsets through the clustering result.

14. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and when the computer program is executed by a processor, the data processing method of any one of claims 1 to 9 or the sample clustering method of any one of claims 10 to 11 is implemented.

15. A computer device, comprising: Comprise: A memory; A processor, and the memory stores a computer program, and when the computer program is executed by the processor, the data processing method of any one of claims 1 to 9 or the sample clustering method of any one of claims 10 to 11 is implemented.

16. A computer program product comprising computer programs or computer instructions, characterized in that, The computer program or computer instructions are executed by the processor to implement the data processing method of any one of claims 1 to 9 or the sample clustering method of any one of claims 10 to 11.

Images