Preprocessing method and device for model input data and electronic equipment

Abstract

The invention provides a model input data preprocessing method and device and electronic equipment, and the method comprises the steps: carrying out the scoring of sample data through an initial prediction model before a target to-be-trained model needs to be trained through the sample data, so as to quantify the complexity of the sample data, and improve the training efficiency. Then binning processing is carried out based on the sample scores of the sample data, different sample weights are assigned in combination with the sample data under different binning paths, in the generated new sample data, the complexity degree of the corresponding sample data is prompted through the sample weights, and if the new sample data set is adopted to train the target to-be-trained model subsequently, the complexity degree of the target to-be-trained model is prompted through the sample weights. The target model to be trained can be helped to quickly know the complexity of the corresponding sample data based on the corresponding sample weight, so that the problem that in the traditional model training process, the model cannot quickly process the sample data which is difficult to classify, and additional computing resources need to be consumed is solved; and therefore, the calculation efficiency of the artificial intelligence model obtained by training can be improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, apparatus and electronic device for preprocessing model input data. Background Technology

[0002] Artificial intelligence (AI) models are models that simulate human intelligence in learning, reasoning, and judgment, autonomously extracting patterns from data and handling specific tasks without requiring manually written rules. With the continuous development of AI technology, AI models are increasingly being applied to various fields, including finance, insurance analysis, and e-commerce. Regardless of the application area, AI models process massive amounts of data and rely heavily on underlying hardware resources.

[0003] However, during the learning process, artificial intelligence models tend to learn from easily categorized sample data, and their ability to distinguish between difficult-to-categorize samples is poor. This results in the artificial intelligence model requiring high computational resources and having poor computational efficiency when processing specific tasks if the input data is difficult-to-categorize sample data. Summary of the Invention

[0004] In view of this, embodiments of this application provide a method, apparatus, and electronic device for preprocessing model input data, which preprocesses the model input data to change the hardware resources occupied by the artificial intelligence model during the training and application stages, thereby improving the computational efficiency of the trained artificial intelligence model.

[0005] Firstly, this application provides a method for preprocessing model input data, wherein the method includes: Obtain the sample dataset to be processed, input the sample dataset to be processed into the initial prediction model, and obtain the sample score output by the initial prediction model for each sample data in the sample dataset; wherein, the sample score is positively correlated with the classification complexity of the sample data; The sample dataset is binned, and sample data whose scores conform to preset binning rules are stored in the same binning path, and the same sample weight is assigned to sample data belonging to the same binning path; wherein, the sample weights corresponding to different binning paths are different, and the sample weights are set based on the application scenario; The sample data under each binning path is combined with the corresponding sample weights to generate a new sample dataset, and the new sample dataset is used as the model input data and output to the target model to be trained.

[0006] Secondly, embodiments of this application provide a preprocessing apparatus for model input data, wherein the apparatus includes: The acquisition module is used to acquire the sample dataset to be processed, input the sample dataset to be processed into the initial prediction model, and acquire the sample score output by the initial prediction model for each sample data in the sample dataset; wherein, the sample score is positively correlated with the classification complexity of the sample data; The binning module is used to bin the sample dataset, storing sample data whose scores conform to preset binning rules into the same binning path, and assigning the same sample weight to sample data belonging to the same binning path; wherein, the sample weights corresponding to different binning paths are different, and the sample weights are set based on the application scenario; The output module combines the sample data under each binning path with the corresponding sample weights to generate a new sample dataset, and outputs the new sample dataset as the model input data to the target model to be trained.

[0007] Thirdly, embodiments of this application provide an electronic device, wherein the electronic device includes: a processor; and a memory storing a program; wherein the program includes instructions, which, when executed by the processor, cause the processor to perform the preprocessing method for model input data described in the first aspect.

[0008] Fourthly, embodiments of this application provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the preprocessing method for model input data described in the first aspect.

[0009] The beneficial effects of this application are: This application provides a method, apparatus, and electronic device for preprocessing model input data. Before training a target model using sample data, the method employs the preprocessing method provided in this application to perform preliminary predictions on the sample data in the sample dataset using an initial prediction model, obtaining corresponding sample scores. Then, according to preset binning rules, the sample data in the sample dataset is binned, storing sample data conforming to the preset binning rules under the same binning path. Different sample weights are assigned to sample data under different binning paths. The sample data under the binning paths are then combined with the corresponding sample weights to generate a new sample dataset. This new sample dataset is used as the model input data for the target model to be trained and output to the target model for training.

[0010] In this embodiment of the application, the sample data is scored by an initial prediction model to quantify the complexity of the sample data. Then, based on the sample scores, binning is performed, and different sample weights are assigned to the sample data under different binning paths. In the new sample data generated in this way, the sample weights indicate the complexity of the corresponding sample data. If the target model is subsequently trained using this new sample dataset, it can help the target model quickly know the complexity of the corresponding sample data based on the corresponding sample weights. This solves the problem that in the traditional model training process, the model cannot quickly process the difficult-to-classify sample data and requires additional computing resources, thereby helping to improve the computational efficiency of the trained artificial intelligence model. Attached Figure Description

[0011] Further details, features, and advantages of this application are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which: Figure 1 This paper illustrates a flowchart of a preprocessing method for model input data provided in this application. Figure 2 This paper illustrates another flowchart of the preprocessing method for model input data provided in this application; Figure 3 This invention provides a schematic diagram of the structure of a preprocessing apparatus for model input data provided in this application. Figure 4 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of this application is shown. Detailed Implementation

[0012] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0013] It should be understood that the steps described in the method embodiments of this application may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this application is not limited in this respect.

[0014] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc., mentioned in this application are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0015] It should be noted that the terms "a" and "a plurality of" used in this application are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0016] To reduce the hardware resources consumed by artificial intelligence models during training and application phases and improve the computational efficiency of the trained AI models, this application provides a method, apparatus, and electronic device for preprocessing model input data. In a first aspect, this application provides a method for preprocessing model input data, applicable to any electronic device with model input data preprocessing capabilities, including but not limited to personal mobile terminals, computers, or servers. Figure 1 As shown, the method includes the following steps: S11. Obtain the sample dataset to be processed, input the sample dataset to be processed into the initial prediction model, and obtain the sample score output by the initial prediction model for each sample data in the sample dataset; wherein, the sample score is positively correlated with the classification complexity of the sample data; S12. Perform binning on the sample dataset, storing sample data whose sample scores conform to preset binning rules into the same binning path, and assigning the same sample weight to sample data belonging to the same binning path; wherein, the sample weights corresponding to different binning paths are different, and the sample weights are set based on the application scenario. S13. Combine the sample data under each binning path with the corresponding sample weights to generate a new sample dataset, and use the new sample dataset as model input data to output to the target model to be trained.

[0017] In this embodiment of the application, the sample data is scored by an initial prediction model to quantify the complexity of the sample data. Then, based on the sample scores, binning is performed, and different sample weights are assigned to the sample data under different binning paths. In the new sample data generated in this way, the sample weights indicate the complexity of the corresponding sample data. If the target model is subsequently trained using this new sample dataset, it can help the target model quickly know the complexity of the corresponding sample data based on the corresponding sample weights. This solves the problem that in the traditional model training process, the model cannot quickly process the difficult-to-classify sample data and requires additional computing resources, thereby helping to improve the computational efficiency of the trained artificial intelligence model.

[0018] The following will provide a detailed explanation of steps S11 to S14 with specific examples: In this application, the sample dataset to be processed is a collection of data containing several sample data entries. The specific sample data depends on the specific application scenario, for example: In the insurance field, the sample data in the sample dataset to be processed can be users' insurance claim information. Artificial intelligence models can be trained using the sample data, and then the trained artificial intelligence models can be used to determine whether users have committed insurance fraud based on users' insurance claim information. In the e-commerce field, the sample data in the sample dataset to be processed can be the transaction operation records of goods. The artificial intelligence model can be trained using the sample data, and then the trained artificial intelligence model can be used to analyze whether there are fraudulent transactions of goods based on the transaction operation records of goods. In the field of medical diagnosis, the sample data in the dataset to be processed can be patient diagnosis records. Artificial intelligence models can be trained using the sample data, and then the trained artificial intelligence models can be used to provide auxiliary clinical decision-making. In the industrial sector, the sample data in the dataset to be processed can be equipment operating parameters. Artificial intelligence models can be trained using the sample data, and then the trained artificial intelligence models can be used to monitor the operation of equipment in the factory.

[0019] In the credit field, the sample data in the dataset to be processed can be credit data. This sample data can be used to train an artificial intelligence model, and then the trained artificial intelligence model can be used to predict whether a user will default on a credit loan.

[0020] The model input data preprocessing method provided in this application can be applied to various technical fields that require the use of artificial intelligence models. As a preferred embodiment, the model input data preprocessing method provided in this application is mainly applied to the credit field, which is related to the applicant's main business.

[0021] As one implementation method, it can be as follows Figure 2 As shown, this application involves two models: an initial prediction model M1 and a target model to be trained M2. The initial prediction model M1 is an artificial intelligence model pre-trained using a sample dataset S (with an initial sample weight of 1). It can be understood that the sample data used to train this initial prediction model M1 is not processed through steps S11 to S13 as described above, but is directly input into the trained model. The initial prediction model M1 can be any type of neural network model that outputs corresponding scoring results based on the data features of the input data, such as a logistic regression model, a graph neural network model, etc. The initial prediction model can be preferably selected based on its high training efficiency and low resource consumption.

[0022] The target model to be trained is the model used in later practical applications. In the credit field, this target model to be trained can be a credit risk identification model, used to identify the credit risk of a user based on the input data. As one implementation method, this target model to be trained can be a binary classification model, which can output whether a user has a risk of default based on the input data. If there is a risk of default, the output result is 1, and if there is no risk of default, the output result is 0.

[0023] In some possible embodiments, the initial prediction model is deployed on edge computing nodes within a distributed computing framework, while the target model to be trained is deployed on a central computing node within the same framework. The distinction between edge and central computing nodes within the distributed computing framework depends on the actual application environment of the target model to be trained. Computing nodes closer to the actual application environment can be considered central computing nodes, while those farther away can be considered edge computing nodes.

[0024] The preprocessing method for model input data provided in this application is mainly applied in edge computing nodes. The sample data to be processed can also be stored in edge computing nodes. An initial prediction model M1 can be deployed on the edge computing nodes to score, bin, and adjust the weights of the sample data. After generating a new sample dataset, the newly generated sample dataset is sent to the central computing node, which then trains the target model M2 based on the new sample dataset. In this way, on the one hand, the transmission latency of the credit business data sample data can be reduced; on the other hand, having the edge computing nodes undertake the initial prediction model modeling and data preprocessing work can provide more computing resources for the target model to be trained, thereby ensuring the application performance of the target model in actual business applications.

[0025] When performing step S11, the edge computing node can obtain the corresponding sample dataset by reading the specified local storage path or reading the sample dataset to be processed from the specified path in the cloud. After obtaining the sample dataset to be processed, each sample data in the sample dataset is input into the locally deployed initial prediction model M1. The initial prediction model M1 learns, summarizes, and judges based on the input sample output to determine the complexity of the sample data. Then, it outputs the corresponding sample score according to the complexity. The sample score is positively correlated with the classification complexity of the sample data. That is, the higher the classification complexity of the sample data, the higher the corresponding sample score, indicating that the sample data is a complex sample.

[0026] For credit business data, the complexity of the sample depends on the needs of the business scenario. For example, the sample data for consumer loans has a higher transaction frequency, compared to the sample data for mortgage loans which has a lower transaction frequency, making the sample data for consumer loans more complex. Similarly, for the same business, the sample data for users with a higher frequency of consumer loans is more complex than the sample data for users with a lower frequency of consumer loans.

[0027] In this embodiment of the application, the initial prediction model M1 scores the sample data in the sample dataset to be processed. This can be figuratively understood as sorting the sample dataset to be processed, quickly dividing and quantifying sample data of different levels of complexity, and quantifying the corresponding complexity through sample scores.

[0028] As one implementation method, the classification complexity of the sample data of this credit business can depend on whether there is a hidden default. For example, in a user's credit data, the user has not yet defaulted, but frequently changes jobs, which indicates that the user has a probability of hidden default. If such sample data of hidden default is directly input into the target training model M2, the model M2 will not be able to perform binary classification. Therefore, this application provides the preprocessing means described in steps S11 to S13 above for such situations. When performing step S11, the initial prediction model M1 outputs the corresponding delinquency probability based on the sample data of hidden default. The delinquency probability is used as the basis for generating sample scores. The lower the delinquency probability, the lower the corresponding sample score, and the higher the delinquency probability, the higher the corresponding sample score. Thus, the sample data with hidden default is initially identified so that the subsequent model M2 can perform binary classification according to the weights of different sample data, which helps to improve the classification accuracy of model M2.

[0029] Further, step S12 performs binning on the sample dataset. Binning, in essence, involves dividing the sample data within the dataset into different subsets. Specifically, according to pre-defined binning rules (which can be understood as partitioning rules), sample data conforming to the binning rules are stored in the same binning path. For example, it can be done as follows: Figure 2 As shown, assuming the data needs to be divided into n subsets, each subset corresponds to a binning rule, the sample data can be divided into the corresponding bins according to whether the sample score of the sample data falls under the binning rule of each subset, and then stored in the corresponding binning path.

[0030] In some possible embodiments, the binning rule can be set through the following steps: Step A1: Based on the sample scores, determine the probability density curve of the sample scores and determine the probability distribution type of the complex sample data; Step A2: Based on the probability distribution type of the complex sample data, determine the number of binning paths corresponding to the binning rule, as well as the upper bound sample score threshold and lower bound sample score threshold corresponding to each binning path.

[0031] In step A1, the range of sample scores can be determined first, and outliers that are significantly outside the range can be filtered out. Then, through mathematical statistics, the probability density curves of all sample scores can be generated using histogram smoothing or kernel density estimation. Based on these probability density curves, the statistical distribution of the complex sample can be determined. The corresponding statistical distribution may be a unimodal distribution or a bimodal distribution.

[0032] Then, proceed to step A2. Based on the probability distribution type of the complex samples under all sample data, determine the number of bins and the upper and lower bounds of each bin, i.e., the upper and lower bounds of the sample score thresholds for each bin. Specifically, based on the principle of balanced coverage and focusing on peaks, the upper and lower bounds of the bins can be taken according to the peak position of the statistical distribution of the complex samples. For example, assuming that the statistical distribution corresponding to the complex samples is a unimodal distribution, the sample scores in the entire sample dataset will be concentrated in a peak interval, gradually becoming sparser on both sides. In this case, when binning the sample data, it is necessary to balance covering the entire interval and focusing on the peak area of ​​the complex sample concentration. More bins can be set for the peak area, and fewer bins can be set for the non-peak area.

[0033] Furthermore, when executing step S12, the sample data falling into the [lower bound sample score threshold, upper bound sample score threshold] are stored in the corresponding binning path according to the upper bound sample score threshold and lower bound sample score threshold corresponding to each binning path.

[0034] In this way, the binning rules can perfectly match the inherent statistical patterns of the sample dataset, and can be adaptively adjusted according to the sample distribution of the dataset in different scenarios. This effectively avoids the problems of subjectively set bin numbers and poor binning accuracy inherent in binning rules. In some possible embodiments, step S12 described above can be performed through the following steps: The sample dataset is subjected to equal-frequency binning. According to the sample scores from low to high, the sample data in the sample dataset are stored in different binning paths, so that the difference in the number of sample data in each binning path is within a preset fluctuation range.

[0035] The preset fluctuation range can be flexibly set according to actual needs, such as ±3 sample data. In the embodiments of this application, it can be as follows: Figure 2 As shown, equal-frequency binning is performed based on the sample scores of the sample data. All sample data in the dataset to be processed are divided into n bins according to their sample scores, ensuring that the number of sample data in each bin is equal or similar, regardless of the range of the sample scores. For example, assuming there are 100,000 credit sample data points, they need to be divided into 4 binning paths according to the sample scores from 0 to 100. Each binning path contains approximately 25,000 sample data points. If the sample scores exhibit a bimodal distribution, it indicates that there are relatively more sample data points with low and high scores. In this case, the binning range for binning path 1 could be 0-30 points, and the binning range for binning path 4 could be 70-100 points. The range widths are different, but the amount of sample data is equal. This approach can capture sample data with both low and high delinquency risk in a balanced manner.

[0036] In some possible embodiments, the sample data in the sample dataset belongs to different business types. For example, even within credit business data, it can be divided into consumer loan business data, business loan business data, and mortgage loan business data. Since different credit businesses have varying degrees of impact on overdue and bad debts, if the sample data is credit sample data, when executing step S12, the business type can be pre-classified. That is, when executing step S12, it can also be achieved through the following steps: Step B1: Determine the credit business type corresponding to the credit sample data, and divide the sample data belonging to the same credit business type into the same business sample data set; Step B2: For sample data belonging to the same business sample data set, perform binning according to the same business binning rules; however, different credit business types correspond to different business binning rules.

[0037] Step B1 can be understood as dividing the samples in the sample dataset according to different credit businesses, saving the sample data belonging to the same credit business separately into the corresponding business sample data set, and then executing step B2. For the sample data belonging to the same business sample data set, steps S11 and S12 are executed to select the same business binning rule for binning the same credit business type. In this way, the influence of sample data from different credit businesses on the risk overdue identification of different credit businesses can be avoided, which helps to improve the prediction accuracy of the target model M2 to be trained in the later stage.

[0038] In the execution of step S12, assigning different sample scores to sample data under different binning paths can be understood as assigning the same weight to sample data under the same binning path. For example, it can be done as follows: Figure 2 As shown, a weight w1 is assigned to the sample data in binning path 1, a weight w2 to the sample data in binning path 2, a weight w3 to the sample data in binning path 3, and so on. This sample weight can be set according to the type of bin. For example, suppose the sample score depends on whether the sample data contains implicit defaults; a higher sample score indicates a higher probability of implicit default. In this case, the sample data can be divided into four binning paths: strong implicit default, weak implicit default, weak explicit default, and strong explicit default. Model M2 can easily distinguish sample data within bins containing strong explicit defaults, but it is more difficult to distinguish between strong implicit defaults, weak implicit defaults, and weak explicit defaults, especially strong implicit defaults. In this case, higher sample weights can be assigned to different types of strong implicit defaults, and then the weights can be decreased sequentially.

[0039] In some possible embodiments, the sample weight is calculated based on the risk contribution and business value of the sample data. The corresponding sample weight can be calculated using the formula: weight w = risk contribution * business value. For example, the risk contribution can be calculated by dividing the average overdue days by the total loan amount, and the business value can be calculated directly using the number of loan cycles. Based on this method, the weight of a bin with strong implicit default can be 3.8, the sample weight of a bin with weak implicit default can be 2.7, the sample weight of a bin with weak explicit default can be 2.2, and the sample weight of a bin with strong explicit default can be 1.3.

[0040] Further, in step S13, the sample data in the bins is combined with the corresponding sample weights. Specifically, this can be done by adding sample weight labels after the sample data to generate new sample data. Alternatively, a field can be added directly to the sample data, and the sample weights can be filled into the added field to generate new sample data.

[0041] Finally, the generated new sample data is output to the target training model M2 for training. The target training model M2 can quickly determine whether the received sample data belongs to complex samples based on the sample weights carried in the new sample data, thereby improving the target training model M2's ability to identify difficult-to-classify samples. This helps to improve the ability of the target training model M2 to identify and process complex samples in practical applications after the training is completed.

[0042] Secondly, this application provides a preprocessing apparatus for model input data, wherein, as Figure 3 As shown, the device 30 includes: The acquisition module 301 is used to acquire the sample dataset to be processed, input the sample dataset to be processed into the initial prediction model, and acquire the sample score output by the initial prediction model for each sample data in the sample dataset; wherein, the sample score is positively correlated with the classification complexity of the sample data; Binning module 302 is used to bin the sample dataset, store sample data whose sample scores conform to preset binning rules into the same binning path, and assign the same sample weight to sample data belonging to the same binning path; wherein, the sample weights corresponding to different binning paths are different, and the sample weights are set based on the application scenario. The output module 303 combines the sample data under each binning path with the corresponding sample weights to generate a new sample dataset, and outputs the new sample dataset as model input data to the target model to be trained.

[0043] In some possible embodiments, the binning rules are set through the following steps: Based on the sample scores, determine the probability density curve of the sample scores and determine the probability distribution type to which the complex sample data belongs; Based on the probability distribution type of the complex sample data, determine the number of binning paths corresponding to the binning rule, as well as the upper bound sample score threshold and lower bound sample score threshold corresponding to each binning path.

[0044] In some possible embodiments, the binning module is further used for: Based on the upper bound sample score threshold and lower bound sample score threshold corresponding to each binning path, the sample data is stored in the corresponding bins according to the sample score range to which the sample score belongs.

[0045] In some possible embodiments, the binning module is further used for: The sample dataset is subjected to equal-frequency binning. According to the sample scores from low to high, the sample data in the sample dataset are stored in different binning paths, so that the difference in the number of sample data in each binning path is within a preset fluctuation range.

[0046] In some possible embodiments, the sample data in the sample dataset is credit sample data, and the binning module is specifically used for: Determine the credit business type corresponding to the credit sample data, and group sample data belonging to the same credit business type into the same business sample data set; For sample data belonging to the same business sample data set, they are binned according to the same business binning rules; however, the business binning rules are different for different credit business types.

[0047] In some possible embodiments, the sample weights are calculated based on the risk contribution and business value of the sample data.

[0048] In some possible embodiments, the initial prediction model is deployed on each edge computing node in a distributed computing architecture, the target model to be trained is deployed on the central computing node in the distributed computing architecture, and the method is applied to each of the edge computing nodes.

[0049] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in this application comply with relevant laws and regulations and do not violate public order and good morals.

[0050] The names of the messages or information exchanged between multiple devices in the embodiments of this application are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0051] Thirdly, exemplary embodiments of this application also provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to cause the electronic device to perform a method according to an embodiment of this application.

[0052] An exemplary embodiment of this application also provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform a method according to an embodiment of this application.

[0053] An exemplary embodiment of this application also provides a computer program product, including a computer program, wherein, when executed by a computer's processor, the computer program is used to cause the computer to perform a method according to an embodiment of this application.

[0054] refer to Figure 4 The present invention describes a structural block diagram of an electronic device 400 that can serve as a server or client of this application, which is an example of a hardware device that can be applied to various aspects of this application. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the application described and / or claimed herein.

[0055] like Figure 4 As shown, the electronic device 400 includes a computing unit 401, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM 402) or a computer program loaded from a storage unit 408 into a random access memory (RAM 403). The RAM 403 may also store various programs and data required for the operation of the electronic device 400. The computing unit 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output interface (I / O interface 405) is also connected to the bus 404.

[0056] Multiple components in electronic device 400 are connected to I / O interface 405, including: input unit 406, output unit 407, storage unit 408, and communication unit 409. Input unit 406 can be any type of device capable of inputting information to electronic device 400. Input unit 406 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device. Output unit 407 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 408 may include, but is not limited to, disks and optical discs. Communication unit 409 allows electronic device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth™ devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0057] The computing unit 401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the various methods and processes described above. For example, in some embodiments, the aforementioned preprocessing method for model input data can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 400 via ROM 402 and / or communication unit 409. In some embodiments, the computing unit 401 can be configured to perform the aforementioned preprocessing method for model input data by any other suitable means (e.g., by means of firmware).

[0058] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0059] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0060] As used in this application, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0061] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0062] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0063] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

Claims

1. A method for preprocessing model input data, characterized in that, The method includes: Obtain the sample dataset to be processed, input the sample dataset to be processed into the initial prediction model, and obtain the sample score output by the initial prediction model for each sample data in the sample dataset; wherein, the sample score is positively correlated with the classification complexity of the sample data; The sample dataset is binned, and sample data whose scores conform to preset binning rules are stored in the same binning path, and the same sample weight is assigned to sample data belonging to the same binning path; wherein, the sample weights corresponding to different binning paths are different, and the sample weights are set based on the application scenario; The sample data under each binning path is combined with the corresponding sample weights to generate a new sample dataset, and the new sample dataset is used as the model input data and output to the target model to be trained.

2. The method according to claim 1, characterized in that, The bin sorting rules are set through the following steps: Based on the sample scores, determine the probability density curve of the sample scores and determine the probability distribution type to which the complex sample data belongs; Based on the probability distribution type of the complex sample data, determine the number of binning paths corresponding to the binning rule, as well as the upper bound sample score threshold and lower bound sample score threshold corresponding to each binning path.

3. The method according to claim 2, characterized in that, The step of storing sample data whose sample scores conform to preset binning rules into the same binning path includes: Based on the upper bound sample score threshold and lower bound sample score threshold corresponding to each binning path, the sample data is stored in the corresponding bins according to the sample score range to which the sample score belongs.

4. The method according to claim 1, characterized in that, The step of binning the sample dataset, which involves storing sample data whose scores conform to preset binning rules into the same binning path, includes: The sample dataset is subjected to equal-frequency binning. According to the sample scores from low to high, the sample data in the sample dataset are stored in different binning paths, so that the difference in the number of sample data in each binning path is within a preset fluctuation range.

5. The method according to claim 1, characterized in that, The sample data in the sample dataset is credit sample data. The binning process for the sample dataset, which stores sample data whose scores conform to preset binning rules into the same binning path, includes: Determine the credit business type corresponding to the credit sample data, and group sample data belonging to the same credit business type into the same business sample data set; For sample data belonging to the same business sample data set, they are binned according to the same business binning rules; however, the business binning rules are different for different credit business types.

6. The method according to claim 1, characterized in that, The sample weights are calculated based on the risk contribution and business value of the sample data.

7. The method according to claim 1, characterized in that, The initial prediction model is deployed on each edge computing node in the distributed computing architecture, the target model to be trained is deployed on the central computing node in the distributed computing architecture, and the method is applied to each of the edge computing nodes.

8. A preprocessing device for model input data, characterized in that, The device includes: The acquisition module is used to acquire the sample dataset to be processed, input the sample dataset to be processed into the initial prediction model, and acquire the sample score output by the initial prediction model for each sample data in the sample dataset; wherein, the sample score is positively correlated with the classification complexity of the sample data; The binning module is used to bin the sample dataset, storing sample data whose scores conform to preset binning rules into the same binning path, and assigning the same sample weight to sample data belonging to the same binning path; wherein, the sample weights corresponding to different binning paths are different, and the sample weights are set based on the application scenario; The output module combines the sample data under each binning path with the corresponding sample weights to generate a new sample dataset, and outputs the new sample dataset as the model input data to the target model to be trained.

9. An electronic device, characterized in that, The electronic device includes: a processor and a memory storing a program; wherein the program includes instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 1-7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-7.

Images