Terminal evaluation model construction method, device and storage medium

Abstract

The present invention discloses a method, device and storage medium for constructing a terminal evaluation model, and relates to the field of information processing technology. The method for constructing a terminal evaluation model includes: obtaining a first sample set corresponding to a reference terminal and a second sample set corresponding to a terminal to be evaluated; determining a migration function corresponding to each indicator and migrating from the reference terminal to the terminal to be evaluated according to the distribution of samples in the first sample set and the second sample set; inputting each indicator value of the sample in the first sample set into the corresponding migration function to obtain the migrated indicator value, and determining the migration sample set of the first sample set according to the migrated indicator value; training the evaluation model of the terminal to be evaluated based on the samples in the migration sample set and the second sample set. The embodiments of the present invention can improve the degree of automation of model construction, and the accuracy and efficiency of model construction through these samples are also improved.

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and in particular to a method, device and storage medium for constructing a terminal evaluation model. Background Art

[0002] At present, the scale of IoT users is constantly expanding, business processes are becoming more and more complex, and the number of end-to-end business links is increasing. In order to improve the end-to-end experience of the business, a reasonable and effective terminal quality evaluation method is needed to determine which terminals have quality problems so that they can be dealt with as soon as possible.

[0003] Due to the wide variety of IoT device scenarios and brands, and their rapid updates, the evaluation methods for devices from different manufacturers and models have both similarities and differences. The quality index evaluation of different terminal manufacturers is different, and different evaluation models need to be trained for different manufacturers.

[0004] In the related art, it is usually necessary to re-collect and label the data of the new equipment in order to re-train the evaluation model corresponding to the new equipment. Summary of the invention

[0005] After analysis, the inventors found that the method of collecting samples and re-labeling terminals from different manufacturers is time-consuming and labor-intensive. In addition, there is currently a lack of a universal evaluation model construction method, which requires a lot of manual parameter adjustment work, resulting in a low degree of automation in this process. In addition, even for equipment of the same model and manufacturer, the evaluation indicators are prone to fluctuations after the equipment is upgraded, making it difficult to quickly obtain a large number of new labeled samples. On the one hand, after the terminal is upgraded, the physical size and meaning of key indicators may change, and retraining the model will lack a large amount of sample accumulation; on the other hand, once the indicators change, the labeled samples are basically invalid under the existing methods, and labeling resources are seriously wasted.

[0006] A technical problem to be solved by the embodiments of the present invention is: how to improve the automation level of building a terminal evaluation model.

[0007] According to a first aspect of some embodiments of the present invention, a method for constructing a terminal evaluation model is provided, comprising: obtaining a first sample set corresponding to a reference terminal, and a second sample set corresponding to a terminal to be evaluated; determining a migration function corresponding to each indicator, which is migrated from the reference terminal to the terminal to be evaluated, according to the distribution of samples in the first sample set and the second sample set; inputting each indicator value of the samples in the first sample set into the corresponding migration function to obtain the migrated indicator value, and determining the migrated sample set of the first sample set according to the migrated indicator value; and training the evaluation model of the terminal to be evaluated based on the migrated sample set and the samples in the second sample set.

[0008] In some embodiments, based on the distribution of samples in the first sample set and the second sample set, determining the migration function corresponding to each indicator for migration from the reference terminal to the terminal to be evaluated includes: for each indicator, dividing the corresponding indicator values ​​of the samples in the first sample set and the second sample set into multiple intervals according to preset dimensions; in each interval, determining the representative values ​​of the indicators of the reference terminal and the terminal to be evaluated respectively; based on the representative values ​​of the indicators corresponding to the reference terminal and the terminal to be evaluated respectively, determining the migration function corresponding to the indicator for migration from the reference terminal to the terminal to be evaluated.

[0009] In some embodiments: the loss function of the migration function corresponding to each indicator is determined according to the migration deviation and the deviation weight corresponding to the indicator; the migration deviation is determined according to the output value calculated by the migration function based on the representative value of the reference terminal and the gap between the representative values ​​of the terminal to be evaluated in the corresponding interval; the deviation weight is determined according to the gap between the representative values ​​of the reference terminal and the terminal to be evaluated in the corresponding interval and the corresponding indicator.

[0010] In some embodiments, at each interval, the representative value of the indicator is the median of the indicator values ​​in the interval.

[0011] In some embodiments, the migration function of the indicator is solved based on the least squares method.

[0012] In some embodiments: the difference between the representative values ​​of the reference terminal and the terminal to be evaluated in the corresponding interval and the corresponding indicator is taken as the fluctuation value. When the fluctuation value is not greater than the preset threshold, the deviation weight is determined according to the output value of the migration function and the difference between the representative values ​​of the terminal to be evaluated in the corresponding interval and the corresponding indicator; when the fluctuation value is greater than the preset threshold, the deviation weight is determined according to the fluctuation value and the preset threshold, wherein the deviation weight is positively correlated with the fluctuation value.

[0013] In some embodiments, the distribution is the distribution of samples in various intervals of collection time, or the distribution of samples in various intervals of indicator values.

[0014] In some embodiments, based on the samples in the migration sample set and the second sample set, training the evaluation model of the terminal to be evaluated includes: constructing the evaluation model of the terminal to be evaluated according to the samples in the migration sample set and the second sample set, and the initial weights of the samples; using the second sample set to test the test error rate of the evaluation model of the terminal to be evaluated; adjusting the weights of the samples in the migration sample set and the second sample set according to the test error rate; updating the evaluation model of the terminal to be evaluated according to the samples with adjusted weights; judging whether the preset conditions are currently met, and if not, repeating the test error rate of the evaluation model of the terminal to be evaluated using the second sample set.

[0015] In some embodiments: for each sample in the second sample set, when the evaluation model of the terminal to be evaluated predicts the sample incorrectly, the adjusted weight of the sample is positively correlated with the test error rate; when the evaluation model of the terminal to be evaluated predicts the sample correctly, the sample is not adjusted; for each sample in the migration sample set, when the evaluation model of the terminal to be evaluated predicts the sample incorrectly, the adjusted weight of the sample is negatively correlated with the test error rate; when the evaluation model of the terminal to be evaluated predicts the sample correctly, the sample is not adjusted.

[0016] In some embodiments, adjusting the weights of samples in the migration sample set and the second sample set according to the test error rate includes: calculating an update parameter, wherein the update parameter is positively correlated with the test error rate; for each sample in the second sample set, when the evaluation model of the terminal to be evaluated incorrectly predicts the sample, adjusting the weight of the sample to the product of the original weight and the update parameter; for each sample in the migration sample set, when the evaluation model of the terminal to be evaluated incorrectly predicts the sample, adjusting the weight of the sample to the quotient of the original weight and the update parameter.

[0017] In some embodiments, the update parameter is determined based on the ratio of the prediction error rate, to 1, and the difference between the prediction error rate.

[0018] In some embodiments, the evaluation model of the terminal to be evaluated is a support vector machine model SVM, a lightweight gradient boosting model lightGbm based on a decision tree algorithm, or a long short-term memory network LSTM model.

[0019] In some embodiments, the construction method further includes: inputting data of the terminal to be tested and evaluated into an evaluation model of the terminal to be evaluated, and obtaining an evaluation result of the terminal to be tested and evaluated.

[0020] In some embodiments, the reference terminal and the terminal to be evaluated are Internet of Things terminals.

[0021] According to a second aspect of some embodiments of the present invention, there is provided a device for constructing a terminal evaluation model, comprising: an acquisition module, configured to acquire a first sample set corresponding to a reference terminal, and a second sample set corresponding to a terminal to be evaluated; a migration function determination module, configured to determine a migration function corresponding to each indicator, which is migrated from the reference terminal to the terminal to be evaluated, according to the distribution of samples in the first sample set and the second sample set; a migration sample set determination module, configured to input each indicator value of the samples in the first sample set into a corresponding migration function to obtain a migrated indicator value, and determine a migration sample set of the first sample set according to the migrated indicator value; and a training module, configured to train the evaluation model of the terminal to be evaluated based on the migration sample set and the samples in the second sample set.

[0022] According to a third aspect of some embodiments of the present invention, there is provided a device for constructing a terminal evaluation model, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute any of the aforementioned methods for constructing a terminal evaluation model based on instructions stored in the memory.

[0023] According to a fourth aspect of some embodiments of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements any of the aforementioned methods for constructing a terminal evaluation model.

[0024] Some embodiments of the above invention have the following advantages or beneficial effects: The method of the embodiment of the present invention realizes the migration of samples from the first terminal to the second terminal by constructing a migration function, so that the evaluation model of the terminal to be evaluated can have more training samples when it is constructed. In addition, the migration process of the samples does not require artificial parameters, so that the generation efficiency and accuracy of the samples are improved. In addition, the distribution of the samples is taken into account during the migration process, so that the migrated samples are more consistent with the characteristics of the samples of the second terminal, further improving the availability of the samples. Therefore, the degree of automation of model construction can be improved, and the accuracy and efficiency of the model constructed by these samples are also improved.

[0025] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0027] Figure 1 A schematic flow chart of a method for constructing a terminal evaluation model according to some embodiments of the present invention is shown.

[0028] Figure 2 A schematic flow chart of a method for determining a migration function according to some embodiments of the present invention is shown.

[0029] Figure 3 A schematic flow chart of a method for constructing an evaluation model according to some embodiments of the present invention is shown.

[0030] Figure 4 A schematic flow chart of a terminal evaluation method according to some embodiments of the present invention is shown.

[0031] Figure 5 A schematic structural diagram of a device for constructing a terminal evaluation model according to some embodiments of the present invention is shown.

[0032] Figure 6 A schematic structural diagram of a terminal evaluation model construction device according to some other embodiments of the present invention is shown.

[0033] Figure 7 A schematic structural diagram of a device for constructing a terminal evaluation model according to some further embodiments of the present invention is shown. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present invention and its application or use. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] The relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless specifically stated otherwise.

[0036] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0037] Technologies, methods, and apparatus known to ordinary technicians in the relevant field may not be discussed in detail, but where appropriate, such technologies, methods, and apparatus should be considered part of the authorization specification.

[0038] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0039] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0040] Figure 1 FIG. 1 is a flow chart showing a method for constructing a terminal evaluation model according to some embodiments of the present invention. Figure 1 As shown, the method for constructing a terminal evaluation model in this embodiment includes steps S102 to S108.

[0041] In step S102, a first sample set corresponding to a reference terminal and a second sample set corresponding to a terminal to be evaluated are obtained. The first sample set and the second sample set respectively include a number of samples collected from the corresponding terminals.

[0042] The terminal to be evaluated is a terminal that needs to use samples of other terminals. In some embodiments, the number of samples in the first sample set is greater than the number of samples in the second sample set. That is, the reference terminal has sufficient data, but the data of the terminal to be evaluated itself is relatively small.

[0043] In some embodiments, the reference terminal and the terminal to be evaluated are terminals of different manufacturers or different models, or terminals of the same manufacturer and model but before and after the upgrade. Therefore, there are certain differences between these two types of terminals, and sample migration is required to enable the terminal to be evaluated to use the sample after the reference terminal is migrated.

[0044] In some embodiments, the reference terminal and the terminal to be evaluated are IoT terminals, such as smart water meters, smart electric meters, etc. The characteristics of IoT terminals include, for example, registration or activation characteristic indicators (such as the number of registration requests, the average registration request duration, etc.), downlink control characteristic indicators (such as the number of downlink CON successes, the average downlink CON successful response duration, the downlink CON flow, etc., CON indicates a request that needs to be confirmed), terminal reporting characteristic indicators (such as the number of uplink NON requests, the average uplink NON heartbeat request duration, the average uplink NON service request duration, NON indicates a request that does not need to be confirmed), platform response characteristic indicators (such as the number of downlink NON requests, the average downlink NON request duration, etc.).

[0045] In some embodiments, the reference terminal and the terminal to be evaluated are terminals with the same purpose, for example, both are smart water meters or smart electric meters. Therefore, there are certain similarities between the reference terminal and the terminal to be evaluated, making the sample usability stronger after migration.

[0046] In some embodiments, the index of each sample in the first sample set and the second sample set is a key index in the original sample. For example, the key index is extracted from all the indexes of the original sample by using methods such as principal component analysis and linear discriminant analysis to determine the samples in the first sample set and the second sample set composed of the key index. Thus, the samples used for terminal quality evaluation can more accurately reflect the characteristics of the terminal.

[0047] In step S104, a migration function corresponding to each indicator and migrating from the reference terminal to the terminal to be evaluated is determined according to the distribution of samples in the first sample set and the second sample set.

[0048] The migration function is used to describe the relationship between the data before and after the migration. In some embodiments, the input of the migration function is the value of the index of the first sample set, and the output is the value of the corresponding index after the migration.

[0049] Since there are differences between the samples of the reference terminal and the terminal to be evaluated, the distribution characteristics of the two need to be considered when constructing the migration function. In some embodiments, the distribution is the distribution of samples in various intervals of collection time, or the distribution of samples in various intervals of indicator values.

[0050] In step S106, each index value of the samples in the first sample set is input into the corresponding migration function to obtain the migration index value, and the migration sample set of the first sample set is determined according to the migration index value. The samples in the migration sample set are data generated according to the first sample set and simulating the samples of the second terminal.

[0051] In step S108, an evaluation model of the terminal to be evaluated is trained based on the migration sample set and samples in the second sample set.

[0052] In some embodiments, the evaluation model is a classification model, and each category corresponds to an evaluation level.

[0053] In some embodiments, the evaluation model is a terminal poor quality evaluation model, through which poor quality terminals and non-poor quality terminals can be identified, so that operations such as replacement and repair can be performed on the poor quality terminals in a timely manner.

[0054] In some embodiments, the evaluation model is a support vector machine model SVM, a lightweight gradient boosting model lightGbm based on a decision tree algorithm, and a long short-term memory network LSTM model. As needed, those skilled in the art can also use other models, which will not be described here.

[0055] The method of the above embodiment realizes the migration of samples from the first terminal to the second terminal by constructing a migration function, so that the evaluation model of the terminal to be evaluated can have more training samples when it is constructed. In addition, the migration process of the samples does not require artificial parameters, so that the generation efficiency and accuracy of the samples are improved. In addition, the distribution of the samples is taken into account during the migration process, so that the migrated samples are more consistent with the characteristics of the samples of the second terminal, further improving the availability of the samples. Therefore, the degree of automation of model construction can be improved, and the accuracy and efficiency of the model constructed through these samples are also improved.

[0056] Reference below Figure 2 An embodiment of the method for determining a migration function according to the present invention is described.

[0057] Figure 2FIG. 2 is a flow chart showing a method for determining a migration function according to some embodiments of the present invention. Figure 2 As shown, for each indicator, the migration function determination method of this embodiment includes steps S202 to S206.

[0058] In step S202, the corresponding index values ​​of the samples in the first sample set and the second sample set are divided into a plurality of intervals according to preset dimensions.

[0059] In some embodiments, the number of intervals of the first sample set is equal to the number of intervals of the second sample set, and each interval of the first sample set corresponds to an interval of the second sample set.

[0060] In some embodiments, when the preset dimension is the collection time, the same time period in different time periods is taken as the same interval, and the same time period refers to the time period with the same time offset in different time periods. For example, when the first sample set covers 10 days, for the data in, all the data collected from 8:00 to 9:00 in 10 days are divided into interval 1, and all the data collected from 9:00 to 10:00 are divided into interval 2, and so on. When the second sample set covers 1 day, the data is also divided by hour. Therefore, although there is a difference in the amount of data in the first sample set and the second sample set, through this division method, the data in the same interval can be matched.

[0061] In some embodiments, when the preset dimension is the amount of data, the data distribution of the collected data can be counted, the horizontal axis of the distribution represents the value of the data, and the vertical axis represents the amount of data under the value; then, the intervals are divided according to the vertical axis, and the collected data is divided into multiple intervals. For example, after counting the data distribution of the first sample set and the second sample set respectively, the vertical axis of the first sample set covers a range of 0 to 100, and the vertical axis of the second sample set covers a range of 0 to 10. The first sample set and the second sample set are both divided into 10 intervals, so that the interval with the vertical axis of the first sample set being [0,10) corresponds to the interval with the vertical axis of the second sample set being [0,1).

[0062] In step S204, representative values ​​of the indicator for the reference terminal and the terminal to be evaluated are determined in each interval.

[0063] In some embodiments, in each interval, the representative value of the indicator is the median of the indicator value in the interval. Other statistical values ​​such as the average value or the data distribution characteristic value may also be used as the representative value as needed.

[0064] By dividing the intervals, the representative values ​​of the reference terminal on the corresponding indicators and the corresponding intervals can be matched. The goal of migration is to make the representative value of the reference terminal equal to the representative value of the terminal to be evaluated on the corresponding indicators and the corresponding intervals after migration. Of course, the "equal" situation is an ideal situation. The migration function will make the migrated value as close as possible to the representative value of the terminal to be evaluated on the corresponding indicators and the corresponding intervals.

[0065] In step S206, according to the representative values ​​of the indicators respectively corresponding to the reference terminal and the terminal to be evaluated, a migration function corresponding to the indicator and migrating from the reference terminal to the terminal to be evaluated is determined.

[0066] In some embodiments, the migration function of the indicator is solved based on the least squares method. For example, the migration function constructed for a certain indicator is f(x), and the loss function is f(x) can be solved by the least squares method, where i represents the identifier of the representative value, n represents the number of representative values ​​of the reference terminal, and y i represents the i-th representative value of the terminal to be evaluated, x i represents the i-th representative value of the reference terminal, f(x i ) represents the migration value of the i-th representative value of the reference terminal. The representative values ​​of the reference terminal and the terminal to be evaluated with the same identifier belong to the corresponding intervals.

[0067] In some embodiments, considering the presence of noise or outliers in the collected data, for each indicator: the loss function of the migration function corresponding to the indicator is determined according to the migration deviation and the deviation weight; the migration deviation is determined according to the output value calculated by the migration function based on the representative value of the reference terminal and the gap between the representative values ​​of the terminal to be evaluated in the corresponding interval; the deviation weight is determined according to the gap between the representative values ​​of the reference terminal and the terminal to be evaluated in the corresponding interval and the corresponding indicator. Thus, by introducing the deviation weight, the size of the migration deviation can be adjusted, making the representation of the loss function more reasonable.

[0068] In some embodiments, the difference between the representative values ​​of the reference terminal and the terminal to be evaluated in the corresponding interval and the corresponding indicator is taken as the fluctuation value. When the fluctuation value is not greater than the preset threshold, the deviation weight is determined according to the output value of the migration function and the difference between the representative values ​​of the terminal to be evaluated in the corresponding interval and the corresponding indicator; when the fluctuation value is greater than the preset threshold, the deviation weight is determined according to the fluctuation value and the preset threshold, wherein the deviation weight is positively correlated with the fluctuation value.

[0069] Formulas (1) and (2) exemplarily show a way of expressing a loss function.

[0070]

[0071]

[0072] In formulas (1) and (2), loss represents the loss value; i represents the identifier of the representative value; n represents the number of representative values ​​of the reference terminal; y i represents the i-th representative value of the terminal to be evaluated, x i represents the i-th representative value of the reference terminal, f(x i ) represents the migration value of the i-th representative value of the reference terminal; α i represents the deviation weight corresponding to the i-th representative value of the reference terminal; |x|=|y i -x i | represents the fluctuation value; k represents the preset threshold, and its specific value can be determined through multiple experiments.

[0073] The above embodiment regards the deviation weight as a "noise factor" to adjust the influence of the outliers on the migration function and the updated optimization function. This loss function is constructed in a way that adjusts the deviation weight according to the distribution deviation of the reference terminal and the terminal sample to be evaluated, that is, the weight of the sample points within the reasonable deviation range is increased, and the weight of the sample points with a larger deviation range is decreased or even negative. Thus, the influence of outliers can be more effectively eliminated, and the accuracy of the migration function is improved.

[0074] Reference below Figure 3 An embodiment of a method for constructing an evaluation model is described.

[0075] Figure 3 FIG. 1 is a flow chart showing a method for constructing an evaluation model according to some embodiments of the present invention. Figure 3 As shown, the construction method of this embodiment includes steps S302 to S308.

[0076] In step S302, an evaluation model of the terminal to be evaluated is constructed according to the samples in the migration sample set and the second sample set, and the initial weights of the samples.

[0077] In some embodiments, the initial weights of the samples in the corresponding set are set according to the number of samples in each set. For example, if the number of samples in the migration sample set is m and the number of samples in the second sample set is n, the weights of the samples in the migration sample set and the second sample set are 1 / m and 1 / n respectively.

[0078] In some embodiments, the number of times a sample is used in one iteration of training is determined by the weight of the sample. The larger the weight, the more times the sample is used.

[0079] Then, steps S304 to S308 are repeated until a preset condition is met. The preset condition is, for example, reaching a preset number of iterations, or the calculated error rate is less than a preset value, etc. If the preset condition is met, the training process is stopped.

[0080] In step S304, the second sample set is used to test the test error rate of the evaluation model of the terminal to be evaluated.

[0081] In some embodiments, the test error rate is the ratio of the number of samples with prediction errors to the total number of samples.

[0082] In step S306, the weights of the samples in the migration sample set and the second sample set are adjusted according to the test error rate.

[0083] In some embodiments, for each sample in the second sample set, when the evaluation model of the terminal to be evaluated predicts the sample incorrectly, the adjusted weight of the sample is positively correlated with the test error rate; for each sample in the migration sample set, when the evaluation model of the terminal to be evaluated predicts the sample incorrectly, the adjusted weight of the sample is negatively correlated with the test error rate. For samples in these two sets, when the evaluation model of the terminal to be evaluated predicts the sample correctly, the sample is not adjusted.

[0084] Therefore, if the current evaluation model makes an incorrect evaluation of a sample in the migration sample set, it means that the sample in the migration sample set will theoretically have an adverse effect on the model, so its weight is weakened. When the model makes an incorrect evaluation of a sample in the second sample set, it means that the influence of the sample in the model construction is weak and its weight needs to be strengthened. Through this adjustment method, the characteristics of data from different sources can be taken into account to adjust the importance of data in the training process, thereby improving the prediction accuracy of the model.

[0085] In some embodiments, the weight is adjusted using the calculated update parameter, wherein the update parameter is positively correlated with the test error rate. For each sample in the second sample set, when the evaluation model of the terminal to be evaluated predicts the sample incorrectly, the weight of the sample is adjusted to the product of the original weight and the update parameter; for each sample in the migration sample set, when the evaluation model of the terminal to be evaluated predicts the sample incorrectly, the weight of the sample is adjusted to the quotient of the original weight and the update parameter.

[0086] In some embodiments, the update parameter is determined based on the ratio of the prediction error rate, to 1, and the difference between the prediction error rate.

[0087] Formula (3) exemplarily shows a calculation formula for the error rate.

[0088]

[0089] In formula (3), t represents the number of training iterations, ∈ t represents the error rate of the tth iteration; i represents the identifier of the sample, wherein the 1st to nth samples are samples in the first sample set, and the n+1th to n+mth samples are samples in the second sample set; represents the weight of the i-th sample in the t-th iteration; h t (x i ) represents the classification result of the model for the i-th sample in the t-th iteration, L(x i ) represents the label value of the i-th sample.

[0090] Based on the definitions of the parameters in formula (3), formulas (4) and (5) exemplarily illustrate a method for updating the weights, where β t represents the updated parameters in the tth iteration.

[0091] β t =∈ t / (1-∈ t ) (4)

[0092]

[0093] In step S308, the evaluation model of the terminal to be evaluated is updated according to the weighted samples.

[0094] During the training process, the method of the above embodiment adjusts the weights of the samples used for training according to the predicted error rate, thereby reducing the impact of slight deviations of the samples on the model and improving the accuracy of model prediction.

[0095] After completing the training of the evaluation model of the terminal to be evaluated, the prediction process can be carried out. Figure 4 An embodiment of the terminal evaluation method of the present invention is described.

[0096] Figure 4 FIG. 2 shows a flow chart of a terminal evaluation method according to some embodiments of the present invention. Figure 4 As shown, the terminal evaluation method of this embodiment includes steps S402 to S404.

[0097] In step S402, the data of the terminal to be tested and evaluated is input into the evaluation model of the terminal to be evaluated.

[0098] In step S404, an evaluation result of the terminal to be evaluated output by the evaluation model of the terminal to be evaluated is obtained.

[0099] Therefore, when the sample data of the terminal to be evaluated is relatively small, an accurate evaluation model of the terminal to be evaluated can be obtained through the methods of the aforementioned embodiments, and the quality of the terminal to be evaluated can be evaluated more accurately.

[0100] Reference below Figure 5 An embodiment of a device for constructing an evaluation model according to the present invention is described.

[0101] Figure 5 FIG. 4 is a schematic diagram showing a structure of a device for constructing a terminal evaluation model according to some embodiments of the present invention. Figure 5 As shown, the terminal evaluation model construction device 50 of this embodiment includes: an acquisition module 510, configured to acquire a first sample set corresponding to a reference terminal, and a second sample set corresponding to a terminal to be evaluated; a migration function determination module 520, configured to determine a migration function corresponding to each indicator, which is migrated from the reference terminal to the terminal to be evaluated, according to the distribution of samples in the first sample set and the second sample set; a migration sample set determination module 530, configured to input each indicator value of the samples in the first sample set into a corresponding migration function to obtain a migrated indicator value, and determine a migration sample set of the first sample set according to the migrated indicator value; a training module 540, configured to train the evaluation model of the terminal to be evaluated based on the migration sample set and the samples in the second sample set.

[0102] In some embodiments, the migration function determination module 520 is further configured to: for each indicator, divide the corresponding indicator values ​​of the samples in the first sample set and the second sample set into multiple intervals according to a preset dimension; in each interval, determine the representative values ​​of the indicators of the reference terminal and the terminal to be evaluated respectively; and determine the migration function corresponding to the indicator, which is migrated from the reference terminal to the terminal to be evaluated, based on the representative values ​​of the indicators corresponding to the reference terminal and the terminal to be evaluated respectively.

[0103] In some embodiments, the loss function of the migration function corresponding to each indicator is determined according to the migration deviation and the deviation weight corresponding to the indicator; the migration deviation is determined according to the output value calculated by the migration function based on the representative value of the reference terminal and the gap between the representative value of the terminal to be evaluated in the corresponding interval; the deviation weight is determined according to the gap between the representative values ​​of the reference terminal and the terminal to be evaluated in the corresponding interval and the corresponding indicator.

[0104] In some embodiments, at each interval, the representative value of the indicator is the median of the indicator values ​​in the interval.

[0105] In some embodiments, the migration function of the indicator is solved based on the least squares method.

[0106] In some embodiments, the difference between the representative values ​​of the reference terminal and the terminal to be evaluated in the corresponding interval and the corresponding indicator is taken as the fluctuation value. When the fluctuation value is not greater than the preset threshold, the deviation weight is determined according to the output value of the migration function and the difference between the representative values ​​of the terminal to be evaluated in the corresponding interval and the corresponding indicator; when the fluctuation value is greater than the preset threshold, the deviation weight is determined according to the fluctuation value and the preset threshold, wherein the deviation weight is positively correlated with the fluctuation value.

[0107] In some embodiments, the distribution is the distribution of samples in various intervals of collection time, or the distribution of samples in various intervals of indicator values.

[0108] In some embodiments, the training module 540 is further configured to: construct an evaluation model of the terminal to be evaluated based on the samples in the migration sample set and the second sample set, and the initial weights of the samples; use the second sample set to test the test error rate of the evaluation model of the terminal to be evaluated; adjust the weights of the samples in the migration sample set and the second sample set according to the test error rate; update the evaluation model of the terminal to be evaluated according to the samples with adjusted weights; determine whether the preset conditions are currently met, and if not, repeat the test error rate of the evaluation model of the terminal to be evaluated using the second sample set.

[0109] In some embodiments, for each sample in the second sample set, when the evaluation model of the terminal to be evaluated predicts the sample incorrectly, the adjusted weight of the sample is positively correlated with the test error rate; when the evaluation model of the terminal to be evaluated predicts the sample correctly, the sample is not adjusted; for each sample in the migration sample set, when the evaluation model of the terminal to be evaluated predicts the sample incorrectly, the adjusted weight of the sample is negatively correlated with the test error rate; when the evaluation model of the terminal to be evaluated predicts the sample correctly, the sample is not adjusted.

[0110] In some embodiments, the training module 540 is further configured to: calculate an update parameter, wherein the update parameter is positively correlated with the test error rate; for each sample in the second sample set, when the evaluation model of the terminal to be evaluated predicts the sample incorrectly, adjust the weight of the sample to the product of the original weight and the update parameter; for each sample in the migration sample set, when the evaluation model of the terminal to be evaluated predicts the sample incorrectly, adjust the weight of the sample to the quotient of the original weight and the update parameter.

[0111] In some embodiments, the update parameter is determined based on the ratio of the prediction error rate, to 1, and the difference between the prediction error rate.

[0112] In some embodiments, the evaluation model of the terminal to be evaluated is a support vector machine model SVM, a lightweight gradient boosting model lightGbm based on a decision tree algorithm, or a long short-term memory network LSTM model.

[0113] In some embodiments, the terminal evaluation model construction device 50 further includes: a prediction module 550 configured to input the data of the terminal to be evaluated to the evaluation model of the terminal to be evaluated, and obtain an evaluation result of the terminal to be evaluated.

[0114] In some embodiments, the reference terminal and the terminal to be evaluated are Internet of Things terminals.

[0115] Figure 6 FIG. 2 shows a schematic diagram of a structure of a device for constructing a terminal evaluation model according to some other embodiments of the present invention. Figure 6 As shown, the terminal evaluation model construction device 60 of this embodiment includes: a memory 610 and a processor 620 coupled to the memory 610, and the processor 620 is configured to execute the terminal evaluation model construction method in any one of the aforementioned embodiments based on the instructions stored in the memory 610.

[0116] The memory 610 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, and other programs.

[0117] Figure 7 FIG. 2 shows a schematic diagram of a structure of a device for constructing a terminal evaluation model according to some other embodiments of the present invention. Figure 7 As shown, the terminal evaluation model construction device 70 of this embodiment includes: a memory 710 and a processor 720, and may also include an input / output interface 730, a network interface 740, a storage interface 750, etc. These interfaces 730, 740, 750 and the memory 710 and the processor 720 may be connected, for example, via a bus 760. Among them, the input / output interface 730 provides a connection interface for input / output devices such as a display, a mouse, a keyboard, and a touch screen. The network interface 740 provides a connection interface for various networked devices. The storage interface 750 provides a connection interface for external storage devices such as SD cards and USB flash drives.

[0118] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements any of the aforementioned methods for constructing a terminal evaluation model.

[0119] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable non-transient storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0120] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0121] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0122] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0123] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for constructing a terminal evaluation model, comprising: Acquire a first sample set corresponding to the reference terminal and a second sample set corresponding to the terminal to be evaluated; Determining, according to the distribution of samples in the first sample set and the second sample set, a migration function corresponding to each indicator and migrating from the reference terminal to the terminal to be evaluated, including: for each indicator, dividing the corresponding indicator values ​​of the samples in the first sample set and the second sample set into a plurality of intervals according to a preset dimension; determining, in each interval, representative values ​​of the indicator of the reference terminal and the terminal to be evaluated; determining, according to the representative values ​​of the indicator corresponding to the reference terminal and the terminal to be evaluated, a migration function corresponding to the indicator and migrating from the reference terminal to the terminal to be evaluated; Inputting each index value of the samples in the first sample set into a corresponding migration function to obtain a migration index value, and determining a migration sample set of the first sample set according to the migration index value; An evaluation model of the terminal to be evaluated is trained based on the samples in the migration sample set and the second sample set.

2. The construction method according to claim 1, wherein: The loss function of the migration function corresponding to each indicator is determined according to the migration deviation and deviation weight corresponding to the indicator; The migration deviation is determined according to the difference between the output value calculated by the migration function based on the representative value of the reference terminal and the representative value of the terminal to be evaluated in the corresponding interval; The deviation weight is determined according to the difference between the representative values ​​of the reference terminal and the terminal to be evaluated in the corresponding interval and the corresponding indicator.

3. The construction method according to claim 1, wherein: In each interval, the representative value of the indicator is the median of the indicator values ​​in the interval.

4. The construction method according to claim 2, wherein: The migration function of the indicator is solved based on the least squares method.

5. The construction method according to claim 2, wherein: The difference between the representative values ​​of the reference terminal and the terminal to be evaluated in the corresponding interval and the corresponding indicator is taken as the fluctuation value. When the fluctuation value is not greater than the preset threshold, the deviation weight is determined according to the output value of the migration function and the difference between the representative values ​​of the terminal to be evaluated in the corresponding interval and the corresponding indicator; When the fluctuation value is greater than a preset threshold, the deviation weight is determined according to the fluctuation value and the preset threshold, wherein the deviation weight is positively correlated with the fluctuation value.

6. The construction method according to any one of claims 1 to 5, wherein: The distribution condition is the distribution condition of the samples in each interval of the collection time, or the distribution condition of the samples in each interval of the index value.

7. The construction method according to claim 1, wherein: The step of training the evaluation model of the terminal to be evaluated based on the migration sample set and the samples in the second sample set includes: constructing an evaluation model for the terminal to be evaluated according to the migration sample set and the samples in the second sample set, and the initial weights of the samples; Using the second sample set to test the test error rate of the evaluation model of the terminal to be evaluated; According to the test error rate, adjusting the weights of the samples in the migration sample set and the second sample set; Updating the evaluation model of the terminal to be evaluated according to the weighted samples; It is determined whether a preset condition is currently satisfied, and if not satisfied, the test error rate of the evaluation model of the terminal to be evaluated is repeatedly tested by using the second sample set.

8. The construction method according to claim 7, wherein: For each sample in the second sample set, when the evaluation model of the terminal to be evaluated makes an error in predicting the sample, the adjusted weight of the sample is positively correlated with the test error rate; When the evaluation model of the terminal to be evaluated correctly predicts the sample, the sample is not adjusted; For each sample in the migration sample set, when the evaluation model of the terminal to be evaluated makes an error in predicting the sample, the adjusted weight of the sample is negatively correlated with the test error rate; When the evaluation model of the terminal to be evaluated predicts the sample correctly, the sample is not adjusted.

9. The construction method according to claim 8, wherein: The adjusting the weights of the samples in the migration sample set and the second sample set according to the test error rate includes: Calculating an update parameter, wherein the update parameter is positively correlated with the test error rate; For each sample in the second sample set, when the evaluation model of the terminal to be evaluated makes an error in predicting the sample, adjusting the weight of the sample to the product of the original weight and the update parameter; For each sample in the migration sample set, when the evaluation model of the terminal to be evaluated makes an error in predicting the sample, the weight of the sample is adjusted to the quotient of the original weight and the update parameter.

10. The construction method according to claim 9, wherein: The update parameter is determined according to the prediction error rate and the ratio of the difference between 1 and the prediction error rate.

11. The construction method according to any one of claims 7 to 10, wherein: The evaluation model of the terminal to be evaluated is a support vector machine model SVM, a lightweight gradient boosting model lightGbm based on a decision tree algorithm, and a long short-term memory network LSTM model.

12. The construction method according to claim 1, further comprising: The data of the terminal to be tested and evaluated is input into the evaluation model of the terminal to be evaluated to obtain the evaluation result of the terminal to be tested and evaluated.

13. The construction method according to claim 1, wherein: The reference terminal and the terminal to be evaluated are Internet of Things terminals.

14. A device for constructing a terminal evaluation model, comprising: An acquisition module, configured to acquire a first sample set corresponding to a reference terminal and a second sample set corresponding to a terminal to be evaluated; The migration function determination module is configured to determine the migration function corresponding to each indicator and migrated from the reference terminal to the terminal to be evaluated according to the distribution of samples in the first sample set and the second sample set, including: for each indicator, according to a preset dimension, respectively dividing the corresponding indicator values ​​of the samples in the first sample set and the second sample set into multiple intervals; in each interval, respectively determining the representative value of the indicator of the reference terminal and the terminal to be evaluated; according to the representative values ​​of the indicator corresponding to the reference terminal and the terminal to be evaluated, respectively, determining the migration function corresponding to the indicator and migrated from the reference terminal to the terminal to be evaluated; a migration sample set determination module, configured to input each index value of the samples in the first sample set into a corresponding migration function to obtain a migration index value, and determine a migration sample set of the first sample set according to the migration index value; The training module is configured to train an evaluation model of the terminal to be evaluated based on the migration sample set and samples in the second sample set.

15. A device for constructing a terminal evaluation model, comprising: Memory; as well as A processor coupled to the memory, the processor being configured to execute the method for constructing a terminal evaluation model according to any one of claims 1 to 13 based on instructions stored in the memory.

16. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method for constructing a terminal evaluation model according to any one of claims 1 to 13 is implemented.

Images