Model configuration method and device based on algorithm screening, equipment, storage medium

Abstract

The application belongs to the field of artificial intelligence, and provides a model configuration method, device, equipment and storage medium based on algorithm screening, which comprises the following steps: obtaining historical data and historical reference results, and inputting the historical data into a target model; obtaining corresponding offline running results by offline running a plurality of candidate algorithms according to the historical data, and determining available algorithms according to the offline running results; obtaining an evaluation sample group and a reference evaluation result, assigning the corresponding available algorithms to each sample flow according to a preset strategy, running the available algorithms to obtain corresponding test evaluation results; and determining a target algorithm according to the test evaluation results and configuring the target model. According to the technical scheme of the embodiment, the candidate algorithms can be screened through the historical data, the number of algorithms screened through the sample flow is reduced, the screening of the target algorithm can be realized under the condition that the sample flow is limited, and the efficiency and accuracy of model configuration are improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of artificial intelligence, and particularly relates to a model configuration method and device based on algorithm screening, equipment and a storage medium. BACKGROUND

[0002] With the development of computer technology, artificial intelligence models have been widely applied in various industries, which can effectively improve work efficiency and user experience. The functions of artificial intelligence models are usually complex, so it is necessary to test the optional algorithms during the development stage, evaluate the effects obtained from different algorithms, and select the algorithm that best meets the requirements. AB testing is a common testing method, which sets up experimental groups and control groups, applies different algorithms in different groups, and determines the target algorithm according to the evaluation results obtained. However, algorithm screening through AB testing usually requires a large amount of sample traffic, and it is difficult to achieve algorithm screening under the condition of limited sample traffic, which affects the efficiency of model development. SUMMARY

[0003] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0004] The embodiments of the present application provide a model configuration method and device based on algorithm screening, which can complete model configuration according to less sample traffic, improve the efficiency and accuracy of model configuration.

[0005] In a first aspect, the embodiments of the present application provide a model configuration method based on algorithm screening, comprising:

[0006] obtaining historical data and historical reference results, and inputting the historical data into a target model;

[0007] obtaining a plurality of candidate algorithms, applying each candidate algorithm to the target model for offline running, and obtaining an offline running result corresponding to each candidate algorithm according to the historical data;

[0008] determining a first similarity between the historical reference results and each offline running result, and determining the offline running result with a first similarity greater than a first preset threshold as a usable offline result;

[0009] determining the candidate algorithm corresponding to the usable offline result as a usable algorithm;

[0010] obtaining an evaluation sample group and a reference evaluation result, inputting the evaluation sample group into the target model, and the evaluation sample group comprising a plurality of sample traffics;

[0011] According to a preset strategy, each sample flow is allocated a corresponding available algorithm, and the corresponding test evaluation result is obtained by running the available algorithm.

[0012] The available algorithm corresponding to the test evaluation result closest to the reference evaluation result is determined as a target algorithm, and the target model is configured according to the target algorithm.

[0013] In some embodiments, the preset strategy is an Interleaving model, and the weights of each sample flow are the same.

[0014] In some embodiments, the available algorithm corresponding to the test evaluation result closest to the reference evaluation result is determined as a target algorithm, and the target model is configured according to the target algorithm, comprising:

[0015] A second preset threshold is obtained.

[0016] A second similarity between the reference evaluation result and each test evaluation result is determined.

[0017] The test evaluation result corresponding to the highest second similarity is determined as a target evaluation result.

[0018] The available algorithm corresponding to the target evaluation result is determined as the target algorithm.

[0019] In some embodiments, after the target model is configured according to the target algorithm, the method further comprises:

[0020] A plurality of functional modules of the target model are determined.

[0021] A plurality of parameter groups are determined according to the target model, and at least one parameter value of the functional module is different between different parameter groups.

[0022] The target model is subjected to AB testing according to a plurality of parameter groups.

[0023] In some embodiments, the target model is subjected to AB testing according to a plurality of parameter groups, and the method further comprises:

[0024] A test index and a reference flow of the AB testing are obtained.

[0025] The reference flow is determined as a test flow of the AB testing.

[0026] The test index is adjusted to change the test flow, and the obtained minimum test flow is determined as a target test flow.

[0027] AB test the target model according to the target test traffic and the plurality of parameter groups.

[0028] In some embodiments, the test index comprises at least one of:

[0029] CTR;

[0030] confidence level;

[0031] statistical power;

[0032] index calculation strategy;

[0033] diversion type;

[0034] standard deviation of the index.

[0035] In some embodiments, the AB test the target model according to the target test traffic and the plurality of parameter groups comprises:

[0036] determining an experimental group and a control group of the AB test;

[0037] adding test results triggered by the target test traffic in each parameter group to the experimental group, or adding test results not triggered by the target test traffic in each parameter group to the control group;

[0038] determining test results of the AB test according to the test results recorded by the experimental group and the control group.

[0039] In a second aspect, embodiments of the present application provide a model configuration device based on algorithm screening, comprising:

[0040] a historical data acquisition unit configured to acquire historical data and historical reference results, and input the historical data into a target model;

[0041] a candidate algorithm acquisition unit configured to acquire a plurality of candidate algorithms, apply each candidate algorithm to the target model for offline running, and obtain an offline running result corresponding to each candidate algorithm according to the historical data;

[0042] an offline running unit configured to determine a first similarity between the historical reference results and each offline running result, and determine an offline running result with a first similarity greater than a first preset threshold as an available offline result;

[0043] an available algorithm determination unit configured to determine the candidate algorithm corresponding to the available offline result as an available algorithm;

[0044] An evaluation sample acquisition unit is configured to acquire an evaluation sample group and a reference evaluation result, input the evaluation sample group into the target model, and the evaluation sample group includes a plurality of sample flows;

[0045] A test unit is configured to assign the available algorithm corresponding to each sample flow according to a preset strategy, run the available algorithm to obtain a corresponding test evaluation result;

[0046] A model configuration unit is configured to determine the available algorithm corresponding to the test evaluation result closest to the reference evaluation result as a target algorithm, and configure the target model according to the target algorithm.

[0047] In a third aspect, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the algorithm screening-based model configuration method according to the first aspect when executing the computer program.

[0048] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program for executing the algorithm screening-based model configuration method according to the first aspect.

[0049] The embodiments of the present application include: acquiring historical data and a historical reference result, inputting the historical data into a target model; acquiring a plurality of candidate algorithms, applying each candidate algorithm to the target model for offline running, and obtaining an offline running result corresponding to each candidate algorithm according to the historical data; determining a first similarity between the historical reference result and each offline running result, determining an offline running result with a first similarity greater than a first preset threshold as an available offline result; determining the candidate algorithm corresponding to the available offline result as an available algorithm; acquiring an evaluation sample group and a reference evaluation result, inputting the evaluation sample group into the target model, and the evaluation sample group includes a plurality of sample flows; assigning the available algorithm corresponding to each sample flow according to a preset strategy, running the available algorithm to obtain a corresponding test evaluation result; determining the available algorithm corresponding to the test evaluation result closest to the reference evaluation result as a target algorithm, and configuring the target model according to the target algorithm. According to the technical solution of the present embodiment, the candidate algorithms can be screened through historical data, the number of algorithms screened through sample flows is reduced, the screening of the target algorithm can be realized in the case of limited sample flows, and the efficiency and accuracy of model configuration are improved.

[0050] Other features and advantages of the present application will be set forth in the description that follows, and in part will be apparent from the description, or can be learned by practice of the application. The purposes and other advantages of the application will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings. BRIEF DESCRIPTION OF DRAWINGS

[0051] The accompanying drawings are included to provide a further understanding of the technical solutions of the present application, and constitute a part of the specification, and are used together with the embodiments of the present application to explain the technical solutions of the present application, and do not constitute a limitation on the technical solutions of the present application.

[0052] Figure 1 is a flowchart of a model configuration method based on algorithm screening provided by an embodiment of the present application;

[0053] Figure 2 is a flowchart of determining a target algorithm provided by another embodiment of the present application;

[0054] Figure 3 is a flowchart of performing AB testing provided by another embodiment of the present application;

[0055] Figure 4 is a flowchart of determining the traffic of AB testing provided by another embodiment of the present application;

[0056] Figure 5 is a flowchart of determining the test result of AB testing provided by another embodiment of the present application;

[0057] Figure 6 is a structural diagram of a model configuration device based on algorithm screening provided by another embodiment of the present application;

[0058] Figure 7 is a device diagram of an electronic device provided by another embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the objectives, technical solutions and advantages of the present application more clear, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and do not limit the present application.

[0060] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "target" and the like in the specification, claims or the above drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0061] The application provides a model configuration method and device based on algorithm screening, equipment and a storage medium, and the method comprises the following steps: obtaining historical data and historical reference results, and inputting the historical data into a target model; obtaining a plurality of candidate algorithms, applying each candidate algorithm to the target model for offline running, and obtaining an offline running result corresponding to each candidate algorithm according to the historical data; determining a first similarity between the historical reference results and each offline running result, and determining the offline running result with the first similarity greater than a first preset threshold as a usable offline result; determining the candidate algorithm corresponding to the usable offline result as a usable algorithm; obtaining an evaluation sample group and a reference evaluation result, inputting the evaluation sample group into the target model, and the evaluation sample group comprises a plurality of sample flows; according to a preset strategy, assigning the usable algorithm corresponding to each sample flow, running the usable algorithm to obtain a corresponding test evaluation result; determining the usable algorithm corresponding to the test evaluation result closest to the reference evaluation result as a target algorithm, and configuring the target model according to the target algorithm. According to the technical scheme of the embodiment, the candidate algorithm can be screened through the historical data, the number of algorithms screened through the sample flow is reduced, the screening of the target algorithm can be realized in the case that the sample flow is limited, and the efficiency and accuracy of model configuration are improved.

[0062] The embodiments of the application can compile, obtain and process related data based on artificial intelligence technology. The artificial intelligence (AI) is the use of digital computers or computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, obtain knowledge and use the knowledge to obtain the best results.

[0063] The basic technology of artificial intelligence generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing, operation / interaction devices, mechatronics, etc. The software technology of artificial intelligence mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0064] Machine Learning (ML) is a multi-disciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, etc. It is a specialized study of how computers simulate or implement human learning behavior to acquire new knowledge or skills, reorganize existing knowledge structure to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent. Its applications are widespread in various fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and rule-based learning.

[0065] It can be understood that the AB test can be run on a personal mobile terminal, on a server, or as a third-party device to provide rapid iteration and trial-and-error of client experiment data and background strategy to obtain an experiment report. The specific AB test can be in the form of a program running in the above-mentioned device, or as a system component in the above-mentioned device, or as a kind of cloud service program. The specific operation mode is determined by the actual scene, which is not limited here.

[0066] The control test, i.e. the AB test, is an effective fine operation method. The AB test is to make two (A / B) or more (A / B / n) versions for a webpage or application interface or process. In the same time dimension, respectively let the visitor groups with the same (or similar) composition randomly access these versions, collect user experience data and business data of each group, and finally analyze and evaluate the best version to be officially adopted. For example, when AB testing a product, two schemes can be developed for the same optimization goal, one part of the users uses the A scheme, and another part of the users uses the B scheme. The conversion rate, click volume, retention rate, etc. of different schemes are counted and compared to judge the pros and cons of different schemes and make decisions, so as to improve the conversion rate.

[0067] As shown in Figure 1 , the flowchart of the model configuration method based on algorithm screening provided by an embodiment of the present application includes but is not limited to the following steps: Figure 1 Step S110, obtaining historical data and historical reference results, inputting the historical data into a target model;

[0068] Step S120, obtaining a plurality of candidate algorithms, applying each candidate algorithm to the target model for offline running, and obtaining the offline running result corresponding to each candidate algorithm according to the historical data;

[0069]

[0070] ​In step S130, a first similarity between the historical reference result and each offline running result is determined, and an offline running result with a first similarity greater than a first preset threshold is determined as a usable offline result.

[0071] In step S140, a candidate algorithm corresponding to the usable offline result is determined as a usable algorithm.

[0072] In step S150, an evaluation sample group and a reference evaluation result are obtained, the evaluation sample group is input into the target model, and the evaluation sample group includes a plurality of sample flows.

[0073] In step S160, a usable algorithm corresponding to each sample flow is allocated according to a preset strategy, and the usable algorithm is run to obtain a corresponding test evaluation result.

[0074] In step S170, a usable algorithm corresponding to a test evaluation result closest to the reference evaluation result is determined as a target algorithm, and the target model is configured according to the target algorithm.

[0075] It should be noted that, since the target model is usually used to achieve a certain prediction, recommendation or the like result by using artificial intelligence technology, for example, content recommendation according to user preferences, before the target model is applied, if a functional module or a prior model having a similar function is available, data of the functional module or the prior model can be obtained as historical data, and the historical data and a corresponding historical reference result can be used as verifiable known data, so that the candidate algorithm can be initially screened, and when the target model runs the candidate algorithm and the result obtained according to the historical data is similar to the historical reference result, it can be determined that the candidate algorithm meets the development requirements within a certain standard, and can be determined as a usable algorithm for further screening.

[0076] It should be noted that, since the historical data is not real-time online data, the candidate algorithm can be verified by offline running for the historical data, so as to improve the running efficiency of the target model and reduce the dependence on network resources. In the case where the historical data and the historical reference result are available, each candidate algorithm can be sequentially configured into the target model, one candidate algorithm is configured into the target model each time, the target model is run according to the historical data to obtain an offline running result corresponding to the candidate algorithm, and the offline running result is compared with the historical reference result, so as to determine a usable offline result.

[0077] It should be noted that after obtaining the offline running result by applying each candidate algorithm, the first similarity between the offline running result and the historical reference result can be determined to be a usable offline result. For example, the historical reference result and the offline running result are both data of multiple indicators, and the first similarity can be determined according to the proportion of the data of each indicator. When the proportion of the data of each indicator of the offline running result is similar to the proportion of the data of each indicator of the historical reference result, or the numerical difference satisfies the first preset threshold, it can be determined that the offline running result is similar to the historical reference result, and the offline running result can be determined to be a usable offline result, and the corresponding candidate algorithm is determined to be a usable algorithm. On the contrary, when the first similarity between the offline running result and the historical reference result is less than the first preset threshold, it means that the operation logic of the candidate algorithm deviates from the development requirements of the target model, and the candidate algorithm is discarded. Thus, the candidate algorithms are first screened according to the historical data, the sample flow screening task is reduced, and the accuracy of algorithm screening is improved in the case of limited sample flow.

[0078] It should be noted that in the case of limited sample flow, in order to realize the control test, the preset strategy can not distinguish between the experimental group and the control group, but configure a corresponding usable algorithm for each sample flow, so that each sample flow obtains a test evaluation result according to the assigned usable algorithm. For example, for recommendation algorithm A and recommendation algorithm B, in the case of sample flow 1, sample flow 2, sample flow 3 and sample flow 4, sample flow 1 and sample flow 3 can be assigned recommendation algorithm A, and sample flow 2 and sample flow 4 can be assigned recommendation algorithm B. Thus, test evaluation result 1 corresponding to sample flow 1, test evaluation result 2 corresponding to sample flow 2, test evaluation result 3 corresponding to sample flow 3, and test evaluation result 4 corresponding to sample flow 4 are obtained. Then, the recommendation accuracy of recommendation algorithm A and recommendation algorithm B is determined according to the four test evaluation results, the algorithm with higher recommendation accuracy is taken as the target algorithm, and the target algorithm is configured to the target model, so that the target model has the target algorithm with the highest reliability.

[0079] It should be noted that the reference evaluation result can be determined according to actual needs. For example, as described above, in the case of four sample flows, the expected result corresponding to each sample flow is determined according to actual needs, and the expected result is taken as the reference evaluation result, so that the screened algorithm can be more in line with the development requirements.

[0080] In addition, in an embodiment, the preset strategy is an Interleaving model, and the weight of each sample flow is the same.

[0081] It should be noted that the interleaving model can realize the control test without dividing the experimental group and the control group, and the principle is roughly as follows: taking the subscription recommendation service as an example, in the traditional AB test, usually two groups of subscription users are selected, one group receives the recommendation result of recommendation algorithm A, and the other group receives the recommendation result of recommendation algorithm B. In the interleaving test, there is only one group of subscription users, and these subscription users will receive the alternating ranking generated by mixing the ranking of algorithm A and B, which makes the subscription users can see the recommendation results of algorithm A and B in a row at the same time, and the subscription users cannot distinguish whether a subscription option is recommended by algorithm A or algorithm B, and then the index statistics can be used to measure whether algorithm A or algorithm B is better, so as to reduce the sample flow required for testing.

[0082] It should be noted that in order to eliminate algorithm bias, the weight of each sample flow can be set to be the same, so as to avoid that a certain sample flow has a high weight and affects the test result.

[0083] In addition, in an embodiment, the step S170 of the embodiment shown in Figure 2 , Figure 1 The step S170 of the embodiment shown in the above also includes but is not limited to the following steps:

[0084] Step S210, obtaining a second preset threshold;

[0085] Step S220, determining a second similarity between the reference evaluation result and each test evaluation result;

[0086] Step S230, determining the test evaluation result corresponding to the highest second similarity as the target evaluation result;

[0087] Step S240, determining the available algorithm corresponding to the target evaluation result as the target algorithm.

[0088] It should be noted that since the sample flows in the same evaluation sample group are assigned different available algorithms for evaluation, after the target model obtains the test evaluation results according to the plurality of available algorithms, the second similarity with the reference evaluation result can be determined by the dispersion degree of data, for example, taking the video recommendation model as the target model, the test evaluation result can be the recommendation result of different videos under different recommendation algorithms, and the recommendation times or the recommendation degree of each video can be used as the test evaluation result, for example, video A is recommended according to recommendation algorithm A, and video B is recommended according to recommendation algorithm B, and the test evaluation result is the recommendation times of video A and video B, and the reference evaluation result is the times required to be recommended by video A and video B, when the recommendation times of video A are close to the recommendation times in the reference evaluation result, it can be determined that the recommendation algorithm A is more in line with the development requirements, and is determined as the target algorithm.

[0089] It should be noted that when there are many sample flows and available algorithms, statistical images, such as common bar charts, can be generated based on the test evaluation results. Then, the reference proportion of each sample flow in the reference evaluation results is compared with the proportion of each item in the bar chart. The closer the proportions are, the higher the second similarity. The test evaluation result with the highest second similarity is determined as the target evaluation result, and the corresponding available algorithm is determined as the target algorithm.

[0090] Additionally, in one embodiment, reference is made to Figure 3 After execution Figure 2 Following step S170 in the illustrated embodiment, the following steps are included, but are not limited to:

[0091] Step S310: Determine multiple functional modules of the target model;

[0092] Step S320: Determine multiple parameter groups based on the target model, where at least one functional module in each parameter group has a different parameter value.

[0093] Step S330: Perform A / B testing on the target model based on multiple parameter groups.

[0094] It should be noted that the target model is usually composed of multiple functional modules, each of which implements a type of function, such as a module for style control or a module for sorting. Multiple parameters of the same functional module have certain correlations. Therefore, each functional module can be used as a parameter group. That is, the control group and the experimental group have the same functional modules, but their corresponding parameter values ​​are different. Each parameter group is used as an experimental layer to achieve control testing and avoid the starvation and bias problems between modules in a single-layer experiment.

[0095] It should be noted that the relationship between functional modules and experimental layers can be referred to in the table below. TextColor and BackgroundColor belong to the same functional module, so they are assigned to the same parameter group and experimental layer. Their values ​​are different in the experimental group and the control group. The same applies to the other groups.

[0096]

[0097]

[0098] Table 1: Relationship between Functional Modules and Experimental Layer

[0099] It should be noted that, given the availability of parameter sets, A / B testing of the target model is a technique well-known to those skilled in the art, and this embodiment does not impose excessive limitations on the specific A / B testing process.

[0100] In addition, in an embodiment, referring to Figure 4 , Figure 3 The step S330 of the embodiment shown further includes but is not limited to the following steps:

[0101] In step S410, test indicators and reference traffic of the AB test are obtained.

[0102] In step S420, the reference traffic is determined as the test traffic of the AB test.

[0103] In step S430, the test indicators are adjusted to change the test traffic, and the minimum test traffic obtained is determined as the target test traffic.

[0104] In step S440, the target model is subjected to the AB test according to the target test traffic and the plurality of parameter groups.

[0105] The test indicators include at least one of the following:

[0106] Click-Through-Rate (CTR);

[0107] confidence level;

[0108] statistical power;

[0109] indicator calculation strategy;

[0110] diversion type;

[0111] standard deviation of the indicator.

[0112] It is worth noting that one or more of the above test indicators can be selected, which can be selected according to actual test requirements. By adjusting the values of the test indicators, the traffic of the target model can be changed, so as to determine the minimum traffic required by the target model. For example, the minimum experimental traffic size is calculated automatically according to the expected indicator change (for example, increasing the CTR by 2%), the confidence level (generally taken as 1-α=0.95), the statistical power (generally taken as 1-β=0.80), the calculation method of the indicator, the diversion type, and the standard deviation of the indicator. The minimum test traffic determined as the target test traffic can effectively reduce the traffic required for the target model test, and can successfully complete the test under the condition that the sample traffic is limited.

[0113] It should be noted that after the target test traffic is determined, the allocation of the target test traffic can be determined according to the number of parameter groups. For example, for a parameter group with more parameter values, more target test traffic can be allocated, so that more test results can be obtained, and the results of the AB test can be more accurate.

[0114] In addition, in an embodiment, referring toFigure 5 , Figure 4 The step S440 of the embodiment shown further includes but is not limited to the following steps:

[0115] A step S510 of determining an experimental group and a control group of the AB test;

[0116] A step S520 of adding a test result triggered by the target test flow in each parameter group to the experimental group, or adding a test result not triggered by the target test flow in each parameter group to the control group;

[0117] A step S530 of determining a test result of the AB test according to the test results recorded by the experimental group and the control group.

[0118] It should be noted that the division of the experimental group and the control group on the basis of the AB test of the target model is a technology familiar to those skilled in the art, and will not be described here for the sake of simplicity.

[0119] It should be noted that for a common AB test, each target test flow is run only once according to one parameter group to obtain one test result, while the embodiment can generate two test results according to the triggering condition and add them to the control group and the test group respectively, for example, the target test flow is a weather query, if the query is triggered in the experimental group, the test result can be added to the experimental group, if the query fails, the test result of the query failure is added to the control group, thereby improving the use efficiency of each target test flow and being able to complete the AB test according to fewer target test flows.

[0120] In addition, with reference to Figure 6 , the embodiment of the present application provides a model configuration device based on algorithm screening, which includes but is not limited to the following units:

[0121] A historical data acquisition unit 610 is configured to acquire historical data and historical reference results, and input the historical data to a target model;

[0122] A candidate algorithm acquisition unit 620 is configured to acquire a plurality of candidate algorithms, apply each candidate algorithm to the target model for offline running, and obtain an offline running result corresponding to each candidate algorithm according to the historical data;

[0123] An offline running unit 630 is configured to determine a first similarity between the historical reference results and each offline running result, and determine an offline running result with a first similarity greater than a first preset threshold as a usable offline result;

[0124] A usable algorithm determination unit 640 is configured to determine a candidate algorithm corresponding to the usable offline result as a usable algorithm;

[0125] The evaluation sample acquisition unit 650 is used to acquire the evaluation sample group and the reference evaluation results, and input the evaluation sample group into the target model. The evaluation sample group includes multiple sample flows.

[0126] Test unit 660 is used to assign a corresponding available algorithm to each sample traffic according to a preset strategy, and run the available algorithm to obtain the corresponding test evaluation results;

[0127] The model configuration unit 670 is used to determine the available algorithm corresponding to the test evaluation result that is closest to the reference evaluation result as the target algorithm, and configure the target model according to the target algorithm.

[0128] like Figure 7 As shown, Figure 7 This is a structural diagram of an electronic device provided in one embodiment of the present invention. The present invention also provides an electronic device 700, comprising: at least one control processor 720 and a memory 710 for communicatively connecting to the at least one control processor 720; the memory 710 stores instructions executable by the at least one control processor 720, the instructions being executed by the at least one control processor 720 to enable the at least one control processor 720 to execute the algorithm-based model configuration method as described above.

[0129] The memory 710, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs, such as the algorithm-based model configuration method in the above embodiments of the present invention. The control processor 720 implements the algorithm-based model configuration method in the above embodiments of the present invention by running the non-transitory software program and instructions stored in the memory 710.

[0130] The memory 710 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data required for executing the algorithm-based model configuration method described in the above embodiments. Furthermore, the memory 710 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. It should be noted that the memory 710 may optionally include memory remotely located relative to the control processor 720, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0131] The non-transient software program and instructions required to implement the algorithm-based model configuration method in the above embodiments are stored in memory. When executed by one or more processors, the algorithm-based model configuration method in the above embodiments is executed, for example, the method described above is executed. Figure 1 Method steps S110 to S170, Figure 2 Method steps S210 to S240, Figure 3 Method steps S310 to S330, Figure 4 Method steps S410 to S440 Figure 5 The method steps S510 to S530 are described above. The apparatus embodiments described above are merely illustrative; the units described as separate components may or may not be physically separate, meaning they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0132] Furthermore, one embodiment of the present invention provides a computer-readable storage medium storing a computer program that is executed by a processor or controller, for example, by a processor in the above-described electronic device embodiment, causing the processor to execute the algorithm-based model configuration method described above, for example, performing the above-described [specific method / method]. Figure 1 Method steps S110 to S170, Figure 2 Method steps S210 to S240, Figure 3 Method steps S310 to S330, Figure 4 Method steps S410 to S440 Figure 5Those skilled in the art can understand that all or some steps in the above disclosed method can be implemented as software, firmware, hardware and appropriate combination thereof. Some or all of the physical components can be implemented as software executed by a processor such as a central processing unit, a digital signal processor or a microprocessor, or as hardware, or as an integrated circuit such as an application specific integrated circuit. Such software can be distributed on a computer readable storage medium, which can include computer storage media (or non-transitory storage media) and communication media (or transitory storage media). As is well known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other storage medium which can be used to store the desired information and which can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.

[0133] The embodiments can be used in a variety of computer or computing environments. For example: personal computers, server computers, hand-held or laptop devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types. The present application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including computer storage media that is storage electronic devices.

[0134] The units described in the embodiments of the present application can be implemented by software, or by hardware, or by a combination of software and hardware. The names of the units described do not constitute a limitation on the units themselves in some cases.

[0135] It should be noted that, although several modules or units for electronic devices for action execution are mentioned in the foregoing detailed description, such division is not mandatory. Indeed, according to an embodiment of the application, the features and functionalities of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functionalities of one module or unit described above can be further divided into several modules or units embodied.

[0136] Through the above description of the embodiments, those skilled in the art can easily understand that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solutions according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.) or a network, and includes several instructions to make a computing electronic device (which can be a personal computer, a server, a touch terminal, or a network electronic device, etc.) execute the method according to the embodiments of the present application.

[0137] The electronic device of the embodiment can include a radio frequency (RF) circuit, a memory, an input unit, a display unit, a sensor, an audio circuit, a wireless fidelity (WiFi) module, a processor, and a power supply, etc. The RF circuit can be used for receiving and sending signals in the process of information or call, especially, receiving the downlink information of the base station and processing it by the processor, and sending the uplink data to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF circuit can also communicate with the network and other devices through wireless communication. The above wireless communication can use any communication standard or protocol, including but not limited to global system of mobile communication (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), long term evolution (LTE), email, short messaging service (SMS), etc. The memory can be used to store software programs and modules, and the processor can execute various function applications and data processing of the electronic device by running the software programs and modules stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.), etc.; the data storage area can store data created according to the use of the electronic device (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, for example, at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device. The input unit can be used to receive input digital or character information, and generate key signal input related to the setting and function control of the electronic device. Specifically, the input unit can include a touch panel and other input devices. The touch panel, also known as a touch screen, can collect touch operations (such as operations on or near the touch panel using a finger, a stylus, etc. or any suitable object or accessory) on or near it, and drive the corresponding connection device according to the pre-set program. Optionally, the touch panel can include a touch detection device and a touch controller.The touch detection device detects a touch position and detects a signal caused by a touch operation, and transmits the signal to the touch controller. The touch controller receives touch information from the touch detection device, converts the touch information into touch coordinates, and sends the touch coordinates to the processor. The touch controller can also receive commands from the processor and execute the commands. In addition, the touch panel can be implemented in various types such as a resistive type, a capacitive type, an infrared type, and a surface acoustic wave type. In addition to the touch panel, the input unit can include other input devices. Specifically, the other input devices can include one or more of a physical keyboard, a function key (such as a volume control key, an on / off key, etc.), a trackball, a mouse, a joystick, etc. The display unit can be used to display input information or provided information and various menus of the electronic device. The display unit can include a display panel, which can be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. Further, the touch panel can cover the display panel, and when the touch panel detects a touch operation thereon or in the vicinity thereof, the touch panel transmits the touch event to the processor to determine the type of the touch event, and then the processor provides a corresponding visual output on the display panel according to the type of the touch event. The touch panel and the display panel are implemented as two independent components to realize the input and output functions of the electronic device, but in some embodiments, the touch panel and the display panel can be integrated to realize the input and output functions of the electronic device. The electronic device can also include at least one sensor such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor can include an ambient light sensor and a proximity sensor, wherein the ambient light sensor can adjust the brightness of the display panel according to the brightness of ambient light, and the proximity sensor can turn off the display panel and / or the backlight when the electronic device is moved to the ear. As one of the motion sensors, the accelerometer sensor can detect the magnitude of acceleration in each direction (generally three axes), and when at rest, it can detect the magnitude and direction of gravity, which can be used for applications such as identifying the posture of the electronic device (such as switching between landscape and portrait screens, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc. As for other sensors that can be configured in the electronic device, such as a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, etc., they will not be described here. The audio circuit, the speaker, and the microphone can provide an audio interface. The audio circuit can convert the received audio data into an electrical signal, transmit the electrical signal to the speaker, and convert the electrical signal into a sound signal output by the speaker. On the other hand, the microphone converts the collected sound signal into an electrical signal, which is received by the audio circuit and converted into audio data. The audio data is output to the processor for processing, and then transmitted to another electronic device via the RF circuit, or output to the memory for further processing.

[0138] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the embodiments disclosed herein. It is intended that the specification and examples be considered as exemplary only, with a true scope of the application being indicated by the following claims.

[0139] It is to be understood that the application is not limited to the precise details of construction and the above-described embodiments and that various modifications and changes can be made by those skilled in the art without departing from the scope of the application. The scope of the application should only be limited by the claims appended hereto.

[0140] The above is a specific description of the preferred embodiment of the present application, but the present application is not limited to the above-described embodiments, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application. These equivalent modifications or replacements are all included in the scope defined by the claims of the present application.

Claims

1. A model configuration method based on algorithm screening, characterized in that, The method comprises the following steps: acquiring historical data and historical reference results, and inputting the historical data into a target model; acquiring a plurality of candidate algorithms, applying each of the candidate algorithms to the target model for offline running, and obtaining offline running results corresponding to each of the candidate algorithms according to the historical data; determining a first similarity between the historical reference results and each of the offline running results, and determining the offline running results with a first similarity greater than a first preset threshold as available offline results; determining the candidate algorithms corresponding to the available offline results as available algorithms; acquiring an evaluation sample group and a reference evaluation result, and inputting the evaluation sample group into the target model, wherein the evaluation sample group comprises a plurality of sample flows; allocating the available algorithms corresponding to each of the sample flows according to a preset strategy, and running the available algorithms to obtain corresponding test evaluation results; determining the available algorithm corresponding to the test evaluation result closest to the reference evaluation result as a target algorithm, and configuring the target model according to the target algorithm; the step of determining the available algorithm corresponding to the test evaluation result closest to the reference evaluation result as a target algorithm comprises the following steps: acquiring a second preset threshold; determining a second similarity between the reference evaluation result and each of the test evaluation results; determining the test evaluation result corresponding to the highest second similarity as a target evaluation result; determining the available algorithm corresponding to the target evaluation result as the target algorithm; after the step of configuring the target model according to the target algorithm, the method further comprises the following steps: determining a plurality of functional modules of the target model; determining a plurality of parameter groups according to the target model, wherein at least one parameter value of the functional modules is different between different parameter groups; performing AB testing on the target model according to the plurality of parameter groups; the step of performing AB testing on the target model according to the plurality of parameter groups further comprises the following steps: acquiring a test index and a reference flow of the AB testing; determining the reference flow as a test flow of the AB testing; adjusting the test index to change the test flow, and determining the minimum test flow obtained as a target test flow; performing AB testing on the target model according to the target test flow and the plurality of parameter groups.

2. The model configuration method based on algorithm screening according to claim 1, characterized in that: The preset strategy is an Interleaving model, and the weights of each of the sample flows are the same.

3. The model configuration method based on algorithm screening according to claim 1, characterized in that, The test index comprises at least one of the following: CTR; confidence level; statistical power; index calculation strategy; diversion type; standard deviation of the index.

4. The model configuration method based on algorithm screening according to claim 1, characterized in that, the step of performing AB testing on the target model according to the target test flow and the plurality of parameter groups comprises the following steps: determining an experimental group and a control group of the AB testing; adding test results triggered by the target test flow in each of the parameter groups to the experimental group, or adding test results not triggered by the target test flow in each of the parameter groups to the control group; determining a test result of the AB testing according to the test results recorded by the experimental group and the control group.

5. An algorithm-filtering-based model configuration apparatus characterized by comprising: The method comprises the following steps: A historical data acquisition unit is configured to acquire historical data and historical reference results, and input the historical data into a target model; A candidate algorithm acquisition unit is configured to acquire a plurality of candidate algorithms, apply each candidate algorithm to the target model for offline running, and obtain offline running results corresponding to each candidate algorithm according to the historical data; An offline running unit is configured to determine a first similarity between the historical reference results and each offline running result, and determine the offline running result with a first similarity greater than a first preset threshold as a usable offline result; A usable algorithm determination unit is configured to determine the candidate algorithm corresponding to the usable offline result as a usable algorithm; An evaluation sample acquisition unit is configured to acquire an evaluation sample group and a reference evaluation result, input the evaluation sample group into the target model, and the evaluation sample group includes a plurality of sample flows; A test unit is configured to assign a corresponding usable algorithm to each sample flow according to a preset strategy, run the usable algorithm to obtain a corresponding test evaluation result; A model configuration unit is configured to determine the usable algorithm corresponding to the test evaluation result closest to the reference evaluation result as a target algorithm, and configure the target model according to the target algorithm; The determination of the usable algorithm corresponding to the test evaluation result closest to the reference evaluation result as the target algorithm includes: acquiring a second preset threshold; determining a second similarity between the reference evaluation result and each test evaluation result; determining the test evaluation result corresponding to the highest second similarity as a target evaluation result; determining the usable algorithm corresponding to the target evaluation result as the target algorithm; After the configuration of the target model according to the target algorithm, the method further includes: determining a plurality of functional modules of the target model; determining a plurality of parameter groups according to the target model, and at least one parameter value of the functional module is different between different parameter groups; performing AB testing on the target model according to the plurality of parameter groups; The AB testing on the target model according to the plurality of parameter groups further includes: acquiring a test index and a reference flow of the AB testing; determining the reference flow as a test flow of the AB testing; adjusting the test index to change the test flow, and determining the minimum test flow as a target test flow; performing AB testing on the target model according to the target test flow and the plurality of parameter groups.

6. An electronic device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the algorithm screening-based model configuration method of any one of claims 1 to 4 when executing the computer program.

7. A computer readable storage medium storing a computer program, characterized in that, The computer program is used to execute the algorithm screening-based model configuration method of any one of claims 1 to 4.

Images