Method, device, electronic device and storage medium for debugging

Abstract

The present application provides a kind of DEBUG management method, device, electronic equipment and storage medium, wherein the DEBUG management method includes: obtaining AB test buried point log and sorting service buried point log;AB test buried point log and sorting service buried point log are parsed, aggregated, and structured data is obtained;The structured data is queried, and ranking interpretation is carried out.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of DEBUG. BACKGROUND

[0002] The online explanation and optimization platform is mainly constructed for the DEBUG of ranking explanation and feature, model and strategy arrangement in intelligent scene of search recommendation, advertisement and other Feeds flow. The existing technology mainly has the following problems:

[0003] Firstly, in intelligent scenes such as search recommendation, the whole process goes through recall, prediction, sorting, rearrangement and other processes, the link is relatively long, and the explainability of the final output list sorting is more concerned by the business, but because different services (recall service, model prediction service, strategy sorting service, business rearrangement service, AB platform service, etc.) are involved in the process, the explanation of the final sorting result will become very complex, and it is necessary to find the logs of each service to splice and then go through all the engineering service logic again to troubleshoot the specific reason.

[0004] Secondly, in the algorithm development process, it is expected that the algorithm can simulate the online process before the model is put into operation, and can edit a specific request (feature) to predict the final sorting list through the uncommissioned model.

[0005] Thirdly, the strategy product of search recommendation often hopes to adjust the online strategy in real time to verify the improvement idea, and also hopes to have a place to verify the influence of the improved strategy on the sorting (without affecting the online) on the historical strategy execution snapshot.

[0006] Fourthly, the personnel who troubleshoot the sorting problem often hope to solve the problem by adjusting the problem point (to achieve the expected sorting), and the existing technical solution cannot achieve this. SUMMARY

[0007] The following presents a summary of various exemplary technical solutions. In the following summary, some simplifications and omissions can be made to highlight and introduce some aspects of the various exemplary technical solutions, but the scope of the present application is not limited. A detailed description of the exemplary technical solutions sufficient to allow those of ordinary skill in the art to make and use the present application concepts will be presented in subsequent sections.

[0008] The technical solution of the present application provides a DEBUG management method, comprising: acquiring AB test and sorting service buried point logs; parsing and aggregating the AB test and sorting service buried point logs to obtain structured data; querying the structured data to perform ranking explanation.

[0009] Optionally, the method further includes: using AB test event logs and sorting service event logs as samples to train the algorithm model and output model data.

[0010] Optionally, the method further includes: processing at least one of the following based on the structured data and model data: feature tuning, model tuning, strategy tuning, feature importance, and model interpretation.

[0011] Optionally, the sorting service logging includes recall information, prediction information, sorting information, and rearrangement information.

[0012] Optionally, the AB test logging includes the request ID and AB information.

[0013] Another technical solution of the present invention provides a DEBUG management device, comprising: a receiving module configured to acquire AB test tracking logs and ranking service tracking logs; a parsing module configured to parse and aggregate the AB test tracking logs and ranking service tracking logs to obtain structured data; and a processing module configured to query the structured data and perform ranking interpretation.

[0014] Optionally, the device further includes an algorithm module configured to use AB test data logging and sorting service data logging as samples to train the algorithm model and output model data.

[0015] Optionally, the processing module is configured to process at least one of the following based on the structured data and model data: feature tuning, model tuning, strategy tuning, feature importance, and model interpretation.

[0016] Optionally, the sorting service logging includes recall information, prediction information, sorting information, and rearrangement information.

[0017] Optionally, the AB test logging includes the request ID and AB information.

[0018] Another technical solution of the present invention provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and running thereon, wherein the processor executes the computer program to implement the steps of the method described in any of the above technical solutions.

[0019] Another technical solution of the present invention provides a computer-readable storage medium, wherein the computer program, when executed by a processor, implements the steps of the method described in any of the above technical solutions.

[0020] The technical solution of the present invention has the following main beneficial effects:

[0021] First: in the intelligent scene such as search recommendation, the scheme can directly fuse the logs of different services (recall service, model prediction service, strategy ranking service, business rearrangement service, AB platform service, etc.) together, and after real-time analysis, filtering, aggregation and other operations, finally store the fused structured data for query, so that all the processes can be linked to display. Realize one-key query of the whole process, and solve the problem of ranking explanation difficulty of search recommendation.

[0022] Second: in the algorithm development process, the ability of historical playback and algorithm platform model management can make the algorithm verify its model through real online features in history. Algorithm development can also be simulated in offline scene to optimize in online.

[0023] Third: through the playback based on the finally fused structured data, the features and strategy data are stripped out for editing, and each service provides callback ability through a unified DEBUG-API protocol, so that the edited strategy, model, feature and other data are returned to the DEBUG interface of each service, which can realize real-time rendering of the ranking list. It can realize that the strategy product can verify and compare the influence of different strategies on ranking by adjusting a specific playback strategy on the playback, so that the strategy product can simulate the online environment to debug its strategy, and after the strategy debugging is completed, it can also be directly published online. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to better understand various exemplary embodiments, reference can be made to the accompanying drawings, in which:

[0025] Figure 1 A flowchart of the method of DEBUG management provided by the embodiments is shown;

[0026] Figure 2 A schematic diagram of an example of the whole process provided by the embodiments is shown;

[0027] Figure 3 A schematic diagram of part of the steps in the method of DEBUG management provided by the embodiments is shown;

[0028] Figure 4 A schematic diagram of part of the steps in the method of DEBUG management provided by the embodiments is shown;

[0029] Figure 5 A background query page provided by the embodiments is shown;

[0030] Figure 6 A DEBUG page provided by the embodiments is shown;

[0031] Figure 7 A ranking explanation page provided by the embodiments is shown;

[0032] Figure 8 A structural schematic diagram of an apparatus for DEBUG management provided by an embodiment is shown.

[0033] For ease of understanding, the same reference numerals have been used to refer to elements having substantially the same or similar structure and / or substantially the same or similar function. DETAILED DESCRIPTION

[0034] The description and drawings teach those skilled in the art sufficient information to permit them to produce the present application. Therefore, skills in the art will be able to design various arrangements without departing from the principles of the present application and the present application encompasses all such arrangements. In addition, all examples cited herein are primarily intended to illustrate the principles of the present application and to provide the skilled person with the best understanding of the concepts in the art to deepen the art, and all examples should be considered not limited to such specific cited examples and conditions. In addition, as used herein, unless otherwise indicated (for example, "or in addition" or "or in the alternative"), the term "or" refers to non-exclusive or (i.e., and / or). Also, the various embodiments described herein are not necessarily mutually exclusive, as some embodiments can be combined with one or more other embodiments to form new embodiments.

[0035] Terminology explanation:

[0036] Ranking explanation: Explanation of the ranking list of intelligent scenarios such as search recommendation.

[0037] Tuning: can be edited through the playback log of historical ranking, and the modified ranking list and explanation can be directly rendered through the online interface.

[0038] The first embodiment provides a method for DEBUG management, Figure 1 A flowchart of a method for DEBUG management provided by an embodiment is shown, and the method comprises:

[0039] S101: Obtain AB test and ranking service logging logs;

[0040] First, data logging is needed, including two aspects: first, AB test platform logging, the AB platform test accesses all business (such as tires, maintenance, etc.) must pass, so as to achieve AB log reuse, directly through the AB platform for logging; Second, search recommendation server logging, in order to keep consistent with each service logging, the search recommendation platform can provide a general logging component, that is, a unified logging SDK, the data log format is standard, compatible with the landing format of the data warehouse, the sample analysis logic of the algorithm, the real-time algorithm model, the feature missing alarm analysis and the analysis and filtering logic of the DEBUG platform.

[0041] The AB testing platform can make two (A / B) or multiple (A / B / N) versions, respectively test different versions. The AB testing platform's buried point log includes: AB information (for example, judge whether to use A or B service), request ID, etc., which can be in JSON format.

[0042] The search recommendation platform is used to sort the search list, and its functions include recall, prediction, sorting, rearrangement, wherein the recall refers to obtaining a request list, such as a tire list, the prediction refers to putting the list into a model to score according to the sorting rule and each weight; the sorting refers to logical processing according to the prediction, and then forming the sorting; the rearrangement refers to adding a strategy adjustment to reorder, for example, originally ranking first, but the strategy does not allow it to rank first, etc. The sorting service buried point log records detailed information of the whole process of the search recommendation, including information of each process of the recall, the prediction, the sorting, and the rearrangement, such as request feature information, recall information, prediction information, sorting score weight information, sorting strategy information, sorting result, called model information, etc. The sorting service buried point log can also adopt the JSON format.

[0043] S102: Analyzing and aggregating the AB testing buried point log and the sorting service buried point log to obtain structured data;

[0044] This step can be completed based on the real-time feature calculation rule engine of Flink, wherein the data format is as shown in Figure 3 There is a time drift problem of different service logs: the log timestamp of the AB platform is inconsistent with the log timestamp of the search recommendation service, which will drift, resulting in that the data cannot be associated, therefore, we set a timestamp lock, which is always used as a time dimension key in the whole event process, and whether the timestamp lock exists can be judged according to the request ID. The task of Flink includes analyzing and filtering the buried point log, then calculating and persisting, data merging operation, obtaining structured data, and outputting to HBASE. For example, three HTABLE tables are output here. Respectively including RequestID (full link request ID) + Subtype (business line scene); userID (user ID) + time (time); phone (telephone), which is also for the convenience of querying these information in the background, and the information can be filtered and searched by RequestID, userID, etc.

[0045] S103: Querying the structured data to perform ranking explanation.

[0046] The DEBUG platform queries structured data, including request feature information, recall information, prediction information, ranking score weight information, ranking strategy information, ranking results, called model information, AB information, etc. Through the structured data, ranking explanation can be performed. According to the structured data, the entire logical process of ranking can be played back, and the basis and logic of ranking can be known.

[0047] Optionally, the acquisition of the AB test buried point log and the ranking service buried point log can also form a training sample of an algorithm model. The sample data can be processed, cleaned, and subjected to feature engineering, and then the algorithm model is trained, the model is evaluated, the parameters are optimized, and the model data that passes the output evaluation is output. The sample data, i.e., the features, include user information, such as user age, and product information, such as price.

[0048] The DEBUG platform can perform feature optimization, model optimization, strategy optimization, feature importance, and model explanation through structured data and model data. Feature optimization depends on the combination of model data and structured data. All features participating in the model scoring of the product ranking at that time are called, and the features are input sample data of algorithm platform model training, such as user basic information, such as user age, product information, such as price, etc. The DEBUG platform can reorder by adjusting the value of a certain feature. Model optimization refers to obtaining a newly trained model from the algorithm platform. When the model is iterated, the newly trained model is used for ranking, i.e., different models can be selected to see whether the ranking achieves the expected effect. Feature importance is the value of the importance of all features in the trained model. The value of the feature importance can be viewed. Model explanation is to view each model and the influence of different models on ranking. Feature importance and model structure can be visualized together with Meta information produced by the algorithm platform. In addition, strategy optimization can be performed. The DEBUG platform can call strategy information, such as rearrangement strategy, which is a strategy for rearrangement in the search recommendation platform according to the business requirements, and adjust weight information, etc. The strategy can be optimized.

[0049] See Figure 4The DEBUG platform performs ranking interpretation by querying structured data in the HBASE for ranking interpretation, which can include an HTable table, and the structured data can be used for ranking interpretation. The DB database can also be queried for model version, feature version, and other mate information, which can include ID, model name, scenario ID, model version, feature version, model path, whether online, and the like. The OSS can also be used to obtain model file data, and the OSS (Object Storage) is an HBASE columnar storage database trained by the algorithm platform. The DEBUG platform can support the debugging interface based on the model file data, model version, feature version, and other mate information, and the debugging interface can support the entire DEBUG capability (including features, models, and strategies) by replaying structured data for ranking interpretation. By replaying the final fused structured data, the feature and strategy data can be stripped out for editing, and each service can provide callback capabilities through a unified DEBUG-API protocol, so that the edited strategy, model, feature, and other data can be returned to the DEBUG interface of each service, and the ranking list can be rendered in real time. The structured data can be edited in a stateful manner, and the platform model management and strategy platform strategy orchestration can integrate the capabilities into one, adjust the strategy, model, feature, and other data, and pass the debugging interface again to present the online effect in an offline manner without affecting the online effect.

[0050] The third embodiment also provides a full-process system structure diagram, as shown in Figure 2 The entire large system includes an APP, a message queue, an AB test platform, an intelligent configuration background, and a big data platform, wherein the intelligent configuration background includes a search recommendation platform, an algorithm platform, a real-time feature rule engine, and a Debug capability module.

[0051] 1. A user sends a viewing request on the APP, for example, the user wants to view a list about tires, or a list about maintenance, search, advertising, or other lists.

[0052] 2. The AB test platform requests A / B shunting information and sends an AB information log to the message queue, wherein the AB information log includes AB information (such as determining whether A or B service is applied), a request ID, and the like, which can be in JSON format.

[0053] 3. The search recommendation platform receives a request for a search recommendation ranking list and sends a ranking service logging point log of a search recommendation whole process to a message queue. The search recommendation whole process includes recall, prediction, ranking, and rearrangement. The recall refers to obtaining a request list, such as a tire list. The prediction refers to putting the list into a model to score according to a ranking rule and various weights. The ranking refers to performing logical processing according to the prediction and then forming a ranking. The rearrangement refers to adding a strategy adjustment and re-ranking, such as an additional strategy that does not allow a first ranking. The ranking service logging point log records detailed information of the search recommendation whole process, including information of each process of the recall, the prediction, the ranking, and the rearrangement, such as request feature information, recall information, prediction information, ranking score weight information, ranking strategy information, a ranking result, and called model information. The ranking service logging point log can also be in a JSON format.

[0054] 4. The big data platform consumes logs from the message queue, where the logs include AB information logs and ranking service logging point logs of the search recommendation whole process. A data warehouse (HIVE) in the big data platform persists all the information logs, including the AB information logs and the ranking service logging point logs of the search recommendation whole process.

[0055] 5. The big data platform forms a training sample of an algorithm model according to the persisted ranking service AB information logs and the logging point logs.

[0056] 6. An algorithm development platform processes, cleans, and performs feature engineering on the sample data, and then performs algorithm model training, model evaluation, parameter tuning, and output of a passed model file. The sample data, i.e., features, include user information, such as user age, and product information, such as price. The algorithm platform also selects a model. The model analyzes a user purchase conversion rate for a product. For example, the model can obtain a score of each product with respect to a user purchase possibility. If the user purchase possibility is high, the score is large, and the product is ranked in front, thereby improving the conversion rate.

[0057] 7. A real-time feature rule engine consumes logs from the message queue, where the logs also include the AB information logs and the ranking service logging point logs of the search recommendation whole process. Then, the real-time feature rule engine analyzes, filters, and aggregates the AB information logs and the ranking service logging point logs of the search recommendation whole process into structured data suitable for a database.

[0058] 8. The real-time feature rule engine outputs the structured data and stores the structured data in a HBASE.

[0059] 9. The DEBUG capability module requests playback data, which has two data, one is structured data in HBASE (including request feature information, recall information, prediction information, ranking score weight information, ranking strategy information, ranking result, called model information, AB information, etc.), and the other is the model file produced by the algorithm platform. The DEBUG platform performs ranking interpretation, feature optimization, model optimization, strategy optimization, feature importance, and model interpretation according to the two data. The ranking interpretation depends on the playback structured data, and the entire logical process of ranking can be played back according to the structured data to know the basis and logic of ranking. Feature optimization depends on the combination of model data and playback structured data. By calling all features participating in the model scoring of the goods ranking at that time, the features are the input of the algorithm platform model training, that is, sample data, such as user basic information, such as user age, and product information, such as price. The DEBUG capability module can reorder by adjusting the value of a certain feature. Model optimization refers to obtaining a new trained model from the algorithm platform. When the model is iterated, ranking is performed according to the new trained model, that is, different models can be selected to see if the ranking achieves the expected effect. Strategy optimization is that the DEBUG capability module can call strategy information. The strategy information refers to the strategy of rearranging the search recommendation platform according to the business requirements when adjusting the weight. Feature importance is the value of the importance of all features in the trained model. The feature importance value can be viewed. Model interpretation is to view each model and the influence of different models on ranking. Feature importance and model structure will be visualized together with the Meta information produced by the algorithm platform.

[0060] Referring to Figure 5 , enter the background query page. Through time, user ID, mobile phone number, request ID, scene, etc. information, request ID, request time, scene, user ID, device ID, vehicle type, city, AB grouping information, etc. can be queried; operations can also be performed, including ranking DEBUG configuration, ranking result viewing, recall result viewing, etc. An example is to query the mobile phone number.

[0061] Referring to Figure 6, enter the DEBUG page, you can query the ranking, ItemId, goods, final score, strategy details (including strategy adjustment, etc.), strategy relative to a certain model (here it is oil_suggest_rt model) ranking change, original use model, original ranking, feature snapshot, additional information, debugging time, ranking, ItemId, algorithm score, etc. And you can also select a model, you can edit the strategy. You can see that the feature snapshot is in JSON format, which contains all the features that the participating model scored for the goods at the time. You can edit the JSON, including user information, product information, whether to click, whether to buy, whether to browse, etc. You can adjust the value of a certain feature, which will go through the logic again to sort and see its impact on the ranking. You can query the logic after the change, which is feature tuning. You can also select a model or add a new trained model, reorder, and refresh the list, which is model tuning. You can also view the feature importance in the model. Feature importance is the value of all features in the trained model. You can view the feature importance value. Model explanation is to view each model and the impact of using different models on sorting.

[0062] Referring to Figure 7 , enter the ranking explanation page, you can query the ranking, ItemId, goods, ranking, final score, strategy details (strategy adjustment, etc.), strategy relative to a certain model (here it is oil_suggest_rt model) ranking change, ranking, ItemId, algorithm score, etc. Strategy details include, for example, oil factory and upgraded viscosity weighting strategy, oil 4L, 1L reordering strategy. These strategies can also be modified, such as modifying the value of the weighting, which is strategy tuning.

[0063] The third embodiment also provides a DEBUG management device, as shown in Figure 8 , the DEBUG management device comprises a receiving module 801, an analysis module 802, and a processing module 803.

[0064] The receiving module 801 is configured to obtain AB test and sorting service log data.

[0065] Data logging includes two aspects: first, AB test platform logging. The AB platform test accesses all business (such as tires, maintenance, etc.) that must be passed through, so as to achieve AB log reuse, and logging is performed directly through the AB platform. Second, search recommendation server logging. In order to keep each service logging consistent, the search recommendation platform can provide a general log logging component, that is, a unified logging SDK. The data log format is standard, compatible with the disk format of the data warehouse, the sample analysis logic of the algorithm, the real-time algorithm model, the feature missing alarm analysis, and the analysis and filtering logic of the DEBUG platform.

[0066] The AB testing platform can make two (A / B) or multiple (A / B / N) versions, respectively test different versions. The AB testing platform's buried point log includes: AB information (such as judging whether A or B service is applied), request ID, etc., which can be in JSON format.

[0067] The search recommendation platform is used to sort the search list, and its functions include recall, prediction, sorting, rearrangement, wherein the recall refers to obtaining the request list, such as the tire list, the prediction refers to putting the list into the model to score according to the sorting rule and each weight; the sorting refers to logical processing according to the prediction, and then forming the sorting; the rearrangement refers to adding strategy adjustment and reordering, such as additional strategies that do not allow the first to be ranked first. The sorting service buried point log records the detailed information of the whole process of the search recommendation, including the information of each process of recall, prediction, sorting, and rearrangement, such as request feature information, recall information, prediction information, sorting score weight information, sorting strategy information, sorting result, and called model information. The sorting service buried point log can also be in JSON format.

[0068] The parsing module 802 is configured to parse and aggregate the AB testing buried point log and the sorting service buried point log to obtain structured data.

[0069] The parsing module 802 can be completed based on the real-time feature calculation rule engine of Flink, and the data format is as shown in Figure 3 There is a time drift problem of different service logs: the log timestamp of the AB platform is inconsistent with the log timestamp of the search recommendation service, which will drift, resulting in that the data cannot be associated, therefore, we set a timestamp lock, which is always used as a time dimension key in the whole event process, and whether the timestamp lock exists can be judged according to the request ID. The task of the parsing module 802 includes parsing and filtering the buried point log, then calculating and persisting, data merging operation, obtaining structured data, and outputting to HBASE. For example, three HTABLE tables are output here. Respectively including RequestID (full link request ID) + Subtype (business line scene); userID (user ID) + time (time); phone (telephone), which is also for the convenience of background query of these information, and the information can be filtered and searched by RequestID, userID, etc.

[0070] The processing module 803 is configured to query the structured data and perform ranking interpretation.

[0071] The processing module 803 queries structured data, including request feature information, recall information, prediction information, ranking score weight information, ranking strategy information, ranking results, called model information, AB information, etc. Through the structured data, ranking explanation can be performed. According to the structured data, the entire logical process of ranking can be played back, and the basis and logic of ranking can be known.

[0072] Optionally, the device also includes an algorithm module. The AB test buried point log and the ranking service buried point log can also form a training sample of an algorithm model. The sample data can be processed, cleaned, and subjected to feature engineering, and then the algorithm model is trained, the model is evaluated, the parameters are optimized, and the model data that passes the output evaluation is output. The sample data, i.e., the features, include user information, such as user age, and product information, such as price.

[0073] The processing module 803 can perform feature optimization, model optimization, strategy optimization, feature importance, and model explanation through structured data and model data. Feature optimization depends on the combination of model data and structured data. All features participating in the model ranking and scoring of the product at that time are retrieved. The features are input sample data of algorithm platform model training, such as user basic information, such as user age, and product information, such as price. The processing module 803 can reorder by adjusting the value of a certain feature. Model optimization refers to obtaining a newly trained model from the algorithm platform. When the model is iterated, ranking is performed according to the newly trained model. Different models can be selected to see if the ranking achieves the expected effect. Feature importance is the value of the importance of all features in the trained model. The value of the feature importance can be viewed. Model explanation is to view each model and the influence of different models on ranking. Feature importance and model structure can be visualized together with Meta information produced by the algorithm platform. In addition, strategy optimization can be performed. The processing module 803 can retrieve strategy information, such as rearrangement strategy, which is a strategy for rearranging the search recommendation platform according to the business requirements, and adjust weight information, etc. The strategy can be optimized.

[0074] The fourth embodiment further provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein the processor implements the steps of the method of any one of the above embodiments, such as steps S101-S103, or the processor implements the functions of the modules / units of the above embodiments, such as the functions of the units 201-203, when executing the computer program. The computer program can be divided into one or more modules / units, which are stored in the memory and executed by the processor. The one or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the electronic device.

[0075] The electronic device can be a mobile terminal such as a smartphone, or a computing device such as a desktop computer, a notebook computer, a palm computer, and a cloud server. The electronic device can include, but is not limited to, a processor and a memory, and can include more or fewer components, or combine some components, for example, the electronic device can further include an input / output device, a network access device, a bus, and the like. The processor can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, and the like. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The memory can be an internal storage unit of the electronic device, such as a hard disk or a memory of the electronic device. The memory can also be an external storage device of the electronic device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, and the like. Further, the memory can include both the internal storage unit and the external storage device of the electronic device.

[0076] The fifth embodiment further provides a computer-readable storage medium, wherein the computer program is executable by the processor to implement the steps of the method of any one of the above embodiments.

[0077] The various function units in the embodiments of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware, or in the form of a software function unit. When the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such an understanding, all or part of the flow of the method in the above-described embodiments can be implemented by a computer program, and the computer program can instruct relevant hardware to complete, and the computer program can be stored in a computer readable storage medium. The computer program includes computer program code, which can be in the form of source code, object code, executable file or some intermediate form. The computer readable medium can include any entity or device, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier wave signal, telecommunication signal and software distribution medium, etc. It should be noted that the computer readable medium can include or exclude contents according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electric carrier wave signal and telecommunication signal.

[0078] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above function units and modules is taken as an example, and in actual application, the above functions can be completed by different function units and modules according to needs, that is, the internal structure of the device is divided into different function units or modules to complete all or part of the functions described above. The function units and modules in the embodiments can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit, and the integrated unit can be realized in the form of hardware or in the form of a software function unit. In addition, the specific names of the function units and modules are only for convenient distinction, and do not limit the protection scope of the present application. The specific working process of the units and modules in the system can refer to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0079] In the above embodiments, the description of each embodiment has its own focus, and the parts not described or recorded in a certain embodiment can be referred to the relevant description of other embodiments. A person of ordinary skill in the art can be aware that units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solutions. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0080] In the embodiments provided in the present application, it should be understood that the disclosed system, electronic device and method can be implemented in other ways. For example, the above-described system, electronic device embodiments are merely schematic, for example, the division of the modules or units is only a logical function division, and there can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed units can be indirect coupling or communication connection through some interface, system or unit, and can be electrical, mechanical or other forms. The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0081] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, a person of ordinary skill in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method of DEBUG management, characterized in that, The method comprises the following steps: obtaining AB test and ranking service logging; parsing and aggregating the AB test and ranking service logging to obtain structured data, wherein the structured data comprises at least one of the following: request feature information, recall information, prediction information, ranking score weight information, ranking strategy information, ranking result, called model information, and AB information, a timestamp lock is set, the timestamp is used as a time dimension throughout the method, whether the timestamp lock exists is determined according to a request ID, and three HTABLE tables are output, which respectively comprise full-link request ID + business line scene, user ID + time, and phone; querying the structured data to perform ranking interpretation; based on the structured data, the entire logical process of ranking is played back to know the basis and logic of ranking, based on model file data and model version feature version mate information, various business DEBUG interfaces and search recommendation DEBUG interfaces are debugged to support the entire DEBUG capability, including features, models, and strategies, ranking interpretation is performed through playback of the structured data; through playback based on the finally fused structured data, feature and strategy data are stripped out for editing, various services provide callback capability through a unified DEBUG-API protocol, so that the edited strategy, model, and feature data are returned to the DEBUG interfaces of various services, ranking lists are rendered in real time, the structured data is edited in a state, platform model management and strategy arrangement of a strategy platform are performed, capabilities are fused together, strategy, model, and feature data are adjusted, the debugging interfaces are debugged again, and online effects are presented in an offline manner.

2. The method of DEBUG management according to claim 1, wherein, The method further comprises the following steps: the ranking service logging comprises recall information, prediction information, ranking information, and rearrangement information.

3. The method of DEBUG management according to claim 1, wherein, The method further comprises the following steps: the AB test logging comprises a request ID and AB information.

4. The method of DEBUG management according to claim 1, wherein, The method further comprises the following steps: the AB test and ranking service logging are used as samples to train an algorithm model, and model data is output.

5. The method of DEBUG management according to claim 4, wherein, The method further comprises the following steps: based on the structured data and the model data, at least one of the following is processed: feature optimization, model optimization, strategy optimization, feature importance, and model interpretation.

6. An apparatus for DEBUG management, the apparatus comprising: The method comprises the following steps: a receiving module is configured to obtain AB test and ranking service logging; a parsing module is configured to parse and aggregate the AB test and ranking service logging to obtain structured data, wherein the structured data comprises at least one of the following: request feature information, recall information, prediction information, ranking score weight information, ranking strategy information, ranking result, called model information, and AB information, a timestamp lock is set, the timestamp is used as a time dimension throughout the method, whether the timestamp lock exists is determined according to a request ID, and three HTABLE tables are output, which respectively comprise full-link request ID + business line scene, user ID + time, and phone; a processing module is configured to query the structured data to perform ranking interpretation; Based on the structured data, the entire sorting process is replayed to understand the basis and logic of the ranking. Using model file data and model version feature version mate information, the DEBUG interfaces of various services and the search and recommendation DEBUG interface are debugged to support the entire DEBUG capability, including features, models, and strategies. Ranking is explained through the replay of structured data. By replaying the final fused structured data, feature and strategy data are extracted for editing. Each service provides callback capabilities through a unified DEBUG-API protocol, allowing the edited strategies, models, and feature data to be sent back to the DEBUG interfaces of each service. The sorted list is rendered in real time. The structured data, in an editable state, is managed by the platform model and orchestrated by the strategy platform, integrating capabilities. The strategies, models, and feature data are adjusted, and the debugging interfaces are re-debugged, allowing the online effects to be displayed offline.

7. The apparatus for DEBUG management according to claim 6, wherein, The sorting service logging includes recall information, prediction information, sorting information, and reordering information.

8. The apparatus for DEBUG management according to claim 6, wherein, Also includes: The AB test log includes the request ID and AB information.

9. The apparatus for DEBUG management according to claim 6, wherein, It also includes an algorithm module, which is configured to use A / B test logging and sorting service logging as samples to train the algorithm model and output model data.

10. The apparatus for DEBUG management according to claim 9, wherein, The processing module is further configured to process at least one of the following based on the structured data and model data: feature tuning, model tuning, strategy tuning, feature importance, and model interpretation.

11. An electronic device, comprising: include: A processor, a memory, and a computer program stored in the memory that runs on the memory, wherein the processor, when executing the computer program, implements the method of any one of claims 1-5.

12. A computer-readable storage medium having stored thereon a computer program, characterized in that When the computer program is executed by a processor, it implements the method of any one of claims 1-5.

Images