An algorithm service evaluation method and device

Abstract

The application relates to an algorithm service evaluation method and device. The method comprises the following steps: acquiring an index value of each algorithm service in a plurality of algorithm services; wherein the index value is determined based on a value of a preset index and a number of algorithm service calls, and the plurality of algorithm services correspond to the same preset index; calculating a normal distribution value of each algorithm service according to the index value of each algorithm service; and determining an algorithm service to be improved in the plurality of algorithm services according to the normal distribution value of each algorithm service. Global evaluation of a plurality of algorithm services is realized, and evaluation of a full link such as algorithm call recall, exposure display and user click is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of algorithm service, and in particular, to an algorithm service evaluation method and device. BACKGROUND

[0002] Currently, the effect of business evaluation algorithm is combined with online traffic A / B test, and some business indicators are observed to realize the increase and decrease. For example, for the advertisement algorithm service, the advertisement revenue, ECPM (Effective Cost Per Mille, effective cost per thousand displays), CTR (Click Through Rate, click through rate) and the like are generally used to measure, wherein:

[0003] ECPM is equal to: (total advertising cost / advertising display quantity total) * 1000;

[0004] CTR is equal to: actual click times / total advertising display quantity.

[0005] However, the premise of these indicators is that the strategy return result has been exposed, so the algorithm calling recall link is missing, and only the effect of a single algorithm service is concerned, and the algorithm service cannot be controlled globally. In fact, some algorithm services have very few recalls, so the advertising display quantity is necessarily very small, and the ECPM indicator and the CTR indicator are calculated according to the advertising display quantity, so it is difficult to find the recall problem, and the effect evaluation appears to be a gap. In order to solve the above shortcomings / problems, a new algorithm service evaluation method is urgently needed SUMMARY

[0006] In order to overcome the problems in the related art, embodiments of the present application provide an algorithm service evaluation method and device. The technical solution is as follows:

[0007] According to a first aspect of the embodiments of the present application, an algorithm service evaluation method is provided, comprising:

[0008] Obtaining an indicator value of each algorithm service in a plurality of algorithm services; wherein the indicator value is determined based on a value of a preset indicator and an algorithm service calling number, and the plurality of algorithm services correspond to the same preset indicator;

[0009] According to the indicator value of each algorithm service, a normal distribution value of each algorithm service is calculated;

[0010] According to the normal distribution value of each algorithm service, a to-be-improved algorithm service in the plurality of algorithm services is determined.

[0011] In one embodiment, the obtaining of the indicator value of each algorithm service in the plurality of algorithm services comprises: calculating the indicator value of the algorithm service according to the following formula:

[0012] (pre-set index value / algorithm service call times) * 1000.

[0013] In one embodiment, the determining the algorithm service to be improved from the plurality of algorithm services according to the normal distribution value of each algorithm service comprises:

[0014] According to the normal distribution value of each algorithm service, a frequency distribution histogram and a normal distribution curve are drawn.

[0015] According to the frequency distribution histogram and the normal distribution curve, the algorithm service to be improved from the plurality of algorithm services is determined.

[0016] In one embodiment, the method further comprises:

[0017] According to the pre-set algorithm service and strategy mapping relationship, the strategy corresponding to the algorithm service is obtained.

[0018] The sum of the values of the pre-set index of the strategy corresponding to the algorithm service is obtained as the value of the pre-set index of the algorithm service.

[0019] The sum of the call quantities of the strategy corresponding to the algorithm service is obtained as the algorithm service call times.

[0020] In one embodiment, the method further comprises:

[0021] The index value of the algorithm service is stored.

[0022] After the algorithm service is updated, the effect of the algorithm service is determined according to the index values before and after the algorithm service is updated; and / or, the index value of the algorithm service is calculated after a pre-set time length, and the effect of the algorithm service is determined according to the new and old index values.

[0023] When the effect of the algorithm service is lower than expected, a monitoring notification is sent.

[0024] According to a second aspect of an embodiment of the present application, an algorithm service evaluation device is provided, comprising:

[0025] A first obtaining module is configured to obtain an index value of each algorithm service from a plurality of algorithm services; wherein the index value is determined based on a pre-set index value and algorithm service call times, and the plurality of algorithm services correspond to the same pre-set index.

[0026] A calculating module is configured to calculate a normal distribution value of each algorithm service according to the index value of each algorithm service.

[0027] A determining module is configured to determine an algorithm service to be improved from the plurality of algorithm services according to the normal distribution value of each algorithm service.

[0028] The first obtaining module is configured to calculate the index value of the algorithm service according to the following formula:

[0029] (pre-set index value / algorithm service call times) * 1000.

[0030] In an embodiment, the determining module is configured to:

[0031] According to the normal distribution value of each algorithm service, draw a frequency distribution histogram and a normal distribution curve;

[0032] According to the frequency distribution histogram and the normal distribution curve, determine the algorithm service to be improved in the plurality of algorithm services.

[0033] In an embodiment, the apparatus further comprises:

[0034] The second obtaining module is configured to obtain the strategy corresponding to the algorithm service according to a pre-set algorithm service and strategy mapping relationship.

[0035] The first obtaining module is configured to: obtain the sum of the pre-set index values of the strategies corresponding to the algorithm service as the pre-set index value of the algorithm service; and obtain the sum of the call quantities of the strategies corresponding to the algorithm service as the algorithm service call times.

[0036] According to a third aspect of an embodiment of the present application, an algorithm service evaluation apparatus is provided, comprising:

[0037] a processor;

[0038] a memory for storing processor-executable instructions;

[0039] The processor is configured to:

[0040] obtain an index value of each algorithm service in a plurality of algorithm services, wherein the index value is determined based on a pre-set index value and algorithm service call times; and wherein the plurality of algorithm services correspond to a same pre-set index;

[0041] According to the index value of each algorithm service, calculate a normal distribution value of each algorithm service;

[0042] According to the normal distribution value of each algorithm service, determine the algorithm service to be improved in the plurality of algorithm services.

[0043] According to a fourth aspect of an embodiment of the present application, a computer readable storage medium is provided, which stores computer instructions, the instructions being executed by a processor to implement the steps of any method in the first aspect of an embodiment of the present application.

[0044] The technical scheme provided by the embodiment of the present application is global multi-algorithm service oriented, and the effect of the algorithm service is evaluated according to distribution; and the index value of the algorithm service does not depend on the exposure amount of the algorithm service, but uses the number of algorithm service calls to realize evaluation of the whole link from algorithm call recall, exposure display, user click (charge) and the like, and realize the connection evaluation of engineering call and business index. For the algorithm service with high call amount, little exposure and little income, the existing ECPM index cannot be perceived, but the index value proposed in the present application can be obviously perceived. The preset index in the index value proposed in the present application can be dynamically set in combination with business characteristics.

[0045] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and are not limiting to the present application. BRIEF DESCRIPTION OF DRAWINGS

[0046] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0047] Figure 1 is a flow chart of an algorithm service evaluation method according to an exemplary embodiment;

[0048] Figure 2 is a flow chart of an algorithm service evaluation method according to an exemplary embodiment;

[0049] Figure 3 is a data flow diagram according to an exemplary embodiment;

[0050] Figure 4 is a data calculation process diagram according to an exemplary embodiment;

[0051] Figure 5a is an index value diagram according to an exemplary embodiment;

[0052] Figure 5b is a normal distribution value diagram according to an exemplary embodiment;

[0053] Figure 5c is an algorithm normal distribution curve diagram according to an exemplary embodiment;

[0054] Figure 6 is a block diagram of an algorithm service evaluation device according to an exemplary embodiment;

[0055] Figure 7 is a block diagram of an algorithm service evaluation device according to an exemplary embodiment;

[0056] Figure 8is a block diagram of an algorithm service evaluation device according to an exemplary embodiment. DETAILED DESCRIPTION

[0057] The exemplary embodiments will be described in detail hereinbelow with reference to the drawings. In the following description, the same drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present invention. Instead, they are merely examples of apparatuses and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0058] The effect of existing combined business evaluation algorithms is mostly achieved through online traffic A / B testing, observing the rise and fall of some business indicators. For example, for an advertising algorithm service, generally, advertising revenue, ECPM (Effective Cost Per Mille), CTR (Click Through Rate), and the like are used as indicators for measurement. However, the premise of these indicators is that the strategy return result has been exposed, thus missing the algorithm call recall link, and only the effect of a single algorithm service is concerned, and the algorithm service cannot be globally controlled.

[0059] Embodiments of the present invention provide an algorithm service evaluation method, which can be applied to a mobile phone, computer, server, or the like terminal for evaluating algorithm services. As shown in Figure 1 The method comprises the following steps 101 to 103:

[0060] In step 101, the indicator value of each algorithm service in a plurality of algorithm services is obtained; wherein the indicator value is determined based on the value of a preset indicator and the number of algorithm service calls, and the plurality of algorithm services correspond to the same preset indicator.

[0061] The plurality of algorithm services in this step correspond to the same preset indicator, that is, the plurality of algorithm services can also be considered as services of the same category, for example, the plurality of algorithm services all belong to advertising strategy services, and thus the preset indicator is set as advertising revenue. For example, the advertising strategy services can include the following algorithm services: shopping cart guess you like, search advertising, product detail page guess you like, order center similar store service, and the like. The preset indicators of these algorithm services are all advertising revenue.

[0062] If the plurality of algorithm services all belong to content recommendation algorithms, then the preset indicator can be set as IPV (item page view, detail page view amount); if the plurality of algorithm services all belong to search algorithms, then the preset indicator can be set as CTR.

[0063] The preset index can be dynamically set according to the business characteristics. For example, if the algorithm service is an advertising strategy service, the preset index can be set as the advertising income. In an embodiment of the present application, the correspondence between the categories of algorithm services and the preset indexes can be set in advance, and then when the index value of each algorithm service is obtained, the corresponding preset index is determined according to the category of the algorithm service.

[0064] In an embodiment of the present application, the index value of the algorithm service can be calculated according to the following formula:

[0065] (value of the preset index / call number of the algorithm service)*1000.

[0066] As described above, if the algorithm service is an advertising strategy service, the preset index is set as the advertising income, and the advertising income can be obtained from the advertising deduction flow information. The preset index in the formula can be set according to the business index that the algorithm service cares most about. The call number of the algorithm service can be obtained from the advertising service log. The index value calculated by the present application can be named as EIPC (effective index per call).

[0067] In step 102, the normal distribution value of each algorithm service is calculated according to the index value of each algorithm service.

[0068] In an embodiment of the present application, the average value and the standard deviation of the EIPC of each algorithm service are calculated, and then the normal distribution value of the EIPC of each service is calculated. The normal distribution value of each algorithm service is calculated, including: the normal distribution value of the algorithm service is calculated according to the following formula:

[0069]

[0070] Wherein, σ is the standard deviation; μ is the data expectation (the average value of the random variable).

[0071] In step 103, the algorithm service to be improved in the plurality of algorithm services is determined according to the normal distribution value of each algorithm service.

[0072] According to the statistical principle, the algorithm effect of the plurality of algorithm services should conform to the normal distribution. Therefore, the plurality of algorithm services are evaluated, and the algorithm service with low EIPC is the algorithm service to be improved.

[0073] The technical scheme provided by the application is oriented to global multiple algorithm services, and the effect of the algorithm service is evaluated according to distribution; and the index value of the algorithm service is not dependent on the exposure amount of the algorithm service, but the algorithm service call frequency is used to realize evaluation of the whole link from algorithm call recall, exposure display, user click (charge) and the like, and realize the connected evaluation of engineering call and business index. For an algorithm service with high call amount, little exposure and little income, the existing ECPM index cannot be perceived, but the index value provided by the application can be obviously perceived. The preset index in the index value provided by the application can be set as a quality score in combination with the business ipv (click pv / exposure uv) and search commodity.

[0074] In an embodiment of the application, step 103 determines the algorithm service to be improved in the multiple algorithm services according to the normal distribution value of each algorithm service, and includes the following steps A1-A2.

[0075] In step A1, a frequency distribution histogram and a normal distribution curve are drawn according to the normal distribution value of each algorithm service.

[0076] In step A2, the algorithm service to be improved in the multiple algorithm services is determined according to the frequency distribution histogram and the normal distribution curve.

[0077] The embodiment can intuitively determine the algorithm to be improved from the multiple algorithm services.

[0078] In an embodiment of the application, one service can correspond to multiple strategies, and a strategy is a specific implementation of a service (the industry often uses different strategies of the same service to observe a / b test effect) ; the method further includes steps B1-B3.

[0079] In step B1, the strategy corresponding to the algorithm service is obtained according to a preset algorithm service and strategy mapping relationship.

[0080] One or more strategies corresponding to the algorithm service can be set in the algorithm service and strategy mapping relationship. The algorithm service id (identifier) and the strategy id (identifier) can be recorded in the algorithm service and strategy mapping relationship. For example, the advertisement search service corresponds to three strategies of a benchmark bucket, new category recall and coarse optimization.

[0081] In step B2, the sum of the values of the preset index of the strategy corresponding to the algorithm service is obtained as the value of the preset index of the algorithm service.

[0082] When one algorithm service corresponds to multiple strategies, the values of the preset index of each strategy are accumulated as the value of the preset index of the algorithm service according to the strategy id aggregation.

[0083] In step B3, the sum of the call amounts of the strategies corresponding to the algorithm service is obtained as the algorithm service call frequency.

[0084] When one service can correspond to multiple policies, the number of invocations of each policy is accumulated as the number of invocations of the algorithm service according to the policy id aggregation.

[0085] In an embodiment of the present application, the method further comprises steps C1-C3:

[0086] In step C1, the index value of the algorithm service is stored;

[0087] In step C2, the index value of the algorithm service is calculated again after the algorithm service is updated, and the effect of the algorithm service is determined according to the index values before and after the algorithm service is updated;

[0088] In step C3, a monitoring notification is sent when the effect of the algorithm service is lower than expected.

[0089] In this embodiment, the index value of a single algorithm service itself can be used as a baseline to evaluate the difference of the algorithm service after iteration relative to itself. For example, a first preset threshold can be set, and when the difference between the index values before and after the algorithm service is updated is greater than the first preset threshold, it is considered that the effect of the algorithm service becomes lower than expected.

[0090] In an embodiment of the present application, the method further comprises steps D1-D3:

[0091] In step D1, the index value of the algorithm service is stored;

[0092] In step D2, the index value of the algorithm service is calculated again after a preset time period, and the effect of the algorithm service is determined according to the new and old index values.

[0093] For example, the index value of the algorithm service can be calculated again the next day.

[0094] In step D3, a monitoring notification is sent when the effect of the algorithm service is lower than expected.

[0095] In this embodiment, the index value of a single algorithm service itself can be used as a baseline to evaluate the effect change of a single algorithm service at different periods. For example, a second preset threshold can be set, and when the difference between the new and old index values of the algorithm service is greater than the second preset threshold, it is considered that the effect of the algorithm service becomes lower than expected.

[0096] The above various embodiments can be used in combination as needed.

[0097] The implementation process will be described in detail below through embodiments.

[0098] Figure 2An algorithm service evaluation method according to an exemplary embodiment is shown. The algorithm service evaluation method can be applied to any suitable terminal, such as a server, etc. As shown in Figure 2 , the algorithm service evaluation method includes the following steps:

[0099] Step 201, obtaining an index value of each algorithm service in a plurality of algorithm services.

[0100] Taking 30 advertising algorithm services as an example, using advertising revenue as a preset index, the calculation formula of the index value of each algorithm service is: EIPC = (advertising revenue / algorithm service call quantity) * 1000.

[0101] Specifically, the index value of the algorithm service can be calculated through big data offline analysis. A specific data flow diagram is shown in Figure 3 , for example, the advertising service system generates an advertising service log, uses a log synchronization tool to obtain an advertising algorithm call log from the advertising service log, uses a database synchronization tool to obtain advertising deduction flow, algorithm service information details, etc. from the advertising business database and the strategy service business database, and finally, based on the advertising algorithm call log, the advertising deduction flow, and the algorithm service information details, the EIPC value of the advertising algorithm service can be calculated.

[0102] As shown in Figure 4 , a specific data acquisition and algorithm service index value calculation diagram is shown:

[0103] From the algorithm service information in Figure 3 , the service and strategy mapping relationship (mapping) can be obtained, one service corresponds to multiple strategies, and the strategy is the specific implementation of the service; the algorithm service information limits the advertising algorithm service type, and the strategy id under each service is taken out. For example, the advertising search service will have three strategies, such as benchmark bucket, new category recall, and coarse optimization. From the advertising algorithm call log in Figure 3 , the call count of each strategy is obtained: according to the strategy id, the call count of the strategy with the same id is aggregated. From the advertising deduction flow in Figure 3 , the income (ad_cost) of each strategy is obtained: according to the strategy id, the income of the strategy with the same id is aggregated. The data of the strategy corresponding to the algorithm service is summarized, and the above data acquisition and summarization can be processed by using the database through the key and the join.

[0104] The aggregated data is rolled up and aggregated to obtain the advertising revenue of the algorithm service and the algorithm service call count, and the EIPC value of the algorithm service is calculated through the formula. As shown in Figure 5aEIPC values of 30 ad algorithm services are shown as an example.

[0105] Step 202, according to the index value of each algorithm service, the normal distribution value of each algorithm service is calculated.

[0106] As shown in Figure 5b EIPC values of 30 ad algorithm services according to Figure 5a , the calculated normal distribution value.

[0107] Step 203, according to the normal distribution value of each algorithm service, the frequency distribution histogram and the normal distribution curve are drawn.

[0108] As shown in Figure 5c the frequency distribution histogram and the normal distribution curve drawn according to the normal distribution value of 30 ad algorithm services Figure 5b are shown in the schematic diagram. Specifically, the frequency distribution histogram and the normal distribution curve are drawn, including: first grouping the original data, calculating the frequency and the normal distribution value of each group, and then drawing the column chart and the line chart. The specific steps are as follows: (1) group coordinate calculation: the EIPC average value of 30 services is 38.389, the standard deviation is 89.972, the number of groups of 30 services is 30, the upper limit of group coordinate = average value + standard deviation*0.05, the calculated value is 42.88, the lower limit of group coordinate = average value- standard deviation*0.05, the calculated value is 33.89, then the group distance = (group coordinate upper limit-group coordinate lower limit) / (group number-1), the calculated value is 0.31, the group coordinate of each service = group coordinate lower limit+ group distance, gradually increasing, to get the horizontal coordinate; (2) frequency calculation: the EIPC value is in the nth group coordinate-EIPC value is in the n-1 group coordinate, where n is the current group number. (3) graph drawing: select double coordinate axis, the horizontal coordinate in the figure is the group coordinate, the left vertical coordinate is the frequency, and the right vertical coordinate is the normal curve value, wherein the column chart corresponds to the left vertical coordinate, that is, the frequency of each service; the value of the normal distribution curve corresponds to the right vertical coordinate, that is, the normal distribution value of each service.

[0109] Step 204, according to the frequency distribution histogram and the normal distribution curve, the algorithm service to be improved in the plurality of algorithm services is determined.

[0110] As an example, the services whose column chart height in the frequency distribution histogram is outside the normal distribution curve are determined as the algorithm services to be improved in the plurality of algorithm services. As shown in Figure 5c , the services outside the curve are n services that need to be further analyzed, such as the first 3 and the last 6 algorithm services from left to right are the services that need to be analyzed and paid attention to.

[0111] According to the statistical principle, the algorithm effect of multiple algorithm services should conform to the normal distribution. 30 algorithm services are evaluated, and the services outside the normal distribution curve need to be analyzed in depth. Further analysis of the EIPC low: it is found that the service call volume is extremely high, but 75% of the recall data of the call are empty, and the recall is less, the exposure and advertising income are less, and the algorithm effect is poor and needs to be improved. Further analysis of the service with high EIPC: it is found that the business characteristics (red packet pop-up window) lead to good effect, and the product iteration increases the traffic of the service and improves the income.

[0112] The following is an embodiment of the device of the application, which can be used to execute the method embodiment of the application.

[0113] Figure 6 is a block diagram of an algorithm service evaluation device according to an exemplary embodiment. The algorithm service evaluation device can be a server or part of a server, or a terminal or part of a terminal. The algorithm service evaluation device can be realized by software, hardware or a combination of both to become part or all of an electronic device. As shown in Figure 6 The algorithm service evaluation device includes:

[0114] The first acquisition module 601 is configured to acquire an index value of each algorithm service in a plurality of algorithm services, wherein the index value is determined based on a value of a preset index and a number of algorithm service calls; and the plurality of algorithm services correspond to the same preset index.

[0115] The calculation module 602 is configured to calculate a normal distribution value of each algorithm service according to the index value of each algorithm service.

[0116] The determination module 603 is configured to determine an algorithm service to be improved in the plurality of algorithm services according to the normal distribution value of each algorithm service.

[0117] In an embodiment, the first acquisition module 601 is configured to calculate the index value of the algorithm service according to the following formula: (value of the preset index / number of algorithm service calls)*1000.

[0118] The determination module is configured to:

[0119] Draw a frequency distribution histogram and a normal distribution curve according to the normal distribution value of each algorithm service.

[0120] Determine the algorithm service to be improved in the plurality of algorithm services according to the frequency distribution histogram and the normal distribution curve.

[0121] In an embodiment, the device further includes:

[0122] The second acquisition module is configured to acquire a strategy corresponding to the algorithm service according to a preset algorithm service and strategy mapping relationship.

[0123] a third obtaining module, configured to obtain a sum of preset index values of the strategy corresponding to the algorithm service as a preset index value of the algorithm service;

[0124] a fourth obtaining module, configured to obtain a sum of calling quantities of the strategy corresponding to the algorithm service as the number of times of calling the algorithm service.

[0125] Figure 7 is a block diagram of an algorithm service evaluation device 70 according to an exemplary embodiment. The device for generating a short link address can be a server or a part of a server, or a terminal or a part of a terminal. The algorithm service evaluation device includes:

[0126] a processor 7001;

[0127] a memory 7002 configured to store instructions executable by the processor 1001;

[0128] The processor 7001 is configured to:

[0129] obtain an index value of each algorithm service in a plurality of algorithm services, wherein the index value is determined based on a preset index value, a number of times of calling the algorithm service; and wherein the plurality of algorithm services correspond to the same preset index;

[0130] According to the index value of each algorithm service, calculate a normal distribution value of each algorithm service;

[0131] According to the normal distribution value of each algorithm service, determine an algorithm service to be improved in the plurality of algorithm services.

[0132] Figure 8 is a block diagram of an algorithm service evaluation device 800 according to an exemplary embodiment. The device can be a computer, a server, etc.

[0133] The device can include one or more of the following components: a processing component 802, a memory component 804, a power supply component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0134] The processing component 802 generally controls the overall operation of the device 800 such as the operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 can include one or more processors 820 to execute instructions to complete all or part of steps of the methods described above. In addition, the processing component 802 can include one or more modules to facilitate interaction between the processing component 802 and other components. For example, the processing component 802 can include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0135] The memory 804 is configured to store various types of data to support the operation of the device 800. Examples of such data include instructions for any application or method operating on the device 800, contact data, phonebook data, messages, pictures, videos, and so on. The memory 804 can be implemented by any type of volatile or nonvolatile storage devices or a combination thereof such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read only memory (PROM), read only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0136] The power supply component 806 supplies electrical power for the various components of the device 800. The power supply component 806 can include a power supply management system, one or more power supplies, and other components associated with generating, managing and distributing electrical power for the device 800.

[0137] The multimedia component 808 includes a screen providing an output interface between the device 800 and a user. In some embodiments, the screen can include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, swiping and gestures on the touch panel. The touch sensors can not only sense a boundary of a touch or swiping action, but also detect duration and pressure associated with the touch or swiping action. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. The front and / or rear camera can receive external multimedia data when the device 800 is in an operation mode, such as a shooting mode or a video mode. Each of the front and rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0138] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC) that is configured to receive an external audio signal when the device 800 is in an operation mode, such as a calling mode, a recording mode and a voice recognition mode. The received audio signal can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0139] The I / O interface 812 provides an interface between the processing component 802 and peripheral interface modules, which can be a keyboard, a click wheel, buttons, and so on. The buttons can include, but are not limited to, a home button, a volume button, a start button and a lock button.

[0140] The sensor component 814 includes one or more sensors to provide status assessments for various aspects of the device 800. For example, the sensor component 814 can detect an open / closed status of the device 800, relative positioning of components, such as a display and keypad of the device 800, a change in position of the device 800 or a component of the device 800, presence or absence of user contact with the device 800, orientation or acceleration / deceleration of the device 800, and temperature changes of the device 800. The sensor component 814 can include proximity sensor(s) configured to detect presence of nearby objects without any physical contact. The sensor component 814 can also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor component 514 can also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0141] The communication component 816 is configured to facilitate wired or wireless communication between the device 800 and another device. The device 800 can access a wireless network based on a communication standard, such as a Bluetooth® speakerphone private network, WIFI, 2G, 3G, 4G, or 5G, or a combination thereof. In an example embodiment, the communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In an example embodiment, the communication component 816 also includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) techniques, infrared data association (IrDA) techniques, ultra-wideband (UWB) techniques, Bluetooth (BT) techniques, and other techniques.

[0142] In an example embodiment, the device 800 can be implemented using one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic elements to perform the above-described methods.

[0143] In an example embodiment, a non-transitory computer-readable storage medium, such as the memory 804 including instructions, is also provided, which can be executed by the processor 820 of the device 800 to complete the above-described methods. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disc, and an optical data storage device, etc.

[0144] A non-transitory computer readable storage medium, when instructions in the storage medium are executed by a processor of an apparatus 800, enables the apparatus 800 to perform the above-mentioned algorithm service evaluation method, the method comprising:

[0145] obtaining an index value of each algorithm service in a plurality of algorithm services, wherein the index value is determined based on a value of a preset index and a number of algorithm service calls; wherein the plurality of algorithm services correspond to the same preset index;

[0146] calculating a normal distribution value of each algorithm service according to the index value of each algorithm service;

[0147] determining an algorithm service to be improved in the plurality of algorithm services according to the normal distribution value of each algorithm service.

[0148] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the application being indicated by the following claims.

[0149] It should be understood that the application is not limited to the precise construction that has been described above and shown in the accompanying drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the application. The scope of the application is limited only by the appended claims.

Claims

1. An algorithmic service evaluation method, characterized by, The method comprises: obtaining an index value of each algorithm service in a plurality of algorithm services; wherein the index value is determined based on a value of a preset index and a number of algorithm service calls, and the plurality of algorithm services correspond to the same preset index; calculating a normal distribution value of each algorithm service according to the index value of each algorithm service; determining an algorithm service to be improved in the plurality of algorithm services according to the normal distribution value of each algorithm service; wherein the obtaining of the index value of each algorithm service in the plurality of algorithm services comprises: calculating the index value of the algorithm service according to the following formula: (value of preset index / number of algorithm service calls)*1000; wherein the method further comprises: obtaining a strategy corresponding to the algorithm service according to a preset algorithm service and strategy mapping relationship; obtaining a sum of values of preset indexes of the strategy corresponding to the algorithm service as the value of the preset index of the algorithm service; obtaining a sum of call quantities of the strategy corresponding to the algorithm service as the number of algorithm service calls.

2. The method of claim 1, wherein, The determining of the algorithm service to be improved in the plurality of algorithm services according to the normal distribution value of each algorithm service comprises: drawing a frequency distribution histogram and a normal distribution curve according to the normal distribution value of each algorithm service; determining the algorithm service to be improved in the plurality of algorithm services according to the frequency distribution histogram and the normal distribution curve.

3. The method of claim 1, wherein, The method further comprises: storing the index value of the algorithm service; calculating the index value of the algorithm service again after the algorithm service is updated, determining the effect of the algorithm service according to the index values before and after the update of the algorithm service, and / or calculating the index value of the algorithm service after a preset time length, determining the effect of the algorithm service according to the two index values; when the effect of the algorithm service is lower than expected, sending a monitoring notification.

4. An algorithm service evaluation apparatus characterized by comprising: The method comprises: a first obtaining module configured to obtain an index value of each algorithm service in a plurality of algorithm services; wherein the index value is determined based on a value of a preset index and a number of algorithm service calls, and the plurality of algorithm services correspond to the same preset index; a calculating module configured to calculate a normal distribution value of each algorithm service according to the index value of each algorithm service; a determining module configured to determine an algorithm service to be improved in the plurality of algorithm services according to the normal distribution value of each algorithm service; wherein the first obtaining module is configured to calculate the index value of the algorithm service according to the following formula: (value of preset index / number of algorithm service calls)*1000; wherein the device further comprises: a second obtaining module configured to obtain a strategy corresponding to the algorithm service according to a preset algorithm service and strategy mapping relationship; a third obtaining module configured to obtain a sum of values of preset indexes of the strategy corresponding to the algorithm service as the value of the preset index of the algorithm service; a fourth obtaining module configured to obtain a sum of call quantities of the strategy corresponding to the algorithm service as the number of algorithm service calls.

5. The apparatus of claim 4, wherein, The determining module is configured to: draw a frequency distribution histogram and a normal distribution curve according to the normal distribution value of each algorithm service; determine the algorithm service to be improved in the plurality of algorithm services according to the frequency distribution histogram and the normal distribution curve.

6. An algorithm service evaluation apparatus characterized by comprising: The device comprises: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to: Obtaining an index value of each algorithm service in a plurality of algorithm services, wherein the index value is determined based on a value of a preset index and a number of algorithm service calls; wherein the plurality of algorithm services correspond to the same preset index; According to the index value of each algorithm service, calculating a normal distribution value of each algorithm service; According to the normal distribution value of each algorithm service, determining an algorithm service to be improved in the plurality of algorithm services; Further comprising: According to a preset algorithm service and strategy mapping relationship, obtaining a strategy corresponding to the algorithm service; Obtaining a sum of values of the preset index of the strategy corresponding to the algorithm service as the value of the preset index of the algorithm service; Obtaining a sum of call quantities of the strategy corresponding to the algorithm service as the number of algorithm service calls; Wherein, the obtaining of the index value of each algorithm service in the plurality of algorithm services comprises: According to the following formula to calculate the index value of the algorithm service: (The value of the preset index / the number of algorithm service calls)*1000.

7. A computer readable storage medium having stored thereon computer instructions, wherein, The instructions are executed by the processor to implement the steps of the method of any one of claims 1-3.

Images