Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Model evaluation method and system

A model and technology to be evaluated, applied in the network field, can solve problems such as the inability to evaluate the model in real time, the high complexity of positioning, and the inability to monitor the online effect of the model in time

Active Publication Date: 2018-10-16
BEIJING SANKUAI ONLINE TECH CO LTD
View PDF4 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] However, offline evaluation cannot monitor the online effect of the model in time, and the positioning complexity of online problems is high, the model cannot be evaluated in real time, and the evaluation accuracy is low

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Model evaluation method and system
  • Model evaluation method and system
  • Model evaluation method and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0022] refer to figure 1 , which shows a flow chart of specific steps of a model evaluation method provided in Embodiment 1 of the present invention.

[0023] Step 101, the data distribution node acquires historical access records in real time, and constructs a model evaluation sample set from the historical access records, and the sample set includes at least one sample.

[0024] Among them, the historical access record is the platform access record of the model application to be evaluated, including the access records of the foreground and background servers. For example, for a food delivery platform, when a user orders a food delivery on the platform, the platform can record information such as merchants and dishes that the user has browsed. Of course, the embodiment of the present invention focuses on the information required for calculating the evaluation index value.

[0025] The embodiments of the present invention can obtain access records in real time, thereby ensuri...

Embodiment 2

[0040] refer to figure 2 , which shows a flow chart of specific steps of a model evaluation method provided in Embodiment 2 of the present invention.

[0041] Step 201, the data distribution node acquires historical access records in real time.

[0042] For this step, reference may be made to the detailed description of step 101, which will not be repeated here.

[0043] Step 202: Slide the preset target window to the position where the historical access record is inserted into the new access record according to the preset single sliding length.

[0044] Wherein, if the length of a single swipe is too small, the latest historical access records cannot be obtained easily; if the length of a single swipe is too large, it is easy to cause no new historical access records to be generated during the swipe. The length of a single slide can be represented by an integer number of historical access records. It can be understood that the single sliding length needs to be set accordi...

Embodiment 3

[0118] refer to image 3 , which shows a structural diagram of a model evaluation system provided by Embodiment 3 of the present invention, the details are as follows.

[0119] A data distribution node 301, at least one first-level computing node 302 associated with the data distribution node 301, at least one second-level computing node 303 associated with each first-level computing node 302, and each second-level computing node 303 The central computing node 304 associated with the node 303, the data distribution node 301 is used to obtain historical access records in real time, and construct a model evaluation sample set from the historical access records, the sample set includes at least one sample; each sample is distributed For at least one first-level computing node 302, the first-level computing node 302 is configured to extract metadata required for calculating preset evaluation indicators from the sample, and distribute the metadata of the sample to at least one One...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a model evaluation method and system. The method comprises the following steps that: a data distribution node obtains a historical access record in real time to construct a model evaluation sample set; each sample is distributed to a plurality of first-level calculation nodes; the first-level calculation nodes extract metadata required for calculating a preset evaluation index from samples, and distributes the metadata to a plurality of second-level calculation nodes; the second-level calculation nodes carry out fragmentation clustering according to the metadata, and sends a fragmentation clustering value to a central calculation node; and the central calculation node collects and aggregates the fragmentation clustering value of each second-level calculation node toobtain an evaluation value. By use of the method, the problems in the prior art that offline evaluation can not monitor the online effect of the model can not be monitored in time, online problem positioning complexity is high, the model can not be evaluated in real time and evaluation accuracy is low can be solved, the model can be evaluated according to real-time data, evaluation accuracy is improved, and the positioning complexity of online problems is lowered.

Description

technical field [0001] Embodiments of the present invention relate to the field of network technologies, and in particular, to a model evaluation method and system. Background technique [0002] In the search recommendation system, the sorting model such as machine learning, after online scoring of merchants, products or content and other elements, sorts the merchants, products or content according to the scores, and finally recommends the merchants, products or content according to the ranking results to the user. Thus, the importance of performance evaluation of ranking models has become increasingly prominent. [0003] In the prior art, the main scheme of offline evaluation includes: firstly, the historical usage records of the business are obtained in batches in units of time periods such as days and weeks; then, the offline model is used to predict and score the offline sample set generated according to the historical usage records , and calculate AUC (Area Under the ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F17/30G06Q10/06
CPCG06Q10/06393
Inventor 李悦
Owner BEIJING SANKUAI ONLINE TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com