Method and device for detecting a model

Abstract

This application discloses a method and apparatus for detecting models, relating to the fields of intelligent detection and system optimization. The method includes: obtaining online request data; determining a first prediction result for the request data using an old version prediction model; determining a second prediction result for the request data using a new version prediction model; comparing the first prediction result with the second prediction result, and determining a detection result for the new version prediction model based on the comparison result. This method can improve the accuracy of the detection prediction model.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, specifically to the field of intelligent detection and system optimization technology, and especially to a method and device for detecting models. Background Art

[0002] With the rapid development of artificial intelligence technology, machine models are widely used in various online scenarios (such as commercial advertisement click-through rate prediction scenarios, data recognition scenarios, data classification scenarios, etc.). The update and replacement of machine models (i.e., replacing the original online model with a new model) has become an inevitable problem. Before replacing the original online model with a new model, the new model needs to be tested to ensure the working performance of the new model. Existing methods for testing models before a new model is put online include: offline testing of the working indicators of the new model, offline testing of the model content of the new model, and testing the working performance of the new model after running the new model online.

[0003] However, the method of offline detection of the working indicators of the model or offline detection of the model content of the new model has the problem of inaccurate detection; the method of testing the working performance of the new model after it is put online has the risk of affecting the operation of the online system. Summary of the Invention

[0004] The present disclosure provides a method, an apparatus, an electronic device, and a computer-readable storage medium for detecting a model.

[0005] According to a first aspect, a method for detecting a model is provided, the method comprising: obtaining online request data; determining a first prediction result of the request data using an old version prediction model; determining a second prediction result of the request data using a new version prediction model; comparing the first prediction result with the second prediction result, and determining a detection result for the new version prediction model based on the comparison result.

[0006] According to the second aspect, a device for detecting a model is provided, which includes: an acquisition unit, configured to acquire online request data; a first prediction unit, configured to use an old version prediction model to determine a first prediction result of the request data; a second prediction unit, configured to use a new version prediction model to determine a second prediction result of the request data; a detection unit, configured to compare the first prediction result with the second prediction result, and determine a detection result for the new version prediction model based on the comparison result.

[0007] According to a third aspect, an embodiment of the present disclosure provides an electronic device comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors implement the method for detecting a model as provided in the first aspect.

[0008] According to a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method for detecting a model provided in the first aspect is implemented.

[0009] The method and device for detecting a model provided by the present disclosure obtain online request data; use an old version prediction model to determine a first prediction result of the request data; use a new version prediction model to determine a second prediction result of the request data; compare the first prediction result with the second prediction result, and determine a detection result for the new version prediction model based on the comparison result. This can improve the accuracy of detecting the new version prediction model and avoid the need to test the new version prediction model after it is put online, which affects the performance of the online system.

[0010] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present application.

[0012] Figure 1 is an exemplary system architecture diagram to which embodiments of the present application may be applied;

[0013] Figure 2 is a flow chart of an embodiment of a method for detecting a model according to the present application;

[0014] Figure 3 is a flow chart of another embodiment of a method for detecting a model according to the present application;

[0015] Figure 4 is a flow chart of another embodiment of a method for detecting a model according to the present application;

[0016] Figure 5 is a schematic structural diagram of a single tower model in the method for detecting a model according to the present application;

[0017] Figure 6 This is a schematic diagram of the structure of the double tower model in the method for detecting the model according to the present application.

[0018] Figure 7 is a structural schematic diagram of an embodiment of a device for detecting a model according to the present application;

[0019] Figure 8 It is a block diagram of an electronic device used to implement the method for detecting a model according to an embodiment of the present application. DETAILED DESCRIPTION

[0020] The following description of exemplary embodiments of the present application is made in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0021] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the method for detecting a model or the apparatus for detecting a model of the present application can be applied.

[0022] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. Network 104 is a medium for providing communication links between terminal devices 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0023] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various simulation processes or processes for testing the system can be installed on terminal devices 101, 102, and 103. Various client applications can also be installed on terminal devices 101, 102, and 103, such as information input applications, video applications, playback applications, audio applications, search applications, shopping applications, financial applications, etc.

[0024] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting receiving server messages, including but not limited to smart phones, tablet computers, e-book readers, electronic players, laptop computers, desktop computers, and the like.

[0025] Terminal devices 101, 102, and 103 can be hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various hardware modules or electronic devices. When terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software or software modules (for example, multiple software modules for providing distributed services), or they can be implemented as a single software or software module. No specific limitation is made here.

[0026] The server 105 may obtain online request data; determine a first prediction result of the request data using an old version prediction model; determine a second prediction result of the request data using a new version prediction model; compare the first prediction result with the second prediction result, and determine a detection result for the new version prediction model based on the comparison result.

[0027] It should be noted that the method for detecting a model provided in the embodiments of the present disclosure is generally executed by the server 105 , and accordingly, the device for detecting a model is generally provided in the server 105 .

[0028] It should be understood that Figure 1 The number of devices, networks, and servers shown in the figure is merely illustrative. Any number of devices, networks, and servers may be used as needed.

[0029] Continue to refer Figure 2 , shows a process 200 of an embodiment of a method for detecting a model according to the present disclosure. The method for detecting a model comprises the following steps:

[0030] Step 201: Obtain online request data.

[0031] In this embodiment, the execution subject of the method for detecting the model (eg Figure 1 The server shown in the figure obtains online request data via wired or wireless means. The online request data is request data obtained based on the real online environment in which the model is applied, such as data initiated online by advertisers to predict click-through rates and conversion rates of ads, and video classification and image recognition requests initiated online by users.

[0032] Step 202: Determine a first prediction result of the requested data using the old version prediction model.

[0033] In this embodiment, an old version prediction model can be used to determine a first prediction result for the request data, specifically including: inputting the request data into the old version prediction model currently in use by the online system to obtain the first prediction result output by the old version prediction model. The old version prediction model can be various types of deep learning models.

[0034] Step 203: Determine a second prediction result of the requested data using the new version prediction model.

[0035] In this embodiment, a new version prediction model can be used to determine the second prediction result of the request data, specifically including: inputting the request data into the new version prediction model to be launched to obtain the second prediction result output by the new version prediction model. The new version prediction model can be various types of deep learning models.

[0036] The old version prediction model and the new version prediction model are relative. The old version prediction model is the prediction model currently used by the online system, while the new version model is the model that will be launched soon to replace the old version prediction model.

[0037] Step 204 : Compare the first prediction result with the second prediction result, and determine the detection result for the new version prediction model according to the comparison result.

[0038] In this embodiment, the first prediction result output by the old version prediction model may be compared with the second prediction result output by the new version prediction model, and whether the new version prediction model passes the test may be determined based on the comparison result.

[0039] The method for detecting a model provided in this embodiment obtains online request data; uses an old version prediction model to determine a first prediction result of the request data; uses a new version prediction model to determine a second prediction result of the request data; compares the first prediction result with the second prediction result, and determines a detection result for the new version prediction model based on the comparison result. This can improve the accuracy of detecting the new version prediction model and avoid testing the new version prediction model after it is put online, which affects the performance of the online system.

[0040] Further references Figure 3 , which shows a process 300 of another embodiment of a method for detecting a model. The process 300 of the method for detecting a model includes the following steps:

[0041] Step 301: Obtain online request data.

[0042] In step 302 , the current detection environment is an online detection environment, and the model currently running online is an old version prediction model. Request data is used to request the model running online, and a first prediction result determined by the old version prediction model is obtained.

[0043] In this embodiment, the current detection environment is an online detection environment, and the model currently running online is an old version prediction model. Request data can be used to request the model running online. At this time, the requested model is the old version prediction model, and the first prediction result determined by the old version prediction model is obtained.

[0044] Step 303: Add configuration data related to the new version prediction model to the request data, use the updated request data to request the model running online, and obtain a second prediction result determined by the new version prediction model.

[0045] In this embodiment, configuration data related to the new version prediction model is added to the request data, for example, the model identifier of the new version model, the version number of the new version model, the configuration file of the new version model, etc. After the above-mentioned configuration data related to the new version prediction model is added to the request data, the request data will be sent to the new version prediction model, that is, the requested model is the new version prediction model, and the second prediction result determined by the new version prediction model is obtained.

[0046] Step 304 : Compare the first prediction result with the second prediction result, and determine the detection result for the new version prediction model according to the comparison result.

[0047] The description of step 301 and step 304 in this embodiment is consistent with the description of step 201 and step 204, and will not be repeated here.

[0048] The method for detecting the model provided in this embodiment is compared with Figure 2 The method described in the embodiment specifically describes the specific steps of using the old version prediction model to determine the first prediction result of the requested data, and using the new version prediction model to determine the second prediction result of the requested data when the detection environment is an online / online detection environment. This can realize the detection of the online operating environment of the new version prediction model when the new version prediction model is not actually online.

[0049] Further reference is made to Figure 4, which shows a process 400 of another embodiment of a method for detecting a model. The process 400 of the method for detecting a model comprises the following steps:

[0050] Step 401: Obtain online request data.

[0051] Step 402: The current detection environment is an offline detection environment. The online operating environment and the offline operating environment are obtained, the offline operating environment is initialized, and a first prediction result of the requested data is determined based on the initialized offline operating environment.

[0052] In this embodiment, the current detection environment is an offline detection environment. The online operating environment is pulled down locally to obtain an offline operating environment. The offline operating environment is initialized to ensure that the model requested by the initialized running detection environment is an old version prediction model rather than other models. The old version prediction model requested based on the initialized offline operating environment is used to determine the first prediction result of the requested data.

[0053] Initializing the offline running environment includes initializing the timeout parameters: In the offline detection method, the online running environment needs to be pulled down locally to simulate the offline operation of the online environment. However, due to the limited resources of the local server, it is impossible to fully simulate the running environment in which the model runs online. For example, the local server is a server in City A, and the old version of the prediction model is stored in the server in City B. At this time, the server in City A needs to request the server in City B to use the old version of the prediction model to predict the requested data in the local environment. Cross-regional interaction will cause timeout problems. Therefore, it is necessary to increase the timeout time allowed in the local offline running environment to avoid misjudgment of poor model performance due to cross-regional interaction (rather than problems with the model itself).

[0054] Initializing the offline operating environment also includes version initialization: in the offline detection method, the online operating environment will be pulled down to the local computer, but the configuration of the online operating environment may have been modified. If the configuration of the online operating environment has been modified, the requested model will not be the old version prediction model, but the experimental model. Therefore, to ensure that the requested model is the old version prediction model, it is necessary to initialize the version of the offline operating environment obtained by pulling down.

[0055] Step 403: Update the offline operating environment using the parameters of the new version prediction model, and determine a second prediction result of the requested data based on the updated offline operating environment.

[0056] In this embodiment, the parameters of the new version of the prediction model can be used to update the offline operating environment, and the second prediction result of the request data can be determined based on the updated offline operating environment. In the offline detection environment, since the content of the request data cannot be directly modified, the configuration of the operating environment is modified to request the new version of the prediction model.

[0057] Step 404 : Compare the first prediction result with the second prediction result, and determine the detection result for the new version prediction model based on the comparison result.

[0058] The description of step 401 and step 404 in this embodiment is consistent with the description of step 201 and step 204, and will not be repeated here.

[0059] The method for detecting the model provided in this embodiment is compared with Figure 2 The method described in the embodiment specifically describes the specific steps of using the old version prediction model to determine the first prediction result of the requested data when the detection environment is an offline detection environment, and using the new version prediction model to determine the second prediction result of the requested data. It can realize the detection of the new version prediction model in an online detection environment.

[0060] contrast Figure 3 The described embodiments and Figure 4 In the described embodiment, when the detection environment is an online detection environment, the model version in the request is specified as a new version by modifying the content of the request data. When the detection environment is an offline detection environment, the configuration of the running environment is updated to specify the request for a new version of the prediction model.

[0061] In the above combination Figure 2 、 Figure 3 and Figure 4 In some optional implementations of the described embodiments, the first prediction result includes the prediction results of the request data in each dimension, and the second prediction result includes the prediction results of the request data in each dimension; the first prediction result is compared with the second prediction result, and the detection result for the new version of the prediction model is determined based on the comparison result, including: for each dimension, the prediction result of the request data included in the first prediction result in the dimension is compared with the prediction result of the request data included in the second prediction result in the dimension, and the comparison result is obtained; based on the multiple comparison results obtained for multiple dimensions, the detection result for the new version of the prediction model is determined.

[0062] In this embodiment, the first prediction result or the second prediction result includes the prediction results of the requested data in various dimensions. For example, if the requested data is a request to predict the advertising delivery results, the various dimensions may include: click dimension, conversion dimension, etc. The estimated results of the requested data in various dimensions may be the click volume prediction results, conversion rate prediction results, etc. after the advertising is delivered.

[0063] For each of the dimensions, the prediction result of the request data included in the first prediction result under the dimension can be compared with the prediction result of the request data included in the second prediction result under the dimension, and the comparison result for the dimension can be obtained.

[0064] For example, for the ad click dimension, the prediction result of the ad click rate included in the first prediction result is compared with the prediction result of the ad click rate included in the second prediction result to obtain the comparison result of the ad click rate prediction; for the ad conversion dimension, the prediction result of the ad conversion rate included in the first prediction result is compared with the prediction result of the ad conversion rate included in the second prediction result to obtain the comparison result of the ad conversion rate prediction.

[0065] Finally, the test result of the new version prediction model is determined based on the multiple comparison results obtained for the multiple dimensions. For example, if the majority of the multiple comparison results indicate that the similarity between the two prediction results meets the similarity threshold, then the new version model can be determined to have passed the test.

[0066] In the above combination Figure 2 Figure 3 and Figure 4 In some optional implementations of the described embodiments, the first prediction result and the second prediction result are compared, and the detection result for the new version prediction model is determined based on the comparison result, including: in response to determining that the fluctuation gap between the first prediction result and the second prediction result is less than a preset fluctuation threshold, determining that the new version prediction model has passed the detection; and / or, in response to determining that the similarity between the distribution of the first prediction result and the distribution of the second prediction result is greater than a preset similarity threshold, determining that the new version prediction model has passed the detection.

[0067] In this embodiment, if the fluctuation difference between the first prediction result and the second prediction result is determined to be less than a preset fluctuation threshold, the new version prediction model can be determined to have passed the test. The fluctuation difference between the first prediction result and the second prediction result is the difference between the first prediction result and the second prediction result. If the difference between the two meets the preset fluctuation threshold, it means that the prediction result of the new version prediction model is consistent with the prediction result of the old version prediction model. After replacing the old version prediction model with the new version prediction model, there will be no impact on the online system. Therefore, the new version prediction model can be determined to have passed the test.

[0068] If it is determined that the similarity between the distributions of the first prediction result and the second prediction result is greater than a preset similarity threshold, then the new version prediction model can be determined to have passed the test. The first prediction result and the second prediction result can be probability distributions, and the similarity between their distributions is the similarity between the first probability distribution function and the second probability distribution function. If the similarity between the two distribution functions meets the preset similarity threshold, then the performance of the new version prediction model is consistent with that of the old version prediction model. After replacing the old version prediction model with the new version prediction model, there will be no impact on the online system, and the new version prediction model can be determined to have passed the test.

[0069] In the above combination Figure 2 Figure 3 and Figure 4 In the described embodiment, the old version model and the new version model can be a single tower model structure (such as Figure 5 As shown), the online request data is input into the new version model of the single-tower model structure to obtain the output of the new version model, the online request data is input into the old version model of the single-tower model structure to obtain the output of the old version model, and the output of the new version model is compared with the output of the old version model. If the comparison result indicates that the gap between the two is less than the threshold, it is determined that the new version model can be put into operation to replace the old version model.

[0070] In the above combination Figure 2 Figure 3 and Figure 4 In the described embodiment, the old version model and the new version model can be a double tower model structure (such as Figure 6 ), in practical applications, the dual-tower model can be divided into a dual-tower model that jointly trains a common module and a dual-tower model that jointly trains a single module. There are two ways to transmit hidden layer information in the dual-tower model that jointly trains modules: the first way is that module 1 transmits the hidden layer information to the upstream, stores the information in plain text or serialized form in a reserved storage space, and sends it down together when the upstream requests module 2; the second way is that module 1 stores the hidden layer information in a memory space (cache), and through a targeted distribution method, module 2 requests the corresponding memory space to obtain the corresponding hidden layer information.

[0071] In a dual-tower model, if hidden layer information is transmitted using the first method, the tower that first requests the hidden layer information is used to simulate the packaging of online vectors (embeddings, vectors converted from discrete features). The generated embeddings are parsed and processed, and then used as request information for the other tower. The embeddings generated by the other tower are then inner-producted to obtain the inner-product result. The inner-product result obtained from the old version of the dual-tower model is compared with the inner-product result obtained from the new version of the dual-tower model. If the comparison result indicates that the difference between the two is less than a threshold, the new version of the model is determined to be ready for online operation to replace the old version.

[0072] In the dual-tower model, if the hidden layer information is transmitted through the second method, the request data is input into one side tower to obtain the output result of the side tower, the output result is input into the other side tower to obtain the output result of the other side tower, and the output result of the other side tower is used as the final output result of the dual-tower model. The output result obtained by the old version model based on the dual-tower model structure is compared with the output result obtained by the new version model based on the dual-tower model structure. If the comparison result indicates that the gap between the two is less than the threshold, it is determined that the new version model can be put into operation to replace the old version model.

[0073] Further references Figure 7 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a device for detecting a model. Figure 2 、 Figure 3 and Figure 4 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0074] like Figure 7As shown, the device 700 for detecting a model in this embodiment includes: an acquisition unit 701, a first prediction unit 702, a second prediction unit 703, and a detection unit 704. The acquisition unit is configured to acquire online request data; the first prediction unit is configured to use the old version prediction model to determine a first prediction result for the request data; the second prediction unit is configured to use the new version prediction model to determine a second prediction result for the request data; and the detection unit is configured to compare the first prediction result with the second prediction result and determine a detection result for the new version prediction model based on the comparison result.

[0075] In some embodiments, the current detection environment is an online detection environment, and the model currently running online is an old version prediction model. The first prediction unit includes: a first prediction module, configured to use request data to request the model running online, and obtain a first prediction result determined by the old version prediction model; the second prediction unit includes: a second prediction module, configured to add configuration data related to the new version prediction model to the request data, use the updated request data to request the model running online, and obtain a second prediction result determined by the new version prediction model.

[0076] In some embodiments, the current detection environment is an offline detection environment, and the first prediction unit includes: a third prediction module, configured to obtain the online operating environment and obtain the offline operating environment, initialize the offline operating environment, and determine the first prediction result of the requested data based on the initialized offline operating environment; the second prediction unit includes: a fourth prediction module, configured to update the offline operating environment using parameters of the new version of the prediction model, and determine the second prediction result of the requested data based on the updated offline operating environment.

[0077] In some embodiments, the first prediction result includes the prediction results of the request data in each dimension, and the second prediction result includes the prediction results of the request data in each dimension; the detection unit includes: a detection module, which is configured to compare the prediction results of the request data included in the first prediction result in each dimension with the prediction results of the request data included in the second prediction result in each dimension, and obtain a comparison result; a determination module, which is configured to determine the detection result for the new version prediction model based on multiple comparison results obtained for multiple dimensions.

[0078] In some embodiments, the detection unit includes: a first detection module, configured to determine that the new version prediction model passes the detection in response to determining that the fluctuation gap between the first prediction result and the second prediction result is less than a preset fluctuation threshold; and / or, a second detection module, configured to determine that the new version prediction model passes the detection in response to determining that the similarity between the distribution of the first prediction result and the distribution of the second prediction result is greater than a preset similarity threshold.

[0079] Each unit in the above-mentioned device 700 is consistent with the reference Figure 2 、 Figure 3 and Figure 4 Therefore, the operations, features and technical effects achieved by the above-described method for detecting a model are also applicable to the apparatus 500 and the units included therein, and will not be described in detail here.

[0080] According to an embodiment of the present application, the present application also provides an electronic device, a readable storage medium and a computer program product.

[0081] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0082] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 806 into a random access memory (RAM) 803. Various programs and data required for the operation of the device 800 can also be stored in the RAM 803. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0083] Various components in device 800 are connected to I / O interface 805, including: input unit 806, such as a keyboard, mouse, etc.; output unit 807, such as various types of displays, speakers, etc.; storage unit 806, such as a magnetic disk, optical disk, etc.; and communication unit 805, such as a network card, modem, wireless communication transceiver, etc. The communication unit 805 allows device 800 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0084] The computing unit 801 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as the method for detecting the model. For example, in some embodiments, the method for detecting the model can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 806. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 805. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the method for detecting the model described above can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform the method for detecting the model by any other appropriate means (e.g., by means of firmware).

[0085] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0086] The program code for implementing the methods of the present application can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flow charts and / or block diagrams are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0087] In the context of the present application, a machine-readable medium can be a tangible medium that can contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0088] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0089] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0090] Computer systems may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The client and server relationship arises through computer programs running on the respective computers and having a client-server relationship to each other.

[0091] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this application can be performed in parallel, sequentially, or in a different order, as long as the desired data of the technical solution disclosed in this application can be achieved. This is not limited herein.

[0092] The above specific embodiments do not constitute a limitation on the scope of protection of this application. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application shall be included within the scope of protection of this application.

Claims

1. A method for detecting a model, comprising: Get online request data; Determining a first prediction result of the request data using an old version prediction model includes: if the current detection environment is an online detection environment, using the request data to request an online running model to obtain the first prediction result; if the current detection environment is an offline detection environment, obtaining an online running environment and an offline running environment, initializing the offline running environment, and determining the first prediction result of the request data based on the initialized offline running environment; Determining a second prediction result for the request data using the new version prediction model includes: when the current detection environment is an online detection environment, adding configuration data related to the new version prediction model to the request data, and using the updated request data to request the online running model to obtain the second prediction result; when the current detection environment is an offline detection environment, updating the offline running environment using parameters of the new version prediction model, and determining the second prediction result for the request data based on the updated offline running environment; Comparing the first prediction result with the second prediction result, and determining the detection result for the new version prediction model based on the comparison result, including: in response to determining that the fluctuation gap between the first prediction result and the second prediction result is less than a preset fluctuation threshold, and determining that the similarity between the distribution of the first prediction result and the distribution of the second prediction result is greater than a preset similarity threshold, determining that the new version prediction model passes the detection.

2. The method according to claim 1, wherein The first prediction result includes the prediction results of the request data in each dimension, and the second prediction result includes the prediction results of the request data in each dimension; The comparing the first prediction result with the second prediction result, and determining a detection result for the new version prediction model according to the comparison result, includes: For each of the dimensions, comparing the prediction result of the request data included in the first prediction result under the dimension with the prediction result of the request data included in the second prediction result under the dimension, and obtaining a comparison result; According to the multiple comparison results obtained for the multiple dimensions, a detection result for the new version prediction model is determined.

3. A device for detecting a model, comprising: an acquisition unit, configured to acquire online request data; A first prediction unit is configured to use an old version prediction model to determine a first prediction result of the request data, including a first prediction module and a third prediction module; the first prediction module is configured to use the online running model with the request data to obtain the first prediction result when the current detection environment is an online detection environment; the third prediction module is configured to use the offline detection environment to obtain the online running environment and the offline running environment, initialize the offline running environment, and determine the first prediction result of the request data based on the initialized offline running environment; The second prediction unit is configured to use the new version prediction model to determine the second prediction result of the request data, including a second prediction module and a fourth prediction module; the second prediction module is configured to use the current detection environment as an online detection environment, add configuration data related to the new version prediction model to the request data, and use the updated request data to request the online running model to obtain the second prediction result; the fourth prediction module is configured to use the current detection environment as an offline detection environment, use the parameters of the new version prediction model to update the offline running environment, and determine the second prediction result of the request data based on the updated offline running environment. A detection unit is configured to compare the first prediction result with the second prediction result, and determine a detection result for the new version prediction model based on the comparison result, wherein the detection unit is further configured to determine that the new version prediction model passes the detection in response to determining that the fluctuation gap between the first prediction result and the second prediction result is less than a preset fluctuation threshold, and determining that the similarity between the distribution of the first prediction result and the distribution of the second prediction result is greater than a preset similarity threshold.

4. The device according to claim 3, wherein The first prediction result includes the prediction results of the request data in each dimension, and the second prediction result includes the prediction results of the request data in each dimension; The detection unit comprises: a detection module configured to compare, for each of the dimensions, a prediction result of the request data included in the first prediction result under the dimension with a prediction result of the request data included in the second prediction result under the dimension, and obtain a comparison result; The determination module is configured to determine the detection result of the new version prediction model based on multiple comparison results obtained for multiple dimensions.

5. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 2.

6. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the method according to any one of claims 1 to 2.

7. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 2.

Images