A model online method and related apparatus

Abstract

Embodiments of the present application disclose a model online method and related device. A to-be-uploaded model, test data and a plurality of offline inference results are obtained, a target system is simulated through an isolation system, online inference is performed according to the test data and the to-be-uploaded model in the isolation system, and a plurality of online inference results are obtained, so that online inference in the target system is simulated while the operation of the target system is not affected, and the safety of the target system is ensured. Since the data and the model used for offline inference and online inference are the same, if the consistency rate of the plurality of offline inference results and the plurality of online inference results meets a preset online condition, it is indicated that the effect of the to-be-uploaded model in the environment of the isolation system and the effect of the to-be-uploaded model in the environment of the system where offline inference is performed are within an allowable error range, and the environment of the target system is verified, so that after the to-be-uploaded model is uploaded to the target system, the effects of the to-be-uploaded model in offline inference and online inference are ensured to be consistent.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a model deployment method and related apparatus. Background Technology

[0002] Machine learning models and similar models are functions that can achieve specific functionalities (mappings) through model training. For example, given a set of data and an algorithm, the model uses the algorithm to learn from the data set through model training, obtaining a mapping from input data to output data, thereby completing the model training.

[0003] After a model has been trained, it needs to be deployed online. Deploying a model online means deploying the model files to the production environment, so that the production environment can provide the services corresponding to the model.

[0004] In related technologies, the model to be launched is usually tested manually. The model is only launched after the test is passed. The model launch not only requires a lot of manpower, but also the problem of manual operation can affect the effect of the model launch. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides a model deployment method and related apparatus, which solves the problem that model deployment not only requires a large amount of manpower, but also suffers from problems caused by manual operation, affecting the effectiveness of model deployment.

[0006] The embodiments of this application disclose the following technical solutions:

[0007] On the one hand, embodiments of this application provide a model deployment method, the method comprising:

[0008] Acquire the model to be deployed, test data, and multiple offline inference results. The test data is the data used by the model to be deployed during offline inference, and the offline inference results are the test results generated by the test data through offline inference of the model to be deployed.

[0009] In the isolation system, online inference is performed based on the test data and the model to be launched, resulting in multiple online inference results. The environment of the isolation system is consistent with the environment of the target system, and the target system is the system where the model to be launched is to be launched.

[0010] Determine the consistency rate between the multiple offline inference results and the multiple online inference results;

[0011] If the consistency rate meets the preset launch conditions, the model to be launched is launched on the target system.

[0012] On the other hand, embodiments of this application provide a model deployment device, the device including an acquisition unit, an inference unit, a determination unit, and a deployment unit;

[0013] The acquisition unit is used to acquire the model to be launched, test data, and multiple offline inference results. The test data is the data used by the model to be launched during the offline inference process, and the offline inference results are the test results generated by the test data through offline inference of the model to be launched.

[0014] The inference unit is used to perform online inference in the isolation system based on the test data and the model to be launched, and obtain multiple online inference results. The environment of the isolation system is consistent with the environment of the target system, and the target system is the system where the model to be launched is to be launched.

[0015] The determining unit is used to determine the consistency rate between the plurality of offline inference results and the plurality of online inference results;

[0016] The online unit is used to launch the model to be launched on the target system if the consistency rate meets the preset online conditions.

[0017] On the other hand, embodiments of this application provide a computer device, the computer device including a processor and a memory:

[0018] The memory is used to store computer programs and to transfer the computer programs to the processor;

[0019] The processor is configured to execute the methods described above according to instructions in the computer program.

[0020] On the other hand, embodiments of this application provide a computer-readable storage medium for storing a computer program for performing the methods described above.

[0021] On the other hand, embodiments of this application provide a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods described above.

[0022] As can be seen from the above technical solution, the process involves acquiring the model to be deployed, test data, and multiple offline inference results. The test data is the data used by the model during offline inference, and the offline inference results are the test results generated by using the test data through offline inference with the model to be deployed. In other words, it involves acquiring the same data and model as the offline inference process. The target system is the system where the model to be deployed will be deployed. The environment of the isolated system is consistent with the environment of the target system. This allows for simulation of the target system through the isolated system, enabling verification of the model's effectiveness after deployment. Specifically, in the isolated system, online inference is performed based on the test data and the model to be deployed, yielding multiple online inference results. This simulates online inference in the target system without affecting its operation, ensuring the target system's security. Since the data and model used for offline and online inference are identical, if the consistency rate between the multiple offline and online inference results meets the preset deployment conditions, it indicates that the model's performance in the isolated system environment is within the acceptable error range compared to its performance in the offline inference system environment. The target system environment passes verification, ensuring that the model's performance is consistent between offline and online inference after deployment on the target system. Therefore, by automatically acquiring the data and models used for offline inference, building an isolated system to execute online inference, and deploying the model to be deployed on the target system after the consistency rate of multiple offline inference results and multiple online inference results meets the preset deployment conditions, the automatic deployment of the model is achieved. This not only reduces the large amount of manpower required for model deployment, but also reduces the number of problems caused by manual operation and improves the effectiveness of model deployment. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This application embodiment provides an illustration of an application scenario for the model deployment method.

[0025] Figure 2 A flowchart illustrating the model deployment method provided in this application embodiment;

[0026] Figure 3 A schematic diagram of a test component required for a periodic task provided in an embodiment of this application;

[0027] Figure 4 A timing diagram of a model deployment method provided in an embodiment of this application;

[0028] Figure 5 A schematic diagram illustrating data preparation as provided in an embodiment of this application;

[0029] Figure 6 A schematic diagram of a model sandbox test provided in an embodiment of this application;

[0030] Figure 7 A schematic diagram of a sandbox test result provided in an embodiment of this application;

[0031] Figure 8 A schematic diagram of model evaluation provided for an embodiment of this application;

[0032] Figure 9 A schematic diagram illustrating a model deployment as provided in an embodiment of this application;

[0033] Figure 10 A schematic diagram illustrating a model deployment as provided in an embodiment of this application;

[0034] Figure 11 A schematic diagram of the structure of an image style conversion device provided in an embodiment of this application;

[0035] Figure 12 This is a schematic diagram of the server structure provided in an embodiment of this application;

[0036] Figure 13 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation

[0037] The embodiments of this application will now be described with reference to the accompanying drawings.

[0038] In related technologies, manually deploying models requires significant manpower and can lead to issues that affect the effectiveness of deployment. This application provides a model deployment method and related apparatus that automatically acquires the data and model used for offline inference, builds an isolated system to execute online inference, and deploys the model to be deployed on the target system after the consistency rate between multiple offline inference results and multiple online inference results meets preset deployment conditions. This achieves automatic model deployment, reducing the manpower required for deployment, minimizing the number of problems caused by manual operations, and improving the effectiveness of model deployment.

[0039] The model deployment method provided in this application can be applied to testing equipment with model deployment capabilities, such as terminal devices and servers. Specifically, terminal devices can be mobile phones, tablets, laptops, desktop computers, smart voice interaction devices, smart home appliances, etc., but are not limited to these. Servers can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing cloud computing services. Terminal devices and servers can be directly or indirectly connected via wired or wireless communication, and this application does not impose any restrictions on this connection.

[0040] To facilitate understanding of the model deployment method provided in this application embodiment, the following example uses a server as the execution subject of the model deployment method to illustrate its application scenarios.

[0041] See Figure 1 This figure is a schematic diagram illustrating an application scenario of the model deployment method provided in an embodiment of this application. For example... Figure 1 As shown, this application scenario includes server 110 and server 120, and server 110 and server 120 can communicate with each other via a network.

[0042] Server 110 includes an offline inference system and a training system. The training system produces a trained model, i.e., the model ready for deployment. This model is then deployed in the offline inference system, where offline inference is performed using test data to obtain multiple offline inference results. Server 110 sends the model ready for deployment, the test data, and the multiple offline inference results to server 120.

[0043] Server 120 includes an isolation system whose environment is identical to that of the target system. The target system is the system where the model to be deployed is about to go live. The isolation system simulates the target system to verify the effectiveness of the model after its deployment. Specifically, within the isolation system, online inference is performed based on test data and the model to be deployed, yielding multiple online inference results. This allows for the simulation of online inference within the target system without affecting its operation, ensuring the target system's security.

[0044] Since the offline and online inference use the same data and models, if the consistency rate of multiple offline inference results and multiple online inference results meets the preset deployment conditions, it indicates that the performance of the model to be deployed in the isolated system environment is within the allowable error range compared to its performance in the offline inference system environment, and the target system environment has passed verification. After the model to be deployed is deployed to the target system, server 120 can guarantee that the performance of the model to be deployed is consistent in both offline and online inference.

[0045] Therefore, by automatically acquiring the data and models used for offline inference, building an isolated system to execute online inference, and deploying the model to be deployed on the target system after the consistency rate of multiple offline inference results and multiple online inference results meets the preset deployment conditions, the automatic deployment of the model is achieved. This not only reduces the large amount of manpower required for model deployment, but also reduces the number of problems caused by manual operation and improves the effectiveness of model deployment.

[0046] The model deployment method provided in this application embodiment can be executed by a server. However, in other embodiments of this application, the terminal device may also have similar functions to the server to execute the model deployment method provided in this application embodiment, or the terminal device and the server may jointly execute the model deployment method provided in this application embodiment. This embodiment does not limit this.

[0047] The following describes in detail the model deployment method provided in this application through method embodiments.

[0048] See Figure 2 This figure is a flowchart illustrating the model deployment method provided in an embodiment of this application. For ease of description, the following embodiments will still use a server as the execution subject of this model deployment method. Figure 2 As shown, the method for deploying this model includes the following steps:

[0049] S201: Obtain the model to be deployed, test data, and multiple offline inference results.

[0050] The model is obtained through training, and its effectiveness is verified through offline inference. This process is repeated until the model achieves satisfactory results before it is deployed to the target system. Because the online environment (the environment of the target system) differs from the offline environment (the environment in which offline inference occurs), even if the model performs well offline, errors and unavailability may occur after deployment, impacting service quality.

[0051] Based on this, the embodiments of this application build an isolation system capable of simulating the target system, and verify the effect of the model in the isolation system after ensuring that only the environment is the only variable.

[0052] It's important to note that offline inference refers to the process of loading a model into an offline environment and then enabling the model's inference capabilities. In an offline environment, the process involves inputting input data into the model and using its inference capabilities to obtain the output results. Specifically, in an offline environment, based on test data and the model to be deployed, multiple offline inference results are obtained through offline inference.

[0053] Here, "model to be deployed" refers to the model of the target system that will soon be deployed. "Target system" refers to the system for which the model to be deployed will be deployed. "Test data" refers to the data used by the model to be deployed during offline inference. "Offline inference results" refers to the test results generated by performing offline inference on the test data using the model to be deployed.

[0054] S202: In the isolated system, online inference is performed based on test data and the model to be deployed, resulting in multiple online inference results.

[0055] An isolation system is a virtual system program that allows models to be validated before deployment within the isolation system, and any changes made during operation can be subsequently deleted. The isolation system creates a sandbox-like, isolated operating environment where programs running within it do not have a permanent impact on the hard drive.

[0056] This application does not impose specific limitations on the isolation system, and those skilled in the art can configure it according to actual needs. For example, the isolation system can be a system built through a sandbox environment, or it can be built using containers or other methods.

[0057] An isolation system is built to mirror the target system's environment, ensuring consistency in operating system, hardware, and configuration. This isolation system simulates the target system to verify the effectiveness of the model after deployment. Specifically, within the isolation system, online inference is performed based on test data and the model, generating multiple inference results. This simulates online inference within the target system without impacting its operation, thus preventing the discovery of errors and unavailability issues online and ensuring service quality.

[0058] Online inference refers to the ability to perform inference functions on a model once it is loaded into an online environment. In an online environment, the process of inputting input data into the model and using its inference capabilities to obtain output results is the online inference process. Specifically, in an online environment, multiple online inference results are obtained based on test data and the model to be deployed.

[0059] S203: Determine the consistency rate between multiple offline inference results and multiple online inference results.

[0060] The consistency rate refers to the ratio of the number of identical results among multiple offline inference results and multiple online inference results to the total number of results. For example, if there are 100 offline inference results and 100 online inference results, and 90 of them are identical, then the consistency rate is 90 / 100*100% = 90%.

[0061] The test data used in the online inference process is the same as that used in the offline inference process. The model to be deployed in the online inference process is the same as that used in the offline inference process. The online inference process is carried out in the isolated system environment, and the offline inference process is carried out in the offline environment.

[0062] Therefore, the difference between online and offline inference processes lies in the different environments in which the model to be deployed is verified. Thus, the consistency rate between multiple offline and online inference results can indicate the difference between the model to be deployed in the offline environment and the isolated system environment. Based on the consistency rate, the model to be deployed can be adjusted in the isolated environment so that the performance of the model to be deployed in the isolated system environment is as close as possible to its performance in the offline environment.

[0063] S204: If the consistency rate meets the preset online conditions, the model to be launched will be launched on the target system.

[0064] If the consistency rate meets the preset online conditions, it means that the difference between the performance of the model to be launched in the isolated system environment and its performance in the offline environment is within the allowable range, and the target system environment passes the verification. Thus, after the model to be launched is launched in the target system, it can be guaranteed that the performance of the model to be launched is consistent in offline inference and online inference.

[0065] Therefore, during the process of validating the model to be deployed in the isolated system, the same test data and model to be deployed are used as in offline inference, and multiple online inference results are obtained through online inference. In other words, the model and data are both quantitative, and only the environment is a variable. The consistency rate can characterize the difference in the performance of the same model in different environments, thus realizing the validation in the online environment.

[0066] The embodiments of this application do not specifically limit the size of the consistency rate, for example, it is 90%, and those skilled in the art can set it according to actual needs.

[0067] As can be seen from the above technical solution, the process involves acquiring the model to be deployed, test data, and multiple offline inference results. The test data is the data used by the model during offline inference, and the offline inference results are the test results generated by using the test data through offline inference with the model to be deployed. In other words, it involves acquiring the same data and model as the offline inference process. The target system is the system where the model to be deployed will be deployed. The environment of the isolated system is consistent with the environment of the target system. This allows for simulation of the target system through the isolated system, enabling verification of the model's effectiveness after deployment. Specifically, in the isolated system, online inference is performed based on the test data and the model to be deployed, yielding multiple online inference results. This simulates online inference in the target system without affecting its operation, ensuring the target system's security. Since the data and model used for offline and online inference are identical, if the consistency rate between the multiple offline and online inference results meets the preset deployment conditions, it indicates that the model's performance in the isolated system environment is within the acceptable error range compared to its performance in the offline inference system environment. The target system environment passes verification, ensuring that the model's performance is consistent between offline and online inference after deployment on the target system.

[0068] Therefore, by automatically acquiring the data and models used for offline inference, building an isolated system to execute online inference, and deploying the model to be deployed on the target system after the consistency rate of multiple offline inference results and multiple online inference results meets the preset deployment conditions, the automatic deployment of the model is achieved. This not only reduces the large amount of manpower required for model deployment, but also reduces the number of problems caused by manual operation and improves the effectiveness of model deployment.

[0069] As one possible implementation, in order to improve the inference accuracy of the model to be deployed, this application also provides a no-load test method to verify the model, as detailed in A1-A3:

[0070] A1: Obtain real-time data.

[0071] Real-time data includes data that comes from the target system in real time. For example, if the target system is a payment system, then real-time data can include transaction data from the payment system and user information from both parties to the transaction.

[0072] It should be noted that, compared to real-time data, the test data mentioned above includes historical data from the target system. For example, test data may include historical transaction data from the payment system, user information from both parties to the transaction, etc.

[0073] It is understood that in the specific embodiments of this application, data such as user information are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0074] A2: Based on real-time data and the model to be launched, perform online inference to obtain multiple real-time inference results.

[0075] Once the model to be deployed is launched on the target system, the target system will have inference capabilities derived from the model, enabling online inference. Therefore, online inference can be performed based on real-time data and the model to be deployed, yielding multiple real-time inference results.

[0076] As one possible implementation, to avoid differences in feature engineering logic between online and offline environments, feature extraction rules from the offline system can be obtained. Real-time data can then be extracted according to these rules, and the resulting feature data can be input into the model to be deployed online for online inference, yielding multiple real-time inference results. The feature extraction rules specify the types of features to be extracted from the test data, such as extracting age, gender, and height features from user data.

[0077] Therefore, although the test data and the real-time data are different, they use the same feature extraction rules. As a result, the input features of the model to be launched in the offline environment are the same as those of the model to be launched in the online environment. This avoids the problem of differences in feature engineering logic between the online and offline environments and ensures that the performance of the model to be launched in the offline environment is not much different from that in the online environment.

[0078] A3: If the distribution of multiple real-time inference results and the distribution of multiple offline inference results meet the preset activation conditions, then the model to be launched will be enabled in the target system.

[0079] Since test data and real-time data differ, the consistency rate cannot be used to verify the model's performance after deployment. Therefore, verification can be performed by comparing the distribution of multiple real-time inference results with the distribution of multiple offline inference results to see if they meet preset activation conditions. This application does not specifically limit the preset activation conditions; for example, the difference between the distribution of multiple real-time inference results and the distribution of multiple offline inference results is less than 2%. Those skilled in the art can set these conditions according to actual needs.

[0080] Continuing with the example of a payment system as the target system, the model to be launched is capable of inferring whether loan users will have the risk of defaulting on payments. Although the expected repayment risk varies for each loan user, based on the law of large numbers, the proportion of people defaulting should be similar. Therefore, the proportion of defaulters to total loan users can be determined based on multiple offline inference results, and the proportion can be determined based on multiple online inference results. If the distributions of the two proportions meet the preset activation condition, i.e., the difference between the two proportions is within the allowable error range, then the model to be launched in the target system.

[0081] It should be noted that in this embodiment, although the model to be deployed is already online in the target system, it is not connected to the decision engine. The model will only connect to the decision engine after the target system activates the model. The decision engine is used to execute corresponding strategies based on the real-time inference results of the model to be deployed. Continuing with the example of a payment system, if the model to be deployed determines that a loan user has a high risk of defaulting, and the model is connected to the decision engine, the loan request can be rejected. If the model to be deployed is not connected to the decision engine, the target system can only determine that a loan user has a high risk of defaulting through the model, but cannot reject the loan request through the decision engine.

[0082] As described above, after the model to be deployed to the target system, online inference is first performed based on real-time data and the model. The effectiveness of the model after deployment to the target system is verified by comparing the distribution of multiple real-time inference results with the distribution of multiple offline inference results to see if they meet preset activation conditions. If the preset activation conditions are met, it indicates that the effectiveness of the model after deployment is within the allowable error range, and therefore the model can be launched on the target system. Thus, verifying the effectiveness of the model through real-time data without making a decision on the corresponding real-time inference structure—that is, running real-time data without input—can increase the amount of input data for the model, thereby improving the inference accuracy of the model.

[0083] As one possible implementation, to avoid discrepancies in feature engineering processing logic that could lead to a failure to meet the preset online deployment conditions, this application provides a specific implementation of S202, namely, a specific implementation of performing online inference based on test data and the model to be deployed, to obtain multiple online inference results, as detailed in B1-B2:

[0084] B1: Extract test data according to feature extraction rules to obtain multiple feature data.

[0085] The feature extraction rules specify the feature categories to be extracted from the test data. These categories can include age, gender, or height. The rules also specify which features to extract based on user information. Furthermore, these rules are identical to those used in offline inference; that is, regardless of whether it's offline or online inference, the test data is extracted using these rules to obtain the feature data.

[0086] B2: Perform online inference based on multiple feature data and the model to be launched, and obtain multiple online inference results.

[0087] For example, multiple feature data can be input into the model to be deployed online, and multiple online inference results can be obtained through online inference.

[0088] Therefore, not only are the test data used for online inference the same as those used for offline inference, but the feature data used for online inference is also the same as those used for offline inference through the same feature extraction rules. This avoids differences in feature engineering processing logic between online and offline inference, and ensures that the only reason for the difference between offline and online inference results is the environment. This allows for targeted adjustments to ensure that the performance of the model after deployment is consistent with its performance in the offline environment.

[0089] As one possible implementation, when the consistency rate fails to meet the preset upper limit condition, it may be because the feature data extracted according to the feature extraction rules includes abnormal features, i.e., features that affect the online inference results. This application provides a method for adjusting feature extraction rules, as detailed in C1-C4:

[0090] C1: Determine the first feature data and the second feature data from multiple feature data.

[0091] In this context, the first feature data and the second feature data are different. For example, multiple feature data include feature data A, feature data B, feature data C, and feature data D. The first feature data includes feature data A, feature data B, and feature data C, and the second feature data includes feature data A, feature data B, feature data C, and feature data D.

[0092] C2: Perform offline inference based on the first feature data and the model to be deployed to obtain the first verification result.

[0093] The first feature data is input into the model to be launched, and the first verification result is obtained through offline inference.

[0094] C3: In the isolated system, online inference is performed based on the second feature data and the model to be deployed to obtain the second verification result.

[0095] In the isolation system, the second feature data is input into the model to be deployed online, and the second verification result is obtained through online inference.

[0096] C4: Adjust the feature extraction rules based on the first and second verification results.

[0097] The consistency rate did not meet the preset online conditions, which may be due to differences in the environment or due to a certain feature data. Therefore, by using the first and second verification results, we can obtain the impact of the first and second feature data on the online environment, and thus adjust the feature extraction rules to ensure that the only reason for the difference between the offline inference results and the online inference results is the environment, and to ensure that the effect of the model to be launched after going online is consistent with the effect in the offline environment.

[0098] It should be noted that if it is determined that the first feature data and the second feature data did not cause the consistency rate to fail to meet the preset upper limit condition, the first feature data and the second feature data can be re-determined from multiple feature data, and the feature data that caused the consistency rate to fail to meet the preset upper limit condition can be continued to be searched, and then the feature extraction rules can be adjusted.

[0099] Therefore, if the consistency rate fails to meet the preset launch conditions, the inference can be re-performed by adjusting the data input to the model to be launched, identifying the feature data that caused the consistency rate to fail to meet the preset launch conditions, and then adjusting the feature extraction rules. This aims to ensure that the only reason for the difference between offline and online inference results is the environment, and to guarantee that the performance of the model after launch is consistent with its performance in the offline environment.

[0100] This application embodiment not only determines that the model to be launched cannot be launched based on the fact that the consistency rate does not meet the preset launch conditions, but can also determine it in the following three ways, which will be explained below.

[0101] Method 1: Based on whether the model to be launched can be loaded successfully.

[0102] As mentioned above, the model to be deployed is placed in the environment of the isolated system so that the isolated system can realize the online inference function of the model. However, the deployment of the model may fail, that is, the model may fail to load. Therefore, the deployment result of the model to be deployed in the isolated system can be obtained. If the deployment result indicates that the model to be deployed in the isolated system has failed, the isolated system cannot realize the online inference function of the model to be deployed, and it is impossible to determine whether the model to be deployed can be launched through the consistency rate. That is, the aforementioned steps S202 and subsequent steps will not be executed.

[0103] Method 2: Based on whether the model to be launched can successfully infer.

[0104] Although the model to be deployed can be successfully deployed in the isolation system, it may fail to perform its own functions, that is, the model to be deployed cannot perform inference. Therefore, if multiple online inference results cannot be obtained, that is, after the test data is input into the model to be deployed, no online inference results can be obtained, it means that the model to be deployed has failed inference. Subsequently, it is impossible to determine whether the model to be deployed can be deployed through the consistency rate, that is, the aforementioned steps S202 and thereafter will no longer be executed.

[0105] Method 3: Based on the stress test results of the model to be launched.

[0106] Stress testing refers to running a deployed model under long-term or full-load conditions. Specifically, during online inference based on test data and the model, it involves acquiring metrics affecting the online inference speed, such as inference time and system resource usage. Inference time refers to the time taken for each piece of test data to generate an online inference result from the model. In computer science, system resources refer to any entity or virtual component in a computer system that limits its computing power, such as the central processing unit (CPU) and memory usage.

[0107] If the metric value exceeds the threshold, it indicates that even if the model to be deployed is launched on the target system, performance issues will lead to insufficient machine resources on the target system, thus affecting service efficiency. Therefore, if the metric value exceeds the threshold, it no longer indicates the step of deploying the model to be deployed on the target system.

[0108] As one possible approach, the model to be deployed to the target system can only be deployed after the consistency rate meets the preset deployment conditions (i.e., verifying data consistency), the deployment result indicates that the model to be deployed to the system has been successfully deployed in the isolated system (i.e., verifying that the model can be loaded), multiple online inference results can be obtained (i.e., verifying that the model can be inferred), and the indicator values ​​affecting the online inference speed do not exceed the indicator thresholds (i.e., verifying that the operating environment can support it). In other words, the model to be deployed to the target system will be deployed only after all four tests are passed. This is to ensure that the effect of the model to be deployed to the target system is consistent with the effect in the offline environment, and that the online inference function can be implemented with good service efficiency.

[0109] As one possible implementation, the testing of the model to be launched can be automatically performed at regular intervals through periodic tasks created for the isolation system. That is, the embodiment of this application provides a specific implementation of S202, which is a specific implementation of online inference based on test data and the model to be launched in the isolation system to obtain multiple online inference results. See D3-D6 for details.

[0110] Before executing D3-D6, D1 and D2 can also be executed, as follows:

[0111] D1: Retrieve the periodic tasks created for the isolated system.

[0112] A periodic task includes a periodic creation time and a model identifier used to indicate the model to be deployed. The periodic creation time identifies when the periodic task creates a test instance; the test instance is an instance generated when the periodic task is tested. For example, if the periodic task is to test the model to be deployed every day at 10:00 AM, then a test instance for deploying the model to be deployed will be created at 10:00 AM on January 15th, and another test instance for deploying the model to be deployed will be created at 10:00 AM on January 16th.

[0113] Therefore, test instances can be created at fixed intervals through periodic tasks, and these test instances are for the test models that are about to be launched.

[0114] D2: If the current time matches the cycle creation time, then create a test instance for the model to be launched based on the model identifier.

[0115] The current time can be obtained from the system time and is used to identify the current system time. If the current time matches the cycle creation time, it means that a test instance for the model to be launched should be created according to the cycle task. That is, test operations should be performed on the cycle task to verify whether the scheduling and operation of the current cycle task in the online environment meets expectations.

[0116] D3: Create a new container based on the test instance to create an isolated system through the container.

[0117] Test instances allow you to create new containers to form isolated systems. A container is created for each test instance corresponding to a model ready for deployment. A container is a form of operating system virtualization. A single container can run everything from small microservices or software processes to large applications. A container contains all the necessary executables, binaries, libraries, and configuration files. However, unlike server or computer virtualization methods, containers do not contain an operating system image. Therefore, containers are more lightweight and portable, with minimal overhead.

[0118] By building an isolation system using containers, the environment of the isolation system is made consistent with the online environment. This allows the online environment to be simulated through the isolation system, and then the online inference implemented in the isolation system can be simulated to achieve testing of the model before it is deployed online.

[0119] D4: Deploy the model deployment files of the model to be launched in the isolated system.

[0120] Based on the model identifier of the model to be deployed in the periodic task, obtain the model deployment file of the model to be deployed. The model deployment file may include binary files, feature files, configuration files, etc. Among them, the binary file is the model file that can be used in the online environment, the feature file is used to describe the feature extraction rules, and the configuration file is used to describe the version and other information of the model to be deployed.

[0121] Deploy the model deployment files in the isolation system, enabling the isolation system to have the online inference capabilities provided by the model to be launched.

[0122] D5: Generate simulated traffic requests based on test data.

[0123] Since online inference can be considered a service in an isolated system, namely a service that provides functionality for the model to be deployed, and the service needs to be invoked through corresponding requests, the data required for simulating traffic can be obtained from the test data. Then, a simulated traffic request can be generated so that the online inference service can be invoked based on the simulated traffic request. In this way, online inference can be performed in the isolated system based on the test data and the model to be deployed, resulting in multiple online inference results.

[0124] D6: Perform online inference based on simulated traffic requests to obtain multiple online inference results.

[0125] As a possible implementation, a simulation test report can also be generated based on multiple online inference results. The report may include the following five aspects: (1) data identifiers for each test data; (2) offline inference results and scores for offline inference; (3) online inference results and scores for online inference; (4) feature extraction rules; and (5) index values ​​obtained from stress testing.

[0126] Adding data identifiers to daily test data can prevent the inability to distinguish test data. For example, if user A transfers money to user B twice, the two parties in the two transactions are the same, and the corresponding two test data cannot be distinguished. Therefore, data identifiers can be added to distinguish multiple transactions with the same parties, so as to determine the consistency rate of multiple online inference results and multiple offline inference results based on the data identifiers.

[0127] Therefore, by continuously creating test instances based on periodic tasks, the model to be deployed is automatically tested at fixed intervals, resulting in multiple online inference results. This standardizes and automates the testing of models before deployment, saving manpower and reducing the number of errors caused by manual operation.

[0128] See Figure 3 This figure is a schematic diagram of a test component required for a periodic task according to an embodiment of this application. The test component is a simple encapsulation of data and methods. Figure 3The system includes a model file deployment component 301, a test data generation component 302, a simulated traffic request component 303, an online inference service component 304, and a result verification component 305.

[0129] The model file deployment component 301 is used to deploy the model deployment file of the model to be launched in the isolation system, so that the online inference service component 304 in the isolation system has the inference function of the model to be launched.

[0130] The test data generation component 302 is used to obtain test data used by the model to be deployed online during offline inference from the offline environment. It should be noted that, due to differences between the offline and online environments, test data obtained from the offline environment can be converted into a format usable in the online environment through data format conversion. This test data can be stored in any local database or in remote storage; this application does not impose any specific limitations on this.

[0131] The simulated traffic request component 303 is used to generate simulated traffic requests based on test data; see D5 for details. It should be noted that simulated traffic requests can also include offline inference results, such as the scoring results of offline inference for the model to be deployed. As one possible implementation, this component runs in a container created for each test instance. Another possible implementation is to score each request and generate a report based on the feature data used, storing it in any local or remote storage.

[0132] The online inference service component 304 is used to perform online inference based on simulated traffic requests, obtaining multiple online inference results (see D6 for details). As one possible implementation, this component's code is completely identical to the online environment, and its runtime environment is a newly created base container consistent with the online environment.

[0133] The result verification component 305 is used to determine the consistency rate between multiple offline inference results and multiple online inference results. As one possible implementation, the stress test results can also be calculated based on metrics such as inference time and system resources obtained from the stress test.

[0134] As one possible implementation, the model to be launched can be deployed through routine push tasks, see E1-E3 for details:

[0135] E1: Get the routine push tasks for the model to be launched.

[0136] Routine push tasks include a periodic push time and a model identifier used to identify models to be deployed. The periodic push time identifies when the routine push task creates a model deployment file push instance; this instance is generated during testing of the routine push task. For example, if the routine push task pushes the model deployment file for the model to be deployed every day at 10:00 AM, then a model deployment file push instance for the model to be deployed will be created at 10:00 AM on January 15th, and another instance will be created at 10:00 AM on January 16th.

[0137] Therefore, a deployment file push instance can be created at fixed intervals through routine push tasks, and this deployment file push instance is for the test model to be launched.

[0138] E2: If the current time matches the periodic push time, then create a model deployment file push instance for the model to be launched based on the model identifier.

[0139] The current time can be obtained from the system time and is used to identify the current system time. If the current time matches the periodic push time, it means that a model deployment file push instance for the model to be launched should be created according to the routine push task. That is, a test operation is performed on the routine push task to verify whether the scheduling and operation of the current routine push task in the online environment meets expectations.

[0140] As one possible implementation, the periodic push time can be later than the periodic creation time. For example, 10:00 AM can be used as the periodic creation time, and 11:00 AM as the periodic push time. The period between 10:00 AM and 11:00 AM can be used as the testing phase for the model to be launched by running empty traffic. At this time, not only does the consistency rate need to meet the preset launch conditions, but the distribution of multiple real-time inference results and the distribution of multiple offline inference results also need to meet the preset activation conditions before the model to be launched is launched on the target system according to the deployment file.

[0141] E3: Push the model deployment file to the target system based on the model deployment file, so that after the consistency rate meets the preset online conditions, the model to be launched is launched on the target system according to the model deployment file.

[0142] Therefore, by continuously creating model deployment file push instances based on routine push tasks for models awaiting deployment, the model deployment files of the models awaiting deployment are pushed to the target system at fixed intervals. Once the consistency rate meets the preset deployment conditions, the models awaiting deployment are deployed to the target system according to the model deployment files. This standardizes and automates the deployment of models awaiting deployment, saving manpower and reducing the number of errors caused by manual operation.

[0143] To facilitate a further understanding of the technical solutions provided in the embodiments of this application, the following uses the server as the execution subject of the model deployment method provided in the embodiments of this application as an example to provide an overall exemplary description of the model deployment method.

[0144] See Figure 4 This figure is a timing diagram of a model deployment method provided in an embodiment of this application. Figure 4 The model deployment system comprises a modeling platform, a unified scheduling system, a payment testing container management system, a distributed storage system, a data push system, and an online service system. These will be described in detail below.

[0145] S1: Users create recurring tasks through the modeling platform.

[0146] Among them, "users" refers to those who use the modeling platform, such as algorithm engineers. "Periodic tasks" are for models awaiting deployment in the isolated system, and include the periodic creation time and a model identifier used to indicate the model to be deployed.

[0147] See Figure 5 This figure is a schematic diagram of data preparation provided in an embodiment of this application. Figure 5 In this process, users need to use the modeling platform to determine the model to be launched, the users, the approving users, the model parameter configuration, feature checks, and other related information.

[0148] S2: The modeling platform sends periodic tasks to the unified scheduling system.

[0149] S3: If the current time matches the periodic creation time, the unified scheduling system will create a test instance for the model to be launched based on the model identifier.

[0150] S4: The unified scheduling system retrieves model deployment files and test data from the storage system based on the model identifier.

[0151] The storage system can be a distributed storage system, in which data is distributed and stored on multiple independent devices.

[0152] S5: The unified scheduling system sends test containers to the modeling platform.

[0153] The test container is created by the unified scheduling system based on the test instance.

[0154] S6: In response to the user's confirmed action, the modeling platform sends a test container creation request to the payment test container management system.

[0155] The modeling platform presents test containers to users so they can adjust and define them.

[0156] S7: Sandbox test of the unified scheduling system execution model.

[0157] See Figure 6 The figure is a schematic diagram of a model sandbox test provided in an embodiment of this application.

[0158] S601: Obtain the model to be launched.

[0159] The model to be deployed can be a pre-trained model or an untrained model, which needs to be trained by modeling and verifying simultaneously. This application does not make any specific restrictions on this.

[0160] S602: Offline inference of the model.

[0161] Offline inference was performed based on test data and the model to be launched, resulting in multiple offline inference results.

[0162] S603: Model deployment file generation.

[0163] This allows for the installation of model deployment files within a sandbox environment (an isolated system environment) to enable online model inference.

[0164] S604: Model Sandbox Test Data Conversion.

[0165] Convert the test data format to obtain test data in a format suitable for the sandbox environment.

[0166] S605: Simulate traffic request.

[0167] Generate simulated traffic requests based on test data in a format suitable for sandbox environments.

[0168] S606: Online Model Inference (Simulated Flow).

[0169] Based on simulated traffic requests, online inference is performed using the model to be deployed, resulting in multiple online inference results.

[0170] S607: Verification of model sandbox test results.

[0171] Determine the consistency rate between multiple online inference results and multiple offline inference results.

[0172] S608: Model deployment configuration generation.

[0173] If the consistency rate meets the preset launch conditions, a model deployment configuration for the model to be launched on the target system is generated so that the model to be launched on the target system can be launched.

[0174] S609: Feature output is online.

[0175] The feature extraction rules are sent to the target system so that the target system can extract feature data from the real-time data according to the feature extraction rules.

[0176] S610: Online model inference (idle flow).

[0177] Online inference is performed based on real-time data, feature extraction rules, and the model to be deployed, resulting in multiple real-time inference results.

[0178] S611: Online model inference (live network traffic).

[0179] If the distribution of multiple real-time inference results and the distribution of multiple offline inference results meet the preset activation conditions, then the model to be launched will be enabled in the target system.

[0180] "Idle traffic" refers to online inference based solely on real-time inference results and the model to be deployed, without connecting to the decision engine. "Live traffic" refers to online inference based on real-time inference results and the model to be deployed, and connected to the decision engine.

[0181] S8: The unified scheduling system sends the sandbox test results to the modeling platform.

[0182] Sandbox test results can include not only the consistency rate, but also the scores for each test data point obtained through offline inference, scores obtained through online inference, feature extraction rules, and metric values ​​obtained from stress testing.

[0183] See Figure 7 The figure is a schematic diagram of a sandbox test result provided in an embodiment of this application. Figure 7 The document shows the model test configuration and the sandbox test results obtained from the model test, such as the test that failed on November 27, 2022.

[0184] Among them, S3-S8 are periodic tasks used to test models to be deployed. Once the model passes the test, the configuration for automated deployment can be automatically generated, such as automatically creating routine push tasks and synchronizing them with the online service system, allowing it to access both idle and production traffic.

[0185] S9: Use the user to determine routine push tasks through the modeling platform.

[0186] This routine push task is for models awaiting deployment. It can be automatically created after testing and sent to users for display, or it can be created by users through the modeling platform. This routine push task includes the periodic push time and a model identifier used to identify models awaiting deployment.

[0187] S10: The modeling platform generates a model service loading configuration file.

[0188] The model service loading configuration file includes configuration information for deploying the model to be launched to the online service system (target system).

[0189] S11: The modeling platform sends the model service loading configuration file to the approving user.

[0190] See Figure 8 This figure is a schematic diagram of a model evaluation provided in an embodiment of this application. Figure 8 In the system, after a user approves the model, the approval process can be displayed to the user, such as Li Si approving at level one and Wang Wu approving at level two, and both approvals being passed. The model's performance can also be displayed on the approval form to demonstrate its capabilities.

[0191] S12: The modeling platform sends the model service loading configuration file to the data push system.

[0192] S13: Modeling platform generates model deployment files and pushes configuration.

[0193] The model deployment file push configuration includes the relevant configuration information for pushing the model to be launched to the online service system.

[0194] See Figure 9 This figure is a schematic diagram of a model deployment provided in an embodiment of this application. Figure 9 The document displays the feature extraction rules. For example, based on the test data and the feature extraction rules, features a through e can be extracted. Furthermore, it displays the feature name, feature type, and feature description for each feature.

[0195] S14: The modeling platform sends the model deployment file push configuration to the data push system.

[0196] See Figure 10 This figure is a schematic diagram of a model deployment provided in an embodiment of this application. Figure 10 The document shows the configuration for pushing model deployment files to models that are about to be deployed, such as the configuration for model deployment file distribution tasks, the configuration for model deployment file distribution tasks, and the configuration for loading live models.

[0197] S15: The modeling platform sends routine push tasks to the unified scheduling system.

[0198] S16: If the current time matches the periodic push time, the unified scheduling system will create a model deployment file push instance.

[0199] S17: The unified scheduling system sends the model deployment file to the data push system.

[0200] S18: The data push system retrieves the model deployment files from the storage system.

[0201] Verify whether the model deployment files have been tampered with by verifying signatures and other methods to ensure the security of the subline service system.

[0202] S19: The data push system pushes the model deployment file to the online service system.

[0203] S16-S19 are examples of pushing model deployment files.

[0204] In addition, interactive tools are provided to troubleshoot problems when the model testing fails. The automated troubleshooting process is as follows:

[0205] a. Compare the model versions of the models to be launched used for online inference and offline inference.

[0206] b. Filter out the different test data used for online reasoning and offline reasoning.

[0207] c. Read the feature data for online inference.

[0208] d. Read the feature data for offline inference.

[0209] e. Compare the feature data of online inference and offline inference.

[0210] f. Read the first feature data used for offline inference.

[0211] g. Read the second feature data used for online inference.

[0212] h. Compare the first feature data and the second feature data.

[0213] i. Adjust the first feature data and the second feature data, and re-perform online and offline inference to see if a certain feature is causing the inconsistency.

[0214] For relevant details, please refer to C1-C4 above, which will not be repeated here. Therefore, through the above automated troubleshooting, differences between online and offline environments can be identified. For example, different environments can lead to different data accuracy. Furthermore, differences in feature engineering processing logic can cause discrepancies between online and offline inference results.

[0215] Once the model tests pass, the model push configuration and service configuration can be automatically generated, and the service configuration will be merged into the code request. After the code merging request passes, the deployment will be pushed to the idle traffic service. Once the idle traffic run is complete, it will be transferred to the production service's live network traffic service. At this point, the model to be deployed on the target system is ready for deployment.

[0216] As can be seen from the above scheme, by introducing model verification (verifying that the model to be launched can be loaded and reasoned), data verification (verifying the consistency between online and offline reasoning of the model to be launched), and runtime environment verification (verifying that system resources such as CPU and memory will not be overloaded), the model to be launched will be pushed online only when all these verifications are passed. Thus, (1) it can avoid the discovery of errors and unavailability issues in the online environment, which would affect the quality of service. (2) it can verify the upper limit of the consistency between online and offline reasoning of the model to be launched, so that the model effect verified offline can be guaranteed in the live network environment. (3) it can effectively measure the reasoning performance of the model to be launched, avoiding the insufficiency of existing machine resources due to performance issues, which would affect service efficiency. (4) it can standardize the model launch, ultimately achieving automation, saving manpower for model launch, and avoiding errors caused by fluctuations in manual operation.

[0217] In response to the model deployment method described above, this application also provides a corresponding model deployment device to enable the above model deployment method to be applied and implemented in practice.

[0218] See Figure 11 This figure is a schematic diagram of the structure of an image style conversion device provided in an embodiment of this application. Figure 11 As shown, the model online device 1100 includes: the device includes an acquisition unit 1101, a reasoning unit 1102, a determination unit 1103 and an online unit 1104;

[0219] The acquisition unit 1101 is used to acquire the model to be launched, test data and multiple offline inference results. The test data is the data used by the model to be launched during the offline inference process, and the offline inference results are the test results generated by the test data through offline inference of the model to be launched.

[0220] The inference unit 1102 is used to perform online inference in the isolation system based on the test data and the model to be launched, and obtain multiple online inference results. The environment of the isolation system is consistent with the environment of the target system, and the target system is the system where the model to be launched is to be launched.

[0221] The determining unit 1103 is used to determine the consistency rate between the plurality of offline inference results and the plurality of online inference results;

[0222] The online unit 1104 is used to launch the model to be launched on the target system if the consistency rate meets the preset online conditions.

[0223] As one possible implementation, the acquisition unit 1101 is further configured to:

[0224] After the model to be launched is put into the target system, real-time data is acquired.

[0225] The reasoning unit 1102 is also used for:

[0226] Based on the real-time data and the model to be deployed, online inference is performed to obtain multiple real-time inference results;

[0227] The device further includes a starting unit for:

[0228] If the distribution of the multiple real-time inference results and the distribution of the multiple offline inference results meet the preset activation conditions, then the model to be launched is activated in the target system.

[0229] As one possible implementation, the acquisition unit 1101 is further configured to:

[0230] Obtain the feature extraction rules used in the offline inference process of the test data, wherein the feature extraction rules are used to indicate the feature categories of the test data to be extracted;

[0231] The reasoning unit 1102 is specifically used for:

[0232] The test data is extracted according to the feature extraction rules to obtain multiple feature data.

[0233] Online inference is performed based on multiple feature data and the model to be deployed, resulting in multiple online inference results.

[0234] As one possible implementation, the device further includes an adjustment unit for:

[0235] If the consistency rate does not meet the preset upper limit condition, a first feature data and a second feature data are determined from the multiple feature data, wherein the first feature data and the second feature data are different;

[0236] Based on the first feature data and the model to be deployed, offline inference is performed to obtain the first verification result;

[0237] In the isolation system, online inference is performed based on the second feature data and the model to be deployed to obtain a second verification result;

[0238] The feature extraction rules are adjusted based on the first verification result and the second verification result.

[0239] As can be seen from the above technical solution, the process involves acquiring the model to be deployed, test data, and multiple offline inference results. The test data is the data used by the model during offline inference, and the offline inference results are the test results generated by using the test data through offline inference with the model to be deployed. In other words, it involves acquiring the same data and model as the offline inference process. The target system is the system where the model to be deployed will be deployed. The environment of the isolated system is consistent with the environment of the target system. This allows for simulation of the target system through the isolated system, enabling verification of the model's effectiveness after deployment. Specifically, in the isolated system, online inference is performed based on the test data and the model to be deployed, yielding multiple online inference results. This simulates online inference in the target system without affecting its operation, ensuring the target system's security. Since the data and model used for offline and online inference are identical, if the consistency rate between the multiple offline and online inference results meets the preset deployment conditions, it indicates that the model's performance in the isolated system environment is within the acceptable error range compared to its performance in the offline inference system environment. The target system environment passes verification, ensuring that the model's performance is consistent between offline and online inference after deployment on the target system. Therefore, by automatically acquiring the data and models used for offline inference, building an isolated system to execute online inference, and deploying the model to be deployed on the target system after the consistency rate of multiple offline inference results and multiple online inference results meets the preset deployment conditions, the automatic deployment of the model is achieved. This not only reduces the large amount of manpower required for model deployment, but also reduces the number of problems caused by manual operation and improves the effectiveness of model deployment.

[0240] As one possible implementation, the acquisition unit 1101 is further configured to:

[0241] Obtain the deployment results of the model to be deployed in the isolation system;

[0242] If the deployment result indicates that the model to be deployed has failed in the isolation system, then the steps of performing online inference based on the test data and the model to be deployed in the isolation system to obtain multiple online inference results and subsequent steps will no longer be executed.

[0243] As one possible implementation, the acquisition unit 1101 is further configured to:

[0244] If the multiple online inference results cannot be obtained, the step of determining the consistency rate between the multiple offline inference results and the multiple online inference results, as well as subsequent steps, will no longer be executed.

[0245] As one possible implementation, the acquisition unit 1101 is further configured to:

[0246] During the online inference process based on the test data and the model to be deployed, the index values ​​that affect the online inference speed are obtained;

[0247] If the indicator value exceeds the indicator threshold, the step of launching the model to be launched on the target system will no longer be executed.

[0248] As one possible implementation, the acquisition unit 1101 is further configured to:

[0249] Obtain the periodic task created for the isolation system, the periodic task including the periodic creation time and the model identifier used to indicate the model to be deployed;

[0250] The apparatus further includes a creation unit for:

[0251] If the current time matches the cycle creation time, a test instance for the model to be launched is created based on the model identifier.

[0252] The reasoning unit 1102 is specifically used for:

[0253] Create a new container based on the test instance, so as to create the isolation system through the container;

[0254] Obtain the model deployment file of the model to be deployed based on the model identifier, and deploy the model deployment file in the isolation system;

[0255] Generate simulated traffic requests based on the test data;

[0256] Online inference is performed based on the simulated traffic requests to obtain multiple online inference results.

[0257] As one possible implementation, the acquisition unit 1101 is further configured to:

[0258] Obtain the routine push task for the model to be launched, the routine push task including the periodic push time and the model identifier;

[0259] The device further includes a creation unit for:

[0260] If the current time matches the periodic push time, then create a model deployment file push instance for the model to be launched based on the model identifier;

[0261] The model deployment file is pushed to the target system according to the model deployment file push instance, so that after the consistency rate meets the preset online conditions, the model to be launched is launched on the target system according to the model deployment file.

[0262] This application also provides a computer device, which is the aforementioned computer device. This computer device can be a server or a terminal device, and the aforementioned model deployment device can be built into the server or terminal device. The computer device provided in this application will be described below from a hardware implementation perspective. Figure 12 The diagram shown is a schematic of the server's structure. Figure 13 The diagram shown is a structural schematic of the terminal device.

[0263] See Figure 12 This figure is a schematic diagram of a server structure provided in an embodiment of this application. The server 1400 can vary considerably due to different configurations or performance. It may include one or more processors 1422, such as a central processing unit (CPU), memory 1432, and one or more application programs 1442 or data storage media 1430 (e.g., one or more mass storage devices). The memory 1432 and storage media 1430 can be temporary or persistent storage. The program stored in the storage media 1430 may include one or more modules (not shown in the figure), each module may include a series of instruction operations on the server. Furthermore, the processor 1422 may be configured to communicate with the storage media 1430 and execute the series of instruction operations in the storage media 1430 on the server 1400.

[0264] Server 1400 may also include one or more power supplies 1426, one or more wired or wireless network interfaces 1450, one or more input / output interfaces 1458, and / or one or more operating systems 1441, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM etc.

[0265] The steps performed by the server in the above embodiments can be based on this Figure 12 The server structure shown.

[0266] CPU 1422 is used to perform the following steps:

[0267] Acquire the model to be deployed, test data, and multiple offline inference results. The test data is the data used by the model to be deployed during offline inference, and the offline inference results are the test results generated by the test data through offline inference of the model to be deployed.

[0268] In the isolation system, online inference is performed based on the test data and the model to be launched, resulting in multiple online inference results. The environment of the isolation system is consistent with the environment of the target system, and the target system is the system where the model to be launched is to be launched.

[0269] Determine the consistency rate between the multiple offline inference results and the multiple online inference results;

[0270] If the consistency rate meets the preset launch conditions, the model to be launched is launched on the target system.

[0271] Optionally, CPU 1422 may also execute method steps of any specific implementation of the model deployment method in the embodiments of this application.

[0272] See Figure 13 The figure is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Figure 13 This diagram illustrates a partial structure of a smartphone related to the terminal device provided in this embodiment. The smartphone includes components such as a radio frequency (RF) circuit 1510, a memory 1520, an input unit 1530, a display unit 1540, a sensor 1550, an audio circuit 1560, a Wi-Fi module 1570, a processor 1580, and a power supply 1590. Those skilled in the art will understand that... Figure 13 The smartphone structure shown does not constitute a limitation on smartphones and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0273] The following is combined with Figure 13 A detailed introduction to the various components of a smartphone:

[0274] The RF circuit 1510 can be used to receive and transmit signals during information transmission or calls. In particular, it receives downlink information from the base station and processes it with the processor 1580; in addition, it transmits uplink data to the base station.

[0275] The memory 1520 can be used to store software programs and modules, and the processor 1580 runs the software programs and modules stored in the memory 1520 to realize various functions and data processing of the smartphone.

[0276] Input unit 1530 can be used to receive input numeric or character information and generate key signal inputs related to user settings and function control of the smartphone. Specifically, input unit 1530 may include touch panel 1531 and other input devices 1532. Touch panel 1531, also known as a touch screen, can collect touch operations on or near the user and drive corresponding connected devices according to a pre-set program. In addition to touch panel 1531, input unit 1530 may also include other input devices 1532. Specifically, other input devices 1532 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0277] The display unit 1540 can be used to display information input by the user or information provided to the user, as well as various menus of the smartphone. The display unit 1540 may include a display panel 1541, which may optionally be configured as a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0278] Smartphones may also include at least one sensor 1550, such as a light sensor, a motion sensor, and other sensors. Other sensors that smartphones may also be equipped with, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be detailed here.

[0279] Audio circuit 1560, speaker 1561, and microphone 1562 provide an audio interface between the user and the smartphone. Audio circuit 1560 converts received audio data into electrical signals and transmits them to speaker 1561, where speaker 1561 converts them into sound signals for output. On the other hand, microphone 1562 converts collected sound signals into electrical signals, which are received by audio circuit 1560, converted into audio data, and then processed by processor 1580 before being transmitted via RF circuit 1510 to, for example, another smartphone, or the audio data can be output to memory 1520 for further processing.

[0280] The processor 1580 is the control center of the smartphone, connecting various parts of the smartphone through various interfaces and lines. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 1520, and by calling data stored in the memory 1520. Optionally, the processor 1580 may include one or more processing units.

[0281] The smartphone also includes a power supply 1590 (such as a battery) that supplies power to various components. Preferably, the power supply can be logically connected to the processor 1580 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0282] Although not shown, smartphones may also include a camera, Bluetooth module, etc., which will not be described in detail here.

[0283] In this embodiment of the application, the memory 1520 included in the smartphone can store program code and transmit the program code to the processor.

[0284] The processor 1580 included in the smartphone can execute the model deployment method provided in the above embodiments according to the instructions in the program code.

[0285] This application also provides a computer-readable storage medium for storing a computer program for executing the model deployment method provided in the above embodiments.

[0286] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the model uploading method provided in the various optional implementations of the above aspects.

[0287] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium can be at least one of the following media: read-only memory (ROM), RAM, magnetic disk, or optical disk, etc., and other media capable of storing program code.

[0288] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The device and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the solution in this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0289] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for deploying a model online, characterized in that, The method includes: Acquire the model to be deployed, test data, and multiple offline inference results. The test data is the data used by the model to be deployed during offline inference, and the offline inference results are the test results generated by the test data through offline inference of the model to be deployed. In the isolation system, online inference is performed based on the test data and the model to be launched, resulting in multiple online inference results. The environment of the isolation system is consistent with the environment of the target system. The isolation system is a virtual system program that simulates the target system. The target system is the system where the model to be launched is to be launched. The online inference is the inference function of the model after loading it in the online environment. Determine the consistency rate between the multiple offline inference results and the multiple online inference results; If the consistency rate meets the preset launch conditions, the model to be launched is launched on the target system.

2. The method according to claim 1, characterized in that, After the target system deploys the model to be deployed, the method further includes: Obtain real-time data; Based on the real-time data and the model to be deployed, online inference is performed to obtain multiple real-time inference results; If the distribution of the multiple real-time inference results and the distribution of the multiple offline inference results meet the preset activation conditions, then the model to be launched is activated in the target system.

3. The method according to claim 1, characterized in that, The method further includes: Obtain the feature extraction rules used in the offline inference process of the test data, and the feature extraction rules are used to indicate the feature categories of the test data to be extracted; The online inference process, based on the test data and the model to be deployed, yields multiple online inference results, including: The test data is extracted according to the feature extraction rules to obtain multiple feature data. Online inference is performed based on multiple feature data and the model to be deployed, resulting in multiple online inference results.

4. The method according to claim 3, characterized in that, If the consistency rate does not meet the preset upper limit condition, the method further includes: A first feature data and a second feature data are determined from a plurality of said feature data, wherein the first feature data is different from the second feature data; Based on the first feature data and the model to be deployed, offline inference is performed to obtain the first verification result; In the isolation system, online inference is performed based on the second feature data and the model to be deployed to obtain a second verification result; The feature extraction rules are adjusted based on the first verification result and the second verification result.

5. The method according to claim 1, characterized in that, The method further includes: Obtain the deployment results of the model to be deployed in the isolation system; If the deployment result indicates that the model to be deployed has failed in the isolation system, then the steps of performing online inference based on the test data and the model to be deployed in the isolation system to obtain multiple online inference results and subsequent steps will no longer be executed.

6. The method according to claim 1, characterized in that, The method further includes: If the multiple online inference results cannot be obtained, the step of determining the consistency rate between the multiple offline inference results and the multiple online inference results, as well as subsequent steps, will no longer be executed.

7. The method according to claim 1, characterized in that, The method further includes: During the online inference process based on the test data and the model to be deployed, the index values ​​that affect the online inference speed are obtained; If the indicator value exceeds the indicator threshold, the step of launching the model to be launched on the target system will no longer be executed.

8. The method according to claim 1, characterized in that, The method further includes: Obtain the periodic task created for the isolation system, the periodic task including the periodic creation time and the model identifier used to indicate the model to be deployed; If the current time matches the cycle creation time, a test instance for the model to be launched is created based on the model identifier. In the isolated system, online inference is performed based on the test data and the model to be deployed, resulting in multiple online inference results, including: Create a new container based on the test instance, so as to create the isolation system through the container; Obtain the model deployment file of the model to be deployed based on the model identifier, and deploy the model deployment file in the isolation system; Generate simulated traffic requests based on the test data; Online inference is performed based on the simulated traffic requests to obtain multiple online inference results.

9. The method according to claim 8, characterized in that, The method further includes: Obtain the routine push task for the model to be launched, the routine push task including the periodic push time and the model identifier; If the current time matches the periodic push time, then create a model deployment file push instance for the model to be launched based on the model identifier; The model deployment file is pushed to the target system according to the model deployment file push instance, so that after the consistency rate meets the preset online conditions, the model to be launched is launched on the target system according to the model deployment file.

10. A model online device, characterized in that, The device includes an acquisition unit, a reasoning unit, a determination unit, and an online unit; The acquisition unit is used to acquire the model to be launched, test data, and multiple offline inference results. The test data is the data used by the model to be launched during the offline inference process, and the offline inference results are the test results generated by the test data through offline inference of the model to be launched. The inference unit is used to perform online inference in the isolation system based on the test data and the model to be launched, and obtain multiple online inference results. The environment of the isolation system is consistent with the environment of the target system. The isolation system is a virtual system program that simulates the target system. The target system is the system where the model to be launched is to be launched. The online inference is to implement the inference function of the model after loading it in the online environment. The determining unit is used to determine the consistency rate between the plurality of offline inference results and the plurality of online inference results; The online unit is used to launch the model to be launched on the target system if the consistency rate meets the preset online conditions.

11. The apparatus according to claim 10, characterized in that, The acquisition unit is further configured to: After the model to be launched is put into the target system, real-time data is acquired. The reasoning unit is also used for: Based on the real-time data and the model to be deployed, online inference is performed to obtain multiple real-time inference results; The device further includes a starting unit for: If the distribution of the multiple real-time inference results and the distribution of the multiple offline inference results meet the preset activation conditions, then the model to be launched is activated in the target system.

12. The apparatus according to claim 10, characterized in that, The acquisition unit is further configured to: Obtain the feature extraction rules used in the offline inference process of the test data, and the feature extraction rules are used to indicate the feature categories of the test data to be extracted; The reasoning unit is used for: The test data is extracted according to the feature extraction rules to obtain multiple feature data. Online inference is performed based on multiple feature data and the model to be deployed, resulting in multiple online inference results.

13. The apparatus according to claim 12, characterized in that, The device further includes an adjustment unit for: If the consistency rate does not meet the preset upper limit condition, a first feature data and a second feature data are determined from the multiple feature data, wherein the first feature data and the second feature data are different; Based on the first feature data and the model to be deployed, offline inference is performed to obtain the first verification result; In the isolation system, online inference is performed based on the second feature data and the model to be deployed to obtain a second verification result; The feature extraction rules are adjusted based on the first verification result and the second verification result.

14. The apparatus according to claim 10, characterized in that, The acquisition unit is further configured to: Obtain the deployment results of the model to be deployed in the isolation system; If the deployment result indicates that the model to be deployed has failed in the isolation system, then the steps of performing online inference based on the test data and the model to be deployed in the isolation system to obtain multiple online inference results and subsequent steps will no longer be executed.

15. A computer device, characterized in that, The computer device includes a processor and memory: The memory is used to store computer programs and to transfer the computer programs to the processor; The processor is configured to execute the method according to any one of claims 1-9 according to instructions in the computer program.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program for performing the method according to any one of claims 1-9.

17. A computer program product comprising a computer program, characterized in that, When it is run on a computer device, it causes the computer device to perform the method described in any one of claims 1-9.

Images