Model verification method and device, electronic equipment, computer readable medium and product

By determining the data path information, obtaining the data to be verified, generating coded information and verifying the model, the problem of insufficient scenario coverage caused by the verification of a single model training framework is solved, and the scope of application and generalization capabilities of model verification are expanded.

CN119089960APending Publication Date: 2024-12-06BEIJING WODONG TIANJUN INFORMATION TECH CO LTD +1

Patent Information

Application Number
CN202310657998.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-06-05
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

In the prior art, the model is only checked through a single model training framework, resulting in less coverage of verification business scenarios and weaker application scope and generalization capabilities of model verification.

Method used

By determining the data path information according to the model verification method, obtaining the data set to be verified, generating the information group to be verified, determining the model to be verified, and verifying it, expanding the verification business scenario and scope of application.

Benefits of technology

Through this method, the scope of application of model verification is expanded and the generalization ability of the verification model is enhanced, and the problem of insufficient scenario coverage caused by the verification of a single model training framework is solved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119089960A_ABST
    Figure CN119089960A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses a model verification method and device, electronic equipment, a computer readable medium and a product. A specific embodiment of the method comprises the following steps: determining data path information according to a model verification mode; obtaining a to-be-verified data set according to the data path information; generating a to-be-verified coding information group set according to the to-be-verified data set; determining a to-be-verified model according to a verification model corresponding to the to-be-verified data set; and according to the to-be-verified coding information group set, verifying the to-be-verified model to obtain model verification information. The embodiment is related to model verification, and the scene range of the covered verification service is expanded, so that the application range of the model verification is expanded, and the generalization ability of the verification model is enhanced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of big data technology, and in particular to a model verification method, device, electronic device, computer-readable medium, and product. Background Art

[0002] With the widespread application of deep learning, model offline training frameworks and model online reasoning frameworks continue to emerge. At present, when verifying a model, the commonly used method is to verify the model to be verified through a single model training framework (model offline training framework or model online reasoning framework).

[0003] However, the inventors have found that when the above method is used to verify the model, the following technical problems often occur: the model to be verified is verified only through a single model training framework, resulting in fewer scenarios for the verification business covered. As a result, the scope of application of the model verification is small and the generalization ability of the verification model is weak.

[0004] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0005] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0006] Some embodiments of the present disclosure propose a model checking method, an apparatus, an electronic device, a computer-readable medium, and a computer program product to solve one or more of the technical problems mentioned in the above background technology section.

[0007] In a first aspect, some embodiments of the present disclosure provide a model verification method, which includes: determining data path information according to a model verification method; obtaining a data set to be verified according to the above data path information; generating a set of coded information groups to be verified according to the above data set to be verified; determining a model to be verified according to a verification model corresponding to the above data set to be verified; and verifying the above model to be verified according to the above coded information group set to be verified to obtain model verification information.

[0008] Optionally, the determining of data path information according to the model verification method includes: in response to determining that the model verification method is online model verification, determining business path information corresponding to the online model verification as data path information.

[0009] Optionally, the above-mentioned determining the data path information according to the model verification method further includes: in response to determining that the above-mentioned model verification method is offline model verification, determining the storage path information corresponding to the offline model verification as the data path information.

[0010] Optionally, the above-mentioned obtaining the data set to be verified based on the above-mentioned data path information includes: in response to determining that the above-mentioned data path information is business path information, obtaining the data set to be verified from the business data storage server corresponding to the above-mentioned business path information; in response to determining that the above-mentioned data path information is storage path information, obtaining the data set to be verified from the storage disk corresponding to the above-mentioned storage path information.

[0011] Optionally, the above-mentioned method generates a set of coded information groups to be verified based on the above-mentioned data set to be verified, including: performing data format conversion processing on the above-mentioned data set to be verified to obtain a converted data set; performing partition processing on the above-mentioned converted data set to obtain a partitioned data group set; for each partitioned data group in the above-mentioned partitioned data group set, performing encoding processing on each partitioned data included in the above-mentioned partitioned data group to generate a partitioned coded data group; and determining each generated coded data group as the set of coded information groups to be verified.

[0012] Optionally, the above-mentioned model to be verified is verified according to the above-mentioned set of coded information groups to be verified to obtain model verification information, including: inputting the above-mentioned set of coded information groups to be verified into the above-mentioned model to be verified to obtain model verification information.

[0013] Optionally, the method also includes: in response to determining that the above-mentioned model verification method is online model verification, sending the above-mentioned model verification information to the above-mentioned business data storage server; in response to determining that the above-mentioned model verification method is offline model verification, storing the above-mentioned model verification information in the above-mentioned storage disk.

[0014] Optionally, before obtaining the data set to be verified according to the data path information, the method further includes: starting the deployment environment according to the deployment mode.

[0015] Optionally, starting the deployment environment according to the deployment mode includes: starting the local deployment environment in response to the deployment mode being local deployment; starting the default deployment environment in response to the deployment mode being default deployment; starting the distributed deployment environment in response to the deployment mode satisfying the distributed deployment environment starting conditions.

[0016] Optionally, the above-mentioned determination of the model to be verified based on the verification model corresponding to the above-mentioned data set to be verified includes: in response to the deployment mode being distributed deployment and being resident deployment and the above-mentioned model service existing, the above-mentioned verification model is determined as the model to be verified; in response to determining that the local deployment environment is started, the above-mentioned verification model is loaded, and the verification model after the loading process is determined as the model to be verified; in response to determining that the default deployment environment is started, the above-mentioned verification model is loaded, and the verification model after the loading process is determined as the model to be verified; in response to determining that the distributed deployment environment is started, the above-mentioned verification model is loaded, and the verification model after the loading process is determined as the model to be verified.

[0017] Optionally, the method also includes: in response to the deployment mode being a distributed deployment and a resident deployment, and the preset forced deletion configuration information indicating that forced deletion is not enabled, destroying the data service environment corresponding to the above-mentioned data set to be verified; in response to the deployment mode being a non-resident deployment, or the above-mentioned deployment mode being a resident deployment and the above-mentioned preset forced deletion configuration information indicating that forced deletion is enabled, destroying the data service environment corresponding to the above-mentioned data set to be verified and the corresponding model service environment.

[0018] In a second aspect, some embodiments of the present disclosure provide a model verification device, which includes: a first determination unit, configured to determine data path information according to a model verification method; an acquisition unit, configured to acquire a set of data to be verified according to the above data path information; a generation unit, configured to generate a set of coded information groups to be verified according to the above data set to be verified; a second determination unit, configured to determine the model to be verified according to the verification model corresponding to the above data set to be verified; and a verification unit, configured to verify the above model to be verified according to the above coded information group set to be verified, to obtain model verification information.

[0019] Optionally, the first determining unit is further configured to: in response to determining that the above-mentioned model verification mode is online model verification, determine the business path information corresponding to the online model verification as the data path information.

[0020] Optionally, the first determining unit is further configured to: in response to determining that the above-mentioned model verification mode is offline model verification, determine the storage path information corresponding to the offline model verification as the data path information.

[0021] Optionally, the acquisition unit is further configured to: in response to determining that the above-mentioned data path information is business path information, obtain the data set to be verified from the business data storage server corresponding to the above-mentioned business path information; in response to determining that the above-mentioned data path information is storage path information, obtain the data set to be verified from the storage disk corresponding to the above-mentioned storage path information.

[0022] Optionally, the generation unit is further configured to: perform data format conversion processing on the above-mentioned data set to be verified to obtain a converted data set; perform partition processing on the above-mentioned converted data set to obtain a partitioned data group set; for each partitioned data group in the above-mentioned partitioned data group set, perform encoding processing on each partitioned data included in the above-mentioned partitioned data group to generate a partitioned encoded data group; and determine each generated encoded data group as a set of encoded information groups to be verified.

[0023] Optionally, the verification unit is further configured to: input the above-mentioned set of coded information groups to be verified into the above-mentioned model to be verified to obtain model verification information.

[0024] Optionally, the model verification device further includes: a sending unit and a storage unit. The sending unit is configured to send the model verification information to the business data storage server in response to determining that the model verification method is online model verification. The storage unit is configured to store the model verification information in the storage disk in response to determining that the model verification method is offline model verification.

[0025] Optionally, before the acquiring unit, the model checking device further includes: a starting unit configured to start the deployment environment according to the deployment mode.

[0026] Optionally, the startup unit is further configured to: in response to the deployment mode being local deployment, start the local deployment environment; in response to the deployment mode being default deployment, start the default deployment environment; in response to the deployment mode satisfying the distributed deployment environment startup condition, start the distributed deployment environment.

[0027] Optionally, the second determination unit is further configured to: in response to the deployment mode being distributed deployment and being resident deployment and the existence of the above-mentioned model service, determine the above-mentioned verification model as the model to be verified; in response to determining that the local deployment environment is started, load the above-mentioned verification model, and determine the verification model after the loading process as the model to be verified; in response to determining that the default deployment environment is started, load the above-mentioned verification model, and determine the verification model after the loading process as the model to be verified; in response to determining that the distributed deployment environment is started, load the above-mentioned verification model, and determine the verification model after the loading process as the model to be verified.

[0028] Optionally, the model verification device further includes: a data service environment destruction unit and a data service environment and model service destruction unit. Among them, the data service environment destruction unit is configured to, in response to the deployment mode being a distributed deployment and being a resident deployment, and the preset forced deletion configuration information indicating that forced deletion is not enabled, destroy the data service environment corresponding to the above-mentioned data set to be verified. The data service environment and model service destruction unit is configured to, in response to the deployment mode being a non-resident deployment, or the above-mentioned deployment mode being a resident deployment and the above-mentioned preset forced deletion configuration information indicating that forced deletion is enabled, destroy the data service environment corresponding to the above-mentioned data set to be verified and the corresponding model service environment.

[0029] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the above-mentioned first aspect.

[0030] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the above-mentioned first aspect is implemented.

[0031] In a fifth aspect, some embodiments of the present disclosure provide a computer program product, including a computer program, which implements the method described in any implementation manner of the above-mentioned first aspect when executed by a processor.

[0032] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the model verification method of some embodiments of the present disclosure, the scenario range of the covered verification business is expanded, thereby expanding the scope of application of the model verification and enhancing the generalization ability of the verification model. Specifically, the reason why the covered verification business scenarios are small, thereby resulting in a small scope of application of the model verification and a weak generalization ability of the verification model is that the verification model is verified only through a single model training framework, resulting in a small number of scenarios for the covered verification business. As a result, the scope of application of the model verification is small and the generalization ability of the verification model is weak. Based on this, the model verification method of some embodiments of the present disclosure, first, determines the data path information according to the model verification mode. Thus, the data path information characterizing the verification mode can be obtained. Thus, it can also be used to distinguish between offline model verification and online model verification. Secondly, according to the above data path information, a set of data to be verified is obtained. Thus, a set of data to be verified corresponding to different verification modes can be obtained. Then, according to the above data set to be verified, a set of coded information groups to be verified is generated. Thus, a set of coded information groups to be verified matching the format of the model to be verified can be obtained. Afterwards, the model to be verified is determined according to the verification model corresponding to the above-mentioned data set to be verified. Thus, the model to be verified in the deployment environment can be obtained. Finally, according to the above-mentioned set of coded information groups to be verified, the above-mentioned model to be verified is verified to obtain model verification information. Thus, model verification information characterizing the model verification performance can be obtained. Also, because the model to be verified can be verified by the verification method, the scope of scenarios of the verification business covered can be expanded, thereby expanding the scope of application of the model verification and enhancing the generalization ability of the verification model. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0034] Figure 1 is a schematic diagram of an application scenario of the model verification method of some embodiments of the present disclosure;

[0035] Figure 2 is a flow chart of some embodiments of the model checking method according to the present disclosure;

[0036] Figure 3 is a deployment diagram of a deployment environment of a model checking method according to some embodiments of the present disclosure;

[0037] Figure 4 are flow charts of other embodiments of the model checking method according to the present disclosure;

[0038] Figure 5 is a schematic diagram of the structure of some embodiments of the model verification device according to the present disclosure;

[0039] Figure 6 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0040] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0041] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0042] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0043] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0044] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0045] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0046] Figure 1 It is a schematic diagram of an application scenario of the model verification method of some embodiments of the present disclosure.

[0047] exist Figure 1In the application scenario, first, the computing device 101 can determine the data path information 103 according to the model verification method 102. Then, the computing device 101 can obtain the data set 104 to be verified according to the above data path information 103. After that, the computing device 101 can generate the coded information group set 105 to be verified according to the above data set 104 to be verified. Secondly, the computing device 101 can determine the model to be verified 106 according to the verification model corresponding to the above data set 104 to be verified. Finally, the computing device 101 can verify the above model to be verified 106 according to the above coded information group set 105 to be verified to obtain model verification information 107.

[0048] It should be noted that the computing device 101 can be hardware or software. When the computing device is hardware, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or as a single server or a single terminal device. When the computing device is embodied as software, it can be installed in the hardware devices listed above. It can be implemented as multiple software or software modules for providing distributed services, or as a single software or software module. No specific limitation is made here.

[0049] It should be understood that Figure 1 The number of computing devices in the embodiment is only illustrative. Any number of computing devices may be provided according to implementation requirements.

[0050] Continue to refer Figure 2 , shows a process 200 of some embodiments of the model checking method according to the present disclosure. The model checking method comprises the following steps:

[0051] Step 201, determine data path information according to the model verification method.

[0052] In some embodiments, the execution body of the model checking method (eg Figure 1 The computing device 101 shown in the figure can determine the data path information according to the model verification method. Among them, the above-mentioned model verification method can characterize whether the model verification is offline verification or online verification. Specifically, the model verification method may include but is not limited to: online model verification and offline model verification. The above-mentioned data path information can be information about the storage path of the model and data to be verified. In practice, in response to determining that the model verification method is offline model verification, the above-mentioned execution subject can determine the path information of the local database as the data path information.

[0053] In some optional implementations of some embodiments, in response to determining that the above-mentioned model verification method is online model verification, the above-mentioned execution subject may determine the business path information corresponding to the online model verification as data path information. The above-mentioned business path information may be path information for storing the business corresponding to the online model verification. Specifically, the above-mentioned business path information may be path information for storing files for storing the above-mentioned business. The above-mentioned business may represent a task of model verification. For example, the above-mentioned business may be a classification task or a prediction task.

[0054] In some optional implementations of some embodiments, in response to determining that the model verification method is offline model verification, the execution subject may determine the storage path information corresponding to the offline model verification as data path information. The storage path information may be information about the path of the disk storing the data and model related to the offline model verification.

[0055] Step 202: Obtain a set of data to be verified according to the data path information.

[0056] In some embodiments, based on the above data path information, the above execution entity can obtain the data set to be verified.

[0057] In practice, the execution subject may obtain the data set to be verified from the local database according to the path corresponding to the data path information through a wired connection or a wireless connection. The local database may be a database storing the data set to be verified. It should be noted that the wireless connection method may include but is not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0058] In some optional implementations of some embodiments, according to the above data path information, the above execution subject may obtain the data set to be verified through the following steps:

[0059] The first step is, in response to determining that the above data path information is business path information, obtaining the data set to be verified from the business data storage server corresponding to the above business path information. The above business data storage server may be a server storing the above data set to be verified. The above business data storage server may be any server. In practice, in response to determining that the above data path information is business path information, the above execution subject may obtain the data set to be verified from the business data storage server according to the path corresponding to the business path information via a wired connection or a wireless connection.

[0060] The second step is, in response to determining that the above data path information is storage path information, obtaining the data set to be verified from the storage disk corresponding to the above storage path information. The above storage disk may be a disk that stores the data set to be verified corresponding to the offline model verification. Specifically, the above disk includes but is not limited to any one of the following: a USB flash drive, a computer built-in hard drive, and an external hard drive. In practice, in response to determining that the above data path information is storage path information, the above execution subject may obtain the data set to be verified from the storage disk according to the path corresponding to the storage path information via a wired connection or a wireless connection.

[0061] Step 203: Generate a set of coded information groups to be verified based on the set of data to be verified.

[0062] In some embodiments, based on the above-mentioned data set to be verified, the above-mentioned execution entity can generate a set of coded information groups to be verified.

[0063] In practice, the execution subject may process the data set to be verified by calling a preset function interface to generate a set of coded information groups to be verified. The preset function interface may be a pre-packaged data processing function.

[0064] Optionally, before obtaining the data set to be verified according to the data path information, the execution subject may also start the deployment environment according to the deployment method. The deployment methods may include but are not limited to: local deployment, default deployment and distributed deployment. The deployment methods may characterize the deployment specifications for model verification of the model to be verified corresponding to the data set to be verified. The deployment specifications of the default deployment method are greater than the deployment specifications of the local deployment method. The deployment specifications of the distributed deployment method are greater than the deployment specifications of the default deployment method. The deployment specifications may be the amount of data processed by the model to be verified corresponding to the data set to be verified. The local deployment environment may be the configuration environment corresponding to the local deployment. The default deployment may be the configuration environment corresponding to the default deployment. The distributed deployment may be the configuration environment corresponding to the distributed deployment. The configuration environment may be the hardware and software configuration required for model verification.

[0065] In some optional implementations of some embodiments, according to the deployment method, the execution subject may start the deployment environment through the following steps:

[0066] In the first step, in response to the deployment mode being local deployment, the local deployment environment is started.

[0067] In the second step, in response to the deployment mode being default deployment, the default deployment environment is started.

[0068] The third step is to start the distributed deployment environment in response to the deployment mode satisfying the distributed deployment environment startup condition. Among them, the above-mentioned distributed deployment environment startup condition can be that the deployment mode is distributed deployment and is non-resident deployment. The above-mentioned distributed deployment environment startup condition can also be that the deployment mode is distributed deployment and is resident deployment and the model service corresponding to the resident deployment does not exist. The above-mentioned resident deployment can be understood as a default deployment with a high frequency of use. For example, when the user performs model verification, the frequency of using the distributed deployment of tensorflow is high. Then the distributed deployment of tensorflow is a resident deployment. Here, the user can be understood as an organization (such as a company) that needs to perform model verification. The above-mentioned model service can be a machine learning model service or a deep learning model service. For example, the above-mentioned deep learning model service can include but is not limited to: CNN (Convolutional Neural Network, Convolutional Neural Network) model service, RNN (Recurrent Neural Network, Recurrent Neural Network) model service and DNN (Deep Neural Networks, Deep Neural Network) model service.

[0069] As an example, the deployment diagram of the above deployment environment can be as follows: Figure 3 As shown. When the deployment mode is local deployment, the local deployment environment is started. When the deployment mode is default deployment, the default deployment environment is started. When the deployment mode is distributed deployment and not resident, the distributed deployment environment is started. When the deployment mode is distributed deployment and resident deployment and the model service does not exist, the distributed deployment environment is started.

[0070] Step 204: determine the model to be verified according to the verification model corresponding to the data set to be verified.

[0071] In some embodiments, the execution subject may determine the model to be verified according to the verification model corresponding to the data set to be verified. In practice, the execution subject may load the verification model and use the loaded verification model as the model to be verified.

[0072] In some optional implementations of some embodiments, according to the verification model corresponding to the above-mentioned data set to be verified, the above-mentioned execution subject may determine the model to be verified by the following steps:

[0073] The first step is to determine the verification model as the model to be verified in response to the deployment mode being distributed deployment and resident deployment and the model service existing.

[0074] In the second step, in response to determining that the local deployment environment is started, the verification model is loaded, and the verification model after the loading process is determined as the model to be verified. In practice, in response to determining that the local deployment environment is started, the verification model is loaded into the local deployment environment, and the verification model after loading into the local deployment environment is determined as the model to be verified.

[0075] In the third step, in response to determining that the default deployment environment is started, the verification model is loaded, and the loaded verification model is determined as the model to be verified. In practice, in response to determining that the default deployment environment is started, the verification model is loaded into the default deployment environment, and the verification model loaded into the default deployment environment is determined as the model to be verified.

[0076] In the fourth step, in response to determining that the distributed deployment environment is started, the verification model is loaded, and the loaded verification model is determined as the model to be verified. In practice, in response to determining that the distributed deployment environment is started, the verification model is loaded into the distributed deployment environment, and the verification model loaded into the distributed deployment environment is determined as the model to be verified.

[0077] Optionally, the above execution entity may further perform the following steps:

[0078] The first step is to, in response to the deployment mode being a distributed deployment and a resident deployment, and the preset forced deletion configuration information indicating that forced deletion is not enabled, destroy the data service environment corresponding to the data set to be verified. Among them, the preset forced deletion configuration information may be information preset on whether to perform forced deletion on the deployment environment corresponding to the model to be verified. The above forced deletion not being enabled can be understood as not forcibly deleting the service environment corresponding to the data set to be verified. The above data service environment may be a service environment for performing relevant data processing on the data set to be verified before it is input into the model to be verified. For example, the above data service environment may be a configuration environment for a server that processes data independently. In practice, in response to the deployment mode being a distributed deployment and a resident deployment, and the preset forced deletion configuration information indicating that forced deletion is not enabled, the above execution subject may destroy the data service environment corresponding to the data set to be verified.

[0079] In the second step, in response to the deployment mode being a non-resident deployment, or the deployment mode being a resident deployment and the preset forced deletion configuration information indicating that forced deletion is turned on, the data service environment and the corresponding model service environment corresponding to the data set to be verified are destroyed. The above-mentioned model service environment may be a service environment for performing model verification on the model to be verified. For example, the above-mentioned model service environment may be a configuration environment for two servers that perform model verification. In practice, in response to the deployment mode being a non-resident deployment, or the deployment mode being a resident deployment and the preset forced deletion configuration information indicating that forced deletion is turned on, the above-mentioned execution subject may destroy the data service environment and the corresponding model service environment corresponding to the data set to be verified.

[0080] Step 205: verify the model to be verified according to the set of coded information groups to be verified, and obtain model verification information.

[0081] In some embodiments, based on the above-mentioned set of coded information groups to be verified, the above-mentioned execution entity can verify the above-mentioned model to be verified to obtain model verification information.

[0082] In some optional implementations of some embodiments, the execution subject may input the set of the coded information group to be verified into the model to be verified to obtain model verification information. The model verification information may characterize the verification performance of the model to be verified. Specifically, the model verification information may be predicted indicator information. For example, the indicator information may be a predicted accuracy rate. The indicator information may be an accuracy rate of 0.98.

[0083] Optionally, the above execution entity may further perform the following steps:

[0084] In the first step, in response to determining that the model verification method is online model verification, the model verification information is sent to the business data storage server. In practice, in response to determining that the model verification method is online model verification, the execution entity can send the model verification information to the business data storage server.

[0085] In the second step, in response to determining that the model verification method is offline model verification, the model verification information is stored in the storage disk. In practice, in response to determining that the model verification method is offline model verification, the execution entity can store the model verification information in the storage disk.

[0086] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the model verification method of some embodiments of the present disclosure, the scenario range of the covered verification business is expanded, thereby expanding the scope of application of the model verification and enhancing the generalization ability of the verification model. Specifically, the reason why the covered verification business scenarios are small, thereby resulting in a small scope of application of the model verification and a weak generalization ability of the verification model is that the verification model is verified only through a single model training framework, resulting in a small number of scenarios for the covered verification business. As a result, the scope of application of the model verification is small and the generalization ability of the verification model is weak. Based on this, the model verification method of some embodiments of the present disclosure, first, determines the data path information according to the model verification mode. Thus, the data path information characterizing the verification mode can be obtained. Thus, it can also be used to distinguish between offline model verification and online model verification. Secondly, according to the above data path information, a set of data to be verified is obtained. Thus, a set of data to be verified corresponding to different verification modes can be obtained. Then, according to the above data set to be verified, a set of coded information groups to be verified is generated. Thus, a set of coded information groups to be verified matching the format of the model to be verified can be obtained. Afterwards, the model to be verified is determined according to the verification model corresponding to the above-mentioned data set to be verified. Thus, the model to be verified in the deployment environment can be obtained. Finally, according to the above-mentioned set of coded information groups to be verified, the above-mentioned model to be verified is verified to obtain model verification information. Thus, model verification information characterizing the model verification performance can be obtained. Also, because the model to be verified can be verified by the verification method, the scope of scenarios of the verification business covered can be expanded, thereby expanding the scope of application of the model verification and enhancing the generalization ability of the verification model.

[0087] Further references Figure 4 , which shows a process 400 of another embodiment of the model checking method. The process 400 of the model checking method includes the following steps:

[0088] Step 401, determine data path information according to the model verification method.

[0089] In some embodiments, the specific implementation of step 401 and the technical effects brought about can be referred to Figure 2 The corresponding step 201 in the embodiments will not be described in detail here.

[0090] Step 402, obtaining a set of data to be verified according to the data path information;

[0091] In some embodiments, the specific implementation of step 402 and the technical effects brought about can be referred to Figure 2 The corresponding step 202 in the embodiments will not be described in detail here.

[0092] Step 403: Perform data format conversion processing on the data set to be verified to obtain a converted data set.

[0093] In some embodiments, the execution subject may perform data format conversion processing on the data set to be verified to obtain a converted data set. In practice, the execution subject may convert the data set to be verified into a converted data set through a preset data format conversion interface. For example, the converted data set may be MGD (Model Guard Data, model verification system processing data). The preset data format conversion interface may be a pre-packaged function that can perform data format conversion on the data set to be verified. The MGD may be data in the format required for model verification.

[0094] Step 404, partition the transformed data set to obtain a partitioned data group set.

[0095] In some embodiments, the execution subject may perform partition processing on the conversion data set to obtain a partition data set. In practice, the execution subject may perform equal partition processing on the conversion data set to obtain a partition data set.

[0096] Step 405: For each partition data group in the partition data group set, encoding processing is performed on each partition data included in the partition data group to generate a partition encoding data group.

[0097] In some embodiments, for each partition data group in the partition data group set, the execution subject may perform encoding processing on each partition data included in the partition data group to generate a partition encoding data group. In practice, for each partition data group in the partition data group set, the execution subject may perform binary encoding processing on each partition data included in the partition data group to generate a partition encoding data group.

[0098] Step 406: determine each generated coded data group as a set of coded information groups to be verified.

[0099] In some embodiments, the execution subject may determine each generated coded data group as a set of coded information groups to be verified. In practice, the execution subject may determine each generated coded data group as a set of coded information groups to be verified.

[0100] Step 407: Determine the model to be verified according to the verification model corresponding to the data set to be verified.

[0101] In some embodiments, the specific implementation of step 407 and the technical effects brought about can be referred to Figure 2 The corresponding step 204 in the embodiments will not be described in detail here.

[0102] Step 408: input the set of coded information groups to be verified into the model to be verified to obtain model verification information.

[0103] In practice, the above-mentioned execution entity can input the set of coded information groups to be verified into the model to be verified to obtain model verification information.

[0104] from Figure 4 It can be seen that Figure 2 Compared with the description of some corresponding embodiments, Figure 4 The process 400 of the model verification method in some corresponding embodiments embodies a method of integrating the data processing of the data set to be verified with the model processing of the model to be verified. Thus, the link of the model going online can be shortened, and the business scope of the connected offline framework can also be expanded.

[0105] Further references Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a model checking device. These device embodiments are similar to Figure 2 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0106] like Figure 5 As shown, the model verification device 500 of some embodiments includes: a first determination unit 501, an acquisition unit 502, a generation unit 503, a second determination unit 504 and a verification unit 505. The first determination unit 501 is configured to determine data path information according to the model verification mode; the acquisition unit 502 is configured to obtain a set of data to be verified according to the above data path information; the generation unit 503 is configured to generate a set of coded information groups to be verified according to the above set of data to be verified; the second determination unit 504 is configured to determine the model to be verified according to the verification model corresponding to the above set of data to be verified; the verification unit 505 is configured to verify the above model to be verified according to the above set of coded information groups to be verified to obtain model verification information.

[0107] Optionally, the first determining unit 501 may be further configured to: in response to determining that the above-mentioned model verification mode is online model verification, determine the business path information corresponding to the online model verification as the data path information.

[0108] Optionally, the first determining unit 501 may be further configured to: in response to determining that the above-mentioned model verification mode is offline model verification, determine the storage path information corresponding to the offline model verification as the data path information.

[0109] Optionally, the acquisition unit 502 can be further configured to: in response to determining that the above-mentioned data path information is business path information, obtain the data set to be verified from the business data storage server corresponding to the above-mentioned business path information; in response to determining that the above-mentioned data path information is storage path information, obtain the data set to be verified from the storage disk corresponding to the above-mentioned storage path information.

[0110] Optionally, the generation unit 503 can be further configured to: perform data format conversion processing on the above-mentioned data set to be verified to obtain a converted data set; perform partition processing on the above-mentioned converted data set to obtain a partitioned data group set; for each partitioned data group in the above-mentioned partitioned data group set, encode the individual partitioned data included in the above-mentioned partitioned data group to generate a partitioned coded data group; and determine the generated individual coded data groups as the coded information group set to be verified.

[0111] Optionally, the verification unit 505 may be further configured to: input the above-mentioned set of coded information groups to be verified into the above-mentioned model to be verified to obtain model verification information.

[0112] Optionally, the model verification device 500 may further include: a sending unit and a storage unit (not shown in the figure). The sending unit is configured to send the model verification information to the business data storage server in response to determining that the model verification mode is online model verification. The storage unit is configured to store the model verification information in the storage disk in response to determining that the model verification mode is offline model verification.

[0113] Optionally, before the acquiring unit 502, the model checking device 500 may further include: a starting unit (not shown in the figure), configured to start the deployment environment according to the deployment mode.

[0114] Optionally, the startup unit can be further configured to: in response to the deployment mode being local deployment, start the local deployment environment; in response to the deployment mode being default deployment, start the default deployment environment; in response to the deployment mode satisfying the distributed deployment environment startup condition, start the distributed deployment environment.

[0115] Optionally, the second determination unit 504 can be further configured to: in response to the deployment mode being distributed deployment and being resident deployment and the above-mentioned model service existing, determine the above-mentioned verification model as the model to be verified; in response to determining that the local deployment environment is started, load the above-mentioned verification model, and determine the verification model after the loading process as the model to be verified; in response to determining that the default deployment environment is started, load the above-mentioned verification model, and determine the verification model after the loading process as the model to be verified; in response to determining that the distributed deployment environment is started, load the above-mentioned verification model, and determine the verification model after the loading process as the model to be verified.

[0116] Optionally, the model verification device 500 may also include: a data service environment destruction unit and a data service environment and model service destruction unit (not shown in the figure). Among them, the data service environment destruction unit is configured to destroy the data service environment corresponding to the above-mentioned data set to be verified in response to the deployment mode being a distributed deployment and a resident deployment, and the preset forced deletion configuration information indicating that forced deletion is not enabled. The data service environment and model service destruction unit is configured to destroy the data service environment and the corresponding model service environment corresponding to the above-mentioned data set to be verified in response to the deployment mode being a non-resident deployment, or the above-mentioned deployment mode being a resident deployment and the above-mentioned preset forced deletion configuration information indicating that forced deletion is enabled.

[0117] It can be understood that the units recorded in the model verification device 500 are similar to the reference Figure 2 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the model checking device 500 and the units included therein, and will not be described in detail here.

[0118] Reference below Figure 6 , which shows an electronic device 600 (eg, Figure 1 Schematic diagram of the structure of the computing device 101). Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0119] like Figure 6As shown, the electronic device 600 may include a processing device 601 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 602 (ROM) or a program loaded from a storage device 608 into a random access memory 603 (RAM). In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0120] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 6 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0121] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0122] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0123] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0124] The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: determines data path information according to the model verification method; obtains a set of data to be verified according to the data path information; generates a set of coded information groups to be verified according to the set of data to be verified; determines the model to be verified according to the verification model corresponding to the set of data to be verified; verifies the model to be verified according to the set of coded information groups to be verified, and obtains model verification information.

[0125] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0126] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0127] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be provided in a processor, for example, may be described as: a processor comprising a first determination unit, an acquisition unit, a generation unit, a second determination unit and a verification unit. The names of these units do not constitute limitations on the units themselves in certain circumstances, for example, the first determination unit may also be described as "a unit for determining data path information according to a model verification method".

[0128] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0129] Some embodiments of the present disclosure further provide a computer program product, including a computer program, which implements any of the above-mentioned model checking methods when executed by a processor.

[0130] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A model verification method, comprising: Determine data path information according to the model verification method; According to the data path information, obtaining a set of data to be verified; Generating a set of coded information groups to be verified according to the set of data to be verified; Determine the model to be verified according to the verification model corresponding to the data set to be verified; According to the set of coded information groups to be verified, the model to be verified is verified to obtain model verification information.

2. The method according to claim 1, wherein: The step of determining data path information according to the model verification method includes: In response to determining that the model verification mode is online model verification, the business path information corresponding to the online model verification is determined as data path information.

3. The method according to claim 2, wherein: The determining of data path information according to the model verification method further includes: In response to determining that the model verification mode is offline model verification, storage path information corresponding to the offline model verification is determined as data path information.

4. The method according to claim 3, wherein: The step of obtaining a set of data to be verified according to the data path information includes: In response to determining that the data path information is business path information, acquiring a set of data to be verified from a business data storage server corresponding to the business path information; In response to determining that the data path information is storage path information, a to-be-verified data set is acquired from a storage disk corresponding to the storage path information.

5. The method according to claim 1, wherein: The step of generating a set of coded information groups to be verified according to the set of data to be verified comprises: Performing data format conversion processing on the data set to be verified to obtain a converted data set; Partitioning the transformed data set to obtain a partitioned data group set; For each partition data group in the partition data group set, encoding processing is performed on each partition data included in the partition data group to generate a partition encoding data group; The generated coded data groups are determined as a set of coded information groups to be verified.

6. The method according to claim 1, wherein: The step of verifying the model to be verified according to the set of coded information groups to be verified to obtain model verification information includes: The set of coded information groups to be verified is input into the model to be verified to obtain model verification information.

7. The method according to claim 4, wherein: The method further comprises: In response to determining that the model verification mode is online model verification, sending the model verification information to the business data storage server; In response to determining that the model verification mode is offline model verification, the model verification information is stored in the storage disk.

8. The method according to claim 1, wherein: Before acquiring the to-be-verified data set according to the data path information, the method further includes: Start the deployment environment according to the deployment mode.

9. The method according to claim 8, wherein: According to the deployment mode, starting the deployment environment includes: In response to the deployment mode being local deployment, starting the local deployment environment; In response to the deployment mode being default deployment, starting a default deployment environment; In response to the deployment mode satisfying the distributed deployment environment startup condition, the distributed deployment environment is started.

10. The method according to claim 9, wherein: The step of determining the model to be verified according to the verification model corresponding to the data set to be verified includes: In response to the deployment mode being distributed deployment and resident deployment and the model service existing, determining the verification model as a model to be verified; In response to determining that the local deployment environment is started, loading the verification model, and determining the loaded verification model as the model to be verified; In response to determining that the default deployment environment is started, loading the verification model, and determining the loaded verification model as a model to be verified; In response to determining that the distributed deployment environment is started, the verification model is loaded, and the loaded verification model is determined as a model to be verified.

11. The method according to claim 10, wherein: The method further comprises: In response to the deployment mode being distributed deployment and being resident deployment, and the preset forced deletion configuration information indicating that forced deletion is not enabled, destroying the data service environment corresponding to the data set to be verified; In response to the deployment mode being non-resident deployment, or the deployment mode being resident deployment and the preset forced deletion configuration information indicating forced deletion is enabled, the data service environment and the corresponding model service environment corresponding to the data set to be verified are destroyed.

12. A model verification device, comprising: A first determining unit is configured to determine data path information according to a model verification method; An acquisition unit, configured to acquire a set of data to be verified according to the data path information; A generating unit, configured to generate a set of coded information groups to be verified according to the set of data to be verified; A second determination unit is configured to determine a model to be verified according to the verification model corresponding to the data set to be verified; The verification unit is configured to verify the model to be verified according to the set of coded information groups to be verified, so as to obtain model verification information.

13. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 11.

14. A computer readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

15. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 11.

Citation Information

Patent Citations

  • A method and a device for combining a performance prediction model and product research and development

    CN109902872A

  • Verification method and device, electronic equipment and computer readable storage medium

    CN113296989A

  • Data verification method and device

    CN114547016A

  • Usage of modeled validations on mobile devices in online and offline scenarios

    US20170177696A1

  • Data verification method and apparatus, and computer device and storage medium

    WO2020258655A1

Cited By

  • Model verification method and apparatus, credit evaluation model verification method and apparatus, and computing device

    CN119762212A