Deviation monitoring methods, devices, equipment, storage media and program products
By acquiring key-value pairs of business data and dynamically selecting models based on preset rules and attribute information, the problem of insufficient flexibility of expert rule models is solved, and more efficient data deviation monitoring is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- INDUSTRIAL AND COMMERCIAL BANK OF CHINA
- Filing Date
- 2023-04-10
- Publication Date
- 2026-05-26
Smart Images
Figure CN116401238B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of big data technology, and in particular to a deviation monitoring method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology
[0002] With the development of digital transformation, data is playing an increasingly important role. In order to ensure data quality, it is usually necessary to monitor the data to verify its quality. Among these measures, data deviation is an important indicator used to characterize data quality.
[0003] In existing technologies, expert rule models are typically used to monitor data deviation. These models are customized by experts according to rules for a specific type of data or a specific use case, and the deviation of the data is monitored through these models.
[0004] However, this expert rule model has poor flexibility in monitoring deviations. Summary of the Invention
[0005] Therefore, it is necessary to provide a highly flexible deviation monitoring method, device, computer equipment, computer-readable storage medium, and computer program product to address the aforementioned technical problems.
[0006] Firstly, this application provides a deviation monitoring method. The method includes:
[0007] Obtain key-value pairs of target external data to be monitored for deviation. This target external data is data generated during business operations. Determine an initial model from multiple candidate models based on preset key-value rules and the key-value pairs. Determine a target model based on the attribute information of the key-value pairs and the initial model. Obtain the deviation of the target external data based on the target external data, the key-value pairs, and the target model.
[0008] In one embodiment, the key-value pair for obtaining the target external data to be monitored for deviation includes: determining the target external data from an external database according to preset filtering conditions, wherein the filtering conditions are related to at least one of the deviation monitoring time period and the business type; and performing unified processing on the target external data to obtain the key-value pair.
[0009] In one embodiment, the process of unifying the target external data to obtain the key-value pair includes: parsing the source data corresponding to the target external data to obtain the data content of the target external data; and unifying the target external data based on the source data and the data content to obtain the key-value pair.
[0010] In one embodiment, the process of unifying the target external data based on the source data and the data content to obtain the key-value pair includes: if the source data is a JSON file, then the source data and the data content are unified based on a serialization tool to obtain the key-value pair; if the source data is a database source file, then the source data and the data content are unified based on a reflection tool to obtain the key-value pair.
[0011] In one embodiment, the attribute information includes the data acquisition time. Determining the target model based on the attribute information of the key-value pair and the initial model includes: comparing the data acquisition time with the model update time of the initial model to determine whether the initial model has failed; if the initial model has failed, then training the initial model using the retained data and the target external data to obtain the target model, wherein the data dimensions of the retained data and the target external data are the same.
[0012] In one embodiment, the method further includes: if the initial model is not invalid, then determining the initial model as the target model.
[0013] Secondly, this application also provides a deviation monitoring device. The device includes:
[0014] The acquisition module is used to acquire key-value pairs of external data of the target to be monitored for deviation. This external data is generated during business operations.
[0015] The determination module is used to determine the initial model from multiple candidate models based on preset key-value rules and the key-value pair;
[0016] The first execution module is used to determine the target model based on the attribute information of the key-value pair and the initial model, and to obtain the deviation of the target external data based on the target external data, the key-value pair and the target model.
[0017] In one embodiment, the acquisition module is specifically used to: determine the target external data from an external database according to preset filtering conditions, wherein the filtering conditions are related to at least one of deviation monitoring time period and business type; and perform unified processing on the target external data to obtain the key-value pair.
[0018] In one embodiment, the acquisition module is specifically used to: parse the source data corresponding to the target external data to obtain the data content of the target external data; and perform unified processing on the target external data based on the source data and the data content to obtain the key-value pair.
[0019] In one embodiment, the acquisition module is specifically used to: if the source data is a JSON file, then based on a serialization tool, perform unified processing on the source data and the data content to obtain the target external data to obtain the key-value pair; if the source data is a database source file, then based on a reflection tool, perform unified processing on the source data and the data content to obtain the target external data to obtain the key-value pair.
[0020] In one embodiment, the first execution module is specifically used to: compare the data acquisition time with the model update time of the initial model to determine whether the initial model has failed; if the initial model has failed, then train the initial model using the retained data and the target external data to obtain the target model, wherein the data dimensions of the retained data and the target external data are the same.
[0021] In one embodiment, the deviation monitoring device further includes a second execution module, which is configured to: determine the initial model as the target model if the initial model has not failed.
[0022] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform any of the steps described in the first aspect above.
[0023] Fourthly, this application also provides a computer-readable storage medium. This computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs any of the steps described in the first aspect above.
[0024] Fifthly, this application also provides a computer program product. This computer program product includes a computer program that, when executed by a processor, implements any of the steps described in the first aspect above.
[0025] The aforementioned deviation monitoring methods, devices, computer equipment, computer-readable storage media, and computer program products first acquire key-value pairs of the target external data to be monitored. This target external data is data generated during business operations. Then, based on preset key-value rules and the key-value pairs, an initial model is determined from multiple candidate models. Finally, based on the attribute information of the key-value pairs and the initial model, a target model is determined. The deviation of the target external data is then obtained based on the target external data, the key-value pairs, and the target model. The initial model in the deviation monitoring method provided in this application is determined from multiple candidate models based on preset key-value rules and key-value pairs. Different target external data can correspond to different initial models; that is, the initial model can be flexibly selected for different target external data. Furthermore, after determining the initial model, it is further adjusted based on the attribute information of the key-value pairs to determine the target model. Finally, the deviation of the target external data is obtained based on the target model, resulting in higher accuracy of the obtained deviation of the target external data. In summary, the deviation monitoring method provided in this application offers high flexibility. Attached Figure Description
[0026] Figure 1 This is a flowchart illustrating a deviation monitoring method in one embodiment;
[0027] Figure 2 This is a schematic diagram of the process for obtaining key-value pairs of external data of the target to be monitored for deviation in one embodiment;
[0028] Figure 3 This is a flowchart illustrating the process of unifying target external data to obtain key-value pairs in one embodiment.
[0029] Figure 4 This is a flowchart illustrating the process of determining the target model in one embodiment;
[0030] Figure 5 This is a flowchart illustrating the deviation monitoring method in another embodiment;
[0031] Figure 6 This is a structural block diagram of a deviation monitoring device in one embodiment;
[0032] Figure 7 This is a structural block diagram of a deviation monitoring device in one embodiment;
[0033] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation
[0034] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0035] With the development of digital transformation, data is playing an increasingly important role. In order to ensure data quality, it is usually necessary to monitor the data to verify its quality. Among these measures, data deviation is an important indicator used to characterize data quality.
[0036] In existing technologies, expert rule models are typically used to monitor data deviation. These models are customized by experts according to rules for a specific type of data or a specific use case, and the deviation of the data is monitored through these models.
[0037] However, this expert rule model has poor flexibility.
[0038] In view of this, this application provides a deviation monitoring method that can effectively improve flexibility.
[0039] The deviation monitoring method provided in this application can be executed by a computer device, which can be a server.
[0040] In one embodiment, such as Figure 1 As shown, a deviation monitoring method is provided, including the following steps:
[0041] Step 101: Obtain key-value pairs of the target external data to be monitored for deviation.
[0042] The external data for this target refers to data generated during business operations.
[0043] The deviation is an indicator used to characterize data quality, referring to the proportion of the absolute value of the difference between the actual data and the predicted data in the predicted data.
[0044] This key-value pair refers to a basic data representation method that can include a key and a value. The key is the number of the stored value, and the value is the data to be stored. This key-value pair can include the data source database, the data source table, the data collection time, and the data granularity information.
[0045] In one possible approach, key-value pairs of external data of the target to be monitored for deviation can be obtained by writing a transformation tool.
[0046] In another possible implementation, key-value pairs of the target external data to be monitored for deviation can be obtained through dependency packages. For example, you can first import the pom dependency, then import the corresponding processing class, and finally call the corresponding method. For example, if the entity is entity, you can finally obtain the key-value pairs of the target external data to be monitored for deviation by map.get(“keyValue”).toString.
[0047] In another possible approach, key-value pairs of external data of the target to be monitored for deviation can be obtained through serialization tools.
[0048] In another possible approach, key-value pairs of external data of the target to be monitored for deviation can be obtained through reflection tools.
[0049] Step 102: Determine the initial model from multiple candidate models based on the preset key-value rules and the key-value pair.
[0050] The preset key-value rules can be set in advance by technical personnel.
[0051] Optionally, based on the preset key-value rule, the computer device can determine the type of the target external data according to the key-value pairs of the target external data.
[0052] In one possible implementation, based on the preset key-value rules, the business type of the target external data can be determined as deposit data through the key-value pairs of the target external data.
[0053] In another possible implementation, based on the preset key-value rules, the business type of the target external data can be determined as withdrawal data through the key-value pairs of the target external data.
[0054] Optionally, based on the preset key-value rule, the computer device can determine the source database of the target external data according to the key-value pairs of the target external data.
[0055] In one possible implementation, based on the preset key-value rules, the source database of the target external data can be determined as database A through the key-value pairs of the target external data.
[0056] In another possible implementation, based on the preset key-value rules, the source database of the target external data can be determined as database B through the key-value pairs of the target external data.
[0057] Optionally, based on the preset key-value rule, the computer device can determine the source data table of the target external data according to the key-value pairs of the target external data.
[0058] In one possible implementation, based on the preset key-value rule, the source data table of the target external data can be determined as source data table A through the key-value pairs of the target external data.
[0059] In another possible implementation, based on the preset key-value rule, the source data table of the target external data can be determined as source data table B through the key-value pairs of the target external data.
[0060] Optionally, based on the preset key-value rule, the computer device can determine the data granularity information of the target external data according to the key-value pairs of the target external data.
[0061] Optionally, the candidate model refers to a neural network model, such as a back propagation neural network (BPNN).
[0062] In an optional embodiment of this application, the initial model determined from a plurality of candidate models may also be a plurality of models.
[0063] In one possible implementation, as described above, based on the preset key-value rules, the type, source database, source data table, and data granularity information of the target external data can be determined through the key-value pairs of the target external data. Therefore, there are multiple models, which are candidate models. After determining the type, source database, source data table, and data granularity information of the target external data based on the preset key-value rules and the key-value pairs, the initial model can be determined from the multiple candidate models.
[0064] In another possible implementation, the initial model can be determined from multiple candidate models using historical retention data, preset key-value rules, and the key-value pairs. The historical retention data refers to the historically retained model evaluation data, which will be explained in detail below and will not be repeated here.
[0065] In another possible implementation, multiple candidate models can be selected as the initial model to obtain a model evaluation of each candidate model based on the deviation of the target external data. This model evaluation, as historical data, can be used to determine the initial model in the future. This will be explained in detail below and will not be repeated here.
[0066] Step 103: Determine the target model based on the attribute information of the key-value pair and the initial model, and obtain the deviation of the target external data based on the target external data, the key-value pair and the target model.
[0067] Optionally, the attribute information can be attribute information of the target external data, and the attribute information of the target external data can be directly read through the key-value pairs of the target external data.
[0068] In one possible implementation, the initial model can be further adjusted based on the attribute information of the key-value pair to determine the target model. Then, the deviation of the target external data and its key-value pairs can be obtained by inputting them into the target model.
[0069] In an optional embodiment of this application, the target external data and the key-value pairs of the target external data are input into the target model. The target model can also output a model evaluation, which is the historical retention data mentioned above. The model evaluation, as historical retention data, can be used to determine the initial model and serve as the basis for determining the initial model. For example, the model evaluation may indicate that the model has a good fit, or it may indicate that the model has a poor fit. The model evaluation refers to whether the model is suitable for deviation calculation based on the target external data.
[0070] In one possible implementation, after the target external data and the key value are input into the target model, the target model will output the deviation of the target external data and the model evaluation of the target model, as shown in Table 1. Here, R-squared refers to the model evaluation index, which is used to judge whether the model fit is good. R-squared ∈ [0.6, 1). The closer R-squared is to the center of the interval, the better the model fit.
[0071] Table 1
[0072] ID date Business Name Business indicator values Business forecast Deviation R-squared Model Evaluation ... ... ... ... ... 0.3 0.86 Good fit ... ... ... ... ... 0.1 0.88 Good fit ... ... ... ... ... 0.5 0.65 Poor fit
[0073] The aforementioned deviation monitoring method, device, computer equipment, storage medium, and computer program product first acquire key-value pairs of the target external data to be monitored, which are data generated during business operations. Then, based on preset key-value rules and the key-value pairs, an initial model is determined from multiple candidate models. Finally, based on the attribute information of the key-value pairs and the initial model, a target model is determined, and the deviation of the target external data is obtained based on the target external data, the key-value pairs, and the target model. The initial model in the deviation monitoring method provided in this application is determined from multiple candidate models based on preset key-value rules and key-value pairs. This initial model is not fixed; different target external data can correspond to different initial models. That is, the initial model can be flexibly selected for different target external data. After determining the initial model, the target model is determined based on the attribute information of the key-value pairs and the initial model. In other words, the initial model is further adjusted based on the attribute information of the key-value pairs of the target external data to determine the target model. Finally, the deviation of the target external data is obtained based on the target model to make the deviation of the target external data more accurate. In summary, the deviation monitoring method provided in this application has high flexibility.
[0074] In one embodiment, such as Figure 2 As shown, the key-value pairs for acquiring external data of the target to be monitored for deviation include the following steps:
[0075] Step 201: Determine the target external data from the external database according to the preset filtering conditions. The filtering conditions are related to at least one of the deviation monitoring time period and business type.
[0076] Optionally, the preset filtering criteria can be set in advance by technical personnel.
[0077] As mentioned above, this screening criterion can be related to the deviation monitoring period.
[0078] In one possible implementation, the preset filtering condition can be the deviation monitoring period. For example, if the preset filtering condition is from January to April, then data from January to April can be determined from an external database as the target external data based on the preset filtering condition.
[0079] In another possible implementation, the preset filtering condition can be the deviation monitoring period. For example, if the preset filtering condition is 2022, then data from 2022 can be determined from an external database as the target external data based on the preset filtering condition.
[0080] As mentioned above, this filtering criterion can also be related to the business type.
[0081] In one possible implementation, the preset filtering condition can also be a business type. For example, if the preset filtering condition is transfer income, then data with the business type of transfer income can be determined from an external database as the target external data based on the preset filtering condition.
[0082] In another possible implementation, the preset filtering condition can also be a business type. For example, if the preset filtering condition is consumer spending, then data with the business type of consumer spending can be determined from an external database as the target external data based on the preset filtering condition.
[0083] In another possible implementation, the preset filtering condition can also be a business type. For example, if the preset filtering condition is credit card consumption, then data with the business type of credit card consumption can be determined from the external database as the target external data based on the preset filtering condition.
[0084] As mentioned above, this filtering condition can also be related to the business type and the deviation monitoring period.
[0085] In one possible implementation, the preset filtering criteria can also be the business type and the deviation monitoring period. For example, if the preset filtering criteria is transfer income from January to April, then data with the business type of transfer income from January to April can be determined from the external database as the target external data based on the preset filtering criteria.
[0086] In another possible implementation, the preset filtering criteria can also be the business type and the deviation monitoring period. For example, if the preset filtering criteria is the transfer revenue in 2022, then the data with the business type of transfer revenue in 2022 can be determined from the external database as the target external data based on the preset filtering criteria.
[0087] Step 202: Perform unified processing on the external data of the target to obtain the key-value pair.
[0088] In one possible implementation, since the target external data is determined based on an external database containing file data, database data, and message queue data, and the data structure of the external database is not uniform, the data structure of the target external data determined from the external database may also be inconsistent. Therefore, it is necessary to unify the target external data to obtain key-value pairs to ensure running efficiency.
[0089] In one embodiment, such as Figure 3 As shown, the process of unifying the external data of the target to obtain the key-value pair includes the following steps:
[0090] Step 301: Parse the source data corresponding to the target's external data to obtain the data content of the target's external data.
[0091] Optionally, the source data refers to data that comes directly from the source file, such as data from the business system database, data from offline files, and IoT data.
[0092] Optionally, the data content may include the data source database, the data source table, the data time, and data granularity information.
[0093] In one possible implementation, the location of the source data for the target external data can be determined through an interface, and then consumed to obtain the data content of the target external data.
[0094] Step 302: Based on the source data and the data content, perform unified processing on the target external data to obtain the key-value pair.
[0095] In an optional embodiment of this application, the process of unifying the target external data based on the source data and the data content to obtain the key-value pair includes: if the source data is a JSON file, then the source data and the data content are unified based on a serialization tool to obtain the key-value pair; if the source data is a database source file, then the source data and the data content are unified based on a reflection tool to obtain the key-value pair.
[0096] In one possible implementation, if the source data is determined to be a JSON file, a serialization tool, such as FastJson, can be used to process the target external data to obtain the key-value pairs of the target external data, as shown in Table 2.
[0097]
[0098] In another possible implementation, if it is determined that the source data is not a JSON file, a reflection tool can be used to process the target external data to obtain the key-value pairs of the target external data. Specifically, each row element of the target external data is named as an object, and one property of the object can correspond to a combination of column name and value. The key-value pairs of the target external data obtained by this method are also shown in Table 2.
[0099] In an optional embodiment of this application, the target external data is uniformly processed according to the source data and the data content to obtain the key value, and then the target external data is persisted, that is, the key-value pair of the target external data is stored in the external database for easy access to the key-value pair of the target external data in the future.
[0100] In one embodiment, such as Figure 4 As shown, the attribute information includes the data acquisition time. Determining the target model based on the attribute information of the key-value pair and the initial model includes the following steps:
[0101] Step 401: Compare the data acquisition time with the model update time of the initial model to determine whether the initial model has failed.
[0102] Optionally, the data acquisition time is also the data time Data_Data in Table 1, which refers to the acquisition time of the external data of the target.
[0103] Optionally, the model update time of the initial model refers to the most recent model update time.
[0104] In one possible implementation, if the difference between the data acquisition time and the model update time of the initial model is greater than or equal to a preset time threshold, the initial model is determined to be invalid. The preset time threshold can be set in advance by technicians. For example, if the acquisition time of the target external data is March 1, 2022, the model update time of the initial model is April 1, 2021, and the preset time threshold is 6 months, then if the difference between the data acquisition time and the model update time of the initial model is greater than the preset time threshold, the initial model can be determined to be invalid.
[0105] In another possible implementation, if the difference between the data acquisition time and the model update time of the initial model is less than a preset time threshold, it is determined that the initial model has not failed. The preset time threshold can be set in advance by technicians. For example, if the acquisition time of the target external data is March 1, 2022, the model update time of the initial model is January 1, 2022, and the preset time threshold is 6 months, then if the difference between the data acquisition time and the model update time of the initial model is less than the preset time threshold, it can be determined that the initial model has not failed.
[0106] Step 402: If the initial model fails, the initial model is trained using the retained data and the target external data to obtain the target model. The data dimensions of the retained data and the target external data are the same.
[0107] As mentioned above, during each operation, after the target external data is processed to obtain key-value pairs, it is also persisted, that is, the key-value pairs are stored in the target external database. The retained data can be filtered from the persisted data in the external database. The retained data is identical to the target external data in all data dimensions except for the data collection time. For example, if the target external data was collected on October 1, 2022, and the preset retention time is 30 days, then the retained data should be data from September 1, 2022 to September 30, 2022, and all other data dimensions are the same as the target external data. The preset retention time can be set by the technicians.
[0108] In one possible implementation, when the initial model is determined to be invalid, the retained data is first determined by filtering the target external data and the retained data over a preset time. The target external data and the retained data are then mixed to obtain training data. The initial model is then trained based on the training data, and the trained initial model is the target model.
[0109] In one embodiment, the method further includes: if the initial model is not invalid, then determining the initial model as the target model.
[0110] The above method first determines whether the initial model has failed. If it has failed, the initial model is trained using the retained data and the target external data to obtain the target model. Since the model is not fixed but constantly updated, it can avoid the problem of poor accuracy in deviation monitoring caused by large fluctuations in data values during special holidays or events.
[0111] In one embodiment, such as Figure 5 As shown, a deviation monitoring method is provided, which includes the following steps:
[0112] Step 501: Determine the target external data from the external database according to the preset filtering conditions. The filtering conditions are related to at least one of the deviation monitoring time period and business type.
[0113] Step 502: Perform data parsing on the source data corresponding to the target's external data to obtain the data content of the target's external data.
[0114] Step 503: If the source data is a JSON file, then based on the serialization tool, the source data and its content are uniformly processed to obtain the key-value pair. If the source data is a database source file, then based on the reflection tool, the source data and its content are uniformly processed to obtain the key-value pair.
[0115] Step 504: Determine the initial model from multiple candidate models according to the preset key-value rules and the key-value pair.
[0116] Step 505: Compare the data acquisition time with the model update time of the initial model to determine whether the initial model has failed.
[0117] Step 506: If the initial model fails, the initial model is trained using the retained data and the target external data to obtain the target model. The data dimensions of the retained data and the target external data are the same.
[0118] Step 507: If the initial model is not invalid, then the initial model is determined to be the target model.
[0119] Step 508: Obtain the deviation of the target's external data based on the target's external data, the key-value pair, and the target model.
[0120] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.
[0121] Based on the same inventive concept, this application also provides a deviation monitoring device for implementing the deviation monitoring method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more deviation monitoring device embodiments provided below can be found in the limitations of the deviation monitoring method described above, and will not be repeated here.
[0122] In one embodiment, such as Figure 6 As shown, a deviation monitoring device is provided, comprising: an acquisition module, a determination module, and a first execution module, wherein:
[0123] The acquisition module is used to acquire key-value pairs of external data of the target to be monitored for deviation. This external data is generated during business operations.
[0124] The determination module is used to determine the initial model from multiple candidate models based on preset key-value rules and the key-value pair;
[0125] The first execution module is used to determine the target model based on the attribute information of the key-value pair and the initial model, and to obtain the deviation of the target external data based on the target external data, the key-value pair and the target model.
[0126] In one embodiment, the acquisition module is specifically used to: determine the target external data from an external database according to preset filtering conditions, wherein the filtering conditions are related to at least one of deviation monitoring time period and business type; and perform unified processing on the target external data to obtain the key-value pair.
[0127] In one embodiment, the acquisition module is specifically used to: parse the source data corresponding to the target external data to obtain the data content of the target external data; and perform unified processing on the target external data based on the source data and the data content to obtain the key-value pair.
[0128] In one embodiment, the acquisition module is specifically used to: if the source data is a JSON file, then based on a serialization tool, perform unified processing on the source data and the data content to obtain the target external data to obtain the key-value pair; if the source data is a database source file, then based on a reflection tool, perform unified processing on the source data and the data content to obtain the target external data to obtain the key-value pair.
[0129] In one embodiment, the first execution module is specifically used to: compare the data acquisition time with the model update time of the initial model to determine whether the initial model has failed; if the initial model has failed, then train the initial model using the retained data and the target external data to obtain the target model, wherein the data dimensions of the retained data and the target external data are the same.
[0130] In one embodiment, such as Figure 7 As shown, another deviation monitoring device 700 is provided. In addition to the modules included in the deviation monitoring device 600, the deviation monitoring device 700 also includes a second execution module 604. The second execution module 604 is used to determine the initial model as the target model if the initial model has not failed.
[0131] Each module in the aforementioned deviation monitoring device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.
[0132] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network. When executed by the processor, the computer program implements a deviation monitoring method.
[0133] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:
[0134] Obtain key-value pairs of target external data to be monitored for deviation. This target external data is data generated during business operations. Determine an initial model from multiple candidate models based on preset key-value rules and the key-value pairs. Determine a target model based on the attribute information of the key-value pairs and the initial model. Obtain the deviation of the target external data based on the target external data, the key-value pairs, and the target model.
[0135] In one embodiment, the processor, when executing a computer program, performs the following steps to obtain the key-value pair of the target external data to be monitored for deviation: determining the target external data from an external database according to preset filtering conditions, wherein the filtering conditions are related to at least one of deviation monitoring time period and business type; and performing unified processing on the target external data to obtain the key-value pair.
[0136] In one embodiment, the target external data is uniformized to obtain the key-value pair. When the processor executes the computer program, it performs the following steps: parsing the source data corresponding to the target external data to obtain the data content of the target external data; and uniformizing the target external data according to the source data and the data content to obtain the key-value pair.
[0137] In one embodiment, the processor performs unified processing on the target external data based on the source data and the data content to obtain the key-value pair. When executing the computer program, the processor performs the following steps: if the source data is a JSON file, the source data and the data content are unified processing on the target external data based on a serialization tool to obtain the key-value pair; if the source data is a database source file, the source data and the data content are unified processing on the target external data based on a reflection tool to obtain the key-value pair.
[0138] In one embodiment, the attribute information includes the data acquisition time. The target model is determined based on the attribute information of the key-value pair and the initial model. When the processor executes the computer program, it performs the following steps: comparing the data acquisition time with the model update time of the initial model to determine whether the initial model is invalid; if the initial model is invalid, the initial model is trained using the retained data and the target external data to obtain the target model, wherein the data dimensions of the retained data and the target external data are the same.
[0139] In one embodiment, the processor executes the following steps when running a computer program: if the initial model is not invalid, then the initial model is determined to be the target model.
[0140] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:
[0141] Obtain key-value pairs of target external data to be monitored for deviation. This target external data is data generated during business operations. Determine an initial model from multiple candidate models based on preset key-value rules and the key-value pairs. Determine a target model based on the attribute information of the key-value pairs and the initial model. Obtain the deviation of the target external data based on the target external data, the key-value pairs, and the target model.
[0142] In one embodiment, the key-value pair for acquiring the target external data to be monitored for deviation is executed by a computer program by a processor in the following steps: determining the target external data from an external database according to preset filtering conditions, wherein the filtering conditions are related to at least one of deviation monitoring time period and business type; and performing unified processing on the target external data to obtain the key-value pair.
[0143] In one embodiment, the target external data is uniformized to obtain the key-value pair. When the computer program is executed by the processor, it performs the following steps: parsing the source data corresponding to the target external data to obtain the data content of the target external data; and uniformizing the target external data according to the source data and the data content to obtain the key-value pair.
[0144] In one embodiment, the target external data is uniformly processed based on the source data and the data content to obtain the key-value pair. When the computer program is executed by the processor, the following steps are implemented: if the source data is a JSON file, the source data and the data content are uniformly processed based on a serialization tool to obtain the key-value pair; if the source data is a database source file, the source data and the data content are uniformly processed based on a reflection tool to obtain the key-value pair.
[0145] In one embodiment, the attribute information includes the data acquisition time. The target model is determined based on the attribute information of the key-value pair and the initial model. When the computer program is executed by the processor, the following steps are implemented: comparing the data acquisition time with the model update time of the initial model to determine whether the initial model has failed; if the initial model has failed, then training the initial model using the retained data and the target external data to obtain the target model, wherein the data dimensions of the retained data and the target external data are the same.
[0146] In one embodiment, when the computer program is executed by the processor, it performs the following steps: if the initial model is not invalid, then the initial model is determined to be the target model.
[0147] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:
[0148] Obtain key-value pairs of target external data to be monitored for deviation. This target external data is data generated during business operations. Determine an initial model from multiple candidate models based on preset key-value rules and the key-value pairs. Determine a target model based on the attribute information of the key-value pairs and the initial model. Obtain the deviation of the target external data based on the target external data, the key-value pairs, and the target model.
[0149] In one embodiment, the key-value pair for acquiring the target external data to be monitored for deviation is executed by a computer program by a processor in the following steps: determining the target external data from an external database according to preset filtering conditions, wherein the filtering conditions are related to at least one of deviation monitoring time period and business type; and performing unified processing on the target external data to obtain the key-value pair.
[0150] In one embodiment, the target external data is uniformized to obtain the key-value pair. When the computer program is executed by the processor, it performs the following steps: parsing the source data corresponding to the target external data to obtain the data content of the target external data; and uniformizing the target external data according to the source data and the data content to obtain the key-value pair.
[0151] In one embodiment, the target external data is uniformly processed based on the source data and the data content to obtain the key-value pair. When the computer program is executed by the processor, the following steps are implemented: if the source data is a JSON file, the source data and the data content are uniformly processed based on a serialization tool to obtain the key-value pair; if the source data is a database source file, the source data and the data content are uniformly processed based on a reflection tool to obtain the key-value pair.
[0152] In one embodiment, the attribute information includes the data acquisition time. The target model is determined based on the attribute information of the key-value pair and the initial model. When the computer program is executed by the processor, the following steps are implemented: comparing the data acquisition time with the model update time of the initial model to determine whether the initial model has failed; if the initial model has failed, then training the initial model using the retained data and the target external data to obtain the target model, wherein the data dimensions of the retained data and the target external data are the same.
[0153] In one embodiment, when the computer program is executed by the processor, it performs the following steps: if the initial model is not invalid, then the initial model is determined to be the target model.
[0154] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.
[0155] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.
[0156] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0157] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.
Claims
1. A deviation monitoring method, characterized in that, The method includes: The target external data to be monitored for deviation is determined from an external database according to preset filtering conditions. The filtering conditions are related to at least one of the deviation monitoring time period and business type. The target external data is data generated during business operation. The target external data is processed to obtain key-value pairs; An initial model is determined from multiple candidate models based on preset key-value rules and the key-value pairs; Determining the target model based on the attribute information of the key-value pairs and the initial model, wherein the attribute information includes data acquisition time, and the determination of the target model based on the attribute information of the key-value pairs and the initial model includes: By comparing the data acquisition time with the model update time of the initial model, it can be determined whether the initial model has become invalid. If the initial model fails, the initial model is trained using the retained data and the target external data to obtain the target model, wherein the data dimensions of the retained data and the target external data are the same; if the initial model does not fail, the initial model is determined to be the target model. The deviation of the target external data is obtained based on the target external data, the key-value pairs, and the target model.
2. The method according to claim 1, characterized in that, The process of unifying the target external data to obtain the key-value pairs includes: Data parsing is performed on the source data corresponding to the target external data to obtain the data content of the target external data; The target external data is uniformly processed based on the source data and the data content to obtain the key-value pairs.
3. The method according to claim 2, characterized in that, The step of unifying the target external data based on the source data and the data content to obtain the key-value pairs includes: If the source data is a JSON file, then based on a serialization tool, the source data and the data content are used to unify the target external data to obtain the key-value pairs; If the source data is a database source file, then based on the reflection tool, the source data and the data content are used to unify the target external data to obtain the key-value pairs.
4. A deviation monitoring device, characterized in that, The device includes: The acquisition module is used to determine the target external data to be monitored from an external database according to preset filtering conditions. The filtering conditions are related to at least one of the deviation monitoring time period and business type. The target external data is data generated during business operation. The target external data is processed to obtain key-value pairs. The determination module is used to determine the initial model from multiple candidate models according to preset key-value rules and the key-value pairs. The first execution module is used to determine the target model based on the attribute information of the key-value pair and the initial model, wherein the attribute information includes the data acquisition time, and to obtain the deviation of the target external data based on the target external data, the key-value pair and the target model; The first execution module is specifically used to compare the data acquisition time with the model update time of the initial model to determine whether the initial model has failed. If the initial model fails, the initial model is trained using the retained data and the target external data to obtain the target model, wherein the data dimensions of the retained data and the target external data are the same; if the initial model does not fail, the initial model is determined to be the target model.
5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 3.
6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 3.
7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 3.