Data processing method and device, equipment, medium and program product

Abstract

The invention provides a data processing method which can be applied to the technical field of artificial intelligence. The data processing method comprises the steps of obtaining attribute information of a to-be-verified target object in response to occurrence of a target event; determining to-be-acquired target data according to the attribute information and generating a data acquisition request; wherein the data acquisition request is used for accessing a specified interface table; acquiring target data from an interface table based on the data acquisition request; wherein the data stored in the interface table is data obtained by processing original data generated by operation of the target object; and verifying the target object based on the target data to obtain a verification result of the target object. The invention further provides a data processing device, equipment, a storage medium and a program product.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and more specifically to a data processing method, device, equipment, medium and program product. Background Art

[0002] With the continuous development of artificial intelligence technology, models have been widely used in many fields, providing convenience for people's lives. For models that have been put into production, their performance may change over time. Therefore, it is necessary to verify the models from time to time to ensure that the models can accurately predict and explain the actual data.

[0003] However, the efficiency of model verification is often affected by computing resources. During the model verification process, original data is often required to calculate verification indicators. Currently, most solutions only improve the efficiency of model verification by increasing computing resources, which is costly and has limited effect on improving the efficiency of some verification methods that require a large amount of original data. Summary of the invention

[0004] In view of the above problems, the present disclosure provides a data processing method, apparatus, device, medium and program product for improving data processing efficiency.

[0005] According to the first aspect of the present disclosure, a data processing method is provided, including: in response to the occurrence of a target event, obtaining attribute information of a target object to be verified; determining the target data to be obtained based on the attribute information and generating a data acquisition request; wherein the data acquisition request is used to access a specified interface table; obtaining the target data from the interface table based on the data acquisition request; wherein the data stored in the interface table is data obtained after processing the original data generated by the operation of the target object; and verifying the target object based on the target data to obtain a verification result of the target object.

[0006] According to an embodiment of the present disclosure, the method also includes: processing the original data based on a specific dimension to obtain first data; wherein the first data is used to reflect the operating status of the target object under the specific dimension, and the data volume of the first data is smaller than the data volume of the original data; and storing the first data in a corresponding interface table according to a specified format.

[0007] According to an embodiment of the present disclosure, in response to the occurrence of a target event, attribute information of a target object to be verified is obtained, including: in response to receiving a user instruction or a preset script being triggered, determining the target object to be verified; obtaining attribute information of the target object to be verified, wherein the attribute information includes an identifier and type of the target object and verification indicators for verifying the target object.

[0008] According to an embodiment of the present disclosure, target data to be acquired is determined based on attribute information and a data acquisition request is generated, including: determining a target data type for calculating a verification indicator based on a verification indicator; determining a data filtering condition based on a target event to obtain the target data to be acquired; wherein different target events correspond to different filtering conditions.

[0009] According to an embodiment of the present disclosure, target data to be acquired is determined based on attribute information and a data acquisition request is generated, including: determining access information of an interface table to be accessed based on the target data; constructing a data acquisition request based on the access information; wherein the data acquisition request includes relevant information of the target data and authentication information.

[0010] According to an embodiment of the present disclosure, target data is obtained from an interface table based on a data acquisition request, including: sending a data acquisition request to the interface table; receiving and parsing response information returned by the interface table to obtain the target data; wherein the response information is generated after the interface table verifies the data acquisition request.

[0011] According to an embodiment of the present disclosure, a target object is verified based on target data to obtain a verification result of the target object, including: determining a calculation formula for calculating a verification index of the target object based on attribute information of the target object; substituting the target data into the calculation formula to obtain a calculation result; and determining the verification result of the target object based on the calculation result.

[0012] The second aspect of the present disclosure provides a data processing device, including: a first acquisition module, used to acquire attribute information of a target object to be verified in response to the occurrence of a target event; a generation module, used to determine the target data to be acquired according to the attribute information and generate a data acquisition request; wherein the data acquisition request is used to access a specified interface table; a second acquisition module, used to acquire the target data from the interface table based on the data acquisition request; wherein the data stored in the interface table is data obtained after processing the original data generated by the operation of the target object; and a verification module, used to verify the target object based on the target data to obtain a verification result of the target object.

[0013] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0014] The fourth aspect of the present disclosure further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the above computer program or instructions are executed by a processor.

[0015] The fifth aspect of the present disclosure further provides a computer program product, including a computer program or instructions, which implement the steps of the above method when the above computer program or instructions are executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0017] Figure 1 The application scenario diagram of the data processing method, apparatus, device, medium and program product according to the embodiments of the present disclosure is schematically shown;

[0018] Figure 2 A flowchart schematically shows a data processing method according to an embodiment of the present disclosure;

[0019] Figure 3 Schematically shows a flow chart of storing data in an interface table according to an embodiment of the present disclosure;

[0020] Figure 4 A flowchart schematically illustrates a method of acquiring attribute information of a target object to be verified in response to a target event occurring according to an embodiment of the present disclosure;

[0021] Figure 5 A flowchart schematically illustrates determining target data to be acquired based on attribute information and generating a data acquisition request according to an embodiment of the present disclosure;

[0022] Figure 6 A flowchart of generating a data acquisition request according to target data according to an embodiment of the present disclosure is schematically shown;

[0023] Figure 7 A flowchart of obtaining target data from an interface table based on a data acquisition request according to an embodiment of the present disclosure is schematically shown;

[0024] Figure 8 A flowchart schematically illustrates a method of verifying a target object based on target data to obtain a verification result of the target object according to an embodiment of the present disclosure;

[0025] Fig. 9 A structural block diagram schematically shows a data processing device according to an embodiment of the present disclosure; and

[0026] Fig.10 A block diagram of an electronic device suitable for implementing a data processing method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0027] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0028] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0029] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0030] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0031] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0032] In the technical solution of the present disclosure, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0033] In the scenario of using personal information for automated decision-making, the methods, devices, and systems provided by the embodiments of the present disclosure provide users with corresponding operation portals for users to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered. The expression "automated decision-making" here refers to the activity of automatically analyzing and evaluating a person's behavioral habits, interests and hobbies, or economic, health, credit status, etc. through computer programs, and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by people who specialize in a certain field, have specialized experience, knowledge and skills, and have reached a certain level of professionalism.

[0034] An embodiment of the present disclosure provides a data processing method, which includes: in response to the occurrence of a target event, obtaining attribute information of a target object to be verified; determining the target data to be obtained based on the attribute information and generating a data acquisition request; wherein the data acquisition request is used to access a specified interface table, and the interface table is used to store processed original data; based on the data acquisition request, obtaining the target data from the interface table; and verifying the target object based on the target data to obtain a verification result of the target object.

[0035] Figure 1 The application scenario diagram of the data processing method, apparatus, device, medium and program product according to the embodiments of the present disclosure is schematically shown.

[0036] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0037] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0038] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0039] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0040] It should be noted that the data processing method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the data processing device provided in the embodiment of the present disclosure can generally be set in the server 105. The data processing method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the data processing device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0041] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0042] The following will be based on Figure 1 The scene described by Figure 2~Figure 6 The data processing method of the disclosed embodiment is described in detail.

[0043] Figure 2 The flowchart of the data processing method according to the embodiment of the present disclosure is schematically shown.

[0044] like Figure 2 As shown, the data processing method of this embodiment includes operations S210 to S240.

[0045] In operation S210 , in response to a target event occurring, attribute information of a target object to be verified is acquired.

[0046] In some embodiments, when a target event occurs, the target object to be verified is determined based on the information of the target event and the attribute information of the target object is obtained. Different objects may correspond to different target events.

[0047] In the specific implementation process, the verification frequency of the model depends on multiple factors such as the model type, application scenario, data change speed, and business needs. For example, models with high real-time requirements (such as online recommendation systems, real-time risk control models, etc.) have a relatively high verification frequency to ensure that the model can adapt to data changes in a timely manner. For models with high stability requirements (such as long-term investment strategy models, long-term weather forecast models, etc.), the verification frequency can be appropriately reduced. Changes in business needs may also cause changes in the verification frequency of the model. For example, when a new product is launched, the model may need additional verification and evaluation. During business peak periods (such as during holiday promotions), the model also needs to be verified more frequently to ensure its stability. Therefore, it is necessary to determine the model to be verified based on the target event.

[0048] In operation S220, target data to be acquired is determined according to the attribute information and a data acquisition request is generated; wherein the data acquisition request is used to access a specified interface table, and the interface table is used to store processed original data.

[0049] In some embodiments, different models have different verification indicators, and the calculation logic and source data of different verification indicators are different. Therefore, it is necessary to determine the target data used to verify the target object based on the attribute information, and generate a corresponding data acquisition request based on the information of the target data. Among them, the addresses and formats of different data storage are different, and correspondingly, the data acquisition requests that need to be constructed to obtain these data will also be different.

[0050] After determining the target object to be acquired, the target data and related information of the target data, such as data type, format, precision, and which interface table of which database the target data is stored in, can be determined based on the identifier, type, and verification index of the target object, for example. A corresponding data acquisition request is generated based on the related information of the target data.

[0051] In the specific implementation process, the target data obtained by the present disclosure is not the original data generated when the target object is running, but the data stored in the interface table after processing the original data. The amount of processed data is smaller, which helps to save computing resources.

[0052] In operation S230, target data is acquired from the interface table based on the data acquisition request.

[0053] In some embodiments, a data acquisition request is sent to a specified interface table to acquire target data from the specified interface table. In this target data acquisition request process, the data in the interface table is also screened, that is, it is not necessary to acquire all the data in the specified interface table, but only to acquire part of the data related to the target event.

[0054] For example, different observation periods correspond to different data acquisition conditions. When the observation period is 1, the current time is 2024-08-31, and the base period is 2020-06-30, the target data obtained are the data corresponding to 2024-08-31 and the data corresponding to 2020-06-30.

[0055] The interface table is a bridge for data exchange between the external system and the database. Asynchronous processing through the JMS queue can effectively improve the flexibility and efficiency of data exchange. In addition, the interface table in the present disclosure is used to store data obtained after processing the original data. The verification index calculation based on the processed data can effectively improve the calculation efficiency of the target object verification index. The external system can process data in batches through the interface table without frequent access to the database, effectively reducing the number and frequency of database accesses, and reducing the pressure and response time of the database.

[0056] In operation S240 , the target object is verified based on the target data to obtain a verification result of the target object.

[0057] In some embodiments, the verification indicators used to verify the model may include, for example, the Population Stability Index (PSI) indicator, the Kolmogorov-Smirnov (KS) indicator, etc., wherein different verification indicators correspond to different calculation methods. By substituting the target data into the corresponding verification indicator calculation formula, the calculation result of the verification indicator is obtained and the verification result of the target object is determined based on the calculation result of the verification indicator.

[0058] The data processing method provided by the present invention improves the query efficiency of target data by processing the original data and storing it in the corresponding interface table. Pre-processing and storing the original data can effectively avoid redundancy and duplication of the original data, help save data processing time and storage space, save computing resources, and improve the efficiency of model verification.

[0059] Figure 3 The flowchart of storing data in the interface table according to an embodiment of the present disclosure is schematically shown.

[0060] like Figure 3 As shown, storing data in the interface table of this embodiment includes operations S310 to S320.

[0061] In operation S310, the original data is processed based on a specific dimension to obtain first data; wherein the first data is used to reflect the operating status of the target object under the specific dimension, and the data volume of the first data is smaller than the data volume of the original data.

[0062] In some implementations, the raw data is the data generated during the operation of the target object, and the raw data is usually stored in the database corresponding to the target object. Taking the target object as a customer evaluation model as an example, the raw data is the evaluation result of each customer. Specific dimensions may include, for example, time dimension, space dimension, attribute, behavior, etc. According to business needs, select dimensional indicators that can reflect the operating status of the target object under specific dimensions, and process the raw data according to the specific dimensions.

[0063] In a specific embodiment, processing the raw data based on a specific dimension may include, for example, data cleaning and data conversion of the raw data, removing duplicate, erroneous, and invalid data, ensuring data accuracy and consistency, and converting the raw data into a format suitable for processing, such as converting a string into a data type, or unifying different time formats into a unified time standard.

[0064] And, processing the processed raw data based on a specific dimension to obtain the first data includes: performing an aggregation operation on the raw data according to the selected dimension indicator. For example, grouping the data according to the time dimension, and calculating the average value, total, maximum value, minimum value and other statistics of each time period. According to business needs, necessary calculation operations are performed to obtain the first data that can reflect the operating status of the target object under a specific dimension. These calculations may include proportion calculation, trend analysis, anomaly detection, etc. The first data is obtained by performing an aggregation operation and / or a calculation operation on the raw data.

[0065] Taking the customer evaluation model A_01 as an example, assuming that the specific dimension is the time dimension, the raw data is processed in units of months to obtain the first data, for example, it may include: when the monthly time is 20240731, the raw data in the interval [20240701, 20240731] is processed and integrated to obtain the first data under the monthly time. The raw data generated during the operation of the A_01 model may be, for example, a customer score. Assuming that a customer score is 201, the score level corresponding to the customer can be determined to be 3 according to the preset information processing rules, and the number of customers with a score level of 3 in the interface table is accumulated and added by 1.

[0066] In operation S320, the first data is stored in a corresponding interface table according to a specified format.

[0067] In some embodiments, the interface table can be, for example, a temporary table for data transfer and storage in the database corresponding to each target object. The structure of the interface table, including field name, data type, field length, etc., can be designed according to the business requirements and data characteristics of the target object. Ensure that the interface table can store the processed first data and facilitate subsequent data access and use. Insert the processed first data into the corresponding interface table in the specified format. This includes writing the data to the correct field of the table and ensuring that the order and format of the data are consistent with the design of the interface table. Format the first data, convert it into a format suitable for storage and use, and store the formatted first data in the corresponding interface table for subsequent data processing and calling.

[0068] Compared with directly accessing the original data in the database, directly obtaining the processed first data from the interface table can effectively improve the data access rate and acquisition efficiency, and because the amount of the first data is smaller than the amount of the original data, the subsequent data processing process based on the first data can also effectively improve the data processing efficiency. In addition, since the data in the interface table is processed data, it will not directly reveal the original sensitive information, which helps to improve confidentiality and security and prevent the risk of data leakage.

[0069] Figure 4 The flowchart schematically shows a method of acquiring attribute information of a target object to be verified in response to occurrence of a target event according to an embodiment of the present disclosure.

[0070] like Figure 4 As shown, in response to the occurrence of a target event, obtaining the attribute information of the target object to be verified in this embodiment includes operations S410 to S420.

[0071] In operation S410 , in response to receiving a user instruction or a preset script being triggered, a target object to be verified is determined.

[0072] In operation S420, attribute information of the target object to be verified is obtained, wherein the attribute information includes an identifier and a type of the target object and a verification index for verifying the target object.

[0073] In some embodiments, in response to receiving a trigger signal of a user instruction or a preset script, the received instruction or script content is parsed to obtain the target object to be verified and obtain the attribute information of the target object. Among them, the preset script is used to enable the server to perform the verification task regularly. The target object can be a model to be verified, or it can be a specific verification indicator of a model to be verified. The attribute information may include, for example, an identifier and type of the target object and a verification indicator for verifying the target object. For example, for model A_01, its corresponding identifier is, for example, 0101. The identifier can be used to determine a database for storing the original data of model A_01 in multiple databases, and can also be used to determine the original data of model A_01 in a multivariate database. The type of model may include, for example, an evaluation model, a prediction model, a classification model, etc. The verification indicator may include, for example, a PSI indicator, a KS indicator, etc. Different models correspond to different verification indicators, and one model may also correspond to multiple verification indicators.

[0074] The data processing method provided by the embodiment of the present disclosure realizes the automation of the target object determination process through preset scripts and user-specified triggers, reduces manual intervention, and effectively improves the efficiency of model verification.

[0075] Figure 5 The flowchart of determining target data to be acquired according to attribute information and generating a data acquisition request according to an embodiment of the present disclosure is schematically shown.

[0076] like Figure 5 As shown, the process of determining target data to be acquired according to attribute information and generating a data acquisition request in this embodiment includes operations S510 to S520.

[0077] In operation S510, a target data type for calculating a verification index is determined based on the verification index.

[0078] In operation S520, a data filtering condition is determined based on the target event to obtain the target data to be acquired; wherein different target events correspond to different filtering conditions.

[0079] In some embodiments, different verification indicators correspond to different data. After clarifying the verification indicators required for verifying the target object, the target data type used to calculate the verification indicators needs to be determined based on the calculation method of the verification indicators. For example, if the verification indicator is used to calculate the accuracy of a classification problem, the target data should include two types: actual labels and predicted labels.

[0080] The target event is usually a user-triggered verification operation (i.e., user instruction) or a system timed verification task (preset script). The data screening conditions corresponding to different target events may be different. The screening conditions may include, for example, time range, user attributes, system status, etc. For example, the data of the corresponding time can be obtained according to the observation period indicated in the target event. Taking the verification indicator as the PSI indicator as an example, when the observation period is 1 and the current time is 2024-08-31, only the data of the current time and the base period data are used to calculate the verification indicator. The screening conditions at this time are: data time is 2024-08-31 and data of the base period. Among them, the base period refers to the data of the period used as a comparison standard. When the observation period is 12 and the current time is 2024-08-31, the calculation of the verification indicator requires the use of 12 periods of data from 2023-08-31 to 2024-08-31 and the base period data.

[0081] By determining the target data type based on the verification indicator and specifying the filtering conditions based on the target event, it is possible to ensure that the data used can accurately reflect the characteristics of the target event, exclude irrelevant or redundant data, quickly locate the target data, enhance the validity of the data, and verify the accuracy of the indicator calculation.

[0082] Figure 6 The flowchart of generating a data acquisition request according to target data according to an embodiment of the present disclosure is schematically shown.

[0083] like Figure 6 As shown, the step of generating a data acquisition request according to target data in this embodiment includes operations S610 to S620.

[0084] In operation S610, access information of an interface table to be accessed is determined based on target data.

[0085] In operation S620, a data acquisition request is constructed according to the access information; wherein the data acquisition request includes relevant information of the target data and authentication information.

[0086] In some embodiments, the interface table to be accessed is determined based on the target data, wherein the interface table stores data processed from the original data to be interacted with the external system. The access information of the interface table may include, for example, the name of the interface table, field information, access rights, etc. A data acquisition request is constructed based on the access information, wherein the data acquisition request generally includes relevant information of the target data (such as field name, data type, etc.) and necessary parameters for accessing the interface table (such as query conditions, paging information, etc.).

[0087] Construct a complete request URL (Uniform Resource Locator) based on the access information of the interface table, and set the request header according to the access information requirements, such as Content-Type, Authorization, etc. For methods such as POST that require sending a request body, prepare the request body data according to the interface requirements. In addition, to ensure data security, the data acquisition request should also include authentication information, which can include user name, password, token, digital signature, etc., to verify the identity and authority of the requester.

[0088] The disclosed embodiment constructs corresponding data access requests based on target information and access information, which has high flexibility and meets the access requirements of different business scenarios. By clarifying the interface table and access information, the target data can be quickly located, reducing the delay and overhead of data access. By adding authentication information, it can be ensured that only authorized users can access the target data, preventing data leakage and illegal access.

[0089] Figure 7 The flowchart of acquiring target data from the interface table based on a data acquisition request according to an embodiment of the present disclosure is schematically shown.

[0090] like Figure 7 As shown, the method of acquiring target data from the interface table based on a data acquisition request in this embodiment includes operations S710 to S720.

[0091] In operation S710, a data acquisition request is sent to an interface table.

[0092] In operation S720, response information returned by the interface table is received and parsed to obtain target data, wherein the response information is generated after the interface table verifies the data acquisition request.

[0093] In some embodiments, the constructed data acquisition request is sent to the specified interface table based on the specified protocol. For example, the request is sent to the URL address of the interface table via the HTTP protocol. After receiving the request, the interface table will verify it, including verifying whether the request method, request parameters, request header, etc. meet the requirements. After the interface table is verified, a response message will be generated and returned to the requester. The requester receives the response message and parses the response content according to the interface document to obtain the target data. The response content usually includes parts such as a status code, a message header, a blank line, and a response body. Among them, the response body may contain data in formats such as JSON and XML, and may need to be processed using corresponding parsing methods.

[0094] The disclosed embodiment constructs and sends a data acquisition request in a programmatic manner, which can realize automatic processing of target data and improve the efficiency and flexibility of acquiring target data.

[0095] Figure 8 The flowchart schematically shows a method of verifying a target object based on target data to obtain a verification result of the target object according to an embodiment of the present disclosure.

[0096] like Figure 8 As shown, in this embodiment, verifying the target object based on the target data to obtain the verification result of the target object includes operations S810 to S830.

[0097] In operation S810, a calculation formula for calculating a verification index of the target object is determined based on attribute information of the target object.

[0098] In operation S820, the target is substituted into a calculation formula to obtain a calculation result.

[0099] In operation S830, a verification result of the target object is determined based on the calculation result.

[0100] In some embodiments, the attribute information of the model may include, for example, the type of model (such as classification model, regression model, target detection model, etc.), input features, output prediction values, and the algorithm used, etc. Such attribute information will directly affect the selection of subsequent verification indicators and the determination of calculation formulas.

[0101] According to the model's attribute information and verification requirements, select appropriate verification indicators and determine the calculation formula of the selected verification indicators. For example, for classification models, commonly used verification indicators include accuracy, precision, recall, F1 score, etc.; for regression models, commonly used verification indicators include mean square error (MSE), root mean square error (RMSE), etc.; for object detection models, commonly used verification indicators include mean average precision (mAP), intersection over union (IoU), etc.

[0102] The target data is brought into the calculation formula of the validation index for calculation, and a series of validation index calculation results are obtained, which are used to evaluate the performance of the model. The calculation results of the validation index are analyzed to evaluate the performance of the model. For example, for a classification model, if the accuracy is high, it means that the model performs well on the classification task; for a regression model, if the RMSE is small, it means that the error between the model's predicted value and the true value is small; for a credit scoring model, if the PSI value is small, it means that the output score distribution of the model remains relatively stable in different time periods or sample sets, and the model performance is stable.

[0103] The data processing method provided in the embodiment of the present disclosure can effectively improve the objectivity and standardization of model evaluation by bringing the target data into the verification index calculation formula and evaluating the model performance according to the calculation results, and provide a strong reference for subsequent model tuning and improvement. And because the data brought into the calculation formula in the present disclosure is processed data, it can effectively improve the calculation efficiency of the model verification index under a long observation period and reduce resource consumption.

[0104] Based on the above data processing method, the present disclosure also provides a data processing device. Fig. 9 The device is described in detail.

[0105] Fig. 9 The structure block diagram of a data processing device according to an embodiment of the present disclosure is schematically shown.

[0106] like Fig. 9 As shown, the data processing device 900 of this embodiment includes a first acquisition module 910 , a generation module 920 , a second acquisition module 930 and a second acquisition module 940 .

[0107] The first acquisition module 910 is used to obtain the attribute information of the target object to be verified in response to the occurrence of the target event. In one embodiment, the first acquisition module 910 can be used to perform the operation S210 described above, which will not be repeated here.

[0108] The generation module 920 is used to determine the target data to be acquired according to the attribute information and generate a data acquisition request; wherein the data acquisition request is used to access the specified interface table. In one embodiment, the generation module 920 can be used to perform the operation S220 described above, which will not be repeated here.

[0109] The second acquisition module 930 is used to acquire target data from the interface table based on the data acquisition request; wherein the data stored in the interface table is data obtained after processing the original data generated by the operation of the target object. In one embodiment, the second acquisition module 930 can be used to perform the operation S230 described above, which will not be repeated here.

[0110] The verification module 940 is used to verify the target object based on the target data to obtain a verification result of the target object. In one embodiment, the verification module 940 can be used to perform the operation S240 described above, which will not be described in detail here.

[0111] According to an embodiment of the present disclosure, any multiple modules of the first acquisition module 910, the generation module 920, the second acquisition module 930, and the second acquisition module 940 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the first acquisition module 910, the generation module 920, the second acquisition module 930, and the second acquisition module 940 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, at least one of the first acquisition module 910 , the generation module 920 , the second acquisition module 930 , and the second acquisition module 940 may be at least partially implemented as a computer program module, and when the computer program module is executed, a corresponding function may be performed.

[0112] Fig.10 A block diagram of an electronic device suitable for implementing a data processing method according to an embodiment of the present disclosure is schematically shown.

[0113] like Fig.10 As shown, the electronic device 1000 according to an embodiment of the present disclosure includes a processor 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage part 1008 to a random access memory (RAM) 1003. The processor 1001 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 1001 may also include an onboard memory for caching purposes. The processor 1001 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0114] In RAM 1003, various programs and data required for the operation of electronic device 1000 are stored. Processor 1001, ROM 1002 and RAM 1003 are connected to each other via bus 1004. Processor 1001 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 1002 and / or RAM 1003. It should be noted that the program can also be stored in one or more memories other than ROM 1002 and RAM 1003. Processor 1001 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0115] According to an embodiment of the present disclosure, the electronic device 1000 may further include an input / output (I / O) interface 1005, which is also connected to the bus 1004. The electronic device 1000 may further include one or more of the following components connected to the input / output (I / O) interface 1005: an input portion 1006 including a keyboard, a mouse, etc.; an output portion 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 1008 including a hard disk, etc.; and a communication portion 1009 including a network interface card such as a LAN card, a modem, etc. The communication portion 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the input / output (I / O) interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage portion 1008 as needed.

[0116] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0117] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 1002 and / or RAM 1003 described above and / or one or more memories other than ROM 1002 and RAM 1003.

[0118] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the data processing method provided by the embodiment of the present disclosure.

[0119] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 1001. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0120] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 1009, and / or installed from the removable medium 1011. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0121] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1009, and / or installed from the removable medium 1011. When the computer program is executed by the processor 1001, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0122] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0123] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0124] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0125] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A data processing method, characterized in that: The method comprises: In response to the occurrence of a target event, obtaining attribute information of a target object to be verified; Determine the target data to be acquired according to the attribute information and generate a data acquisition request; wherein the data acquisition request is used to access a specified interface table; Acquire target data from the interface table based on the data acquisition request; wherein the data stored in the interface table is data obtained by processing the original data generated by the operation of the target object; The target object is verified based on the target data to obtain a verification result of the target object.

2. The data processing method according to claim 1, characterized in that: The method further comprises: Processing the original data based on a specific dimension to obtain first data; wherein the first data is used to reflect the operating status of the target object under the specific dimension, and the data volume of the first data is smaller than the data volume of the original data; The first data is stored in the corresponding interface table according to a specified format.

3. The data processing method according to claim 1, characterized in that: The step of obtaining the attribute information of the target object to be verified in response to the occurrence of the target event includes: In response to receiving a user instruction or a preset script being triggered, determining a target object to be verified; Acquire attribute information of the target object to be verified, wherein the attribute information includes an identifier and a type of the target object and a verification indicator for verifying the target object.

4. The data processing method according to claim 1, characterized in that: The step of determining the target data to be acquired according to the attribute information and generating a data acquisition request includes: Determining a target data type for calculating the verification indicator based on the verification indicator; Determine a data screening condition based on the target event to obtain the target data to be acquired; Among them, different target events correspond to different screening conditions.

5. The data processing method according to claim 1 or 4, characterized in that: The step of determining the target data to be acquired according to the attribute information and generating a data acquisition request includes: Determine access information of an interface table to be accessed based on the target data; A data acquisition request is constructed according to the access information; wherein the data acquisition request includes relevant information of the target data and authentication information.

6. The data processing method according to claim 5, characterized in that: The acquiring target data from the interface table based on the data acquisition request includes: Sending the data acquisition request to the interface table; Receive and parse the response information returned by the interface table to obtain the target data; The response information is generated after the interface table verifies the data acquisition request.

7. The data processing method according to claim 1, characterized in that: The verifying the target object based on the target data to obtain a verification result of the target object includes: Based on the attribute information of the target object, determining a calculation formula for calculating the verification index of the target object; Substituting the target data into the calculation formula to obtain a calculation result; A verification result of the target object is determined based on the calculation result.

8. A data processing device, characterized in that: The device comprises: A first acquisition module, used for acquiring attribute information of a target object to be verified in response to the occurrence of a target event; A generating module, used to determine the target data to be acquired according to the attribute information and generate a data acquisition request; wherein the data acquisition request is used to access a specified interface table; A second acquisition module is used to acquire target data from the interface table based on the data acquisition request; wherein the data stored in the interface table is data obtained by processing the original data generated by the operation of the target object; and The verification module is used to verify the target object based on the target data to obtain a verification result of the target object.

9. An electronic device, comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images