Data model optimization method and device, computer equipment and storage medium

Abstract

The invention discloses a data model optimization method and device, computer equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: firstly, performing prediction through a data model to obtain prediction data and a prediction label thereof, comparing the prediction label with a real labeling label, and correcting the prediction label; and then, constructing an iterative training sample set by using the input data carrying the annotation label and the prediction data carrying the correction label. Thirdly, updating optimization parameters of the data model according to a set target value; and iteratively training the data model by continuously using the iterative training sample set until the model parameters reach the target optimization parameters, thereby outputting the updated data model. The invention further relates to the technical field of block chains, and the input data is stored on the block chain node. According to the method, model prediction and data label correction are combined, and the prediction performance and generalization ability of the model are improved through repeated iterative training.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to a data model optimization method, device, computer equipment and storage medium. Background Art

[0002] After the data model is successfully deployed, ensuring its stable operation and continuously optimizing performance becomes a key task. Although the model performance test has been carried out in the verification test of the development stage, which has guaranteed the stability of the model performance to a certain extent, in the face of complex and changeable actual business scenarios, the difference between development data and real-time production data often leads to deviations in prediction results. This requires that after the model is deployed, it is still necessary to continue to collect and analyze production data, and continuously narrow this deviation through algorithm optimization, parameter adjustment and other means to ensure that the model can run accurately and efficiently in the production environment and bring real value to the enterprise.

[0003] Therefore, building an efficient model effect feedback mechanism to objectively evaluate model performance and promote model iterative training has become an urgent problem to be solved. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a data model optimization method, device, computer equipment and storage medium to build an efficient model effect feedback mechanism to objectively evaluate model performance and promote model iterative training.

[0005] In order to solve the above technical problems, the present application embodiment provides a method for optimizing a data model, which adopts the following technical solution:

[0006] A method for optimizing a data model, comprising:

[0007] Obtain input data, input the input data into the data model, and obtain output prediction data of the data model;

[0008] Get the annotation labels of the input data, and get the predicted labels of the predicted data;

[0009] Correct the predicted label according to the annotated label to obtain the corrected label;

[0010] Using input data carrying the annotated label as first sample data and using predicted data carrying the corrected label as second sample data, an iterative training sample set is constructed;

[0011] Update the optimization parameters of the data model according to the set target values ​​to obtain the target optimization parameters;

[0012] The data model is iteratively trained using the iterative training sample set until the model parameters reach the target optimization parameters, and the updated data model is output.

[0013] Furthermore, after the steps of obtaining input data, inputting the input data into the data model, and obtaining the predicted data output by the data model, the method further includes:

[0014] Determine whether the prediction data output by the data model is in the form of a key-value pair, where the body of the prediction data is the key position of the key-value pair, and the prediction label corresponding to the prediction data is the value position of the key-value pair;

[0015] If the prediction data output by the data model is not in the form of a key-value pair, the prediction data output by the data model is obtained by obtaining the production log of the data model.

[0016] Furthermore, the step of obtaining the prediction data output by the data model by obtaining the production log of the data model specifically includes:

[0017] Get the production log fragment generated by the data model's prediction processing of the input data, where the production log is a JSON type log;

[0018] Perform JSON path parsing on the production log fragment to obtain the JSON log path parsing result;

[0019] Match and map the pre-configured log core fields with the JSON log path parsing results, and obtain the data model output prediction data based on the matching results.

[0020] Furthermore, the pre-configured log core fields are matched and mapped with the JSON log path parsing results, and the steps of obtaining the data model output prediction data according to the matching results specifically include:

[0021] Traverse the JSON log path parsing results, match the log core fields with the fields in the JSON log path parsing results, and obtain the matching log fields;

[0022] Find the field value corresponding to the matching log field in the JSON log path parsing result;

[0023] Build field key-value pairs based on matching log fields and field values, and use the field key-value pairs as the data model to output prediction data.

[0024] Furthermore, the predicted label is corrected according to the annotated label to obtain the corrected label, which specifically includes:

[0025] Obtaining an input data segment corresponding to the annotation label, and constructing a first data label combination based on the annotation label and the input data segment;

[0026] Obtaining a predicted data segment corresponding to the predicted label, and constructing a second data label combination based on the predicted label and the predicted data segment;

[0027] Determine whether the first data label combination and the second data label combination are consistent;

[0028] If the first data label combination and the second data label combination are inconsistent, the predicted label is corrected according to the first data label combination to obtain a corrected label.

[0029] Furthermore, the step of correcting the predicted label according to the first data label combination to obtain the corrected label specifically includes:

[0030] Identifying a data segment matching the input data segment in the predicted data to obtain a matching data segment;

[0031] Determine a matching relationship between the input data segment and the annotated label in the first data label combination to obtain a true label matching relationship;

[0032] The matching relationship between the matching data segment and the predicted label is corrected according to the real label matching relationship to complete the correction of the predicted label and obtain the corrected label.

[0033] Further, the data model is iteratively trained using the iterative training sample set until the model parameters reach the target optimization parameters, and the step of outputting the updated data model specifically includes:

[0034] Input the iterative training sample set into the data model to obtain the iterative prediction result;

[0035] Compare the iterative prediction results with the preset standard results, and calculate the iterative prediction error of the data model;

[0036] Use the directional propagation algorithm to propagate the prediction error in the data model;

[0037] The gradient descent algorithm is used to iteratively update the model parameters until the model parameters reach the target optimization parameters, and the updated data model is output.

[0038] In order to solve the above technical problems, the embodiment of the present application further provides a data model optimization device, which adopts the following technical solution:

[0039] A data model optimization device, comprising:

[0040] A data prediction module is used to obtain input data, input the input data into a data model, and obtain output prediction data of the data model;

[0041] The label acquisition module is used to obtain the annotation labels of the input data and the predicted labels of the predicted data;

[0042] A label correction module is used to correct the predicted label according to the annotated label to obtain a corrected label;

[0043] A sample construction module, used to construct an iterative training sample set using input data carrying annotated labels as first sample data and predicted data carrying correction labels as second sample data;

[0044] A parameter setting module is used to update the optimization parameters of the data model according to the set target values ​​to obtain the target optimization parameters;

[0045] The iterative training module is used to iteratively train the data model using the iterative training sample set until the model parameters reach the target optimization parameters, and output the updated data model.

[0046] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0047] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the data model optimization method as described in any one of the above items when executing the computer-readable instructions.

[0048] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0049] A computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the data model optimization method as described in any one of the above.

[0050] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0051] The present application discloses a method, device, computer equipment and storage medium for optimizing a data model, and belongs to the field of artificial intelligence technology. First, the prediction data and its prediction label are obtained by prediction through the data model, and the prediction label is compared with the real annotation label to correct the prediction label. Subsequently, an iterative training sample set is constructed using the input data carrying the annotation label and the prediction data carrying the correction label. Then, the optimization parameters of the data model are updated according to the set target value. The data model is iteratively trained by continuously using the iterative training sample set until the model parameters reach the target optimization parameters, thereby outputting the updated data model. The present application significantly improves the prediction accuracy of the data model by introducing a correction mechanism for the prediction label and constructing an iterative training sample set using the corrected data. Through continuous iterative training, the model parameters are gradually optimized to the target value, and finally a data model with better performance is output, which effectively enhances the generalization ability of the model and improves the accuracy and efficiency of data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0053] Figure 1 An exemplary system architecture diagram is shown in which the present application can be applied;

[0054] Figure 2 A flow chart showing an embodiment of a method for optimizing a data model according to the present application;

[0055] Figure 3 A schematic structural diagram of an embodiment of a data model optimization device according to the present application is shown;

[0056] Figure 4 A schematic structural diagram of an embodiment of a computer device according to the present application is shown. DETAILED DESCRIPTION

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0058] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0059] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0060] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0061] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0062] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0063] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0064] It should be noted that the data model optimization method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the data model optimization device is generally set in the server / terminal device.

[0065] It should be understood that Figure 1 The numbers of terminal devices, networks and servers in the embodiment are merely illustrative. The above system may have any number of terminal devices, networks and servers according to implementation requirements.

[0066] Continue to refer Figure 2 , shows a flow chart of an embodiment of a method for optimizing a data model according to the present application. The method for optimizing a data model comprises the following steps:

[0067] S201, obtaining input data, and inputting the input data into a data model, and obtaining output prediction data of the data model.

[0068] Specifically, the system obtains input data to be processed, inputs the input data into the data model, and obtains output prediction data from the data model. Taking the document recognition scenario as an example, the input data usually refers to various document images that need to be recognized or parsed, such as invoices, receipts, contracts, bank statements, etc. These data come from a variety of channels, such as scanned copies, mobile phone photos, PDF documents, etc.

[0069] In the document recognition task, the data model is usually a trained deep learning model, such as a convolutional neural network (CNN) or a model combined with a recurrent neural network (RNN), which is used to automatically extract features from the image and identify key information (such as date, amount, recipient, payer, etc.). After the model processes the input data, it outputs the prediction results. In the document recognition scenario, these prediction results usually include the recognition results of various fields on the document.

[0070] S202, obtaining the annotation labels of the input data and obtaining the prediction labels of the prediction data.

[0071] Specifically, the annotation label refers to the real or known information of each sample in the data set, which is usually provided manually by domain experts or strictly trained data annotators. In the document recognition scenario, the annotation label includes key information such as the date, amount, invoice number, and payee on the document. The predicted label refers to the prediction result calculated by the model through its internal algorithm and parameters after receiving new, unlabeled data samples. In document recognition, the predicted label is the model's automatic recognition result of key information on the input document image.

[0072] S203: Correct the predicted label according to the annotated label to obtain a corrected label.

[0073] Specifically, the predicted label is the model's automatic recognition result of the input data sample, reflecting the model's understanding and generalization ability of the current task. However, due to the complexity of the model and the uncertainty of the data, the predicted label often has certain errors or uncertainties, which may come from various factors such as overfitting, underfitting, and noisy data of the model.

[0074] In order to further improve the accuracy and reliability of the model, the system needs to calibrate the predicted labels. The purpose of calibration is to reduce the difference between the predicted labels and the annotated labels, so that the predicted results are closer to the actual situation. Through calibration, we can find out where the model is deficient and adjust the parameters and structure of the model accordingly to optimize its performance.

[0075] There are many ways to correct the predicted labels based on the annotated labels, depending on the requirements of the task and the characteristics of the data. For example, the predicted labels can be corrected by direct replacement, partial correction, probability correction, and feedback learning.

[0076] S204: construct an iterative training sample set using the input data carrying the annotated label as the first sample data and the predicted data carrying the corrected label as the second sample data.

[0077] The input data with annotated labels constitute the initial sample set of the training process. These sample data contain all the key information that the model needs to learn, and the true labels of the input data are known, so they can be used to train the model and evaluate its performance.

[0078] The predicted data with correction labels are used as the second sample data here. These data were originally based on the prediction results of the model, but after correction, they contain information that is closer to the actual situation. Adding these data to the training sample set can further enrich the diversity of training data and help the model learn more useful information.

[0079] Specifically, the system uses input data carrying annotated labels as first sample data and predicted data carrying correction labels as second sample data to construct an iterative training sample set to facilitate subsequent iterative model training.

[0080] S205, updating the optimization parameters of the data model according to the set target values ​​to obtain target optimization parameters.

[0081] Specifically, in machine learning and deep learning, setting a target value (also called the minimum value of the objective function, loss function, or cost function) is the core goal of model training. This target value usually represents the degree of difference between the model's predicted results and the actual results, such as the cross entropy loss in classification tasks, the mean square error in regression tasks, etc. The purpose of setting a target value is to guide the model's learning process so that the model can gradually reduce the prediction error and improve the prediction accuracy.

[0082] During the training process, optimization parameters are those model parameters that are adjusted by the optimization algorithm to reduce the target value. The selection and adjustment of optimization parameters are crucial to the performance of the model. Through the optimization algorithm, an optimal set of parameter values ​​can be found so that the prediction error of the model on a given data set is minimized.

[0083] S206, iteratively train the data model using the iterative training sample set until the model parameters reach the target optimization parameters, and output the updated data model.

[0084] The iterative training sample set is a dynamically updated dataset that contains the initial labeled data and the data that is subsequently predicted and corrected by the model. This dataset is continuously expanded during the training process, providing the model with more learning samples and change patterns.

[0085] Specifically, the model is trained multiple times through iterative training sample sets, and the model parameters are continuously adjusted to reduce the value of the objective function (such as the loss function). In each iteration, the model uses a portion of the data in the iterative training sample set for training, and updates the model parameters based on the training results. This process is repeated until the model parameters reach the target optimization parameters, and the updated data model is output.

[0086] In the above embodiments, the present application significantly improves the prediction accuracy of the data model by introducing a correction mechanism for the prediction labels and using the corrected data to construct an iterative training sample set. Through continuous iterative training, the model parameters are gradually optimized to the target values, and ultimately a data model with better performance is output, which effectively enhances the generalization ability of the model and improves the accuracy and efficiency of data processing.

[0087] Furthermore, after the steps of obtaining input data, inputting the input data into the data model, and obtaining the predicted data output by the data model, the method further includes:

[0088] Determine whether the prediction data output by the data model is in the form of a key-value pair, where the body of the prediction data is the key position of the key-value pair, and the prediction label corresponding to the prediction data is the value position of the key-value pair;

[0089] If the prediction data output by the data model is not in the form of a key-value pair, the prediction data output by the data model is obtained by obtaining the production log of the data model.

[0090] In this embodiment, in the machine learning or data processing process, the output results of the data model are specifically processed to obtain prediction data by judging the output data format and taking an alternative method under specific conditions (i.e., when the output is not in the expected key-value pair form).

[0091] Determine whether the prediction data is in the form of key-value pairs. This format is very useful in many scenarios because the key-value pair format allows the prediction results (entities) to be directly associated with their corresponding prediction labels or attributes (values), which facilitates subsequent processing and analysis.

[0092] When the prediction data is not in the form of key-value pairs, an alternative method is needed to obtain the prediction data. The system obtains the prediction data by obtaining the production log of the data model, where the data model records its output results in the log file during the production process.

[0093] In the above embodiment, a step of judging and processing the output format is added when processing the output data of the data model, which improves the flexibility of the process to a certain extent.

[0094] Furthermore, the step of obtaining the prediction data output by the data model by obtaining the production log of the data model specifically includes:

[0095] Get the production log fragment generated by the data model's prediction processing of the input data, where the production log is a JSON type log;

[0096] Perform JSON path parsing on the production log fragment to obtain the JSON log path parsing result;

[0097] Match and map the pre-configured log core fields with the JSON log path parsing results, and obtain the data model output prediction data based on the matching results.

[0098] In this embodiment, first, it is necessary to clarify the source of the production log, including the file system, database, message queue (such as Kafka, RabbitMQ) or cloud storage service (such as Amazon S3, Google Cloud Storage), etc.

[0099] Then, according to the location and format of log storage, choose appropriate tools and methods to obtain these logs regularly or in real time. For example, you can use the log file rotation mechanism combined with periodic tasks (such as cron jobs) to grab log files on the file system; or use database query statements to extract log data from the database; you can also use the listener of the message queue to capture log messages in real time, etc.

[0100] Next, you can use a json parsing library (such as Python's json module, JavaScript's json.parse(), etc.) to easily access and extract specific fields in the json object by path. The parser supports dynamic path parsing, which means it can handle any json path defined in the configuration interface or configuration file.

[0101] Finally, write the matching mapping logic, which is responsible for matching and mapping the pre-configured log core fields with the JSON log path parsing results, and obtaining the data model output prediction data based on the matching results.

[0102] In the above embodiment, by directly parsing the JSON type production log generated by the data model, using JSON path parsing to accurately locate key information, and matching it with the preset log core field mapping, the output prediction data of the data model is efficiently and accurately extracted, thereby improving data processing efficiency and accuracy, and simplifying the data acquisition process.

[0103] Furthermore, the pre-configured log core fields are matched and mapped with the JSON log path parsing results, and the steps of obtaining the data model output prediction data according to the matching results specifically include:

[0104] Traverse the JSON log path parsing results, match the log core fields with the fields in the JSON log path parsing results, and obtain the matching log fields;

[0105] Find the field value corresponding to the matching log field in the JSON log path parsing result;

[0106] Build field key-value pairs based on matching log fields and field values, and use the field key-value pairs as the data model to output prediction data.

[0107] In this embodiment, the system needs to define and configure the log core fields in advance. These fields represent the key information expected to be extracted from the log, that is, the relevant fields of the predicted data output by the data model. By traversing the json log path parsing results, the log core fields are matched with the fields in the parsing results. The matching process is based on the field name. Only when the field name in the log matches the configured log core field, the matching log field is determined. After confirming the match, it is necessary to find the field value corresponding to the matching log field in the json log path parsing results. The matched log field name is used as the key (key), and its corresponding field value is used as the value (value) to construct a field key-value pair.

[0108] In the above embodiment, through pre-configuration and precise matching, key information can be efficiently extracted from complex JSON logs, reducing unnecessary data processing time. At the same time, by configuring the core fields of the log, it can flexibly adapt to the requirements of different data models and log formats, improving the maintainability and scalability of the system.

[0109] Furthermore, the predicted label is corrected according to the annotated label to obtain the corrected label, which specifically includes:

[0110] Obtaining an input data segment corresponding to the annotation label, and constructing a first data label combination based on the annotation label and the input data segment;

[0111] Obtaining a predicted data segment corresponding to the predicted label, and constructing a second data label combination based on the predicted label and the predicted data segment;

[0112] Determine whether the first data label combination and the second data label combination are consistent;

[0113] If the first data label combination and the second data label combination are inconsistent, the predicted label is corrected according to the first data label combination to obtain a corrected label.

[0114] In this embodiment, the input data segment corresponding to the annotation label is obtained, and the annotation label is usually a real result that has been verified by humans or other reliable methods, representing the real attribute or category of the data segment. By combining the annotation label with the corresponding input data segment, a first data label combination is constructed.

[0115] The predicted data segment corresponding to the predicted label is obtained. The predicted label is automatically generated by the data model based on the input data and may have certain errors. The predicted label and the predicted data segment are combined into a second data label combination, which reflects the current prediction result of the model.

[0116] Compare the first data label combination and the second data label combination to see if they are consistent. "Consistent" here means that if the annotation label is the same as the predicted label, the corresponding input data segments are also the same or similar. Through this comparison, possible errors or inconsistencies in the predicted labels can be identified.

[0117] If it is found that the first data label combination and the second data label combination are inconsistent, that is, under the premise that the labeled label and the predicted label are the same, the input data segment and the predicted data segment are different, then the predicted label is corrected based on the first data label combination (that is, the actual situation).

[0118] In the above embodiments, by comparing and correcting the prediction labels, the prediction accuracy of the data model can be significantly improved. During the model training process, using the corrected prediction labels as feedback can further optimize the performance of the model and reduce future prediction errors.

[0119] Furthermore, the step of correcting the predicted label according to the first data label combination to obtain the corrected label specifically includes:

[0120] Identifying a data segment matching the input data segment in the predicted data to obtain a matching data segment;

[0121] Determine a matching relationship between the input data segment and the annotated label in the first data label combination to obtain a true label matching relationship;

[0122] The matching relationship between the matching data segment and the predicted label is corrected according to the real label matching relationship to complete the correction of the predicted label and obtain the corrected label.

[0123] In this embodiment, first, a data segment that matches the input data segment used to generate the annotated label is searched in the prediction data set. Then, the correspondence between the annotated label (true label) and the input data segment, that is, the true label matching relationship, is clarified. This is the basis of the correction process, because the purpose of the correction is to make the predicted label as close to the true label as possible. According to the true label matching relationship, the predicted label of the matching data segment is corrected. Specifically, the predicted label of the matching data segment is compared with the corresponding true label. If the two are inconsistent, correction is performed according to the true label matching relationship.

[0124] In the above embodiment, by correcting the predicted labels, the prediction error can be reduced and the prediction accuracy of the data model can be improved. The corrected labels are closer to the actual situation, thereby enhancing the quality of the overall data.

[0125] Further, the data model is iteratively trained using the iterative training sample set until the model parameters reach the target optimization parameters, and the step of outputting the updated data model specifically includes:

[0126] Input the iterative training sample set into the data model to obtain the iterative prediction result;

[0127] Compare the iterative prediction results with the preset standard results, and calculate the iterative prediction error of the data model;

[0128] Use the directional propagation algorithm to propagate the prediction error in the data model;

[0129] The gradient descent algorithm is used to iteratively update the model parameters until the model parameters reach the target optimization parameters, and the updated data model is output.

[0130] In this embodiment, the prepared iterative training sample sets are input into the data model. These sample sets usually contain a large amount of input data and corresponding target outputs (or standard results), which are used to train the model to learn the mapping relationship between input and output. Iterative training means that this process is repeated many times, each time using a different training sample set or a different subset of the same sample set to gradually optimize the model parameters.

[0131] The data model performs forward propagation based on the input iterative training sample set and outputs iterative prediction results. These prediction results are the model's response to the current input data and are used to compare with the preset standard results, which include annotation labels and correction labels.

[0132] The iterative prediction results are compared with the preset standard results, and the iterative prediction error of the data model is calculated through the model loss function (such as mean square error, cross entropy, etc.). The prediction error reflects the degree of difference between the model prediction results and the actual results, and is a key indicator for optimizing the model.

[0133] The prediction error is transferred in the data model using the directional propagation (often called backpropagation) algorithm. This step is the core of deep learning, allowing the algorithm to start from the output layer and calculate the contribution of each layer to the final error (i.e., gradient) layer by layer, thereby determining the update direction of the model parameters.

[0134] Use gradient descent (or other optimization algorithms, such as Adam, RMSprop, etc.) to iteratively update model parameters. According to the gradient calculated by back propagation, adjust the model parameters to reduce the prediction error. This process will be repeated until the model parameters reach the target optimization parameters (such as the loss function value is lower than a certain threshold, the training round reaches the preset upper limit, etc.), at which time the model is considered to have been well trained.

[0135] When the model parameters reach the target optimization parameters, the training process ends. At this time, the updated data model is output, which has learned the mapping relationship between the input data and the target output and can be used for subsequent prediction or classification tasks.

[0136] In the above embodiment, through iterative training, the data model can gradually learn the characteristics of the input data and optimize its own parameters to reduce prediction errors, thereby improving the performance of the model.

[0137] In this embodiment, the present application discloses a method for optimizing a data model, which belongs to the field of artificial intelligence technology. First, the predicted data and its predicted label are obtained by prediction through the data model, and the predicted label is compared with the real labeled label to correct the predicted label. Subsequently, an iterative training sample set is constructed using the input data carrying the labeled label and the predicted data carrying the corrected label. Then, the optimization parameters of the data model are updated according to the set target value. The data model is iteratively trained by continuously using the iterative training sample set until the model parameters reach the target optimization parameters, thereby outputting an updated data model. By introducing a correction mechanism for the predicted label and constructing an iterative training sample set using the corrected data, the prediction accuracy of the data model is significantly improved. Through continuous iterative training, the model parameters are gradually optimized to the target value, and finally a data model with better performance is output, which effectively enhances the generalization ability of the model and improves the accuracy and efficiency of data processing.

[0138] In this embodiment, the optimization method of the data model is executed on the electronic device (eg Figure 1 The server shown in the figure) can receive instructions or obtain data through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0139] It should be emphasized that in order to further ensure the privacy and security of the above input data, the above input data can also be stored in a node of a blockchain.

[0140] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0141] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0142] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0143] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0144] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0145] Further references Figure 3 , as a response to the above Figure 2 In order to realize the method shown in the figure, the present application provides an embodiment of an optimization device for a data model, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0146] like Figure 3 As shown, the data model optimization device 300 described in this embodiment includes:

[0147] The data prediction module 301 is used to obtain input data, input the input data into the data model, and obtain output prediction data of the data model;

[0148] The label acquisition module 302 is used to obtain the annotation labels of the input data and obtain the predicted labels of the predicted data;

[0149] The label correction module 303 is used to correct the predicted label according to the marked label to obtain a corrected label;

[0150] A sample construction module 304 is used to construct an iterative training sample set using input data carrying annotated labels as first sample data and prediction data carrying correction labels as second sample data;

[0151] The parameter setting module 305 is used to update the optimization parameters of the data model according to the set target values ​​to obtain the target optimization parameters;

[0152] The iterative training module 306 is used to iteratively train the data model using the iterative training sample set until the model parameters reach the target optimization parameters, and output the updated data model.

[0153] Furthermore, the data model optimization device 300 further includes an output form determination unit, which is specifically used for:

[0154] Determine whether the prediction data output by the data model is in the form of a key-value pair, where the body of the prediction data is the key position of the key-value pair, and the prediction label corresponding to the prediction data is the value position of the key-value pair; if the prediction data output by the data model is not in the form of a key-value pair, obtain the prediction data output by the data model by obtaining the production log of the data model.

[0155] Furthermore, the data prediction module 301 is specifically used for:

[0156] Obtain the production log fragment generated by the data model's predictive processing of the input data, where the production log is a JSON type log; perform JSON path parsing on the production log fragment to obtain the JSON log path parsing result; match and map the pre-configured log core fields with the JSON log path parsing result, and obtain the data model output prediction data based on the matching result.

[0157] Furthermore, the data prediction module 301 is further used for:

[0158] Traverse the json log path parsing results, match the log core fields with the fields in the json log path parsing results, and obtain the matching log fields; find the field values ​​corresponding to the matching log fields in the json log path parsing results; build field key-value pairs based on the matching log fields and field values, and use the field key-value pairs as the data model to output the predicted data.

[0159] Furthermore, the label correction module 303 is specifically used for:

[0160] Obtain an input data segment corresponding to the annotated label, and construct a first data label combination based on the annotated label and the input data segment; obtain a predicted data segment corresponding to the predicted label, and construct a second data label combination based on the predicted label and the predicted data segment;

[0161] Determine whether the first data label combination and the second data label combination are consistent; if the first data label combination and the second data label combination are inconsistent, correct the predicted label according to the first data label combination to obtain a corrected label.

[0162] Furthermore, the label correction module 303 is further used to:

[0163] Identify the data segment that matches the input data segment in the predicted data to obtain the matching data segment; determine the matching relationship between the input data segment and the annotated label in the first data label combination to obtain the true label matching relationship; correct the matching relationship between the matching data segment and the predicted label according to the true label matching relationship to complete the correction of the predicted label and obtain the corrected label.

[0164] Furthermore, the iterative training module 306 is specifically used for:

[0165] Input the iterative training sample set into the data model to obtain the iterative prediction result; compare the iterative prediction result with the preset standard result, and calculate the iterative prediction error of the data model; use the directional propagation algorithm to transfer the prediction error in the data model; use the gradient descent algorithm to iteratively update the model parameters until the model parameters reach the target optimization parameters, and output the updated data model.

[0166] In this embodiment, the present application discloses an optimization device for a data model, which belongs to the field of artificial intelligence technology. First, the prediction data and its prediction label are obtained by prediction through the data model, and the prediction label is compared with the real annotation label to correct the prediction label. Subsequently, an iterative training sample set is constructed using the input data carrying the annotation label and the prediction data carrying the correction label. Then, the optimization parameters of the data model are updated according to the set target value. The data model is iteratively trained by continuously using the iterative training sample set until the model parameters reach the target optimization parameters, thereby outputting an updated data model. By introducing a correction mechanism for the prediction label and constructing an iterative training sample set using the corrected data, the prediction accuracy of the data model is significantly improved. Through continuous iterative training, the model parameters are gradually optimized to the target value, and finally a data model with better performance is output, which effectively enhances the generalization ability of the model and improves the accuracy and efficiency of data processing.

[0167] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0168] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 4 with a memory 41, a processor 42, and a network interface 43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0169] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0170] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk equipped on the computer device 4, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of the optimization method of the data model, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0171] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run computer-readable instructions stored in the memory 41 or process data, such as computer-readable instructions for running the optimization method of the data model.

[0172] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0173] In this embodiment, the present application discloses a computer device, which belongs to the field of artificial intelligence technology. First, the predicted data and its predicted label are obtained by prediction through the data model, and the predicted label is compared with the real labeled label to correct the predicted label. Subsequently, an iterative training sample set is constructed using the input data carrying the labeled label and the predicted data carrying the corrected label. Then, the optimization parameters of the data model are updated according to the set target value. The data model is iteratively trained by continuously using the iterative training sample set until the model parameters reach the target optimization parameters, thereby outputting the updated data model. By introducing a correction mechanism for the predicted label and constructing an iterative training sample set using the corrected data, the prediction accuracy of the data model is significantly improved. Through continuous iterative training, the model parameters are gradually optimized to the target value, and finally a data model with better performance is output, which effectively enhances the generalization ability of the model and improves the accuracy and efficiency of data processing.

[0174] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the data model optimization method as described above.

[0175] In this embodiment, the present application discloses a computer-readable storage medium, which belongs to the field of artificial intelligence technology. First, the predicted data and its predicted label are obtained by prediction through the data model, and the predicted label is compared with the real labeled label to correct the predicted label. Subsequently, an iterative training sample set is constructed using the input data carrying the labeled label and the predicted data carrying the corrected label. Then, the optimization parameters of the data model are updated according to the set target value. The data model is iteratively trained by continuously using the iterative training sample set until the model parameters reach the target optimization parameters, thereby outputting the updated data model. By introducing a correction mechanism for the predicted label and constructing an iterative training sample set using the corrected data, the prediction accuracy of the data model is significantly improved. Through continuous iterative training, the model parameters are gradually optimized to the target value, and finally a data model with better performance is output, which effectively enhances the generalization ability of the model and improves the accuracy and efficiency of data processing.

[0176] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0177] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0178] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A method for optimizing a data model, characterized in that: include: Obtain input data, and input the input data into the data model to obtain output prediction data of the data model; Obtaining a label for the input data and obtaining a predicted label for the predicted data; Correcting the predicted label according to the annotated label to obtain a corrected label; Using the input data carrying the annotated label as first sample data and the predicted data carrying the correction label as second sample data, constructing an iterative training sample set; Updating the optimization parameters of the data model according to the set target values ​​to obtain target optimization parameters; The data model is iteratively trained using the iterative training sample set until the model parameters reach the target optimization parameters, and the updated data model is output.

2. The method for optimizing a data model according to claim 1, wherein: After the step of obtaining input data, inputting the input data into the data model, and obtaining the predicted data output by the data model, the method further includes: Determine whether the prediction data output by the data model is in the form of a key-value pair, wherein the body of the prediction data is the key position of the key-value pair, and the prediction tag corresponding to the prediction data is the value position of the key-value pair; If the prediction data output by the data model is not in the form of a key-value pair, the prediction data output by the data model is obtained by obtaining the production log of the data model.

3. The data model optimization method according to claim 2, characterized in that: The step of obtaining the prediction data output by the data model by obtaining the production log of the data model specifically includes: Obtaining a production log segment generated by the data model performing prediction processing on the input data, wherein the production log is a JSON type log; Performing json path parsing on the production log segment to obtain a json log path parsing result; The pre-configured log core field is matched and mapped with the json log path parsing result, and the data model output prediction data is obtained according to the matching result.

4. The method for optimizing a data model according to claim 3, characterized in that: The step of matching and mapping the pre-configured log core field with the JSON log path parsing result and obtaining the data model output prediction data according to the matching result specifically includes: Traversing the json log path parsing result, matching the log core field with the field in the json log path parsing result to obtain a matching log field; Searching for a field value corresponding to the matching log field in the json log path parsing result; A field key-value pair is constructed based on the matching log field and the field value, and the field key-value pair is used as the data model to output prediction data.

5. The method for optimizing a data model according to claim 1, wherein: The step of correcting the predicted label according to the annotated label to obtain the corrected label specifically includes: Acquire an input data segment corresponding to the annotation label, and construct a first data label combination based on the annotation label and the input data segment; Acquire a predicted data segment corresponding to the predicted label, and construct a second data label combination based on the predicted label and the predicted data segment; Determining whether the first data tag combination and the second data tag combination are consistent; If the first data label combination and the second data label combination are inconsistent, the predicted label is corrected according to the first data label combination to obtain the corrected label.

6. The method for optimizing a data model according to claim 5, characterized in that: The step of correcting the predicted label according to the first data label combination to obtain the corrected label specifically includes: identifying a data segment matching the input data segment in the predicted data to obtain a matching data segment; Determine a matching relationship between the input data segment and the annotated label in the first data label combination to obtain a true label matching relationship; The matching relationship between the matching data segment and the predicted label is corrected according to the real label matching relationship to complete the correction of the predicted label and obtain the corrected label.

7. The data model optimization method according to claim 1, characterized in that: The step of iteratively training the data model using the iterative training sample set until the model parameters reach the target optimization parameters and outputting the updated data model specifically includes: Inputting the iterative training sample set into the data model to obtain an iterative prediction result; Comparing the iterative prediction result with a preset standard result, and calculating the iterative prediction error of the data model; Propagate the prediction error in the data model using a directional propagation algorithm; The model parameters are iteratively updated using a gradient descent algorithm until the model parameters reach the target optimization parameters, and the updated data model is output.

8. A data model optimization device, characterized in that: include: A data prediction module, used to obtain input data, input the input data into the data model, and obtain output prediction data of the data model; A label acquisition module, used to acquire the annotation label of the input data and acquire the prediction label of the prediction data; A label correction module, used to correct the predicted label according to the marked label to obtain a corrected label; A sample construction module, used to construct an iterative training sample set using the input data carrying the annotated label as first sample data and the predicted data carrying the correction label as second sample data; A parameter setting module, used to update the optimization parameters of the data model according to the set target values ​​to obtain target optimization parameters; The iterative training module is used to iteratively train the data model using the iterative training sample set until the model parameters reach the target optimization parameters, and output the updated data model.

9. A computer device, characterized in that: It comprises a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the data model optimization method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the data model optimization method according to any one of claims 1 to 7.

Images