Data processing method and device, electronic equipment and computer readable storage medium

Abstract

The present disclosure relates to a data processing method and device, electronic equipment and computer readable storage medium. The data processing method comprises: obtaining input data and output data of a to-be-processed link as main input data and main output data, wherein the to-be-processed link comprises a plurality of sequentially connected sub-links; determining a reference sub-link from the plurality of sub-links of the to-be-processed link, and processing the main input data based on the to-be-processed link to obtain simulated main output data, wherein the output data of the reference sub-link is obtained by executing the reference sub-link, and the output data of other sub-links is obtained by a pre-trained prediction model; determining whether an end condition is met; in response to the end condition being met, determining an approximate main output data from all simulated main output data of the to-be-processed link based on the main output data; and determining the reference sub-link corresponding to the approximate main output data as a target sub-link.

Description

Technical Field

[0001] The present disclosure relates to the field of traffic replay technology, and in particular to a data processing method and device, an electronic device, and a computer-readable storage medium. Background Art

[0002] For online applications, the system typically replays traffic on specific links regularly to proactively troubleshoot issues. However, for occasional online issues, most systems only record the input and return data of the link at the time the problem occurred, without fully recording the detailed data within the link. Therefore, for such issues, manual analysis is usually the only way to locate the problem, failing to fully utilize computing resources to improve the efficiency of troubleshooting occasional issues, resulting in low resource utilization. Summary of the Invention

[0003] The present disclosure provides a data processing method and apparatus, an electronic device, and a computer-readable storage medium to at least solve a problem in the related art of how to improve resource utilization when troubleshooting occasional problems.

[0004] According to a first aspect of the present disclosure, a data processing method is provided, which includes: obtaining input data and output data of a link to be processed as main input data and main output data, wherein the link to be processed includes multiple sub-links connected in sequence; determining a reference sub-link from the multiple sub-links of the link to be processed, and processing the main input data based on the link to be processed to obtain simulated main output data, wherein the output data of the reference sub-link is obtained by executing the reference sub-link, and the output data of other sub-links in the link to be processed except the reference sub-link are obtained by a pre-trained prediction model, and the prediction model is used to predict the output data of the corresponding sub-link; judging whether an end condition is met, wherein the end condition is a condition indicating that there is no need to determine a new reference sub-link from the multiple sub-links of the link to be processed; in response to meeting the end condition, determining an approximate main output data from all the simulated main output data of the link to be processed based on the main output data; and determining the reference sub-link corresponding to the approximate main output data as the target sub-link.

[0005] Optionally, in response to satisfying the termination condition, based on the main output data, an approximate main output data is determined from all the simulated main output data of the link to be processed, including: in response to satisfying the termination condition, determining the degree of similarity between all the simulated main output data of the link to be processed and the main output data; and determining, from all the simulated main output data of the link to be processed, one with the greatest similarity and greater than a first preset threshold as the approximate main output data.

[0006] Optionally, the determining a reference sub-link from the multiple sub-links of the link to be processed includes: determining a read-only sub-link from the multiple sub-links of the link to be processed as the reference sub-link.

[0007] Optionally, in response to satisfying the end condition, after determining an approximate main output data from all the simulated main output data of the link to be processed based on the main output data, the data processing method further includes: in response to the inability to determine the approximate main output data from all the simulated main output data of the link to be processed, taking the write-type sub-link in the link to be processed as a new link to be processed, and repeating the steps from obtaining the input data and output data of the link to be processed to determining the approximate main output data until the approximate main output data is determined or a new link to be processed cannot be determined, wherein, when the degree of similarity between all the simulated main output data of the link to be processed and the main output data is less than or equal to the first preset threshold, it is confirmed that the approximate main output data cannot be determined.

[0008] Optionally, the termination condition includes at least one of the following: the simulated main output data corresponding to all read-only sub-links of the link to be processed have been obtained, and the degree of similarity between the currently obtained simulated main output data and the main output data exceeds a second preset threshold, wherein the second preset threshold is greater than the first preset threshold.

[0009] Optionally, each sub-link has a corresponding prediction model, and the pre-trained prediction model is trained through the following steps: obtaining multiple sample data of the sub-link to be trained in the production environment, the sample data including sample input data and sample output data; determining a candidate prediction model based on the multiple sample data; continuing to obtain new sample data of the sub-link to be trained in the production environment; inputting the newly obtained sample input data into the candidate prediction model to obtain predicted output data; correcting the candidate prediction model based on the predicted output data and the corresponding sample output data; repeating the steps from continuing to obtain new sample data to correcting the candidate prediction model until the training end condition is met, and using the candidate prediction model finally obtained as the pre-trained prediction model.

[0010] Optionally, the repeated execution of the steps from continuing to acquire new sample data to correcting the candidate prediction model includes: in response to the cumulative number of sample data used reaching a sample threshold, based on all the predicted output data and the corresponding sample output data, counting the difference probabilities of multiple output fields of the sub-link to be trained, wherein the difference probability of each output field is the proportion of the difference prediction data of the corresponding output field in all the predicted output data, wherein the similarity between the corresponding output field of the difference prediction data and the corresponding output field of the sample output data corresponding to the difference prediction data is less than a similarity threshold; taking the output field whose difference probability is greater than or equal to the difference threshold as the difference field; repeating the steps from continuing to acquire new sample data to correcting the candidate prediction model, and using other fields of the multiple output fields except the difference field when correcting the candidate prediction model.

[0011] According to a second aspect of the present disclosure, a data processing device is provided, comprising: an acquisition unit configured to acquire input data and output data of a link to be processed as primary input data and primary output data, wherein the link to be processed includes multiple sub-links connected in sequence; an imitation unit configured to determine a reference sub-link from the multiple sub-links of the link to be processed, and obtain simulated primary output data based on the primary input data, wherein the output data of the reference sub-link is obtained by executing the reference sub-link, and the output data of other sub-links in the link to be processed except the reference sub-link are obtained by a pre-trained prediction model, and the prediction model is used to predict the output data of the corresponding sub-link; a judgment unit configured to judge whether an end condition is satisfied, wherein the end condition is a condition indicating that there is no need to determine a new reference sub-link from the multiple sub-links of the link to be processed; a determination unit configured to determine an approximate primary output data from all the simulated primary output data of the link to be processed based on the primary output data in response to satisfying the end condition; and a positioning unit configured to determine the reference sub-link corresponding to the approximate primary output data as a target sub-link.

[0012] Optionally, the determination unit is further configured to: in response to satisfying the end condition, determine the degree of similarity between all the simulated main output data of the link to be processed and the main output data; and determine, from all the simulated main output data of the link to be processed, the one with the greatest similarity and greater than a first preset threshold as the approximate main output data.

[0013] Optionally, the imitation unit is further configured to: determine a read-only sub-link from the multiple sub-links of the link to be processed as the reference sub-link.

[0014] Optionally, the acquisition unit is further configured to: in response to the determination unit being unable to determine the approximate main output data, take the write-type sub-link in the link to be processed as a new link to be processed, repeat the step of acquiring the input data and output data of the link to be processed as the main input data and main output data, and run the imitation unit and the determination unit again until the determination unit determines the approximate main output data or the acquisition unit is unable to determine a new link to be processed, wherein the determination unit is further configured to: when the degree of similarity between all the simulated main output data of the link to be processed and the main output data is less than or equal to the first preset threshold, confirm that the approximate main output data cannot be determined.

[0015] Optionally, the termination condition includes at least one of the following: the simulated main output data corresponding to all read-only sub-links of the link to be processed have been obtained, and the degree of similarity between the currently obtained simulated main output data and the main output data exceeds a second preset threshold, wherein the second preset threshold is greater than the first preset threshold.

[0016] Optionally, each sub-link has a corresponding prediction model, and the pre-trained prediction model is trained through the following steps: obtaining multiple sample data of the sub-link to be trained in the production environment, the sample data including sample input data and sample output data; determining a candidate prediction model based on the multiple sample data; continuing to obtain new sample data of the sub-link to be trained in the production environment; inputting the newly obtained sample input data into the candidate prediction model to obtain predicted output data; correcting the candidate prediction model based on the predicted output data and the corresponding sample output data; repeating the steps from continuing to obtain new sample data to correcting the candidate prediction model until the training end condition is met, and using the candidate prediction model finally obtained as the pre-trained prediction model.

[0017] Optionally, the repeated execution of the steps from continuing to acquire new sample data to correcting the candidate prediction model includes: in response to the cumulative number of sample data used reaching a sample threshold, based on all the predicted output data and the corresponding sample output data, counting the difference probabilities of multiple output fields of the sub-link to be trained, wherein the difference probability of each output field is the proportion of the difference prediction data of the corresponding output field in all the predicted output data, wherein the similarity between the corresponding output field of the difference prediction data and the corresponding output field of the sample output data corresponding to the difference prediction data is less than a similarity threshold; taking the output field whose difference probability is greater than or equal to the difference threshold as the difference field; repeating the steps from continuing to acquire new sample data to correcting the candidate prediction model, and using other fields of the multiple output fields except the difference field when correcting the candidate prediction model.

[0018] According to a third aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and at least one memory storing computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, prompt the at least one processor to execute the data processing method according to the present disclosure.

[0019] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the data processing method according to the present disclosure.

[0020] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising computer instructions, which implement the data processing method according to the present disclosure when executed by at least one processor.

[0021] The technical solutions provided by the embodiments of the present disclosure bring at least the following beneficial effects:

[0022] According to the data processing method and data processing device of the embodiment of the present disclosure, multiple playbacks are used to check the sub-links of the link to be processed. In each playback, for other sub-links other than the reference sub-link currently being checked, a pre-trained prediction model is used to obtain output data, thereby more accurately reproducing the output data of the corresponding sub-link when no problem occurs in the production environment, supplementing the complete link data, and at the same time performing a playback operation on the reference sub-link in a simulation environment, which helps to ensure that the playback operation results of the reference sub-link are reliable and the playback of the link to be processed proceeds smoothly, thereby obtaining more reliable simulated main output data of the link to be processed. On this basis, by determining the one that best matches the actual main output data from all the simulated main output data obtained by playback, the corresponding playback can be regarded as a valid playback of the link to be processed, and then the corresponding reference sub-link can be determined as the target sub-link with the problem. The target sub-link can be determined quickly, conveniently and reliably without the need for manual intervention analysis, thereby achieving efficient data processing of occasional problems.

[0023] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0025] Figure 1 is a flowchart illustrating a data processing method according to an exemplary embodiment of the present disclosure;

[0026] Figure 2 is a logical diagram illustrating a data processing method according to a specific embodiment of the present disclosure;

[0027] Figure 3 is a block diagram illustrating a data processing apparatus according to an exemplary embodiment of the present disclosure;

[0028] Figure 4 is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0030] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0031] It should be noted that the phrase "at least one of the items" in this disclosure includes three types of parallel situations: "any one of the items", "a combination of any multiple items of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. For another example, "performing at least one of step 1 and step 2" includes the following three parallel situations: (1) performing step 1; (2) performing step 2; and (3) performing steps 1 and 2.

[0032] It should be noted that the user information and data involved in this disclosure are all information and data authorized by the users or fully authorized by all parties.

[0033] For online applications, the implementation of a specific function often relies on multiple simpler sub-functions, and each sub-function often relies on multiple simpler sub-functions. These different levels of functions combine to achieve the specific function. Based on this, the program that implements a function can be considered a link, and the programs that implement its sub-functions can be considered sub-links of this link, and sub-links can further contain more detailed sub-links.

[0034] To ensure the normal operation of the link, the system typically regularly replays traffic for the entire link or some sub-links to proactively troubleshoot issues. Traffic represents requests within a specific time period. Traffic replay records the requests sent to application A and then forwards them to application B, ensuring that the request parameters received by application B are consistent with those of application A. This allows application B to re-request the request received by application A. Traffic replay can proactively check whether there are any issues with the replayed link.

[0035] For occasional online issues, most systems only record the input parameters and return data of the link (referred to as the pending link below for ease of explanation) at the time the issue occurs. Detailed link data for each sublink within the pending link is not fully recorded. Therefore, problem location cannot be analyzed through link data analysis, and it is difficult to locate the problem through traffic replay. To briefly explain, a service, from development to launch, and then to routine maintenance and upgrades, requires the creation of corresponding environments to create services in different environments to meet the needs of different stages. Typical environments include: development, test, staging, and production (also known as online). The development environment is specifically used for development and debugging. The production environment officially provides external services. The test environment is typically a clone of the production environment's configuration for pre-launch testing. However, the test data is typically stored in a dedicated test database, isolated from the production database. The staging environment can be considered a mirror image of the production environment. Traffic replay is difficult to use to locate problems because directly replaying traffic in the online production environment could affect the production environment. If traffic replay is performed in an offline simulation environment (such as a development environment, a test environment, etc.), there is a problem of incomplete link data, which makes it difficult to quickly and accurately obtain the input data of the sub-link in the link to be processed when replaying the sub-link. Among them, it is difficult to quickly obtain the input data, which may cause the system to wait too long and report an error, and the traffic replay is interrupted. It is difficult to accurately obtain the input data, which will in turn affect the accuracy of the output data. In addition, for write-type sub-links, a write operation will be performed during playback, that is, certain fields in its input data will be written to the system, such as writing to a database of the system. At this time, if the data written to the database is wrong, it may cause the traffic replay to be interrupted. Therefore, it is usually only possible to manually look through the code for analysis to locate the sub-link with problems in the link to be processed. The computing resources are not fully utilized to improve the efficiency of troubleshooting occasional problems, and there is a problem of low resource utilization.

[0036] According to the data processing method and device of the exemplary embodiment of the present disclosure, multiple playbacks are used to check the sub-links of the link to be processed. In each playback, for other sub-links other than the reference sub-link currently being checked, a pre-trained prediction model is used to obtain output data, thereby more accurately reproducing the output data of the corresponding sub-link when no problem occurs in the production environment, supplementing the complete link data, and directly executing the reference sub-link to achieve its playback, which helps to ensure that the playback operation results of the reference sub-link are reliable and the playback of the link to be processed proceeds smoothly, thereby obtaining more reliable simulated main output data of the link to be processed. On this basis, by determining the one that best matches the actual main output data from all the simulated main output data obtained through playback, the corresponding playback can be regarded as a valid playback of the link to be processed, and then the corresponding reference sub-link can be determined as the target sub-link with the problem. The target sub-link can be determined quickly, conveniently and reliably without the need for manual intervention analysis, which fully improves resource utilization and achieves efficient problem location of occasional problems.

[0037] Below, we will refer to Figures 1 to 4 A data processing method and a data processing apparatus according to exemplary embodiments of the present disclosure are described in detail.

[0038] Figure 1 is a flow chart illustrating a data processing method according to an exemplary embodiment of the present disclosure. It should be understood that the data processing method according to the exemplary embodiment of the present disclosure can be implemented in a terminal device such as a smartphone, a tablet computer, or a personal computer (PC), or can also be implemented in a device such as a server.

[0039] Reference Figure 1 In step 101, the input data and output data of the link to be processed are obtained as main input data and main output data, wherein the link to be processed includes multiple sub-links connected in sequence.

[0040] It should be understood that sequential connection means that the execution order and data transfer relationship of each sub-link are fixed, and it is not limited to serial connection one by one. In other words, multiple sub-links can be serially connected or a composite connection that combines serial and parallel connections. The data transfer relationship between two adjacent sub-links may be that the subsequent sub-link directly uses the output data of the preceding sub-link, or there may be data processing logic between the two, that is, the output data of the preceding sub-link is first processed and converted according to the data processing logic, and then the converted data is used as the input data of the subsequent sub-link. This disclosure does not impose any restrictions on this.

[0041] In step 102, a reference sub-link is determined from multiple sub-links of the link to be processed, and main input data is processed based on the link to be processed to obtain simulated main output data, wherein the output data of the reference sub-link is obtained by executing the reference sub-link, and the output data of other sub-links in the link to be processed except the reference sub-link are obtained by a pre-trained prediction model, and the prediction model is used to predict the output data of the corresponding sub-link.

[0042] It should be understood that for the location of occasional problems, the process of processing the main input data based on the link to be processed is the process of replaying the traffic of the link to be processed, and during the replay process, different sub-links process data in different ways. Specifically, each sub-link of the link to be processed has a corresponding prediction model. By pre-training the prediction model, the prediction model can be used to predict the output data of the corresponding sub-link, thereby more accurately reproducing the output data of the corresponding sub-link when no problem occurs in the production environment. In addition, by determining a reference sub-link and applying the prediction model to the other sub-links in the link to be processed except the reference sub-link to obtain output data, the complete link data can be supplemented more accurately. At the same time, directly executing the reference sub-link can achieve the replay of the reference sub-link, which helps to ensure that the replay operation results of the reference sub-link are reliable and the replay of the link to be processed proceeds smoothly, thereby obtaining more reliable simulated main output data of the link to be processed and achieving targeted traffic replay for the determined reference sub-link. As an example, when executing the reference sub-link, it can be executed in a simulation environment. The simulation environment is an environment similar to the production environment, which can be various forms of test environments or development environments. This is not limited by the present disclosure. In other words, the reference sub-link is replayed in a simulation environment to avoid dirty data in the production environment.

[0043] In step 103, it is determined whether the termination condition is met, wherein the termination condition is a condition indicating that there is no need to determine a new reference sub-link from the multiple sub-links of the link to be processed. In other words, it is a condition that indicates that sufficient reference sub-links have been fully determined. The termination condition will be further explained later and will not be expanded here. It should be understood that, corresponding to the termination condition being met, step 102 is repeated to determine a new reference sub-link and obtain the corresponding simulated main output data, thereby achieving multiple playbacks of the link to be processed. By determining the reference sub-links multiple times, targeted traffic playback can be performed for each determined reference sub-link, which facilitates the inspection of each determined reference sub-link and ensures the feasibility of the solution.

[0044] In step 104 , in response to the end condition being met, an approximate main output data is determined from all simulated main output data of the link to be processed based on the main output data.

[0045] Optionally, step 104 includes: in response to the termination condition being met, determining the degree of similarity between all simulated main output data of the link to be processed and the main output data; and determining, from among all simulated main output data of the link to be processed, the one with the greatest degree of similarity that exceeds a first preset threshold value as the approximate main output data. By explicitly calculating the degree of similarity between the simulated main output data and the main output data, quantitative judgment can be achieved, helping to improve feasibility and reliability. Furthermore, by requiring the degree of similarity between the approximate main output data and the main output data to exceed the first preset threshold value, sufficient similarity between the approximate main output data and the main output data can be ensured, thereby improving the reliability of data processing.

[0046] In step 105, the reference sub-link corresponding to the approximate main output data is determined as the target sub-link.

[0047] As previously mentioned, when replaying traffic on the link to be processed under any reference sublink, it is assumed that no issues have occurred on sublinks other than the reference sublink, and only the reference sublink is replayed under a simulated environment. Based on this, by determining an approximate primary output data from all the simulated primary output data obtained through playback, the one that best matches the actual primary output data can be found. This corresponding playback can be considered a valid playback of the link to be processed, and the corresponding reference sublink can be identified as the target sublink with the issue. This allows for quick, convenient, and reliable identification of the target sublink without manual analysis, achieving efficient data processing for occasional issues.

[0048] Next, we will further introduce how to process data.

[0049] Optionally, determining a reference sub-link from the multiple sub-links of the link to be processed in step 102 includes: determining a read-only sub-link from the multiple sub-links of the link to be processed as a reference sub-link. Considering that the sub-links include read-only and write types, the former can only read data in addition to obtaining output data and will not affect the data stored in the system, while the latter can also perform write operations in addition to obtaining output data (refer to the previous introduction). Therefore, there are problems such as errors caused by inconsistent written data and interrupted playback of the link to be processed. By only using read-only sub-links as reference sub-links, smooth playback of the link to be processed can be fully guaranteed.

[0050] Optionally, corresponding to the embodiment in step 104 where the degree of similarity between the approximate main output data and the main output data must be greater than a first preset threshold, when the degree of similarity between all simulated main output data of the link to be processed and the main output data is less than or equal to the first preset threshold, it is confirmed that the approximate main output data cannot be determined.

[0051] In some embodiments, the process may be terminated at this point, confirming that the problem location has failed.

[0052] In other embodiments, corresponding to the embodiment in which only read-only sub-links are used as reference sub-links, after step 104, the data processing method according to the exemplary embodiment of the present disclosure further includes: in response to the inability to determine the approximate main output data from all simulated main output data of the link to be processed, taking the write-type sub-link in the link to be processed as a new link to be processed, and repeating steps 101 to 104 until the approximate main output data is determined or the new link to be processed cannot be determined. By further probing the write-type sub-link when the approximate main output data cannot be determined, taking the write-type sub-link as a new link to be processed, and further replaying the traffic of the new link to be processed, it is possible to perform analysis in a recursive manner, which can not only ensure smooth playback while increasing the possibility of locating the target sub-link, but also check sub-links at different levels, which helps to improve data processing efficiency.

[0053] For example, when determining new links to be processed, each link can be identified one by one according to the execution order of each write-type sub-link in the link to be processed. If traffic playback has been performed on all write-type sub-links and approximate primary output data still cannot be determined, it is considered that no new links to be processed can be determined, and the process ends.

[0054] As an example, when repeating step 101, the prediction model can be used to predict the output data of each sublink in the original link to be processed that precedes the selected write-type sublink, thereby obtaining the input data of the selected write-type sublink. The selected write-type sublink can then be executed in a simulation environment to simulate its operation in a production environment and obtain its output data. Furthermore, a separate storage area can be created, and the write operation of the selected write-type sublink during playback can be modified to write to this separate storage area to reduce the risk of data inconsistencies, playback interruptions, and failure to obtain output data.

[0055] Optionally, the termination conditions may include at least one of the following: the simulated main output data corresponding to all read-only sublinks of the link to be processed has been obtained, and the degree of similarity between the currently obtained simulated main output data and the main output data exceeds a second preset threshold, wherein the second preset threshold is greater than the first preset threshold. The first condition ensures that, for embodiments in which only read-only sublinks are used as reference sublinks, all read-only sublinks have been specifically replayed, effectively reducing the risk of sublinks being missed. The second condition allows, when simulated main output data that is very close to the main output data has been obtained, to directly deem this simulation as a valid replay of the link to be processed, significantly reducing the number of replays and the amount of computation, thereby reducing the computational load and improving data processing efficiency. It should be understood that the first preset threshold is used to ensure that the determined approximate main output data is sufficiently similar to the main output data, and is the lower limit requirement for determining a simulated main output data as approximate main output data. The second preset threshold is used to indicate that a simulated main output data is so close to the main output data that no other simulated main output data need to be considered. Therefore, the second preset threshold must be greater than the first preset threshold.

[0056] As an example, when determining the reference sub-links, the sub-links of the read-only type in the link to be processed may be determined one by one as reference sub-links according to the execution order of the sub-links.

[0057] It should be noted that for the case where no distinction is made between read-only and write types, the termination condition may include that the simulated main output data corresponding to all sub-links of the link to be processed have been obtained, and may also include that a preset number of simulated main output data have been obtained. This disclosure does not impose any restrictions on this.

[0058] Next, we will introduce the training process of the prediction model.

[0059] Optionally, each sub-link has a corresponding prediction model, and the pre-trained prediction model is trained through the following steps: obtaining multiple sample data of the sub-link to be trained in the production environment, the sample data including sample input data and sample output data; determining a candidate prediction model based on the multiple sample data; continuing to obtain new sample data of the sub-link to be trained in the production environment; inputting the newly obtained sample input data into the candidate prediction model to obtain predicted output data; correcting the candidate prediction model based on the predicted output data and the corresponding sample output data; repeating the steps from continuing to obtain new sample data to correcting the candidate prediction model until the training end condition is met, and using the final candidate prediction model as the pre-trained prediction model.

[0060] First of all, it should be noted that although the problems that occur in the link to be processed are sporadic, the link itself is an application link (hereinafter referred to as the target link) used to implement a specific function. Therefore, the execution logic of the target link is clear, but the data generated each time it is executed is different. Therefore, during pre-training, a prediction model dedicated to each sub-link of the target link can be trained separately. When sporadic problems occur in the target link, the pre-trained prediction links can be applied to achieve traffic replay. When training the sub-link to be trained, by first collecting multiple sample data generated by the sub-link to be trained in the production environment, a candidate prediction model can be preliminarily determined. Thereafter, by continuously using new sample data to correct the candidate prediction model, the candidate prediction model can gradually become stable. The candidate prediction model can also be adaptively adjusted in combination with the new data generated in the production environment, which helps to ensure the timeliness of the trained prediction model and improve the prediction accuracy.

[0061] It should be understood that for the sub-links to be trained, sample data can be collected by random sampling to ensure that the amount of data can meet the training needs, and it is not necessary to collect all the operating data. As an example, detailed link data of the target link can be regularly obtained from the production environment through offline tasks, and then each sub-link can be trained offline. During offline training, for the sub-links to be trained, part of the sample data can be used to determine the candidate prediction model first, and then another part of the sample data can be used to gradually correct the candidate prediction model, so that there is no need to repeatedly obtain data from the production environment. Furthermore, when the candidate prediction model has been determined, after new sample data is obtained later, the new sample data can be directly used to correct the existing prediction model, which is equivalent to correcting the latest candidate prediction model without re-determining the candidate prediction model. The above are all exemplary descriptions, and the present disclosure does not limit the specific training cycle arrangement of the prediction model.

[0062] It should also be understood that determining and revising the prediction model is based on comparing the predicted output data with the corresponding sample output data. The specific method is a mature technology in this field and will not be described in detail here. As an example, the training end condition is, for example, that the similarity between the predicted output data and the corresponding sample output data is greater than or equal to the training threshold. Accordingly, when revising the candidate prediction model, the similarity between the predicted output data and the corresponding sample output data can be calculated, and then a determination can be made as to whether the similarity is greater than or equal to the training threshold. If so, the training end condition is met and the pre-trained prediction model is obtained. If not, the candidate prediction model is revised based on the similarity.

[0063] Furthermore, as an example, different comparison strategies can be adopted in the process of comparing the predicted output data and the corresponding sample output data, depending on whether the output data of the sub-link to be trained is clear. Specifically, whether the output data is clear refers to whether the output data will change when the input data is constant, that is, whether other variables, such as the current time, random values, etc., are introduced into the process of calculating the output data. For the case where the output result of the sub-link to be trained is clear, a full matching strategy can be adopted. Only when each field in the predicted output data is equal to the corresponding field of the corresponding sample output data, the prediction result is considered accurate; for the case where the output result of the sub-link to be trained is unclear, a similarity strategy can be adopted to ensure that the predicted output data and the corresponding sample output data reach a certain similarity. The specific value can be determined according to the actual situation.

[0064] Optionally, for the above-mentioned situation where the output result of the sub-link to be trained is unclear, when training the prediction model, repeatedly executing the steps from continuing to obtain new sample data to revising the candidate prediction model may include: in response to the cumulative number of sample data used reaching a sample threshold, based on all predicted output data and corresponding sample output data, counting the difference probabilities of multiple output fields of the sub-link to be trained, wherein the difference probability of each output field is the proportion of the difference prediction data of the corresponding output field in all predicted output data, wherein the similarity between the corresponding output field of the difference prediction data and the corresponding output field of the sample output data corresponding to the difference prediction data is less than the similarity threshold; taking the output field with a difference probability greater than or equal to the difference threshold as the difference field; repeatedly executing the steps from continuing to obtain new sample data to revising the candidate prediction model, and using other fields of the multiple output fields except the difference field when revising the candidate prediction model. For example, if the sample threshold is 10,000 and the difference probability is 70%, then, after accumulating 10,000 sample data, if the similarity between field A of the predicted output data and field A of the corresponding sample output data for 7,000 or more sample data is less than the similarity threshold, it is considered that 7,000 or more sample data cannot be successfully predicted in field A. In other words, the difference probability of field A has reached the difference threshold, and field A is considered a difference field. In conjunction with the above, this means that field A is considered to be related to variables other than the input data, making it difficult to accurately predict field A. By calculating the difference probability for each of the multiple output fields of the sub-link to be trained and subsequently using the other fields of the multiple output fields except the difference field when revising the candidate prediction model, that is, not using the difference field with a high difference probability, this can reduce the possibility that the use of the difference field will reduce the similarity between the predicted output data and the corresponding sample output data, leading to an underestimation of the prediction accuracy of the current candidate prediction model, and thus reducing the possibility of the candidate prediction model being repeatedly revised. This customized training method for the difference field helps reduce the amount of correction computation, saves computational overhead, and improves the training efficiency of the prediction model.

[0065] Furthermore, for the difference fields, the patterns therein can be analyzed, for example, whether the difference fields are related to time can be analyzed, and the prediction results of the candidate prediction model for the difference fields can be determined based on the patterns, thereby fully improving the accuracy of the pre-trained prediction model.

[0066] Next, the data processing method of the present disclosure is introduced through a specific embodiment.

[0067] Figure 2 It is a logical diagram illustrating a data processing method according to a specific embodiment of the present disclosure.

[0068] Reference Figure 2, the input data of the link to be processed that occasionally encounters problems in the production environment is the main input data MInput, and the output data is the main output data MOutput. When passing through each sub-link, the function f'(x) corresponding to the sub-link is used to perform a prediction calculation on the output data to obtain the simulated output data of the i-th sub-link, referred to as simulated output data i, and recorded as SOutput-i'=f'(SInput-i)≈SOutput-i. Among them, SInput-i' represents the simulated input data of the i-th sub-link during playback, referred to as simulated input data i, and SOutput-i represents the output data of the i-th sub-link in the production environment, referred to as output data i. The above formula shows that after training, the function f'(x) can make the simulated output data SOutput-i' of the sub-link approximately equal to the actual output data SOutput-i of the corresponding sub-link in the production environment. For SInput-i', if no processing is required on the primary input data before executing the first sublink, then SInput-1' = SInput-1 = MInput. This means that simulated input data 1 is equal to input data 1 and, in turn, the primary input data. The simulated input data SInput-i' of other sublinks can be derived from the simulated output data of the preceding sublink (e.g., SOutput-(i-1)') and the data processing logic between the two sublinks. If no data processing is required between the two sublinks, the simulated input data SInput-i' of a sublink is equal to the simulated output data of its preceding sublink (e.g., SOutput-(i-1)'). Furthermore, after training, the function f'(x) can make the simulated input data SInput-i' approximately equal to the actual input data SInput-i of the corresponding sublink in a production environment. In this specific embodiment, the link to be processed has N sub-links. By taking each sub-link as a reference sub-link in turn, N traffic replays can be achieved. The replay operation is as follows:

[0069] frequency Skip reference sublink Simulate main output data First playback SInput1' MOutput'-1 Second playback SInput2' MOutput'-2 Third playback SInput3' MOutput'-3 ... ... ... Nth playback SInputN' MOutput'-N

[0070] For the skipped reference sub-link, its simulated input data SInput-i' is used to replay the reference sub-link in a simulation environment to obtain simulated output data SOutput-i', instead of being calculated by the function f'(x).

[0071] After the playback is completed, the simulated main output data MOutput'-i of each playback is compared with the main output data MOutput of the production environment in turn, and the one with the highest similarity is calculated to be MOutput'-k. If MOutput'-k=MOutput'-N, then the Nth sub-link is determined as the target sub-link.

[0072] Figure 3is a block diagram illustrating a data processing apparatus according to an example embodiment of the present disclosure. It should be understood that the data processing apparatus according to the example embodiment of the present disclosure can be implemented in a terminal device such as a smart phone, a tablet computer, a personal computer (PC) in software, hardware or a combination of software and hardware, or in a device such as a server.

[0073] Referring to Figure 3 The data processing apparatus 300 includes an acquisition unit 301, an imitation unit 302, a judgment unit 303, a determination unit 304, and a positioning unit 305.

[0074] The acquisition unit 301 can acquire input data and output data of a to-be-processed link as main input data and main output data, where the to-be-processed link includes a plurality of sequentially connected sub-links.

[0075] The imitation unit 302 can determine a reference sub-link from the plurality of sub-links of the to-be-processed link, and process the main input data based on the to-be-processed link to obtain imitated main output data, where the output data of the reference sub-link is obtained by executing the reference sub-link, and the output data of other sub-links of the to-be-processed link except the reference sub-link is obtained by a pre-trained prediction model, and the prediction model is used to predict the output data of the corresponding sub-link.

[0076] The judgment unit 303 can judge whether an end condition is met, where the end condition is a condition indicating that there is no need to determine a new reference sub-link from the plurality of sub-links of the to-be-processed link.

[0077] The determination unit 304 can, in response to the end condition being met, determine an approximate main output data from all imitated main output data of the to-be-processed link based on the main output data.

[0078] The positioning unit 305 can determine the reference sub-link corresponding to the approximate main output data as a target sub-link.

[0079] Optionally, the determination unit 304 can further: in response to the end condition being met, determine a similarity degree of each of all imitated main output data of the to-be-processed link to the main output data; and determine one of all imitated main output data of the to-be-processed link with a maximum similarity degree and greater than a first preset threshold as the approximate main output data.

[0080] Optionally, the imitation unit 302 can further: determine a read-only type sub-link from the plurality of sub-links of the to-be-processed link as the reference sub-link.

[0081] Optionally, the acquisition unit 301 may also: in response to the determination unit 304 being unable to determine the approximate main output data, take the write-type sub-link in the link to be processed as a new link to be processed, repeat the step of obtaining the input data and output data of the link to be processed as the main input data and main output data, and run the imitation unit 302 and the determination unit 304 again until the determination unit 304 determines the approximate main output data or the acquisition unit 301 is unable to determine a new link to be processed, wherein the determination unit 304 may also: when the similarity between all the simulated main output data of the link to be processed and the main output data is less than or equal to the first preset threshold, confirm that the approximate main output data cannot be determined.

[0082] Optionally, the termination condition includes at least one of the following: the simulated main output data corresponding to all read-only sub-links of the link to be processed have been obtained, and the similarity between the currently obtained simulated main output data and the main output data exceeds a second preset threshold, wherein the second preset threshold is greater than the first preset threshold.

[0083] Optionally, each sub-link has a corresponding prediction model, and the pre-trained prediction model is trained through the following steps: obtaining multiple sample data of the sub-link to be trained in the production environment, the sample data including sample input data and sample output data; determining a candidate prediction model based on the multiple sample data; continuing to obtain new sample data of the sub-link to be trained in the production environment; inputting the newly obtained sample input data into the candidate prediction model to obtain predicted output data; correcting the candidate prediction model based on the predicted output data and the corresponding sample output data; repeating the steps from continuing to obtain new sample data to correcting the candidate prediction model until the training end condition is met, and using the final candidate prediction model as the pre-trained prediction model.

[0084] Optionally, the steps from continuing to acquire new sample data to revising the candidate prediction model are repeated, including: in response to the number of cumulatively used sample data reaching a sample threshold, based on all predicted output data and corresponding sample output data, counting the difference probabilities of multiple output fields of the sub-link to be trained, wherein the difference probability of each output field is the proportion of the difference prediction data of the corresponding output field in all predicted output data, wherein the similarity between the corresponding output field of the difference prediction data and the corresponding output field of the sample output data corresponding to the difference prediction data is less than a similarity threshold; taking the output field with a difference probability greater than or equal to the difference threshold as the difference field; repeating the steps from continuing to acquire new sample data to revising the candidate prediction model, and using other fields of the multiple output fields except the difference field when revising the candidate prediction model.

[0085] Regarding the apparatus in the above embodiment, the specific manner in which each unit performs operations has been described in detail in the embodiment of the method, and will not be elaborated on here.

[0086] Figure 4 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure.

[0087] Reference Figure 4 The electronic device 400 includes at least one memory 401 and at least one processor 402. The at least one memory 401 stores a set of computer-executable instructions. When the computer-executable instruction set is executed by the at least one processor 402, the data processing method according to the exemplary embodiment of the present disclosure is executed.

[0088] As an example, the electronic device 400 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above-mentioned instruction set. Here, the electronic device 400 is not necessarily a single electronic device, but may also be any device or circuit that can execute the above-mentioned instructions (or instruction set) individually or in combination. The electronic device 400 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device that is interconnected with a local or remote (e.g., via wireless transmission) interface.

[0089] In electronic device 400, processor 402 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0090] The processor 402 can execute instructions or codes stored in the memory 401, wherein the memory 401 can also store data. Instructions and data can also be sent and received over the network via the network interface device, wherein the network interface device can use any known transmission protocol.

[0091] Memory 401 may be integrated with processor 402, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. Furthermore, memory 401 may comprise a separate device, such as an external disk drive, a storage array, or any other storage device usable by a database system. Memory 401 and processor 402 may be operatively coupled or may communicate with each other, for example, via an I / O port, a network connection, or the like, such that processor 402 can access files stored in memory.

[0092] In addition, the electronic device 400 may further include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 400 may be connected to each other via a bus and / or a network.

[0093] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium may also be provided, which, when the instructions in the computer-readable storage medium are executed by at least one processor, prompts the at least one processor to perform the data processing method according to the exemplary embodiment of the present disclosure. Examples of computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-RLTH , BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or ultra-fast digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device configured to store the computer program and any associated data, data files and data structures in a non-transitory manner and provide the computer program and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0094] According to exemplary embodiments of the present disclosure, a computer program product may also be provided. The computer program product includes computer instructions. When the computer instructions are executed by at least one processor, the at least one processor is prompted to perform the data processing method according to exemplary embodiments of the present disclosure.

[0095] According to the data processing method and device, electronic device, and computer-readable storage medium of the exemplary embodiments of the present disclosure, multiple playbacks are used to check the sub-links of the link to be processed. In each playback, for other sub-links other than the reference sub-link currently being checked, a pre-trained prediction model is used to obtain output data, thereby more accurately reproducing the output data of the corresponding sub-link when no problem occurs in the production environment, supplementing the complete link data, and at the same time performing a playback operation on the reference sub-link in a simulation environment, which helps to ensure that the playback operation results of the reference sub-link are reliable and the playback of the link to be processed proceeds smoothly, thereby obtaining more reliable simulated main output data of the link to be processed. On this basis, by determining the one that best matches the actual main output data from all the simulated main output data obtained through playback, the corresponding playback can be regarded as a valid playback of the link to be processed, and then the corresponding reference sub-link can be determined as the target sub-link with the problem. The target sub-link can be determined quickly, conveniently, and reliably without the need for manual intervention analysis, thereby achieving efficient data processing of occasional problems.

[0096] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0097] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A data processing method, characterized in that: The data processing method includes: Acquire input data and output data of a link to be processed as main input data and main output data, wherein the link to be processed includes a plurality of sub-links connected in sequence; Determining a reference sub-link from multiple sub-links of the link to be processed, and processing the main input data based on the link to be processed to obtain simulated main output data, wherein the output data of the reference sub-link is obtained by executing the reference sub-link, and the output data of other sub-links in the link to be processed except the reference sub-link are obtained by using a pre-trained prediction model, and the prediction model is used to predict the output data of the corresponding sub-links; Determining whether an end condition is satisfied, wherein the end condition is a condition indicating that there is no need to determine a new reference sub-link from the multiple sub-links of the link to be processed; In response to the end condition being met, determining, based on the main output data, an approximate main output data from all the simulated main output data of the link to be processed, wherein a degree of similarity between the approximate main output data and the main output data is greater than a first preset threshold; The reference sub-link corresponding to the approximate main output data is determined as the target sub-link.

2. The data processing method according to claim 1, wherein: In response to the end condition being met, determining, based on the main output data, an approximate main output data from all the simulated main output data of the link to be processed, includes: In response to the end condition being met, determining a degree of similarity between all the simulated main output data of the link to be processed and the main output data; From all the simulated main output data of the link to be processed, one with the greatest similarity and greater than the first preset threshold is determined as the approximate main output data.

3. The data processing method according to claim 2, wherein: The determining a reference sub-link from the multiple sub-links of the link to be processed includes: A read-only sub-link is determined from the multiple sub-links of the link to be processed as the reference sub-link.

4. The data processing method according to claim 3, wherein: After determining, in response to the end condition being met, an approximate main output data from all the simulated main output data of the link to be processed based on the main output data, the data processing method further includes: In response to the inability to determine the approximate main output data from all the simulated main output data of the link to be processed, the write-type sub-link in the link to be processed is used as a new link to be processed, and the steps from obtaining the input data and output data of the link to be processed to determining the approximate main output data are repeatedly performed until the approximate main output data is determined or a new link to be processed cannot be determined, wherein, when the degree of similarity between all the simulated main output data of the link to be processed and the main output data is less than or equal to the first preset threshold, it is confirmed that the approximate main output data cannot be determined.

5. The data processing method according to claim 2, wherein: The termination condition includes at least one of the following: the simulated main output data corresponding to all read-only sub-links of the link to be processed have been obtained, and the similarity between the currently obtained simulated main output data and the main output data exceeds a second preset threshold, wherein the second preset threshold is greater than the first preset threshold.

6. The data processing method according to any one of claims 1 to 5, characterized in that: Each sub-link has a corresponding prediction model, and the pre-trained prediction model is trained through the following steps: Acquire multiple sample data of the sub-link to be trained in a production environment, wherein the sample data includes sample input data and sample output data; Determining a candidate prediction model based on the multiple sample data; Continue to obtain new sample data of the sub-link to be trained in the production environment; Inputting the newly acquired sample input data into the candidate prediction model to obtain predicted output data; Modifying the candidate prediction model according to the predicted output data and the corresponding sample output data; Repeat the steps from continuing to obtain new sample data to revising the candidate prediction model until the training end condition is met, and use the candidate prediction model finally obtained as the pre-trained prediction model.

7. The data processing method according to claim 6, wherein: The step of repeatedly performing the steps from continuing to acquire new sample data to correcting the candidate prediction model includes: In response to the cumulative amount of used sample data reaching a sample threshold, statistically analyzing difference probabilities of multiple output fields of the sub-link to be trained based on all the predicted output data and the corresponding sample output data, wherein the difference probability of each output field is a proportion of difference predicted data of the corresponding output field in all the predicted output data, and wherein a similarity between the corresponding output field of the difference predicted data and the corresponding output field of the sample output data corresponding to the difference predicted data is less than a similarity threshold; The output field whose difference probability is greater than or equal to the difference threshold is used as the difference field; Repeat the steps from continuing to acquire new sample data to revising the candidate prediction model, and use other fields of the multiple output fields except the difference field when revising the candidate prediction model.

8. A data processing device, characterized in that: The data processing device includes: an acquiring unit configured to acquire input data and output data of a link to be processed as main input data and main output data, wherein the link to be processed includes a plurality of sub-links connected in sequence; an imitation unit configured to determine a reference sub-link from multiple sub-links of the link to be processed, and process the main input data based on the link to be processed to obtain simulated main output data, wherein the output data of the reference sub-link is obtained by executing the reference sub-link, and the output data of other sub-links in the link to be processed except the reference sub-link are obtained by using a pre-trained prediction model; a judging unit configured to judge whether an end condition is satisfied, wherein the end condition is a condition indicating that there is no need to determine a new reference sub-link from the multiple sub-links of the link to be processed; a determining unit configured to, in response to the end condition being met, determine, based on the main output data, an approximate main output data from all the simulated main output data of the link to be processed, wherein a degree of similarity between the approximate main output data and the main output data is greater than a first preset threshold; The positioning unit is configured to determine the reference sub-link corresponding to the approximate main output data as the target sub-link.

9. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer-executable instructions, Wherein, when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to perform the data processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to perform the data processing method according to any one of claims 1 to 7.

Images