Data processing method and device, computer device and storage medium
By taking traceability measures at different stages of data processing and obtaining trace data, the problems of large sample requirements and long learning time of AI systems are solved, and efficient AI model learning and rule output are achieved.
Patent Information
- Application Number
- CN202310967058.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-02
- Publication Date
- 2025-11-21
- Estimated Expiration
- 2043-08-02
AI Technical Summary
In existing technologies, AI systems require a large number of samples and a long learning time when learning artificial intelligence models, resulting in high system processing overhead, large sample requirements, and long learning time.
By defining the data processing stage for the data to be processed and processing the data accordingly based on the tracking measures at that stage, and obtaining the data, including proactively alerting abnormal data during the data query stage, using tools to track data during the data processing stage, recording row and column selection operations during the analysis and decision-making stage, and extracting and labeling decision motivations during the decision-making selection stage, the sample size requirement and learning time are reduced.
By reducing the required sample size and shortening the learning time, system overhead is reduced, while the learning efficiency of the AI model and the interpretability of the rule output are improved.
Smart Images
Figure CN117235516B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of computer, and particularly relates to a data processing method and device, computer equipment and storage medium. BACKGROUND
[0002] In the aspect of network optimization, in the related art, an artificial intelligence (AI) model learning is mainly performed by using full-process service data. The AI system needs to extract rules from the service data, and different problems need sufficient sample quantity to train the rules, so that the sample demand quantity is large, the learning time is long, and the system processing cost is large. SUMMARY
[0003] The present application provides a data processing method and device, computer equipment and storage medium. The specific solutions are as follows:
[0004] An embodiment of the present application provides a data processing method, comprising:
[0005] obtaining to-be-processed data;
[0006] determining a trace processing stage to which the to-be-processed data belongs;
[0007] performing trace processing on the to-be-processed data according to a trace measure corresponding to the trace processing stage, to obtain trace data corresponding to the to-be-processed data.
[0008] An embodiment of the present application provides a data processing device, comprising:
[0009] a first obtaining module, configured to obtain to-be-processed data;
[0010] a first determining module, configured to determine a trace processing stage to which the to-be-processed data belongs;
[0011] a second obtaining module, configured to perform trace processing on the to-be-processed data according to a trace measure corresponding to the trace processing stage, to obtain trace data corresponding to the to-be-processed data.
[0012] An embodiment of the present application provides computer equipment, comprising a processor and a memory;
[0013] The processor runs a program corresponding to an executable program code stored in the memory by reading the executable program code, to implement the road anomaly recording method in the above embodiment.
[0014] An embodiment of the present application provides a non-transitory computer readable storage medium, which stores a computer program. When the program is executed by a processor, the road anomaly recording method in the above embodiment is implemented.
[0015] Additional aspects and advantages of the application will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following and / or can be learned by practice of the application. BRIEF DESCRIPTION OF DRAWINGS
[0016] The above and / or additional aspects and advantages of the application will become apparent and be well understood from a review of the description of an embodiment, taken in conjunction with the drawings, in which:
[0017] Figure 1 A flowchart of a data processing method provided for an embodiment of the application;
[0018] Figure 2 A flowchart of another data processing method provided for an embodiment of the application;
[0019] Figure 3 A schematic diagram of active reminding of abnormal data provided for an embodiment of the application;
[0020] Figure 4 A schematic diagram of learning differences between a non-trace table and a trace table provided for an embodiment of the application;
[0021] Figure 5 A flowchart of another data processing method provided for an embodiment of the application;
[0022] Figure 6 A flowchart of a data collection and automatic classification storage process provided for an embodiment of the application;
[0023] Figure 7 A data trace and application schematic diagram provided for an embodiment of the application;
[0024] Figure 8 A trace processing stage distribution diagram provided for an embodiment of the application;
[0025] Figure 9 A schematic diagram of trace data storage according to a three-layer five-stage model provided for an embodiment of the application;
[0026] Figure 10 A structural schematic diagram of a data processing device provided for an embodiment of the application. DETAILED DESCRIPTION
[0027] Embodiments of the application are described in detail below with reference to the attached drawing figures, wherein the same or like elements have the same or similar reference numerals and thereby have the same or similar function. The embodiments described below are examples intended to provide an explanation of the application and are not intended to restrict the application.
[0028] The data processing method, device, computer device and storage medium of the embodiments of the present application are described below with reference to the drawings.
[0029] Figure 1 A flowchart of a data processing method provided by the embodiments of the present application is shown.
[0030] As shown in Figure 1 , the data processing method comprises:
[0031] In step 101, the data to be processed is obtained.
[0032] In the present application, the data to be processed can refer to data that can be processed by leaving traces. The data to be processed can be obtained based on user operation, or can be directly read from a file, etc. The present application does not limit this.
[0033] For example, when a user performs data query, the computer device can obtain data satisfying the query condition from the business data according to the query condition, that is, obtain the query result. The query result can be considered as the data to be processed. Alternatively, the user can perform data analysis based on the query result, for example, can select part of the data from the query result for analysis. The data selected for analysis can be considered as the data to be processed, etc. Alternatively, the data that can be processed by leaving traces can also be obtained through other ways.
[0034] In step 102, the trace processing stage to which the data to be processed belongs is determined.
[0035] In the present application, the trace processing stage can include but is not limited to the data query stage, the data processing stage, the analysis and decision-making stage, the decision-making selection stage, etc.
[0036] In the present application, the trace processing stage to which the data to be processed belongs can be determined according to the obtaining method or source of the data to be processed, the role of the data to be processed, etc. For example, if the data to be processed is obtained by query, it can be determined that the trace processing stage to which the data to be processed belongs is the data query stage. For another example, if the role of the data to be processed is to optimize the network, that is, decision-making or measures, it can be determined that the trace processing stage to which the data to be processed belongs is the decision-making selection stage.
[0037] In step 103, the data to be processed is processed by leaving traces according to the trace processing stage corresponding to the trace processing stage, so as to obtain the trace data corresponding to the data to be processed.
[0038] In the present application, different trace measures can be adopted for the to-be-processed data according to different trace processing stages to which the to-be-processed data belongs. For example, the trace measure corresponding to the trace processing stage to which the to-be-processed data belongs can be active reminding of abnormal data when the trace processing stage to which the to-be-processed data belongs is a data query stage, and the trace measure corresponding to the trace processing stage to which the to-be-processed data belongs can be row and column checking when the trace processing stage to which the to-be-processed data belongs is an analysis and decision stage.
[0039] The trace data can include the to-be-processed data and corresponding trace information.
[0040] In the present application, trace processing of the to-be-processed data can be performed according to the trace measure corresponding to the trace processing stage, so as to meet the trace processing requirements of different trace processing stages and obtain trace data of different trace processing stages.
[0041] In the embodiments of the present application, the to-be-processed data is obtained, and the trace processing stage to which the to-be-processed data belongs is determined. The to-be-processed data is processed according to the trace measure corresponding to the trace processing stage, and the trace data corresponding to the to-be-processed data is obtained. Thus, the to-be-processed data is processed according to the trace measure corresponding to the trace processing stage to which the to-be-processed data belongs, so as to meet different trace processing requirements, obtain trace data of different trace processing stages, and reduce the sample size required for learning, shorten the learning time, and save system overhead based on the trace data of different trace processing stages for AI model learning.
[0042] Figure 2 Another flowchart of a data processing method provided by the embodiments of the present application is shown.
[0043] As shown in Figure 2 The data processing method includes the following steps.
[0044] In step 201, to-be-processed data is obtained.
[0045] In the present application, when a user queries related data on the platform side, the computer device can obtain data satisfying the query condition from the business data according to the query condition, that is, obtain the query result. The query result can be regarded as to-be-processed data.
[0046] In step 202, it is determined that the trace processing stage to which the to-be-processed data belongs is a data query stage.
[0047] In the present application, since the to-be-processed data is obtained based on data query, it can be determined that the trace processing stage to which the to-be-processed data belongs is a data query stage.
[0048] In step 203, abnormal data in the to-be-processed data is determined.
[0049] In this application, the data to be processed can be compared with the stored sample data to determine whether there is abnormal data in the data to be processed, or the data to be processed can be compared with a preset data range. If some data is not within the preset data range, the data can be considered as abnormal data.
[0050] Step 204: Provide a notification for abnormal data.
[0051] One approach is to highlight abnormal data on the page when the data to be processed is returned and displayed, to alert the user. This can be done by highlighting abnormal data, making the font of abnormal data bold, or displaying abnormal data in a special color (such as red).
[0052] As another implementation method, when returning and displaying the data to be processed, a prompt message indicating the existence of abnormal data can be displayed. The prompt message may include information such as the abnormal data and the location of the abnormal data in the data to be processed.
[0053] It should be noted that other methods may also be used to indicate abnormal data in this application, but this application will not address that.
[0054] Step 205: If a marking operation on abnormal data is detected, obtain the trace data based on the marked abnormal data.
[0055] In this application, users can mark abnormal data based on the prompts. Therefore, when the marking operation is detected, the marked abnormal data can be identified, and trace data can be obtained based on the marked abnormal data. The trace data may include the marked abnormal data within the data to be processed.
[0056] To facilitate understanding, the following will be combined with... Figure 3 To explain, Figure 3 This is a schematic diagram illustrating an active alert for abnormal data provided in an embodiment of this application. For example... Figure 3 As shown, users can perform data queries, and then the query results are returned. The query results can then be processed. Specifically, the queried data can be compared with the stored model sample data. When it is found that the data returned by the query is outside the normal distribution range, the abnormal data in the query results will be highlighted when the query results are displayed, which can be easily marked by the user.
[0057] For example, when querying a certain indicator (such as wireless connection rate) of a certain base station over a certain period of time, the position of the queried wireless connection rate in the distribution map of wireless connection rates of all base stations in the background database can be compared. If it is found that the wireless connection rate is in the bottom 5% or the top 5% of the wireless connection rate of the entire network, the wireless connection rate will be highlighted on the webpage to remind the user to pay attention.
[0058] In an embodiment of the present application, if it is determined that the trace processing stage to which the to-be-processed data belongs is the data query stage, the abnormal data in the to-be-processed data can be determined, the abnormal data is prompted, and in a case where a marking operation on the abnormal data is detected, the trace data is acquired according to the marked abnormal data. Thus, in the data query stage, the abnormal data in the query result can be prompted, and the user is facilitated to mark, so that the trace data in the data query stage can be acquired according to the marking operation of the user.
[0059] In an embodiment of the present application, the user can import data to be analyzed based on the query result, and the imported data can be determined as to-be-processed data. Since the to-be-processed data is acquired through the user import operation, it can be determined that the trace processing stage to which the to-be-processed data belongs is the data processing stage. After the user imports the data to be analyzed, the user can use a tool to analyze the data, and the tool can be, for example, a sorting management tool, a filtering tool, a clear filtering tool, a batch modification tool, a batch deletion tool, a column-based deduplication tool, and the like.
[0060] In the present application, the tool used to analyze the to-be-processed data can be determined by detecting a selection operation of the user on the tool, and the trace data can be acquired according to the tool used. The trace data can include, for example, which tables or which data processing tools are used, the functions of the tools used, the order of the tools used, and the like.
[0061] For example, the data queried by the user can be directly imported into the data analysis module, and the user can edit and process the imported data in the data analysis module. All operation actions can be traced through online tools.
[0062] In an embodiment of the present application, if it is determined that the trace processing stage to which the to-be-processed data belongs is the data analysis stage, the tool used to analyze the to-be-processed data can be determined, and the trace data can be acquired according to the tool used. Thus, in the data analysis stage, all operations can be traced through tools, so that the trace data in the data analysis stage can be acquired according to the tool used.
[0063] In an embodiment of the present application, the content finally output by the data analysis can be determined as to-be-processed data. Thus, according to the source of the to-be-processed data, it can be determined that the trace processing stage to which the to-be-processed data belongs is the analysis decision stage. When a row-column check operation on the to-be-processed data is detected, one or more of the checked rows, columns, and cells and the check order can be determined, and the trace data can be acquired according to the one or more of the checked rows, columns, and cells and the check order. The trace data can include, for example, which rows, columns, and cells of which table are checked and the check order.
[0064] The AI system can know how the expert thinks when dealing with a problem through the checked records and sequence of the traces. For ease of understanding, the following will be described in combination with Figure 4 , Figure 4 A schematic diagram of learning differences between a table without traces and a table with traces is provided for the embodiments of the present application. As shown in the figure, for the table without traces, the AI system finds the rules from the A1-D6 columns, and a large number of samples need to be learned to output the rules. For the table with traces, the expert tells the AI system the cells that affect its decision and the sequence of the cells, which are B3->C2->D3. The AI system learns the differences between the cells B3, C2, D3 and other cell contents, that is, the detailed experience is passed to the AI system through the traces, which can save the learning time of the AI system and reduce the number of samples that need to be learned. Figure 4
[0065] In the embodiments of the present application, if it is determined that the trace processing stage to which the to-be-processed data belongs is the analysis and decision stage, the checked row, column, cell or one or more of them and the checking sequence can be determined in the case of detecting the row and column checking operation on the to-be-processed data, and the trace data can be acquired according to the checked row, column, cell or one or more of them and the checking sequence. Thus, in the analysis and decision stage, the trace data of the data analysis stage can be acquired according to the checked row, column, cell or one or more of them and the checking sequence.
[0066] Figure 5 A flowchart of another data processing method provided for the embodiments of the present application is shown in the figure.
[0067] As shown in the figure, the data processing method includes the following steps. Figure 5
[0068] Step 501: acquiring to-be-processed data.
[0069] In the present application, the full amount of data obtained through the query in the automation process can be used as the to-be-processed data.
[0070] Step 502: determining that the trace processing stage to which the to-be-processed data belongs is the decision selection stage.
[0071] In the present application, the trace processing stage to which the to-be-processed data belongs can be determined as the decision selection stage according to the acquisition method or source of the to-be-processed data.
[0072] Step 503: extracting a first rule measure from the to-be-processed data.
[0073] In the present application, the to-be-processed data can be preprocessed, and the preprocessed data can be cleaned. After cleaning, feature extraction is performed to obtain feature data. The first rule measure can be extracted from the feature data by using a decision model.
[0074] The decision model can be an AI model for providing a decision to a user, the first rule measure can include a first rule and a first measure corresponding to the first rule, and the first measure can be considered as an optimization measure corresponding to the first rule. For example, the first rule is 4G weak coverage, and the first measure is adjusting an antenna.
[0075] To facilitate subsequent calculation, the first rule measure can be represented by an array, including an array of the first rule and an array of the first measure.
[0076] Step 504, determining the similarity between the first rule measure and each second rule measure in the rule base of the decision model.
[0077] In this application, the second rule measure in the rule base of the decision model can be learned by the decision model from a large number of samples. Each second rule measure can include a second rule, a second measure corresponding to the second rule, and the like, and the second measure can be considered as an optimization measure corresponding to the second rule.
[0078] As an implementation manner, the second rule measure in the rule base can be put into an array for vectorization, for example, the rule measure in the rule base is shown in Table 1:
[0079] Table 1: Rule measure example in rule base
[0080]
[0081]
[0082] In this application, the second rule measure in the rule base can be put into an array, as shown below:
[0083] X array [X1=main reason, X2=4G RB priority MCS selection, X3=weak coverage ratio, X4=uplink PRB utilization rate, X5=downlink PRB utilization rate, X6=reference signal power]; and Y array [optimization measure]. Wherein, the X array represents the second rule, and the Y array represents the second measure. Each row of data in the second and third rows in the above Table 1 can be regarded as a second rule measure, and each row of data includes a second rule and a corresponding second measure.
[0084] Then, the above array can be subjected to data binning operation and normalization operation.
[0085] When a user makes a new decision selection, the decision model can extract the X array and the Y array [optimization measure] (i.e. the first rule measure) from the user sample (i.e. the to-be-processed data), and compare the similarity with the vectorized second rule measure in the rule base.
[0086] For example, the similarity calculation formula based on vector is shown as follows:
[0087]
[0088] wherein S x , S y are the sample standard deviations of x and y, respectively, x, y are the first and second regular measures after normalization processing, and the range of p(x, y) is [-1, 1], the greater the absolute value, the higher the similarity.
[0089] In step 505, the trace data is obtained according to the similarity corresponding to each second regular measure.
[0090] In the present application, if the similarity corresponding to each second regular measure in the rule library is less than the preset threshold, the first measure and the second measure in the second regular measure with the highest similarity can be provided for the user to select, prompting the user to select the measure and label the decision motive, and if the labeling operation of the decision motive is detected, the target decision selected by the user and the decision motive corresponding to the target decision input by the user can be obtained, and the trace data can be obtained according to the target decision and the decision motive, wherein the trace data can include the first rule, the target decision, the decision motive of the target decision, etc.
[0091] If the similarity corresponding to any second regular measure in the rule library is greater than the preset threshold, the first measure can be provided to the user, and the trace data can be obtained according to the first measure, wherein the trace data can include the first rule, the first measure, etc.
[0092] If the similarity corresponding to any second regular measure in the rule library is greater than the preset threshold, the second measure in the second regular measure can be provided to the user, and the trace data can be obtained according to the second measure in the second regular measure, wherein the trace data can include the first measure, the second measure in the second regular measure, etc.
[0093] In the present application, when the similarity of all second regular measures in the rule library and the new sample of the user is less than the preset threshold, that is, the similarity is very low, the AI system can prompt the user to label the decision motive or consideration point, and the decision motive or consideration point labeled by the user is taken as a new feature column and included in the sample library for learning, so as to realize the update iteration of the decision model. The AI system can include the decision model, etc.
[0094] In an embodiment of the present application, the trace data can also be classified and stored. Optionally, in the present application, after the trace data is obtained by using the above trace measures to perform trace processing on the to-be-processed data, the trace data can be classified and stored.
[0095] In the present application, the layer of the trace data can be determined from the user layer, the service layer and the network layer according to the purpose of the trace data, and the stage of the trace data can be determined from the perception stage, the behavior stage, the decision stage, the execution stage and the evaluation stage according to the trace processing stage of the trace data, and the trace data can be classified and stored according to the layer and the stage.
[0096] For example, the trace data is obtained in the data query stage, and it can be considered that the trace data is in the behavior stage. For another example, the trace data includes the checked rows and columns and the checking sequence, and it can be considered that the trace data is in the decision stage. For another example, the trace data includes the target decision input by the user and the decision motive of the target decision, and it can be considered that the trace data is in the decision stage.
[0097] It can be seen that the trace data can be classified and stored from two dimensions of the layer and the stage in the present application.
[0098] In the present application, the trace data can be processed and classified by the intermediate layer (sensing classifier), and the data is classified and stored in different layers (user layer or service layer or network layer) and different stages (perception stage or behavior stage or decision stage or execution stage or evaluation stage) by the intermediate layer (sensing classifier). The data collection and automatic classification and storage process is shown in Figure 6 .
[0099] The data classification process is as follows:
[0100] In the data collection module, all data fields (indicators or parameters or root causes) contacted by network optimization daily are defined in advance and put into the data dictionary table. The data dictionary table includes the field English name, the field Chinese name, the field type, the field length, manual / automatic, whether it is required to fill in, the attribution node and the like.
[0101] For example, a certain data in the data dictionary table includes: the field English name is FAULTTIME, the field Chinese name is fault time, the field type is VARCHAR2, the field length is 30, the manual / automatic is automatic, whether it is required to fill in is yes, and the attribution node is basic information.
[0102] The intermediate layer (sensing classifier) can classify the fields transmitted by the data collection module according to the three-layer five-stage model and store them in a large table (hereinafter referred to as a wide table); through the wide table, the specific content of a work order in different stages (perception stage or behavior stage or decision stage or execution stage or evaluation stage) can be very conveniently queried.
[0103] For example, the example of the user in the analysis stage (query) stored in the wide table is shown in Table 2.
[0104] Table 2 Analysis stage (query) result example
[0105]
[0106] Wherein, the work order number, that is, task_id is AG 130369, the RRC connection average number can be a query condition, and the network layer_analysis stage indicates that the queried data is stored in the network layer and stored in the analysis stage.
[0107] An example of the user stored in the wide table in the execution stage (query) is shown in Table 3, wherein the modified parameters or the values before and after the parameter modification can be directly presented in the wide table through the work order number (task_id).
[0108] Table 3: Example of execution stage (query) result
[0109] Work Order Number Network Layer_Execution Phase task_id Normalized PDSCH Power Offset AG 270609 [2022-08-17 11 :01 :44 | Centralized Parameter Platform | 0 -> 2] AG 280309 [2022-08-18 17 :27 :12 | Centralized Parameter Platform | 0 -> 6]
[0110] In the present application, the trace data is classified according to a three-layer (user layer or service layer or network layer) five-stage (perception or behavior or decision or execution or evaluation stage) model and stored in a wide table. The data is losslessly stored in the wide table, and the AI model only needs to check one table to obtain all the field information to be learned, and the explainability of the output rule learned by the AI model is stronger.
[0111] In order to facilitate understanding of the data processing method of the present application, the following will be described in combination with Figures 7-9 .
[0112] As Figure 7 shown, the passive trace full-service data, automatic monitoring of service data, data analysis online, row and column check trace function, user marking automatic reminder, etc. can be trace processed to obtain trace data, which is classified and stored according to three layers and five stages. Then, the stored data is preprocessed, and the preprocessed data is cleaned and feature extracted. The extracted feature data is learned by an AI model, and the obtained AI model is evaluated and applied.
[0113] Among them, the passive trace full-service data can include full-service data, and data obtained when querying, analyzing, and decision selecting the full-service data.
[0114] Among them, when the service data automatic monitoring finds that the service data is abnormal, the user is reminded to pay attention; the data analysis online means that the operation of the data analysis process can be traced through the online tool; the row and column check trace function can mean that the user can check the rows or columns that have an impact on the analysis and decision one by one; and the user marking automatic reminder means that when the user makes a decision analysis or scheme selection, the user can be reminded to mark the motivation or consideration point.
[0115] AsFigure 8 As shown, the trace processing stage has a data query stage, a data processing stage, an analysis decision stage, and a decision selection stage, wherein the trace measure of the data query stage is the active reminder of abnormal data, the trace measure of the data processing stage is the full-line data analysis, the trace measure of the analysis decision stage is the row and column selection trace function, and the trace measure of the decision selection stage is to remind the user to mark the motivation or consideration point when the user makes a decision or selects a scheme.
[0116] The scheme of the present application increases the "active trace" mechanism on the basis of "passive trace", and on the basis of business sample trace, makes it easier for the AI system to learn and acquire expert experience and decision motivation (the motivation or consideration point of the user for different decisions for different problems), reduces the sample size required for learning, shortens the learning time, and saves system overhead.
[0117] As shown, Figure 9 The scheme of the present application classifies and stores the trace data in three levels (user level or business level or network level) and five stages (perception or behavior or decision or execution or evaluation stage), and after data processing, the AI system is trained, the output rule has strong explainability, the business expert is easy to interpret and cooperate with the AI expert, and the AI system is more effective Experience is passed to the AI system for rule optimization and system iteration and evolution.
[0118] In order to realize the above-mentioned embodiments, an embodiment of the present application further provides a data processing device. Figure 10 A structural schematic diagram of a data processing device provided by an embodiment of the present application.
[0119] As shown, Figure 10 The data processing device 1000 includes:
[0120] The first acquisition module 1010 is configured to acquire the to-be-processed data.
[0121] The first determination module 1020 is configured to determine a trace processing stage to which the to-be-processed data belongs.
[0122] The second acquisition module 1030 is configured to perform trace processing on the to-be-processed data according to a trace measure corresponding to the trace processing stage, to acquire trace data corresponding to the to-be-processed data.
[0123] In a possible implementation manner of an embodiment of the present application, the trace processing stage is a data query stage, and according to the trace measure corresponding to the trace processing stage, the second acquisition module 1030 is configured to:
[0124] determine abnormal data in the to-be-processed data;
[0125] prompt the abnormal data;
[0126] In a case where the marking operation on the abnormal data is detected, the trace data is acquired according to the marked abnormal data.
[0127] In a possible implementation manner of the embodiment of the present application, the trace processing stage is the data processing stage, and the second acquisition module 1030 is configured to:
[0128] determine a tool for analyzing the to-be-processed data;
[0129] acquire the trace data according to the tool.
[0130] In a possible implementation manner of the embodiment of the present application, the trace processing stage is the analysis decision stage, and the second acquisition module 1030 is configured to:
[0131] In a case where the row-column check operation on the to-be-processed data is detected, determine one or more of the checked row, column, and cell and the checking sequence;
[0132] acquire the trace data according to the one or more of the checked row, column, and cell and the checking sequence.
[0133] In a possible implementation manner of the embodiment of the present application, the trace processing stage is the decision selection stage, and the second acquisition module 1030 is configured to:
[0134] extract a first rule measure from the to-be-processed data, wherein the first rule measure includes a first rule and a first measure corresponding to the first rule;
[0135] determine a similarity between the first rule measure and each second rule measure in a rule library of a decision model, wherein the second rule measure includes a second rule and a second measure corresponding to the second rule;
[0136] acquire the trace data according to the similarity corresponding to each second rule measure.
[0137] In a possible implementation manner of the embodiment of the present application, the second acquisition module 1030 is configured to:
[0138] in a case where the similarity corresponding to each second rule measure in the rule library is less than a preset threshold, provide the second measure in the second rule measure with the highest similarity and the first measure, and prompt to select the measure and mark a decision motive;
[0139] In a case where the marking operation on the decision motive is detected, acquire a target decision and a decision motive corresponding to the target decision;
[0140] acquire the trace data according to the target decision and the decision motive;
[0141] In a case where the similarity corresponding to any second rule measure is greater than a preset threshold, a first measure or a second measure in any second rule measure is provided;
[0142] According to the first measure or the second measure in any second rule measure, trace data is acquired.
[0143] In a possible implementation manner of the embodiment of the present application, the apparatus can further include:
[0144] The second determination module is configured to determine a layer to which the trace data belongs from a user layer, a service layer and a network layer according to a use of the trace data.
[0145] The third determination module is configured to determine a stage in which the trace data is located from a perception stage, a behavior stage, a decision stage, an execution stage and an evaluation stage according to a trace processing stage to which the trace data belongs.
[0146] The storage module is configured to store the trace data according to the layer to which the trace data belongs and the stage in which the trace data is located.
[0147] In a possible implementation manner of the embodiment of the present application, the apparatus can further include:
[0148] The reading module is configured to read the stored trace data.
[0149] The training module is configured to learn a decision model by using the read trace data.
[0150] It should be noted that the above explanation and description of the data processing method embodiment also applies to the data processing apparatus of the embodiment, and thus will not be described here again.
[0151] In the embodiment of the present application, by acquiring to-be-processed data and determining a trace processing stage to which the to-be-processed data belongs, the to-be-processed data is processed according to a trace measure corresponding to the trace processing stage, and trace data corresponding to the to-be-processed data is acquired. Thus, the to-be-processed data is processed by using a corresponding trace measure based on the trace processing stage to which the to-be-processed data belongs, so that different trace processing requirements can be met, trace data of different trace processing stages can be acquired, an AI model can be learned based on the trace data of different trace processing stages, the sample amount required for learning can be reduced, the learning time can be shortened, and system overheads can be saved.
[0152] In order to implement the above-mentioned embodiments, the embodiment of the present application further proposes a computer device including a processor and a memory;
[0153] The processor runs a program corresponding to an executable program code stored in the memory by reading the executable program code, so as to implement the data processing method as described in the above-mentioned embodiments.
[0154] To achieve the above-mentioned embodiments, the embodiments of the present application further provide a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data processing method as described in the above-mentioned embodiments.
[0155] In the description of the present specification, the terms "first", "second" are only used for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise specifically limited.
[0156] Although the embodiments of the present application have been shown and described above, it can be understood that the above-mentioned embodiments are exemplary and cannot be understood as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above-mentioned embodiments within the scope of the present application.
Claims
1. A data processing method, characterized in that, include: Obtain the data to be processed; Determine the logging stage to which the data to be processed belongs; Based on the trace retention measures corresponding to the trace retention stage, the data to be processed is subjected to trace retention processing to obtain the trace retention data corresponding to the data to be processed. Based on the purpose of the trace data, the layer to which the trace data belongs is determined from the user layer, business layer, and network layer; Based on the trace processing stage to which the trace data belongs, the stage in which the trace data is located is determined from the perception stage, behavior stage, decision-making stage, execution stage, and evaluation stage; The trace data is classified and stored according to the level and stage in which it is located.
2. The method as described in claim 1, characterized in that, The trace retention process is a data query process. The step of performing trace retention processing on the data to be processed according to the trace retention measures corresponding to the trace retention process to obtain the trace data corresponding to the data to be processed includes: Identify the abnormal data in the data to be processed; The abnormal data will be displayed as a notification; If a marking operation on the abnormal data is detected, the trace data is obtained based on the marked abnormal data.
3. The method as described in claim 1, characterized in that, The trace retention process is a data processing stage. The step of performing trace retention processing on the data to be processed according to the trace retention measures corresponding to the trace retention stage to obtain the trace data corresponding to the data to be processed includes: Determine the tools to be used for analyzing the data to be processed; The trace data is obtained according to the tools used.
4. The method as described in claim 1, characterized in that, The trace retention process is an analysis and decision-making stage. The step of performing trace retention processing on the data to be processed according to the trace retention measures corresponding to the trace retention stage to obtain the trace data corresponding to the data to be processed includes: If a row or column selection operation is detected on the data to be processed, determine one or more of the selected rows, columns, and cells and the selection order. The trace data is obtained based on one or more of the selected rows, columns, and cells, and the order in which they are selected.
5. The method as described in claim 1, characterized in that, The trace retention processing stage is a decision-making and selection stage. The step of performing trace retention processing on the data to be processed according to the trace retention measures corresponding to the trace retention processing stage to obtain the trace data corresponding to the data to be processed includes: Extract a first rule measure from the data to be processed, wherein the first rule measure includes a first rule and a first measure corresponding to the first rule; Determine the similarity between the first rule measure and each second rule measure in the rule base of the decision model, wherein the second rule measure includes a second rule and a second measure corresponding to the second rule; The trace data is obtained based on the similarity corresponding to each of the second rule measures.
6. The method as described in claim 5, characterized in that, The step of obtaining the trace data based on the similarity corresponding to each of the second rule measures includes: If the similarity of each second rule measure in the rule base is less than a preset threshold, the second measure of the second rule measure with the highest similarity and the first measure are provided, and the selection of measures is prompted and the decision motivation is marked. Upon detecting the annotation operation of decision motivation, obtain the target decision and the decision motivation corresponding to the target decision; Based on the stated target decision and the stated decision motivation, the trace data is obtained; If the similarity corresponding to any second rule measure is greater than the preset threshold, provide the first measure or the second measure among the second rule measures. The trace data is obtained according to the first measure or the second measure in any of the second rule measures.
7. The method according to any one of claims 1-6, characterized in that, Also includes: Read the stored trace data; The decision-making model is learned using the retrieved trace data.
8. A data processing apparatus, characterized in that, include: The first acquisition module is used to acquire data to be processed. The first determination module is used to determine the trace processing stage to which the data to be processed belongs; The second acquisition module is used to perform trace processing on the data to be processed according to the trace processing measures corresponding to the trace processing stage, so as to obtain the trace data corresponding to the data to be processed. The second determining module is used to determine the layer to which the trace data belongs from the user layer, business layer, and network layer based on the purpose of the trace data. The third determining module is used to determine the stage of the trace data based on the trace processing stage to which the trace data belongs, from the perception stage, behavior stage, decision-making stage, execution stage and evaluation stage; The storage module is used to classify and store the trace data according to the level and stage in which the trace data is located.
9. The apparatus as claimed in claim 8, characterized in that, The trace retention process is a data query process. Based on the trace retention measures corresponding to the trace retention process, the second acquisition module is used for: Identify the abnormal data in the data to be processed; The abnormal data will be displayed as a notification; If a marking operation on the abnormal data is detected, the trace data is obtained based on the marked abnormal data.
10. The apparatus as claimed in claim 8, characterized in that, The trace retention stage is a data processing stage, and the second acquisition module is used for: Determine the tools to be used for analyzing the data to be processed; The trace data is obtained according to the tools used.
11. The apparatus as claimed in claim 8, characterized in that, The trace processing stage is an analysis and decision-making stage, and the second acquisition module is used for: If a row or column selection operation is detected on the data to be processed, determine one or more of the selected rows, columns, and cells and the selection order. The trace data is obtained based on one or more of the selected rows, columns, and cells, and the order in which they are selected.
12. The apparatus as claimed in claim 8, characterized in that, The trace processing stage is a decision-making stage, and the second acquisition module is used for: Extract a first rule measure from the data to be processed, wherein the first rule measure includes a first rule and a first measure corresponding to the first rule; Determine the similarity between the first rule measure and each second rule measure in the rule base of the decision model, wherein the second rule measure includes a second rule and a second measure corresponding to the second rule; The trace data is obtained based on the similarity corresponding to each of the second rule measures.
13. The apparatus as claimed in claim 12, characterized in that, The second acquisition module is used for: If the similarity of each second rule measure in the rule base is less than a preset threshold, the second measure of the second rule measure with the highest similarity and the first measure are provided, and the selection of measures is prompted and the decision motivation is marked. Upon detecting the annotation operation of decision motivation, obtain the target decision and the decision motivation corresponding to the target decision; Based on the stated target decision and the stated decision motivation, the trace data is obtained; If the similarity corresponding to any second rule measure is greater than the preset threshold, provide the first measure or the second measure among the second rule measures. The trace data is obtained according to the first measure or the second measure in any of the second rule measures.
14. The apparatus according to any one of claims 8-13, characterized in that, Also includes: The read module is used to read the stored trace data; The training module is used to learn the decision model using the read trace data.
15. A computer device, characterized in that, Including processor and memory; The processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the method as described in any one of claims 1-7.
16. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.
Citation Information
Patent Citations
Data processing method and device
CN110457348A