Internet-based big data analysis method and system

Abstract

The invention provides a big data analysis method and system based on the Internet, and belongs to the technical field of data analysis, and the method comprises the steps: sequentially carrying out the result analysis of to-be-analyzed data based on each analysis index in an analysis index set, and obtaining the result defect data based on each analysis index; performing comparative analysis on the actual log of the task execution logic and the standard log to obtain logic abnormal data; according to the occurrence position of the result defect data under each analysis index in the to-be-analyzed data, a first label is given to each position unit, meanwhile, mapping association is carried out on the occurrence position and the occurrence moment of the logic abnormal data, a second label is given to each position unit, and defect level judgment is carried out on the to-be-analyzed data. The defect level can be accurately determined, a basis is provided for follow-up reasonable defect processing, and the service experience of a user is effectively guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of data analysis, and in particular to a big data analysis method and system based on the Internet. Background Art

[0002] As the Internet and related service industries are showing a trend of stabilizing and improving development, more and more Internet-based service tasks are generated. The purpose of implementing service tasks is to provide users with a better service experience. However, after the task is executed, only a single comparative judgment is made on the results of the task according to the set standards to determine the existing defects. The judgment method is too rough, which makes it impossible to accurately determine the defect level, and thus fail to reasonably deal with the defects caused by the task, indirectly reducing the service experience.

[0003] Therefore, the present invention proposes an Internet-based big data analysis method and system. Summary of the invention

[0004] The present invention provides an Internet-based big data analysis method and system, which is used to comprehensively determine the first label and the second label of each location unit from two aspects: result analysis and logical comparison analysis, so as to accurately determine the defect level, provide a basis for subsequent reasonable handling of the defects, and effectively guarantee the user's service experience.

[0005] The present invention provides an Internet-based big data analysis method, comprising: Step 1: Obtain the data to be analyzed for the target task based on the Internet platform, and at the same time, obtain the analysis indicator set matching the target task; Step 2: performing result analysis on the data to be analyzed in turn based on each analysis indicator in the analysis indicator set to obtain result defect data based on each analysis indicator; Step 3: Retrieve the task execution logic of the target task, and compare and analyze the actual log of the task execution logic with the standard log to obtain logic abnormality data; Step 4: assign a first label to each location unit according to the location of the result defect data under each analysis indicator in the data to be analyzed, and at the same time, map and associate the location with the time of occurrence of the logical abnormal data, and assign a second label to each location unit; Step 5: Determine the defect level of the data to be analyzed based on all the first tags and the second tags.

[0006] Preferably, obtaining an analysis indicator set matching the target task includes: Acquire a unique task identifier of the target task from the Internet platform; An analysis indicator set matching the unique task identifier is obtained from the identifier-analysis comparison table.

[0007] Preferably, performing result analysis on the data to be analyzed in turn based on each analysis indicator in the analysis indicator set includes: Obtaining an indicator parameter set for each analysis indicator, and performing mapping extraction based on the indicator parameter set and the data to be analyzed to obtain actual analysis data for each indicator parameter under the corresponding analysis indicator; Calculate the basic defect ratio based on each actual analysis data; ;in, represents the extracted value of the actual analysis data corresponding to the i-th indicator parameter; Indicates the reference value corresponding to the i-th indicator parameter; They represent the left adjustment value and right adjustment value of the corresponding i-th indicator parameter respectively; min represents the minimum value symbol; ln represents the logarithmic function symbol; According to the basic defect ratio, a ratio sequence based on the corresponding analysis index is constructed, and compared with the standard sequence of the corresponding analysis index to lock the abnormal parameters; Actual analysis data consistent with the abnormal parameters are obtained as result defect data.

[0008] Preferably, calling the task execution logic of the target task includes: Based on the unique task identifier of the target task, a matching task execution logic is obtained from an identifier-logic comparison table.

[0009] Preferably, comparing and analyzing the actual log of the task execution logic with the standard log includes: Input the actual log into the vector construction model to obtain an actual indicator vector and an actual time vector, wherein each element in the actual indicator vector represents a logical execution result involved in an execution analysis indicator, and each element in the actual time vector represents the logical execution duration of a logical node; Based on the standard log, retrieve the indicator threshold vector and the time threshold vector consistent with the task execution logic from the logic log database; Comparing the indicator threshold vector with the actual indicator vector to obtain an indicator difference vector; Determine the logic attribute of each logic node in the task execution logic, and obtain the code file of the corresponding logic node from the attribute-file database; Obtaining a standard time offset according to the file conversion time standard of the code file; Retrieving the historical conversion time of the code file under different historical tasks to obtain the error deviation; ;in, Indicates the error deviation based on the corresponding code file; Indicates the historical conversion time of the corresponding code file under the j3th historical task; Indicates the file conversion time standard for the corresponding code file; Indicates all The number of items in the normal distribution with probability 0.8; Indicates the total number of historical tasks; Indicates that based on all The variance of Based on the standard time offset and the error offset, the time threshold vector is adjusted bidirectionally to obtain a first threshold vector and the second threshold vector , and respectively with the actual time vector Compare and score to get the time difference vector ,in, represents the first threshold of the i2th element in the first threshold vector; represents the second threshold value of the i2th element in the second threshold value vector; m1 represents the total number of elements in the time-based threshold value vector; Indicates the logical execution time of the i2th element in the actual time vector; Indicates the difference duration of the i2th element in the time difference vector; in, Indicates the error bias corresponding to the i2th element; Indicates the standard time deviation corresponding to the i2th element; represent the proportion coefficients, and ; According to the indicator arrangement principle, the time difference vector and the indicator difference vector are difference-mapped to obtain the current difference value of the same arrangement position, and the abnormal indicator is determined to obtain the log data matching the abnormal indicator as the logical abnormal data.

[0010] Preferably, assigning a first label to each occurrence position includes: According to the occurrence position of the result defect data under each analysis indicator in the data to be analyzed, the analysis indicators involved in each position unit are counted, and the cell array of the corresponding position unit is constructed. ,in, Indicates the result defect data involved in the j1th position unit; Represents the involved index under the j1th position unit the number of represents the unit arrangement number of the j1th position unit; The cell array Match them with the array-label comparison table in sequence to obtain the first label of the corresponding position unit.

[0011] Preferably, assigning a second label to each location unit comprises: Determine the theoretical sub-execution logic of each location unit and the theoretical execution time of the theoretical sub-execution logic according to the standard mapping relationship between the target task and the task execution logic; Extracting the sub-exception data of the theoretical sub-execution logic and the occurrence time of the sub-exception data from the logic exception data; The theoretical execution time is temporally associated with the occurrence time, and a second label is assigned to the corresponding position unit.

[0012] Preferably, determining the defect level of the data to be analyzed based on all the first tags and the second tags includes: Inputting the first label and the second label of each position unit into the label analysis model in sequence to obtain the first coefficient of the corresponding position unit; According to the first coefficient of each position unit and the continuous evaluation coefficient of each position unit, a second coefficient of the corresponding position unit is obtained; Based on all the second coefficients and the unit weights of the position units, the defect coefficients of the data to be analyzed are obtained, and the defect levels are obtained by matching with the coefficient-level comparison table.

[0013] The present invention provides an Internet-based big data analysis system, comprising: An indicator set acquisition module is used to acquire the data to be analyzed of the target task based on the Internet platform, and at the same time, acquire the analysis indicator set matching the target task; A result analysis module, used to perform result analysis on the data to be analyzed in turn based on each analysis indicator in the analysis indicator set to obtain result defect data based on each analysis indicator; A logic anomaly acquisition module is used to retrieve the task execution logic of the target task, and compare and analyze the actual log of the task execution logic with the standard log to obtain logic anomaly data; A label assigning module, used to assign a first label to each position unit according to the position of occurrence of the result defect data under each analysis indicator in the data to be analyzed, and at the same time, map and associate the occurrence position with the occurrence time of the logical abnormal data, and assign a second label to each position unit; The level determination module is used to determine the defect level of the data to be analyzed based on all the first tags and the second tags.

[0014] Compared with the prior art, the present invention has the following beneficial effects: The first label and the second label of each location unit are comprehensively determined from the results analysis and the logical comparison analysis, so that the defect level can be accurately determined, which provides a basis for the subsequent reasonable handling of the defects and effectively guarantees the user's service experience.

[0015] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0016] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings: Figure 1 This is a flow chart of a big data analysis method based on the Internet in an embodiment of the present invention; Figure 2 The figure is a structural diagram of an Internet-based big data analysis system in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0019] The present invention provides a big data analysis method based on the Internet, such as Figure 1 As shown, including: Step 1: Obtain the data to be analyzed for the target task based on the Internet platform, and at the same time, obtain the analysis indicator set matching the target task; Step 2: performing result analysis on the data to be analyzed in turn based on each analysis indicator in the analysis indicator set to obtain result defect data based on each analysis indicator; Step 3: Retrieve the task execution logic of the target task, and compare and analyze the actual log of the task execution logic with the standard log to obtain logic abnormality data; Step 4: assign a first label to each location unit according to the location of the result defect data under each analysis indicator in the data to be analyzed, and at the same time, map and associate the location with the time of occurrence of the logical abnormal data, and assign a second label to each location unit; Step 5: Determine the defect level of the data to be analyzed based on all the first tags and the second tags.

[0020] In this embodiment, the service task refers to the task performed by the users of the enterprise on the Internet platform. For example, the service task on the Internet platform is: the user A of the enterprise enters the search task of information A1 on the Internet platform, and the Internet platform is connected with multiple cross-platform communications, mainly to facilitate information acquisition, and the cross-platform is to obtain the results based on the search task and finally summarize them on the Internet platform, and the cross-platform can be platform 1, platform 2, platform 3, etc.

[0021] In this embodiment, the data to be analyzed of the target task refers to the information related to the search task retrieved from the Internet platform, and in the process of retrieval, there may be changes to the data, incomplete data retrieval, etc., which requires the result analysis of the scheduled data (data to be analyzed). It should be noted that in the process of task implementation, there will be pre-set standard implementation results to compare and analyze the actual retrieved results to determine the existing defects, so as to facilitate the determination of the defect level based on the results.

[0022] In this embodiment, the Internet platform is pre-built and can execute different tasks, and the platform is similar to the construction method of APP, mini-programs, etc.

[0023] In this embodiment, the analysis indicator set includes at least one analysis indicator, and the analysis indicator sets corresponding to different target tasks are different. For example, the analysis indicator set of the search task includes: a numerical consistency analysis indicator based on the result, a data missing analysis indicator based on the result, etc.

[0024] For example, the data to be analyzed is obtained from data 01 obtained from platform 1, data 02 obtained from platform 2, and data 03 obtained from platform 1.

[0025] In this embodiment, it is assumed that data 01 corresponds to value a1, data 02 corresponds to value a2, and data 03 corresponds to value a3. The standard result based on the numerical consistency analysis index of the result is: b1, a2, a3. At this time, the result defect data is data 01 corresponding to value a1.

[0026] In this embodiment, the task execution logic is obtained based on the comparison table, and the logic is pre-planned. For example, at time t1, it is first obtained from platform 2, at time t2, it is then obtained from platform 1, and at time t3, it is finally obtained from platform 3, with a sequence. However, in the actual acquisition process, according to the task execution logic, at time t1, it is first obtained from platform 2, at time t3, it is then obtained from platform 1, and at time t4, it is finally obtained from platform 3. The logical exception data is the log data involved in the actual acquisition process. The log data here is the sequence in which the data is obtained from different platforms at different times.

[0027] In this embodiment, the occurrence position refers to the position of the result defect data in the data to be analyzed, for example, the occurrence position of data 01 in data 01 -- data 02 -- data 03, wherein the occurrence position is constructed by a number of position units.

[0028] In this embodiment, the first tag is mainly related to the result defect data and the related indicators.

[0029] In this embodiment, the second tag is mainly related to the defect data corresponding to the occurrence time.

[0030] It should be noted that during the execution of the task execution logic, the different data obtained at different times and the log data generated are all pre-defined, that is, there is a standard mapping relationship between the appearance position and the appearance time. For example, data 01 is obtained from platform 2 at time t1, then at this time t1 - data 01 (the appearance position is the abnormal position).

[0031] In this embodiment, the defect level is determined based on the defects corresponding to the first label and the second label.

[0032] The beneficial effect of the above technical solution is: the first label and the second label of each location unit are comprehensively determined from the two aspects of result analysis and logical comparison analysis, so as to accurately determine the defect level, provide a basis for subsequent reasonable handling of defects, and effectively guarantee the user's service experience.

[0033] The present invention provides an Internet-based big data analysis method, which obtains an analysis indicator set matching the target task, including: Acquire a unique task identifier of the target task from the Internet platform; An analysis indicator set matching the unique task identifier is obtained from the identifier-analysis comparison table.

[0034] In this embodiment, each task will automatically generate a corresponding unique task identifier based on the Internet platform, which is unique mainly to facilitate matching and analyzing indicator sets.

[0035] In this embodiment, the identifier-analysis comparison table includes different identifiers and a number of analysis indicators that match the identifiers, that is, each analysis indicator set includes at least one analysis indicator.

[0036] The beneficial effect of the above technical solution is: by obtaining the analysis indicator set matching the identifier from the comparison table, a convenient basis is provided for analyzing the data to be analyzed.

[0037] The present invention provides an Internet-based big data analysis method, which sequentially performs result analysis on the data to be analyzed based on each analysis indicator in the analysis indicator set, including: Obtaining an indicator parameter set for each analysis indicator, and performing mapping extraction based on the indicator parameter set and the data to be analyzed to obtain actual analysis data for each indicator parameter under the corresponding analysis indicator; Calculate the basic defect ratio based on each actual analysis data; ;in, represents the extracted value of the actual analysis data corresponding to the i-th indicator parameter; Indicates the reference value corresponding to the i-th indicator parameter; They represent the left adjustment value and right adjustment value of the corresponding i-th indicator parameter respectively; min represents the minimum value symbol; ln represents the logarithmic function symbol; According to the basic defect ratio, a ratio sequence based on the corresponding analysis index is constructed, and compared with the standard sequence of the corresponding analysis index to lock the abnormal parameters; Actual analysis data consistent with the abnormal parameters are obtained as result defect data.

[0038] In this embodiment, each analysis indicator has a number of indicator parameters, and the number of indicator parameters constitute an indicator parameter set, wherein the indicator parameter set is obtained based on an indicator-parameter comparison table, which includes different sub-analysis indicators and a number of indicator parameters matching the analysis indicators. For example, analysis indicator 1 includes indicator parameter v1, indicator parameter v2, etc., because the data matched by the corresponding analysis indicator may include several parameter information. Therefore, the basic defect ratio can be effectively determined by analyzing the actual numerical value through the indicator parameter set.

[0039] In this embodiment, They are all pre-determined. The left and right adjustment amounts corresponding to the values ​​of different indicator parameters are different. They are obtained based on the parameter-adjustment comparison table, which contains different indicator parameters and the left and right adjustment values ​​matching the parameters, which is convenient for direct retrieval.

[0040] In this embodiment, the ratio sequence={basic defect ratios of different analysis indicators}, wherein the order of the analysis indicators based on the ratio sequence is pre-planned, and the standard sequence is pre-planned in a specified order.

[0041] For example, the standard sequence of indicator 1 = {ratio e1 of parameter v1 ratio e2 of parameter v2 ...} The ratio sequence of indicator 1 = {ratio e01 of parameter v1, ratio e02 of parameter v2...} Comparison and locking of abnormal parameters refers to determining the absolute value of the difference between e1 and e0. If it is greater than 0.1, it is considered an abnormal parameter. Otherwise, it is not an abnormal parameter. Similarly, the absolute value of the difference of the ratio under the same parameter is calculated.

[0042] In this embodiment, the actual analysis data consistent with the abnormal parameters refers to the data corresponding to the parameter v1, which can be regarded as the result defect data.

[0043] In this embodiment, for the numerical consistency analysis index, the index involves the standard numerical value r1 for parameter v1, the actual analysis data y1 for parameter v2, and the actual analysis data y1 corresponds to the standard numerical value r2, etc., that is, "the detection result of sensor 1 includes: detection sensitivity, and the sensitivity is p1, detection error, and the error is p2", then, parameter v1 corresponds to the detection sensitivity, the actual analysis data y1 has a sensitivity of p1, and the corresponding extraction value is p1.

[0044] The beneficial effect of the above technical solution is: by acquiring the indicator parameter set to analyze the basic defect ratio of each actual analysis data in the data to be analyzed, and then comparing and analyzing it with the standard sequence, the abnormal data can be effectively locked to ensure the accuracy of the result defect data acquisition.

[0045] The present invention provides an Internet-based big data analysis method, which calls the task execution logic of the target task, including: Based on the unique task identifier of the target task, a matching task execution logic is obtained from an identifier-logic comparison table.

[0046] In this embodiment, the identifier-logic comparison table includes different identifiers and pre-planned logic matching the identifiers. For example, it is first obtained from platform 2 at time t1, then obtained from platform 1 at time t2, and finally obtained from platform 3 at time t3.

[0047] The beneficial effect of the above technical solution is: obtaining logic from the comparison table through identification, providing a basis for data analysis from two aspects.

[0048] The present invention provides an Internet-based big data analysis method, which compares and analyzes the actual log of the task execution logic with the standard log, including: Input the actual log into the vector construction model to obtain an actual indicator vector and an actual time vector, wherein each element in the actual indicator vector represents a logical execution result involved in an execution analysis indicator, and each element in the actual time vector represents the logical execution duration of a logical node; Based on the standard log, retrieve the indicator threshold vector and the time threshold vector consistent with the task execution logic from the logic log database; Comparing the indicator threshold vector with the actual indicator vector to obtain an indicator difference vector; Determine the logic attribute of each logic node in the task execution logic, and obtain the code file of the corresponding logic node from the attribute-file database; Obtaining a standard time offset according to the file conversion time standard of the code file; Retrieving the historical conversion time of the code file under different historical tasks to obtain the error deviation; ;in, Indicates the error deviation based on the corresponding code file; Indicates the historical conversion time of the corresponding code file under the j3th historical task; Indicates the file conversion time standard for the corresponding code file; Indicates all The number of items in the normal distribution with probability 0.8; Indicates the total number of historical tasks; Indicates that based on all The variance of Based on the standard time offset and the error offset, the time threshold vector is adjusted bidirectionally to obtain a first threshold vector and the second threshold vector , and respectively with the actual time vector Compare and score to get the time difference vector ,in, represents the first threshold of the i2th element in the first threshold vector; represents the second threshold value of the i2th element in the second threshold value vector; m1 represents the total number of elements in the time-based threshold value vector; Indicates the logical execution time of the i2th element in the actual time vector; Indicates the difference duration of the i2th element in the time difference vector; in, Indicates the error bias corresponding to the i2th element; Indicates the standard time deviation corresponding to the i2th element; represent the proportion coefficients, and ; According to the indicator arrangement principle, the time difference vector and the indicator difference vector are difference-mapped to obtain the current difference value of the same arrangement position, and the abnormal indicator is determined to obtain the log data matching the abnormal indicator as the logical abnormal data.

[0049] In this embodiment, the vector construction model is pre-trained, and the neural network model is trained based on samples of different logs and expert analysis results of the logs. At this time, the indicator-based vector and the time-based vector can be directly obtained.

[0050] In this embodiment, a logical node refers to a node in the task execution logic, and the task execution logic is composed of multiple logical segments, and each logical segment corresponds to a logical node, mainly to carry out the orderly execution of the logical segments and facilitate the subsequent reasonable sorting of the results.

[0051] In this embodiment, the logic database contains several indicator threshold vectors and time threshold vectors that match the standard logs, so as to directly obtain them as reference standards to facilitate the analysis of defects that exist in practice.

[0052] In this embodiment, the indicator difference vector=the indicator threshold vector-the actual indicator vector.

[0053] In this embodiment, the logical attribute refers to the requirement that the corresponding logical node needs to execute, for example, the requirement to retrieve information 01 from platform 1.

[0054] In this embodiment, the attribute-file database includes execution requirements and code files matching the requirements, and the code files are pre-set, for example, the code of information 01 is retrieved from platform 1.

[0055] In this embodiment, the code file needs to be read or converted into a format consistent with the Internet platform during the conversion process. During this process, there will be certain conversion deviations due to differences in reading signals or errors in format conversion. Therefore, the corresponding standard time offset is matched from the file-time comparison table, and the comparison table contains different code files and standard time offsets matching the code files, which are obtained by expert evaluation after the code files are set.

[0056] In this embodiment, historical tasks refer to different service tasks based on the Internet platform, and during the historical execution process, the same code file may be applied to multiple different service tasks. At this time, the error bias based on the code file is comprehensively determined by retrieving the actual conversion time (historical conversion time) of the same code file under different historical tasks.

[0057] In this embodiment, the bidirectional adjustment refers to adjusting each element in the time threshold vector downward and upward to obtain the first threshold vector and the second threshold vector.

[0058] In this embodiment, is a variable, and different code files may have different corresponding coefficients.

[0059] In this embodiment, the indicator arrangement principle refers to the order in which the analysis indicators in the analysis indicator set analyze the data to be analyzed and the results are arranged according to the preset order of the indicators, that is, the principle of maintaining consistency.

[0060] In this embodiment, the same arrangement position is based on the arrangement position of the indicator difference vector. Each indicator may involve multiple logical nodes, so as to obtain the difference average and difference variance of the time differences of the logical nodes involved in the same indicator. The current difference value = the indicator difference of the corresponding element in the indicator difference vector × ln(2 + the total number of all logical nodes involved under the same indicator × (corresponding difference average + difference variance) / the maximum time difference among all logical nodes involved under the same indicator).

[0061] The indicator corresponding to the current difference value being greater than the preset difference value is considered an abnormal indicator.

[0062] The log data matched by the abnormal indicator refers to the logical data of the logical node that matches the indicator, and then the logical abnormal data obtained by synthesis.

[0063] The beneficial effects of the above technical solution are: obtaining indicators and time vectors through model analysis, and then determining the indicator difference vector based on the results through comparative analysis with the standard, and in order to reduce the error calculation caused by time differences, the existing error bias is comprehensively determined based on historical tasks, and the accuracy of the error is ensured through two-way adjustment, and then logical abnormality data is obtained through difference mapping, thereby ensuring the accuracy of data acquisition and providing an effective basis for subsequent data defect analysis.

[0064] The present invention provides an Internet-based big data analysis method, which assigns a first label to each occurrence position, including: According to the occurrence position of the result defect data under each analysis indicator in the data to be analyzed, the analysis indicators involved in each position unit are counted, and the cell array of the corresponding position unit is constructed. ,in, Indicates the result defect data involved in the j1th position unit; Represents the involved index under the j1th position unit the number of represents the unit arrangement number of the j1th position unit; The cell array Match them with the array-label comparison table in sequence to obtain the first label of the corresponding position unit.

[0065] In this embodiment, the array-tag comparison table includes different arrays and tags matching the arrays, and the tags include relevant information of the corresponding arrays.

[0066] In this embodiment, the occurrence position refers to the position of the defective data in the data to be analyzed. For example, there are data 01, data 02, and data 03. For example, data 01 occupies 2 units, data 02 occupies 3 units, and data 03 occupies 2 units. At this time, indicator 1 occupies the first unit under data 01, and indicator 2 occupies the first unit under data 01 and 2 units under data 03.

[0067] At this time, the first unit under data 01 involves indicators 1 and 2, and the number of indicators involved is 2. The unit arrangement number is 1 (the numbers are pre-set). At this time, the result defect data is the data involved in the first unit under data 01. Similarly, the unit array can be obtained.

[0068] The beneficial effect of the above technical solution is: the occurrence positions are counted to obtain the unit array under the position unit, and the first label is obtained by matching the comparison table, which provides a basis for subsequent defect determination.

[0069] The present invention provides an Internet-based big data analysis method, which assigns a second label to each location unit, including: Determine the theoretical sub-execution logic of each location unit and the theoretical execution time of the theoretical sub-execution logic according to the standard mapping relationship between the target task and the task execution logic; Extracting the sub-exception data of the theoretical sub-execution logic and the occurrence time of the sub-exception data from the logic exception data; The theoretical execution time is temporally associated with the occurrence time, and a second label is assigned to the corresponding position unit.

[0070] In this embodiment, the standard mapping relationship is pre-planned and includes the location unit, the indicator involved in the location unit and the logical node matching the indicator, and the standard mapping relationship corresponding to different tasks is different. At this time, the logical node of the corresponding location unit can be directly mapped (theoretical sub-execution logic, theoretical execution time).

[0071] In this embodiment, because the logic exception data includes the actual logic exception and the actual time that occurred during the actual execution process, the difference in the corresponding time can be obtained through comparative analysis, and then the second label can be assigned.

[0072] In this embodiment, the second label is obtained by matching the time association result (the logical node of the sub-execution logic, the time difference under the node) from the result-label comparison table, which contains labels set by experts that match the results, and the label includes the time association result.

[0073] The beneficial effect of the above technical solution is: by starting from the logic and execution time to achieve the comparison between theory and practice, it is convenient to obtain the time-based label.

[0074] The present invention provides an Internet-based big data analysis method, which determines the defect level of the data to be analyzed based on all first tags and second tags, including: Inputting the first label and the second label of each position unit into the label analysis model in sequence to obtain the first coefficient of the corresponding position unit; Based on all the first coefficients and the unit weights of the position units, the defect coefficients of the data to be analyzed are obtained, and the defect levels are obtained by matching with the coefficient-level comparison table.

[0075] In this embodiment, the same position unit has both a first label and a second label, and the purpose of setting the label is to perform a more convenient and standardized analysis of the unit. This is implemented based on a label analysis model, which is based on different combinations of labels and expert analysis results (coefficients) of the labels for sample pair neural network model training, which facilitates direct acquisition of the first coefficient.

[0076] In this embodiment, the defect coefficient=the sum of the first coefficient of the corresponding position unit×the unit weight.

[0077] It should be noted that the unit weights of different location units under each task are pre-planned, and the sum of the weights of all location units is 1.

[0078] In this embodiment, the coefficient-level comparison table includes defect coefficients and defect levels matching the coefficients, and is pre-set and can be directly retrieved for use.

[0079] The beneficial effect of the above technical solution is: the first coefficient is effectively obtained by combining the two labels, and combined with the unit weight, the defect coefficient is directly calculated and then matched to obtain the defect level, which provides a basis for subsequent reasonable handling of defects and effectively guarantees the user's service experience.

[0080] The present invention provides an Internet-based big data analysis system, such as Figure 2 As shown, including: An indicator set acquisition module is used to acquire the data to be analyzed of the target task based on the Internet platform, and at the same time, acquire the analysis indicator set matching the target task; A result analysis module, used to perform result analysis on the data to be analyzed in turn based on each analysis indicator in the analysis indicator set to obtain result defect data based on each analysis indicator; A logic anomaly acquisition module is used to retrieve the task execution logic of the target task, and compare and analyze the actual log of the task execution logic with the standard log to obtain logic anomaly data; A label assigning module, used to assign a first label to each position unit according to the position of occurrence of the result defect data under each analysis indicator in the data to be analyzed, and at the same time, map and associate the occurrence position with the occurrence time of the logical abnormal data, and assign a second label to each position unit; The level determination module is used to determine the defect level of the data to be analyzed based on all the first tags and the second tags.

[0081] The beneficial effect of the above technical solution is: the first label and the second label of each location unit are comprehensively determined from the two aspects of result analysis and logical comparison analysis, so as to accurately determine the defect level, provide a basis for subsequent reasonable handling of defects, and effectively guarantee the user's service experience.

[0082] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A big data analysis method based on the Internet, characterized in that: include: Step 1: Obtain the data to be analyzed for the target task based on the Internet platform, and at the same time, obtain the analysis indicator set matching the target task; Step 2: performing result analysis on the data to be analyzed in turn based on each analysis indicator in the analysis indicator set to obtain result defect data based on each analysis indicator; Step 3: Retrieve the task execution logic of the target task, and compare and analyze the actual log of the task execution logic with the standard log to obtain logic abnormality data; Step 4: assign a first label to each location unit according to the location of the result defect data under each analysis indicator in the data to be analyzed, and at the same time, map and associate the location with the time of occurrence of the logical abnormal data, and assign a second label to each location unit; Step 5: Determine the defect level of the data to be analyzed based on all the first tags and the second tags.

2. The Internet-based big data analysis method according to claim 1, characterized in that: Obtain an analysis indicator set that matches the target task, including: Acquire a unique task identifier of the target task from the Internet platform; An analysis indicator set matching the unique task identifier is obtained from the identifier-analysis comparison table.

3. The Internet-based big data analysis method according to claim 1, characterized in that: Performing result analysis on the data to be analyzed in turn based on each analysis indicator in the analysis indicator set includes: Obtaining an indicator parameter set for each analysis indicator, and performing mapping extraction based on the indicator parameter set and the data to be analyzed to obtain actual analysis data for each indicator parameter under the corresponding analysis indicator; Calculate the basic defect ratio based on each actual analysis data; ;in, represents the extracted value of the actual analysis data corresponding to the i-th indicator parameter; Indicates the reference value corresponding to the i-th indicator parameter; They represent the left adjustment value and right adjustment value of the corresponding i-th indicator parameter respectively; min represents the minimum value symbol; ln represents the logarithmic function symbol; According to the basic defect ratio, a ratio sequence based on the corresponding analysis index is constructed, and compared with the standard sequence of the corresponding analysis index to lock the abnormal parameters; Actual analysis data consistent with the abnormal parameters are obtained as result defect data.

4. The Internet-based big data analysis method according to claim 2, characterized in that: Retrieving the task execution logic of the target task includes: Based on the unique task identifier of the target task, a matching task execution logic is obtained from an identifier-logic comparison table.

5. The Internet-based big data analysis method according to claim 1, characterized in that: Compare and analyze the actual log of the task execution logic with the standard log, including: Input the actual log into the vector construction model to obtain an actual indicator vector and an actual time vector, wherein each element in the actual indicator vector represents a logical execution result involved in an execution analysis indicator, and each element in the actual time vector represents the logical execution duration of a logical node; Based on the standard log, retrieve the indicator threshold vector and the time threshold vector consistent with the task execution logic from the logic log database; Comparing the indicator threshold vector with the actual indicator vector to obtain an indicator difference vector; Determine the logic attribute of each logic node in the task execution logic, and obtain the code file of the corresponding logic node from the attribute-file database; Obtaining a standard time offset according to the file conversion time standard of the code file; Retrieving the historical conversion time of the code file under different historical tasks to obtain the error deviation; ;in, Indicates the error deviation based on the corresponding code file; Indicates the historical conversion time of the corresponding code file under the j3th historical task; Indicates the file conversion time standard for the corresponding code file; Indicates all The number of items in the normal distribution with probability 0.8; Indicates the total number of historical tasks; Indicates that based on all The variance of Based on the standard time offset and the error offset, the time threshold vector is adjusted bidirectionally to obtain a first threshold vector and the second threshold vector , and respectively with the actual time vector Compare and score to get the time difference vector ,in, represents the first threshold of the i2th element in the first threshold vector; represents the second threshold value of the i2th element in the second threshold value vector; m1 represents the total number of elements in the time-based threshold value vector; Indicates the logical execution time of the i2th element in the actual time vector; Indicates the difference duration of the i2th element in the time difference vector; in, Indicates the error bias corresponding to the i2th element; Indicates the standard time deviation corresponding to the i2th element; represent the proportion coefficients, and According to the indicator arrangement principle, the time difference vector and the indicator difference vector are differentially mapped to obtain the current difference value of the same arrangement position, and the abnormal indicator is determined to obtain the log data matching the abnormal indicator as the logical abnormal data.

6. The Internet-based big data analysis method according to claim 1, characterized in that: Assign a first label to each occurrence, including: According to the occurrence position of the result defect data under each analysis indicator in the data to be analyzed, the analysis indicators involved in each position unit are counted, and the cell array of the corresponding position unit is constructed. ,in, Indicates the result defect data involved in the j1th position unit; Represents the involved index under the j1th position unit the number of represents the unit arrangement number of the j1th position unit; The cell array Match them with the array-label comparison table in sequence to obtain the first label of the corresponding position unit.

7. The Internet-based big data analysis method according to claim 1, characterized in that: Each location unit is assigned a second label, including: Determine the theoretical sub-execution logic of each location unit and the theoretical execution time of the theoretical sub-execution logic according to the standard mapping relationship between the target task and the task execution logic; Extracting the sub-exception data of the theoretical sub-execution logic and the occurrence time of the sub-exception data from the logic exception data; The theoretical execution time is temporally associated with the occurrence time, and a second label is assigned to the corresponding position unit.

8. The Internet-based big data analysis method according to claim 1, characterized in that: Determining the defect level of the data to be analyzed based on all the first tags and the second tags includes: Inputting the first label and the second label of each position unit into the label analysis model in sequence to obtain the first coefficient of the corresponding position unit; According to the first coefficient of each position unit and the continuous evaluation coefficient of each position unit, a second coefficient of the corresponding position unit is obtained; Based on all the second coefficients and the unit weights of the position units, the defect coefficients of the data to be analyzed are obtained, and the defect levels are obtained by matching with the coefficient-level comparison table.

9. A big data analysis system based on the Internet, characterized in that: include: An indicator set acquisition module is used to acquire the data to be analyzed of the target task based on the Internet platform, and at the same time, acquire the analysis indicator set matching the target task; A result analysis module, used to perform result analysis on the data to be analyzed in turn based on each analysis indicator in the analysis indicator set, to obtain result defect data based on each analysis indicator; A logic anomaly acquisition module is used to retrieve the task execution logic of the target task, and compare and analyze the actual log of the task execution logic with the standard log to obtain logic anomaly data; A label assigning module, used to assign a first label to each position unit according to the position of occurrence of the result defect data under each analysis indicator in the data to be analyzed, and at the same time, map and associate the occurrence position with the occurrence time of the logical abnormal data, and assign a second label to each position unit; The level determination module is used to determine the defect level of the data to be analyzed based on all the first tags and the second tags.

Images