Data analysis method and device, equipment and storage medium

Abstract

The invention discloses a data analysis method and device, equipment and a storage medium, and relates to the technical field of data analysis, and the method comprises the steps: analyzing a user analysis demand according to a data big model and prompt word information corresponding to a preset grammar rule, and obtaining a demand operation type and a data processing sequence; performing instruction generation according to a preset grammar rule, the demand operation type and the data processing sequence to obtain a domain-specific language; and analyzing the to-be-analyzed data according to the domain-specific language and the data analysis engine to obtain a target data analysis result. Complex data analysis and visualization requirements of a user are expressed in a concise language description through a set grammar rule, a large model only needs to generate a domain-specific language, and then the domain-specific language is analyzed by an analysis engine, so that the use cost of the large model is reduced, and meanwhile, the data analysis efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data analysis, and particularly relates to a data analysis method and device, equipment and a storage medium. BACKGROUND

[0002] Data analysis and visualization reports are one of the most common functions in a system. When a user wants to perform data analysis or data visualization, the system developer needs to customize and develop according to the requirements. Using large model technology can complete different data analysis and visualization requirements in real time according to the user's description, greatly improving the work efficiency. However, the existing method for data analysis using large models has the problems of unstable output results, low execution efficiency and high use cost. Therefore, how to reduce the use cost of large models while improving the data analysis efficiency has become a problem to be solved. SUMMARY

[0003] The main purpose of the present application is to provide a data analysis method, device, equipment and storage medium, which aims to solve the technical problem of how to reduce the use cost of large models while improving the data analysis efficiency.

[0004] To achieve the above-mentioned purpose, the present application provides a data analysis method, which comprises the following steps: According to the prompt word information corresponding to the data large model and the preset syntax rule, the user analysis requirement is analyzed to obtain the requirement operation type and the data processing sequence; According to the preset syntax rule, the requirement operation type and the data processing sequence, an instruction is generated to obtain a domain-specific language; According to the domain-specific language and the data analysis engine, the to-be-analyzed data is analyzed to obtain a target data analysis result.

[0005] In an embodiment, the step of generating an instruction according to the preset syntax rule, the requirement operation type and the data processing sequence to obtain a domain-specific language comprises: According to the preset syntax rule and the requirement operation type, an operator matching is performed to obtain a plurality of data operators; According to the data processing sequence, a pipeline operation is performed on the plurality of data operators to obtain connected data operators; According to the connected data operators, an instruction is generated to obtain a domain-specific language.

[0006] In an embodiment, the step of analyzing the to-be-analyzed data according to the data processing instruction and the data analysis engine to obtain a target data analysis result comprises: The data processing instruction is disassembled to obtain a plurality of data processing steps; The data parsing engine is matched with each data processing step according to the data processing order, and a data parsing component corresponding to each data processing step is obtained. The target extraction data is extracted according to each data processing step and the data to be analyzed. The target data analysis result is obtained by analyzing the target extraction data according to the data processing order and the data parsing component.

[0007] In an embodiment, the step of analyzing the target extraction data according to the data processing order and the data parsing component to obtain the target data analysis result comprises: The current step to be processed is determined according to the data processing order. The parsing component corresponding to the current step to be processed and the extraction data corresponding to the current step to be processed are determined according to the data parsing component and the target extraction data. The target data analysis result is obtained by analyzing the extraction data corresponding to the current step to be processed according to the parsing component corresponding to the current step to be processed.

[0008] In an embodiment, the step of analyzing the extraction data corresponding to the current step to be processed according to the parsing component corresponding to the current step to be processed to obtain the target data analysis result comprises: The first data analysis result is obtained by analyzing the extraction data corresponding to the current step to be processed according to the parsing component corresponding to the current step to be processed. The second data analysis result is obtained by analyzing the extraction data corresponding to the target step to be processed according to the parsing component corresponding to the target step to be processed when there is a target step to be processed after the current step to be processed. The first data analysis result and the second data analysis result are summarized to obtain the target data analysis result when there is no step to be processed after the target step to be processed.

[0009] In an embodiment, the step of disassembling the data processing instruction to obtain a plurality of data processing steps further comprises: When there is a visualization step in the plurality of data processing steps, the data to be analyzed is visualized and rendered according to the rendering operator corresponding to the visualization step to obtain a visualization chart and visualization data. The target data analysis result is determined according to the visualization chart and the visualization data.

[0010] In an embodiment, the step of disassembling the data processing instruction to obtain a plurality of data processing steps comprises: Pipeline symbol recognition is performed on the data processing instruction to obtain a pipeline symbol recognition result. The data processing instruction is disassembled according to the pipeline symbol recognition result to obtain a plurality of data processing steps.

[0011] In addition, to achieve the above object, the present application also provides a data analysis device, which comprises: The analysis module is configured to analyze the to-be-analyzed data according to the domain-specific language and a data analysis engine to obtain a target data analysis result. The analysis module is configured to analyze the to-be-analyzed data according to the domain-specific language and a data analysis engine to obtain a target data analysis result. The analysis module is configured to analyze the to-be-analyzed data according to the domain-specific language and a data analysis engine to obtain a target data analysis result.

[0012] In addition, to achieve the above object, the present application also provides a data analysis device, which comprises: a memory, a processor, and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the data analysis method as described above.

[0013] In addition, to achieve the above object, the present application also provides a storage medium, which is a computer-readable storage medium, and the storage medium stores a computer program, the computer program being executed by a processor to implement the steps of the data analysis method as described above.

[0014] In addition, to achieve the above object, the present application also provides a computer program product, which comprises a computer program, the computer program being executed by a processor to implement the steps of the data analysis method as described above.

[0015] The present application analyzes the user's analysis requirements according to the prompt word information corresponding to the data large model and the preset syntax rule, obtains the demand operation type and the data processing sequence, generates the instruction according to the preset syntax rule, the demand operation type and the data processing sequence, obtains the domain-specific language, and analyzes the to-be-analyzed data according to the domain-specific language and the data analysis engine to obtain the target data analysis result. Through the set syntax rule, the user's complex data analysis and visualization requirements are expressed in simple language, and the large model only needs to generate the domain-specific language, and then the analysis engine analyzes the domain-specific language, thereby reducing the use cost of the large model and improving the data analysis efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0016] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application.

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings required by the embodiments or prior art description will be briefly introduced as follows. Obviously, those skilled in the art can obtain other drawings according to these drawings without any creative effort.

[0018] Figure 1 A flowchart is provided for the data analysis method of Embodiment One of the present application. Figure 2 A flowchart is provided for the data analysis method of Embodiment Two of the present application. Figure 3 A syntax analysis flowchart is provided for the data analysis domain-specific language of Embodiment Two of the present application. Figure 4 A brief flowchart is provided for the data analysis method of Embodiment One of the present application. Figure 5 A module structure diagram is provided for the data analysis device of the present application. Figure 6 A device structure diagram is provided for the hardware running environment involved in the data analysis method of the present application.

[0019] The purpose implementation, functional features and advantages of the present application will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0020] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and do not limit the present application.

[0021] In order to better understand the technical solutions of the present application, the following will be described in detail in conjunction with the drawings and specific embodiments of the specification.

[0022] The main solution of the embodiments of the present application is: according to the prompt word information corresponding to the data large model and the preset syntax rule, the user analysis demand is parsed to obtain the demand operation type and the data processing sequence; according to the preset syntax rule, the demand operation type and the data processing sequence, the instruction is generated to obtain the domain-specific language; according to the domain-specific language and the data analysis engine, the data to be analyzed is analyzed to obtain the target data analysis result.

[0023] Data analysis and visualization reports are one of the most common functions in a system, and when a user wants to perform data analysis or data visualization, the system developer needs to customize the development according to the requirements. Using large model technology can complete different data analysis and visualization requirements in real time according to the user's description, greatly improving work efficiency. However, the existing method of using large models for data analysis has the problems of unstable output results, low execution efficiency, and high use cost. Therefore, how to reduce the use cost of large models while improving the efficiency of data analysis has become a problem to be solved.

[0024] The present application analyzes the user's analysis requirements according to the prompt word information corresponding to the data large model and the preset syntax rule, obtains the requirement operation type and the data processing sequence; generates instructions according to the preset syntax rule, the requirement operation type and the data processing sequence, obtains the domain-specific language; analyzes the data to be analyzed according to the domain-specific language and the data analysis engine, and obtains the target data analysis result. Through the set syntax rule, the user's complex data analysis and visualization requirements are expressed in simple language description, and the large model only needs to generate the domain-specific language, and then the analysis engine analyzes the domain-specific language, which realizes reducing the use cost of the large model while improving the efficiency of data analysis.

[0025] It should be noted that the execution subject of the present embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a data analysis device capable of realizing the above functions. The following takes the data analysis device as an example to illustrate the present embodiment and the following embodiments.

[0026] Based on this, the present application provides a data analysis method, referring to Figure 1 , Figure 1 The flowchart of the first embodiment of the data analysis method of the present application is shown in the figure.

[0027] In the present embodiment, the data analysis method comprises steps S10-S30: Step S10, analyze the user's analysis requirements according to the prompt word information corresponding to the data large model and the preset syntax rule, and obtain the requirement operation type and the data processing sequence; It should be noted that the embodiment designs a set of domain-specific language for the data analysis and visualization scene, which can express the user's complex data analysis and visualization requirements in a concise language, so that the user only needs to generate the DSL (Domain Specific Language) when using the large model, and then the parsing engine parses the DSL and analyzes it by using the pandas library (Python Data Analysis Library, which is a core library for data operation and analysis). Compared with directly writing pandas code, the embodiment has greater advantages in stability, accuracy and execution efficiency of task execution.

[0028] It should be noted that the syntax structure in the embodiment includes: (1) basic syntax, operator structure: VERB ARGUMENT [OPTIONS]; pipeline operation: multiple operators are connected by |, and data flows from left to right. (2) Operator details: [1]. FILTER (filtering): syntax: FILTER [condition expression], supporting comparison (==, >, >=, <=,!=), logical operation (AND, OR), and parentheses priority ((A OR B) AND C); [2]. GROUP... AGG... (grouping and aggregation): syntax: GROUP column name AGG column name: function [condition] AS result name, supporting aggregation functions such as sum, mean, max, min and count; [3]. LINES (line number statistics): syntax: LINES option AS result name, options: BY grouping column, *; [4]. UNIQUE (unique value query): syntax: UNIQUE column name [option], options include LIST, indicating returning a unique value list, and returning the number of unique values when not set; [5]. SELECT (column selection): syntax: SELECT column name [AS alias]; [6]. SORT (sorting): syntax: SORT column name [ASC|DESC]; [7]. LIMIT (quantity limitation): syntax: LIMIT N; [8]. COMPUTE (new column calculation / grouping calculation): syntax: COMPUTE new column name = expression [BY grouping column], supporting basic arithmetic, aggregation functions and grouping calculation; [9]. PIVOT (data pivot / row-to-column): syntax: PIVOT id_vars = column name 1, column name 2 value_vars = column name 1, column name 2 var_name = new column name value_name = value column name;

[10] . VISUALIZE (visualization): syntax: VISUALIZE chart type [parameter table], supporting types such as line, bar, pie and scatter, and flexible parameters.

[0029] It can be understood that the preset syntax rule refers to a preset syntax rule for converting the user's data analysis intention into a corresponding domain-specific language, including basic syntax (operator structure and pipeline operation) and operator details, and the prompt word information refers to information for the large model to learn the syntax rule, the data large model refers to a large model for data analysis, the user analysis requirement refers to the data analysis and visualization requirement proposed by the user through natural language, i.e., the user's data analysis intention, the requirement operation type refers to the operator type required in the user's data analysis intention, and the data processing sequence refers to the sequence of data processing.

[0030] In specific implementation, the user's data analysis intention is parsed through the optimized data large model, and then the key information in the user's data analysis is identified to determine the corresponding operator type, i.e., the requirement operation type, and the sequence of data processing is determined according to the identification sequence, which is also the connection sequence of multiple operators. For example, the user asks: "Statistical total sales of various products in the African region", and the large model first identifies "Africa" to determine the corresponding operator as FILTER.

[0031] Step S20, generating instructions according to the preset syntax rule, the requirement operation type, and the data processing sequence to obtain a domain-specific language; It can be understood that the domain-specific language refers to a domain-specific language (Domain Specific Language, DSL) for data analysis.

[0032] In specific implementation, the preset syntax rule is used to match the corresponding operator for the operator type required in the user's data analysis intention, and multiple operators are connected according to the sequence of data processing to obtain a domain-specific language, wherein the connected operators are processed in the order from left to right. For example, the user asks: "Statistical total sales of various products in the African region", and the processing process is: according to the DSL syntax and the data sample, the large model generates the corresponding DSL description: FILTER region=='Africa' | GROUP category AGG sales:sum AS total sales | SELECT category, total sales.

[0033] In a feasible implementation, step S20 can include steps A11-A13: Step A11, matching operators according to the preset syntax rule and the requirement operation type to obtain multiple data operators; It should be noted that the data operator refers to an operator corresponding to the user's data analysis requirement.

[0034] In practice, the optimized data analysis big data model has learned the set syntax rules. Then, after identifying the type of operator required in the user's data analysis intent, it matches the corresponding operator for the type of operator required in the user's data analysis intent, thus obtaining multiple operators.

[0035] Step A12: Pipe multiple data operators according to the data processing order to obtain concatenated data operators; In practice, multiple data operators are connected according to the data processing order, that is, multiple data operators are connected by the pipe symbol (|). For example, FILTER Region == 'Africa' | GROUP Category AGG Sales: sum AS Total Sales | SELECT Category, Total Sales.

[0036] Step A13: Generate instructions based on the concatenated data operators to obtain the domain-specific language.

[0037] It is understandable that the concatenated data operators include multiple data processing steps and the order of data processing, and each data processing step corresponds to an operator. Therefore, a complete domain-specific language (DSL) can be generated based on the concatenated data operators.

[0038] Step S30: Analyze the data to be analyzed according to the domain-specific language and data parsing engine to obtain the target data analysis results.

[0039] It is understandable that the data parsing engine refers to the DSL syntax parsing engine, the data to be analyzed refers to the analysis data provided by the user, and the target data analysis results include analysis data and visualization charts, etc.

[0040] In its implementation, the DSL syntax parsing engine processes the following steps: Step-by-step parsing: The complete DSL instruction is divided into multiple steps using the '|' operator and parsed progressively. Operator mapping: A dedicated parsing component is defined for each operator based on the pandas library (e.g., FILTER is mapped to Pandas query). Parameter extraction: Regular expression parsing is performed on the parameters of each operation to extract column names, functions, conditions, etc. Sequential execution: Data is processed sequentially according to the DSL order, with the result of the previous step serving as the input for the next step. Visualization rendering: When the VISUALIZE instruction is encountered, the pyecharts library is used for visualization rendering.

[0041] The embodiment optimizes the data large model according to the prompt word information corresponding to the preset syntax rule, obtains an optimized data large model, analyzes the user analysis requirement according to the optimized data large model, obtains a requirement operation type and a data processing sequence, generates an instruction according to the preset syntax rule, the requirement operation type and the data processing sequence, obtains a domain-specific language, and analyzes the to-be-analyzed data according to the domain-specific language and a data analysis engine, and obtains a target data analysis result. The set syntax rule is used to describe the complex data analysis and visualization requirement of the user in a simple language, the large model only needs to generate the domain-specific language, and then the analysis engine analyzes the domain-specific language, so that the data analysis efficiency is improved while the use cost of the large model is reduced.

[0042] Based on the first embodiment of the present application, the same or similar contents as the above embodiment one can be referred to the above description, and will not be repeated hereinafter. On this basis, please refer to Figure 2 , the data analysis method further includes steps S31-S34 in step S30: Step S31, the data processing instruction is disassembled to obtain a plurality of data processing steps; It can be understood that the data processing step refers to the processing step divided by the DSL syntax analysis engine.

[0043] In specific implementation, the DSL syntax analysis engine in the embodiment divides the complete DSL instruction into multiple steps, analyzes step by step, and performs data analysis according to the data processing sequence, and the processing result of the previous step is used as the input of the next step, and the data analysis is iterated, and the processing results of the steps are summarized after there is no subsequent processing step.

[0044] In a feasible implementation, step S31 can include steps B11-B12: Step B11, the data processing instruction is identified by a pipe symbol to obtain a pipe symbol identification result; It can be understood that the pipe symbol identification result refers to the identification result of the position and quantity of the pipe symbol.

[0045] In specific implementation, the DSL syntax analysis engine divides the complete DSL instruction into multiple steps according to the pipe symbol, therefore, the pipe symbol in the complete DSL instruction needs to be identified to determine the position and quantity of the pipe symbol, that is, to obtain the pipe symbol identification result.

[0046] Step B12, the data processing instruction is disassembled according to the pipe symbol identification result to obtain a plurality of data processing steps.

[0047] In a specific implementation, the DSL syntax parsing engine determines the number of pipes to determine the number of splitting steps, and determines the splitting position of the steps according to the position of the pipes, and then splits the complete DSL instruction into multiple steps according to the pipes.

[0048] In a feasible implementation, after step S41, steps C11-C12 can be further included. Step C11, when there is a visualization step in the plurality of data processing steps, visualizing and rendering the to-be-analyzed data according to a rendering operator corresponding to the visualization step to obtain a visualization chart and visualization data; It can be understood that the visualization chart refers to an interactive graph rendered and generated by calling the pyecharts library through the VISUALIZE instruction, and the structured data is converted into visual expression that can be directly understood by humans. The visualization data refers to a preprocessed structured data set, which is the basis for generating the visualization chart. In the DSL workflow, it specifically refers to the intermediate result processed by the pre-order operator (such as GROUP and PIVOT), which is stored in the memory in the form of a two-dimensional table.

[0049] In a specific implementation, the DSL syntax parsing engine performs a visualization step judgment on the plurality of steps, that is, whether the VISUALIZE instruction is detected, and when it is determined that there is a visualization step, performs a visual rendering operation on the analysis data provided by the user according to the rendering operator (VISUALIZE), and then obtains the corresponding visualization chart and visualization data.

[0050] Step C12, determining a target data analysis result according to the visualization chart and the visualization data.

[0051] In a specific implementation, the visualization chart and the visualization data are summarized to obtain the final data analysis result, and the final data analysis result is returned to the user.

[0052] Step S32, when there is no visualization step in the plurality of data processing steps, performing component matching on each data processing step according to the data parsing engine to obtain a data parsing component corresponding to each data processing step; It can be understood that the data parsing component refers to defining a special parsing component for each operator based on the pandas library.

[0053] In a specific implementation, when there is no visualization step in the plurality of data processing steps, it is indicated that each data processing step is iteratively processed, and then a corresponding parsing component needs to be matched for each data processing step based on the pandas library to determine the data parsing component corresponding to each data processing step, for example, FILTER is mapped to Pandas query.

[0054] Step S33, data extraction is performed according to each data processing step and the data to be analyzed, to obtain target extraction data; It can be understood that the target extraction data refers to column names, functions, conditions and the like corresponding to each data processing step.

[0055] In a specific implementation, each data processing step corresponds to an operator, and the range of processing data is defined in each operator, thereby extracting the analysis data provided by the user to determine the column names, functions, conditions and the like corresponding to each data processing step.

[0056] Step S34, according to the data processing sequence and the data analysis component, the target extraction data is analyzed to obtain a target data analysis result.

[0057] It can be understood that the data is processed in the order of the processing sequence (i.e., the execution order of the data processing steps), until all data processing steps are completed, and the analysis results corresponding to all data processing steps are summarized to obtain the final data analysis result.

[0058] In a feasible implementation, step S34 can include steps D11-D13: Step D11, determining a current processing step according to the data processing sequence; It can be understood that the current processing step refers to the currently executed data processing step, and by analyzing the data processing sequence, the first data processing step divided by the DSL syntax analysis engine is taken as the current processing step, i.e., the current processing step is executed first and then the subsequent data processing steps are executed.

[0059] Step D12, determining the analysis component corresponding to the current processing step and the extraction data corresponding to the current processing step according to the data analysis component and the target extraction data; It can be understood that the data analysis component contains the analysis components of all data processing steps, and the target extraction data also contains the data required by all data processing steps, so when the current processing step is executed, the analysis component corresponding to the current processing step is matched based on the data analysis component, and the data corresponding to the current processing step is matched based on the target extraction data, i.e., the analysis component corresponding to the current processing step and the extraction data are obtained.

[0060] Step D13, according to the analysis component corresponding to the current processing step, the extraction data corresponding to the current processing step is analyzed to obtain a target data analysis result.

[0061] It can be understood that the parsing component in the DSL syntax parsing engine analyzes the extracted data corresponding to the current to-be-processed step to obtain a data analysis result, and then determines whether there is a subsequent data processing step. If there is a subsequent data processing step, the data analysis is repeated until there is no subsequent data processing step. The final data analysis result is obtained by summarizing the results of all processing steps.

[0062] In a possible implementation, step D13 can include steps E11-E13. Step E11, analyzing the extracted data corresponding to the current to-be-processed step according to the parsing component corresponding to the current to-be-processed step to obtain a first data analysis result. It can be understood that the first data analysis result refers to the data analysis result corresponding to the current to-be-processed step.

[0063] In specific implementation, the parsing component in the DSL syntax parsing engine analyzes the extracted data corresponding to the current to-be-processed step to obtain the data analysis result corresponding to the current to-be-processed step.

[0064] Step E12, when there is a target to-be-processed step after the current to-be-processed step, analyzing the extracted data corresponding to the target to-be-processed step according to the parsing component corresponding to the target to-be-processed step to obtain a second data analysis result. It can be understood that the target to-be-processed step refers to the subsequent step of the current to-be-processed step, and the second data analysis result refers to the data analysis result corresponding to the target to-be-processed step.

[0065] In specific implementation, when there is a target to-be-processed step after the current to-be-processed step, the parsing component and the extracted data corresponding to the target to-be-processed step are determined based on the data parsing component and the target extracted data, and then the data analysis of the extracted data corresponding to the target to-be-processed step is performed by the parsing component corresponding to the target to-be-processed step to obtain the data analysis result corresponding to the target to-be-processed step.

[0066] Step E13, when there is no to-be-processed step after the target to-be-processed step, the first data analysis result and the second data analysis result are summarized to obtain a target data analysis result.

[0067] In specific implementation, the to-be-processed step refers to a step waiting for processing. After the data analysis of a step is performed, it is determined whether there is a subsequent step, and then it is determined whether the data processing is completed. When there is no to-be-processed step, it indicates that the data processing is completed. The data analysis results corresponding to the processed steps are summarized to obtain the final data analysis result.

[0068] It should be noted that, asFigure 3 The processing flow of the DSL syntax parsing engine in this embodiment is shown in the following table: Step-by-step parsing: The complete DSL instruction is divided into multiple steps according to the | symbol, and each step is parsed; Operator mapping: Based on the pandas library, a special parsing component is defined for each operator (such as FILTER mapped to Pandas' query); Parameter extraction: Regular parsing is performed on the parameters of each operation to extract column names, functions, conditions, etc. Sequential execution: Process the data in the order of DSL, and use the result of the previous step as the input of the next step; Visual rendering: When the VISUALIZE instruction is encountered, use the pyecharts library for visual rendering.

[0069] When the DSL is parsed, the DSL is first disassembled into multiple steps, and each step is processed by a corresponding component. The first component receives the original data, and the processed data is passed to the next component as input. This process continues until the last component is processed, and the result is output to the user. The following is a description of the parsing implementation of each component: [1]. The parsing method of the FILTER component: based on the query method of the pandas library; [2]. The parsing method of the GROUP... AGG... component: based on the groupby and agg methods of the pandas library; [3]. The parsing method of the LINES component: based on the groupby and len methods of the pandas library; [4]. The parsing method of the UNIQUE component: based on the unique method of the pandas library; [5]. The parsing method of the SELECT component: based on the DataFrame value retrieval of the pandas library; [6]. The parsing method of the SORT component: based on the sort_values method of the pandas library; [7]. The parsing method of the LIMIT component: based on the head method of the pandas library; [8]. The parsing method of the COMPUTE component: based on the groupby and transform methods of the pandas library; [9]. The parsing method of the PIVOT component: based on the melt method of the pandas library;

[10] . The parsing method of the VISUALIZE component: line, bar, pie, scatter chart implementation based on the pyecharts library.

[0070] It should be noted that the user asks: use the column chart to compare the average profit and average cost of each region, and after renaming the average profit and average cost as the type, group and display. The processing process: according to the DSL syntax and the data sample, the corresponding DSL description is generated by the large model: GROUP region AGG profit: mean AS average profit, cost: mean AS average cost | PIVOT id_vars=region value_vars=average profit, average cost var_name=type value_name=value | VISUALIZE bar x=region measure_column=value title=average profit and average cost of each region color=type. Disassemble the DSL and execute each step in order: use the GROUP component to group the data according to the region, and then calculate the average value of the profit and cost respectively; use the PIVOT component to convert the average profit and average cost from column to row and name the column as type; use the VISUALIZE component to draw a column chart, with region as the horizontal coordinate, and different values are displayed according to the type grouping; finally, the system returns the result to the user.

[0071] The embodiment obtains a plurality of data processing steps by disassembling the data processing instruction; when there is no visualization step in the plurality of data processing steps, the data processing steps are matched with the data analysis engine to obtain the data analysis component corresponding to each data processing step; the target extraction data is obtained by extracting data according to the data processing steps and the data to be analyzed; and the target data analysis result is obtained by analyzing the target extraction data according to the data processing order and the data analysis component. In the above manner, the large model is responsible for generating the domain-specific language for data analysis, instead of pandas code, avoiding the potential risks of directly running unknown code in the production environment; and the large model is only responsible for generating the DSL, and the generated content is relatively small, so the overall generation efficiency of the DSL is higher than that of the generated code, thereby improving the response speed of the function.

[0072] For example, in order to help understand the implementation process of the data analysis method obtained after combining the above embodiment one, please refer to Figure 4 , Figure 4A brief flowchart of a data analysis method is provided, specifically: receiving user input starts, first checking the input for compliance and format, including checking whether it is a legal DSL syntax and whether it meets the preset format rules, if not, entering the error handling process; after passing the verification, parsing and identifying the syntax elements such as operators and parameters, generating a syntax tree, then splitting the execution unit according to the syntax tree, and scheduling the execution engine. The execution engine reads the execution unit line by line, judges whether it is a visualization instruction, if so, calls the visualization component to process and render; if it is a normal operation, extracts the parameters and maps them to the corresponding processing logic, performs calculation based on the output results of the previous step, and returns the intermediate data. If there is an error during execution, enter the error handling, if there is no error, continue to flow, finally integrate all processing results, complete data output or visualization display.

[0073] It should be noted that the above examples are only for understanding the present application and do not constitute a limitation on the data analysis method of the present application. Based on this technical concept, more forms of simple transformation are within the protection scope of the present application.

[0074] The present application also provides a data analysis device, please refer to Figure 5 , the data analysis device comprises: The parsing module 10 is configured to parse the user analysis demand according to the prompt word information corresponding to the data large model and the preset syntax rule, and obtain the demand operation type and the data processing sequence; The generation module 20 is configured to generate instructions according to the preset syntax rule, the demand operation type and the data processing sequence, and obtain the domain-specific language; The analysis module 30 is configured to analyze the data to be analyzed according to the domain-specific language and the data parsing engine, and obtain the target data analysis result.

[0075] Optionally, the generation module 20 is further configured to: According to the preset syntax rule and the demand operation type, the operator matching is performed to obtain a plurality of data operators; According to the data processing sequence, the pipeline operation is performed on the plurality of data operators to obtain the connected data operators; According to the connected data operators, the instruction generation is performed to obtain the domain-specific language.

[0076] Optionally, the analysis module 30 is further configured to: The data processing instruction is disassembled to obtain a plurality of data processing steps; When there is no visualization step in the plurality of data processing steps, the component matching is performed on each data processing step according to the data parsing engine to obtain the data parsing component corresponding to each data processing step; According to each data processing step and the data to be analyzed, target extraction data is obtained through data extraction; According to the data processing sequence and the target extraction data, target data analysis results are obtained through analysis by the data parsing component.

[0077] Optionally, the analysis module 30 is further configured to: According to the data processing sequence, determine the current step to be processed; According to the data parsing component and the target extraction data, determine the parsing component corresponding to the current step to be processed and the extraction data corresponding to the current step to be processed; According to the parsing component corresponding to the current step to be processed, analyze the extraction data corresponding to the current step to be processed, and obtain target data analysis results.

[0078] Optionally, the analysis module 30 is further configured to: According to the parsing component corresponding to the current step to be processed, analyze the extraction data corresponding to the current step to be processed, and obtain first data analysis results; When there is a target step to be processed after the current step to be processed, according to the parsing component corresponding to the target step to be processed, analyze the extraction data corresponding to the target step to be processed, and obtain second data analysis results; When there is no step to be processed after the target step to be processed, the first data analysis results and the second data analysis results are summarized to obtain target data analysis results.

[0079] Optionally, the analysis module 30 is further configured to: When there is a visualization step in the plurality of data processing steps, according to the rendering operator corresponding to the visualization step, visualize and render the data to be analyzed to obtain a visualization chart and visualization data; According to the visualization chart and the visualization data, determine target data analysis results.

[0080] Optionally, the analysis module 30 is further configured to: Pipe symbol recognition is performed on the data processing instruction to obtain a pipe symbol recognition result; According to the pipe symbol recognition result, the data processing instruction is disassembled to obtain a plurality of data processing steps.

[0081] The data analysis device provided in the present application adopts the data analysis method in the above embodiments, and can solve the technical problem of how to reduce the use cost of a large model while improving the data analysis efficiency. Compared with the prior art, the data analysis device provided in the present application has the same beneficial effects as the data analysis method provided in the above embodiments, and other technical features in the data analysis device are the same as the features disclosed in the above embodiments, which will not be repeated here.

[0082] The present application provides a data analysis device, comprising: at least one processor; and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the data analysis method in Embodiment I.

[0083] Reference will now be made to the drawings, and specific examples thereof will be illustrated. Figure 6 The data analysis device in the embodiments of the present application can include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. Figure 6 The data analysis device shown is only an example, and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0084] As Figure 6As shown, the data analysis device can include a processing apparatus 1001 (e.g., a central processor, a graphics processor, etc.) that can perform various appropriate actions and processes according to programs stored in a read only memory (ROM) 1002 or loaded from a storage apparatus 1003 into a random access memory (RAM) 1004. Various programs and data required for operation of the data analysis device are also stored in the RAM 1004. The processing apparatus 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: input apparatuses 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output apparatuses 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; the storage apparatus 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication apparatus 1009. The communication apparatus 1009 can allow the data analysis device to communicate with other devices wirelessly or by wire to exchange data. Although the data analysis device with various systems is shown in the figure, it should be understood that all the shown systems are not required to be implemented or possessed. More or fewer systems can be alternatively implemented or possessed.

[0085] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program codes for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication apparatus, or installed from the storage apparatus 1003, or installed from the ROM 1002. When the computer program is executed by the processing apparatus 1001, the above-mentioned functions defined in the methods of the embodiments disclosed in the present application are performed.

[0086] The data analysis device provided by the present application adopts the data analysis method in the above-mentioned embodiments, and can solve the technical problem of how to reduce the use cost of large models while improving the data analysis efficiency. Compared with the prior art, the data analysis device provided by the present application has the same beneficial effects as the data analysis method provided by the above-mentioned embodiments, and other technical features in the data analysis device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0087] It should be understood that various aspects of the disclosure can be implemented in hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any appropriate manner in any one or more embodiments or examples.

[0088] The above description is merely illustrative of the application and is not intended to limit the scope of the application. Any variations and modifications that can be made by any person skilled in the art within the spirit and scope of the application are intended to be encompassed by the application. Therefore, the scope of the application should be determined by the appended claims.

[0089] The application provides a computer readable storage medium having stored thereon computer readable program instructions (i.e., a computer program) for performing the data analysis method in the above embodiments.

[0090] The computer readable storage medium provided by the application may, for example, be a U disk, but is not limited to an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination thereof. More specific examples of the computer readable storage medium can include, but are not limited to, an electric connection with one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any appropriate combination thereof. In the present embodiment, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer readable storage medium can be transmitted by any appropriate medium, including but not limited to an electric wire, an optical cable, an RF (Radio Frequency), etc., or any appropriate combination thereof.

[0091] The above computer readable storage medium can be contained in the data analysis device; or can exist separately without being assembled into the data analysis device.

[0092] The computer readable storage medium carries one or more programs, when the one or more programs are executed by the data analysis device, the data analysis device is caused to: analyze the user analysis demand according to prompt word information corresponding to the data large model and a preset syntax rule, and obtain a demand operation type and a data processing sequence; generate an instruction according to the preset syntax rule, the demand operation type, and the data processing sequence, and obtain a domain-specific language; and analyze the to-be-analyzed data according to the domain-specific language and a data analysis engine, and obtain a target data analysis result.

[0093] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0094] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0095] The modules involved in the embodiments of the present application can be implemented in a software manner or in a hardware manner. In some cases, the name of the module does not constitute a limitation on the module itself.

[0096] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e. computer programs) for executing the above-mentioned data analysis method, and can solve the technical problem of how to reduce the use cost of large models while improving the data analysis efficiency. Compared with the prior art, the beneficial effects of the computer readable storage medium provided by the present application are the same as those of the data analysis method provided by the above-mentioned embodiments, which will not be repeated here.

[0097] The present application also provides a computer program product comprising a computer program, which, when executed by a processor, implements the steps of the data analysis method as described above.

[0098] The computer program product provided by the present application can solve the technical problem of how to reduce the use cost of large models while improving the data analysis efficiency. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as those of the data analysis method provided by the above-mentioned embodiments, which will not be repeated here.

[0099] The above-mentioned is only part of the embodiments of the present application, and does not limit the patent scope of the present application. Any equivalent structural transformation, direct / indirect application in other related technical fields based on the technical concept of the present application, and the contents of the specification and drawings are included in the patent protection scope of the present application.

Claims

1. A data analysis method, characterized by, The data analysis method comprises: According to the prompt word information corresponding to the data large model and the preset syntax rule, the user analysis demand is analyzed, and the demand operation type and data processing sequence are obtained; According to the preset syntax rule, the demand operation type and the data processing sequence, instruction generation is carried out, and the domain specific language is obtained; According to the domain specific language and data analysis engine, the analysis of the data to be analyzed is carried out, and the target data analysis result is obtained.

2. The method of claim 1, wherein, The step of generating instructions according to the preset syntax rule, the demand operation type and the data processing sequence to obtain the domain specific language comprises: According to the preset syntax rule and the demand operation type, the operator matching is carried out, and a plurality of data operators are obtained; According to the data processing sequence, the pipeline operation is carried out on the plurality of data operators, and the connected data operators are obtained; According to the connected data operator, the instruction generation is carried out to obtain the domain specific language.

3. The method of claim 1, wherein, The step of analyzing the data to be analyzed according to the data processing instruction and the data analysis engine to obtain the target data analysis result comprises: The data processing instruction is disassembled, and a plurality of data processing steps are obtained; When there is no visualization step in the plurality of data processing steps, the component matching is carried out on each data processing step according to the data analysis engine, and the data analysis component corresponding to each data processing step is obtained; According to each data processing step and the data to be analyzed, the data extraction is carried out, and the target extraction data is obtained; According to the data processing sequence and the data analysis component, the target extraction data is analyzed, and the target data analysis result is obtained.

4. The method of claim 3, wherein, The step of analyzing the target extraction data according to the data processing sequence and the data analysis component to obtain the target data analysis result comprises: According to the data processing sequence, the current processing step is determined; According to the data analysis component and the target extraction data, the analysis component corresponding to the current processing step and the extraction data corresponding to the current processing step are determined; According to the analysis component corresponding to the current processing step, the extraction data corresponding to the current processing step is analyzed, and the target data analysis result is obtained.

5. The method of claim 4, wherein, The step of analyzing the extraction data corresponding to the current processing step according to the analysis component corresponding to the current processing step to obtain the target data analysis result comprises: According to the analysis component corresponding to the current processing step, the extraction data corresponding to the current processing step is analyzed, and the first data analysis result is obtained; When there is a target processing step after the current processing step, the extraction data corresponding to the target processing step is analyzed according to the analysis component corresponding to the target processing step, and the second data analysis result is obtained; When there is no processing step after the target processing step, the first data analysis result and the second data analysis result are summarized, and the target data analysis result is obtained.

6. The method of claim 3, wherein, The step of disassembling the data processing instruction to obtain a plurality of data processing steps further comprises: When the visualization step is present in the plurality of data processing steps, visual rendering is performed on the data to be analyzed according to a rendering operator corresponding to the visualization step, to obtain a visualization chart and visualization data; A target data analysis result is determined according to the visualization chart and the visualization data.

7. The method of claim 3, wherein, The step of decomposing the data processing instruction to obtain the plurality of data processing steps comprises: Pipe symbol recognition is performed on the data processing instruction to obtain a pipe symbol recognition result; The data processing instruction is decomposed according to the pipe symbol recognition result to obtain the plurality of data processing steps.

8. A data analysis device, characterized by The apparatus comprises: The parsing module is configured to parse the user analysis demand according to prompt word information corresponding to a data large model and a preset syntax rule, to obtain a demand operation type and a data processing sequence; The generation module is configured to generate an instruction according to the preset syntax rule, the demand operation type, and the data processing sequence, to obtain a domain-specific language; The analysis module is configured to analyze the data to be analyzed according to the domain-specific language and a data parsing engine, to obtain a target data analysis result.

9. A data analysis device, characterized by The device comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the data analysis method according to any one of claims 1 to 7.

10. A storage medium, characterized by The storage medium is a computer-readable storage medium, and the storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the data analysis method according to any one of claims 1 to 7.

Images