Method and system for parsing large-scale xml data

Abstract

The application discloses a method and system for parsing large-scale xml data, and relates to the technical field of big data.In the method, a set of xml files is read from a file system; data cleaning preprocessing is performed on the set of xml files to obtain a set of preprocessed xml files; the set of preprocessed xml files comprises a plurality of preprocessed xml subfiles; the plurality of preprocessed xml subfiles are taken as a plurality of xml nodes; the plurality of xml nodes are parsed based on a Spark data processing framework to obtain first parsed data; the first parsed data is structured data; at least one parsed data output request is received, the parsed data output request carrying a data output condition; and the first parsed data is output according to an output mode corresponding to the data output condition. The technical scheme provided by the application can effectively improve the efficiency of parsing and output when large-scale xml data is processed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, in particular to a large-scale xml data parsing method and system. BACKGROUND

[0002] XML (Extensible Markup Language) is a commonly used data exchange format. In the big data environment, XML is often used to store and transmit structured data as a common carrier format.

[0003] In the big data warehouse, when the xml file set is used as the data source, the efficiency of parsing xml data is required. However, due to the fact that the xml file set usually contains a large number of tags and nested structures, the traditional parsing method has performance bottlenecks when processing and outputting large-scale xml data, and has the problem of low efficiency. Therefore, how to improve the efficiency of parsing and output when processing large-scale xml data has become a problem to be solved.

[0004] Therefore, a large-scale xml data parsing method is needed to solve the problems existing in the current technology. SUMMARY

[0005] The present application provides a large-scale xml data parsing method and system, which can effectively improve the efficiency of parsing and output when processing large-scale xml data.

[0006] In a first aspect, the present application provides a large-scale xml data parsing method, which comprises: reading an xml file set from a file system; the xml file set includes a plurality of xml subfiles; performing data cleaning preprocessing on the xml file set to obtain a preprocessed xml file set; the preprocessed xml file set includes a plurality of preprocessed xml subfiles; a plurality of preprocessed xml subfiles are used as a plurality of xml nodes; based on the Spark data processing framework, a plurality of xml nodes are parsed to obtain first parsed data; the first parsed data is structured data; receiving at least one parsed data output request, the parsed data output request carrying a data output condition; according to the output mode corresponding to the data output condition, the first parsed data is output.

[0007] By adopting the technical solutions, the multiple xml sub-files are preprocessed to avoid invalid data and repeated data from affecting subsequent xml data parsing, so that a large number of xml file sets can be parsed at the same time; the multiple preprocessed xml sub-files are taken as multiple xml nodes, and the multiple xml nodes are parsed based on the Spark data processing framework to obtain first parsed data; the parsing process can fully utilize the computing capacity and memory computing advantages of the Spark distributed data processing engine, thereby effectively improving the efficiency of parsing large-scale xml data. The first parsed data is output according to the output mode corresponding to the data output condition, thereby effectively improving the efficiency of outputting large-scale xml data.

[0008] Optionally, the preprocessed xml sub-file includes multiple initial attributes; the preprocessed xml sub-file is parsed based on the Spark data processing framework to obtain first parsed data, specifically including: the multiple xml nodes, the multiple initial attributes, and file content in the multiple preprocessed xml sub-files are converted based on the Spark data processing framework to obtain second parsed data; the second parsed data is converted into third parsed data based on a preset first function; the third parsed data is converted into the first parsed data based on a preset second function.

[0009] Optionally, the multiple xml nodes, the multiple initial attributes, and file content in the multiple preprocessed xml sub-files are converted based on the Spark data processing framework to obtain second parsed data, specifically including: whether the multiple initial attributes include nested attributes is judged; if yes, each nested attribute is split into multiple sub-attributes, and each initial attribute other than the nested attribute is taken as a sub-attribute; if not, each initial attribute is taken as a sub-attribute; the i th xml node is mapped to the i th row of a DataFrame based on the Spark data processing framework, and the j th sub-attribute is mapped to the j th column of the DataFrame; wherein, i∈[1,n], j∈[1,m], n and m are both positive integers greater than 1; file content corresponding to the j th sub-attribute in the i th preprocessed xml sub-file is converted into a preset field structure based on the Spark data processing framework; the preset field structure includes a corresponding relationship between the i th row and the j th column and the file content; the preset field structure is stored in the i th row and the j th column of the DataFrame to obtain the second parsed data; the second parsed data is DataFrame data.

[0010] By adopting the technical scheme, whether the initial attribute contains a nested attribute is judged, if yes, the nested attribute is split into multiple sub-attributes, so that the attribute is more complete and accurate, and the generated DataFrame data is more complete and accurate.

[0011] Optionally, the second analysis data is converted into third analysis data based on a preset first function, specifically including: the second analysis data is converted into third analysis data based on a df.as[ds] function; the second analysis data is DataFrame data, and the third analysis data is DataSet data.

[0012] Optionally, the third analysis data is converted into the first analysis data based on a preset second function, specifically including: the third analysis data is converted into the first analysis data based on a ds.write().mode(SaveMode.Overwrite).save(tablename) function; the third analysis data is DataSet data, and the first analysis data is Hive table data.

[0013] Optionally, after the multiple preprocessed xml sub-files are parsed based on the Spark data processing framework to obtain first analysis data, the method further includes: receiving an analysis and processing request of the analysis data, the analysis and processing request carrying a data analysis and processing condition; based on the analysis and processing condition, the first analysis data is analyzed and processed by using Spark SQL and / or Hive SQL to obtain an analysis and processing result.

[0014] By adopting the technical scheme, the first analysis data is analyzed and processed by using Spark SQL and / or Hive SQL based on the analysis and processing condition, and Spark SQL and Hive SQL have various functions of analyzing and processing the first analysis data, so that the first analysis data can be analyzed and processed on a large scale.

[0015] Optionally, the xml file set is preprocessed to obtain a preprocessed xml file set, specifically including: a data cleaning operation is performed on each xml sub-file to remove invalid data and repeated data in the xml sub-file, and multiple preprocessed xml sub-files are obtained.

[0016] In a second aspect of the present application, a system for parsing large-scale XML data is provided, and the system comprises an XML file set reading module, a preprocessing module, a parsing module, a processing module, and an output module; the XML file set reading module is configured to read an XML file set from a file system; the XML file set comprises a plurality of XML subfiles; the preprocessing module is configured to preprocess the XML file set to obtain a preprocessed XML file set; the preprocessed XML file set comprises a plurality of preprocessed XML subfiles; the parsing module is configured to parse the plurality of preprocessed XML subfiles based on a Spark data processing framework to obtain first parsed data; the first parsed data is structured data; the processing module is configured to receive at least one parsed data output request, wherein the parsed data output request carries a data output condition; and the output module is configured to output the first parsed data according to an output mode corresponding to the data output condition.

[0017] In a third aspect of the present application, an electronic device is provided, which comprises a processor, a memory, a user interface, and a network interface; the memory is configured to store instructions; the user interface and the network interface are configured to communicate with other devices; and the processor is configured to execute the instructions stored in the memory, so that the electronic device performs the method according to any one of the first aspect of the present application.

[0018] In a fourth aspect of the present application, a computer-readable storage medium is provided, which stores a computer program capable of being loaded and executed by a processor to perform the method according to any one of the first aspect of the present application.

[0019] In summary, the one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0020] 1. The plurality of XML subfiles are preprocessed to avoid the influence of invalid data and duplicate data on subsequent parsing of XML data, so that a large number of XML file sets can be parsed at the same time; the plurality of preprocessed XML subfiles are taken as a plurality of XML nodes, and the plurality of XML nodes are parsed based on a Spark data processing framework to obtain first parsed data; the parsing process can fully utilize the computing power and memory computing advantages of a Spark distributed data processing engine, thereby effectively improving the efficiency of parsing large-scale XML data; and the first parsed data is output according to an output mode corresponding to the data output condition, thereby effectively improving the efficiency of outputting large-scale XML data.

[0021] 2. By judging whether the initial attribute contains a nested attribute, if so, the nested attribute is split into multiple sub-attributes, so that the attribute is more complete and accurate, and the generated DataFrame data is more complete and accurate.

[0022] 3. By analyzing the processing conditions, the first parsed data is analyzed and processed by using Spark SQL, and / or the first parsed data is analyzed and processed by using Hive SQL, and Spark SQL and Hive SQL have various functions of analyzing and processing the first parsed data, so that the first parsed data can be analyzed and processed on a large scale. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 is one of the flowcharts of the large-scale xml data parsing method provided by the embodiments of the present application;

[0024] Figure 2 is the second flowchart of the large-scale xml data parsing method provided by the embodiments of the present application;

[0025] Figure 3 is the third flowchart of the large-scale xml data parsing method provided by the embodiments of the present application;

[0026] Figure 4 is the structural schematic diagram of the large-scale xml data parsing system provided by the embodiments of the present application;

[0027] Figure 5 is the structural schematic diagram of the electronic device disclosed by the embodiments of the present application.

[0028] Mark explanation: 1, xml file set reading module; 2, preprocessing module; 3, parsing module; 4, processing module; 5, output module; 500, electronic device; 501, processor; 502, communication bus; 503, user interface; 504, network interface; 505, memory. DETAILED DESCRIPTION

[0029] In order for those skilled in the art to better understand the technical solutions in the specification, the technical solutions in the specification will be described clearly and completely in the embodiments of the specification in combination with the drawings in the embodiments of the specification. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments.

[0030] In the description of the embodiments of this application, words such as "for example" or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "for example" or "for instance" in the embodiments of this application should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "for example" or "for instance" is intended to present the relevant concepts in a concrete manner.

[0031] In the description of the embodiments of the present application, the term "multiple" means two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals. In addition, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0032] This application provides a large-scale XML data parsing method. Figure 1 , which shows one of the flow charts of a large-scale XML data parsing method provided in an embodiment of the present application. The method includes steps S11-S16, which are as follows:

[0033] Step S11: Read an XML file set from the file system; the XML file set includes multiple XML sub-files.

[0034] In the above steps, the server reads the XML file collection from the file system.

[0035] Specifically, this technical solution implements parsing of XML data based on the Spark distributed data processing engine. First, the server uses Spark's API to load a collection of XML files from the file system, such as the SparkSQL method spark.read.format("xml").load("file.xml") Other methods for reading XML files from the file system are possible, and this application does not impose any specific restrictions on these methods.

[0036] Step S12: performing data cleaning preprocessing on the XML file set to obtain a preprocessed XML file set; the preprocessed XML file set includes a plurality of preprocessed XML sub-files.

[0037] In the above steps, the server preprocesses the XML file set to obtain a preprocessed XML file set.

[0038] Specifically, in the technical solution, the xml file set is preprocessed, that is, invalid data and repeated data in the xml file set are removed, so as to avoid the influence of invalid data and repeated data on subsequent xml data analysis. The following embodiments are specific ways of preprocessing the xml file set.

[0039] In a possible implementation, the step S12 specifically includes the following steps: performing a data cleaning operation on each xml sub-file to remove invalid data and repeated data in the xml sub-file, thereby obtaining a plurality of preprocessed xml sub-files.

[0040] Specifically, in the technical solution, in the present application, the xml file set includes a plurality of xml sub-files. Taking patent files as an example, a large xml file set can include thousands or tens of thousands of xml sub-files of patent file types. Therefore, the xml file set is preprocessed, that is, all sub-files in the xml file set are preprocessed, and the preprocessed xml file set includes a plurality of preprocessed xml sub-files. The specific way of removing invalid data and repeated data in the xml sub-file can be set according to specific conditions, which is not limited in the present application.

[0041] Step S13: The plurality of preprocessed xml sub-files are taken as a plurality of xml nodes.

[0042] In the above step, the server takes the plurality of preprocessed xml sub-files as a plurality of xml nodes.

[0043] Specifically, in the technical solution, when analyzing the xml file set containing a large number of xml sub-files, the plurality of preprocessed xml sub-files can be taken as a plurality of xml nodes.

[0044] Step S14: The plurality of xml nodes are analyzed based on a Spark data processing framework to obtain first analysis data; the first analysis data is structured data.

[0045] In the above step, the server analyzes the plurality of xml nodes based on the Spark data processing framework to obtain the first analysis data.

[0046] Specifically, in this technical solution, the present invention parses XML data based on the Spark distributed data processing engine, and can make full use of the following characteristics of the Spark distributed data processing engine: High performance: Utilizing Spark's distributed computing capabilities and memory computing advantages, the speed and efficiency of XML data parsing are greatly improved. Scalability: Due to Spark's horizontal expansion capabilities, this method can process large-scale XML data and is suitable for big data environments. Flexibility: Through Spark's rich data processing and analysis functions, flexible data operations and mining can be performed on the parsing results. Accuracy: XML data is cleaned in the data preprocessing stage to ensure the accuracy of the parsing results. Based on the Spark data processing framework, multiple preprocessed XML sub-files are parsed, and the first parsed data obtained is structured data, that is, tabular data. After the XML data is parsed into tabular data, it is convenient to further analyze and process the tabular data, and when the tabular data is output, the XML data can be displayed more intuitively. The following embodiment is a specific method for parsing multiple XML nodes to obtain the first parsed data.

[0047] In one possible implementation, refer to Figure 2 , which shows the second flow chart of a large-scale XML data parsing method provided by an embodiment of the present application. Step S14 specifically includes steps S141-S143:

[0048] Step S141: Based on the Spark data processing framework, the file contents of the multiple XML nodes, the multiple initial attributes, and the multiple pre-processed XML sub-files are converted to obtain second parsed data.

[0049] Specifically, in this technical solution, the server converts the file contents of multiple XML nodes, multiple initial attributes, and multiple pre-processed XML sub-files based on the Spark data processing framework to obtain second parsed data. Among them, the pre-processed XML sub-file includes multiple initial attributes. Taking patent document data as an example, the initial attributes can be patent numbers, titles, names, etc., and the initial attributes of each XML sub-file are the same. The second parsed data is DataFrame data. DataFrame is a two-dimensional table data structure that is commonly used in fields such as data processing, data analysis, and machine learning. It is an important data structure in the Pandas library (Python data processing library). DataFrame is similar to a data table in a spreadsheet or database, and can contain different types of data, such as numbers, strings, Boolean values, etc., and the data type of each column can be different. The following embodiment is a specific way to implement step S141.

[0050] In one possible implementation, refer toFigure 3 , which shows the third flow chart of a large-scale XML data parsing method provided by an embodiment of the present application. Step S141 specifically includes steps S1411-S1416:

[0051] Step S1411: Determine whether the multiple initial attributes contain nested attributes.

[0052] Specifically, in this technical solution, the server determines whether multiple initial attributes contain nested attributes. A nested attribute refers to an initial attribute that includes multiple, more detailed sub-attribute features. For example, in the above example, a person's name is a nested attribute, and its sub-attributes can include applicant, right holder, agent, and so on.

[0053] Step S1412: If yes, split each nested attribute into multiple sub-attributes, and use the original attributes other than each nested attribute as sub-attributes.

[0054] Specifically, in this technical solution, if the server determines that multiple initial attributes contain nested attributes, each nested attribute is split into multiple sub-attributes, and the initial attributes outside each nested attribute are used as sub-attributes. For example, if there are five initial attributes, namely A, B, C, D, and E; among them, A is a nested attribute, and the multiple sub-attributes obtained by splitting are A1, A2, and A3; therefore, after splitting the initial attribute A, the four initial attributes B, C, D, and E are used as sub-attributes, and the final sub-attributes are A1, A2, A3, B, C, D, and E, and subsequent processing is performed.

[0055] Step S1413: If not, treat each initial attribute as a sub-attribute.

[0056] Specifically, in this technical solution, if the server determines that the multiple initial attributes do not contain nested attributes, each initial attribute is used as a sub-attribute.

[0057] Step S1324: Based on the Spark data processing framework, map the i-th xml node to the i-th row of the DataFrame, and map the j-th sub-attribute to the j-th column of the DataFrame; where i∈[1,n], j∈[1,m], n and m are both positive integers greater than 1.

[0058] Specifically, in this technical solution, based on the Spark data processing framework, the server maps the i-th XML node to the i-th row of a DataFrame and the j-th sub-attribute to the j-th column of the DataFrame. For example, if there are n XML sub-files, there are n XML nodes, each containing j attributes. Therefore, the server maps the i-th XML node to the i-th row of a DataFrame and the j-th sub-attribute to the j-th column of the DataFrame, thus constructing the DataFrame's table structure.

[0059] Step S1415: Based on the Spark data processing framework, the file content corresponding to the jth sub-attribute in the i-th preprocessed XML sub-file is converted into a preset field structure; the preset field structure includes the correspondence between the i-th row and the j-th column and the file content.

[0060] Specifically, in this technical solution, the server converts the file content corresponding to the jth sub-attribute in the i-th preprocessed xml sub-file into a preset field structure based on the Spark data processing framework. The file content of all xml sub-files is identified based on the Spark data processing framework, and the file content corresponding to each sub-attribute in each xml sub-file is converted into a preset field structure. The preset field structure can be in the form of key-value, where the key indicates that the data is obtained from the j-th attribute of the i-th xml sub-file, and the value is the file content corresponding to the sub-attribute. At the same time, the key-value form can also clearly indicate that the file content corresponding to the j-th sub-attribute in the i-th preprocessed xml sub-file should be stored in the i-th row and j-th column of the DataFrame.

[0061] Step S1416: The preset field structure is stored in the i-th row and j-th column of the DataFrame to obtain second parsed data; the second parsed data is DataFrame data.

[0062] Specifically, in this technical solution, the server stores the preset field structure in the i-th row and j-th column of the DataFrame to obtain the second parsed data. That is, the server stores all the preset field structures in the DataFrame to obtain the DataFrame data.

[0063] Step S142: converting the second parsed data into third parsed data based on a preset first function.

[0064] Specifically, in the technical solution, the server converts the second parsed data into third parsed data based on a preset first function. The third parsed data is DataSet data. DataSet is also structured data, which is an extension of DataFrame and aims to provide more rich functions and better type safety. DataSet is a strongly typed (statically typed) dataset, unlike DataFrame, which performs type checking at compile time and can catch more type errors. DataSet specifies the data structure of the dataset by defining classes, allowing data to be represented and processed in a more structured manner. The following is a specific way to implement step S133.

[0065] In one possible implementation, step S133 specifically includes the following steps: converting the second parsed data into third parsed data based on the df.as[ds] function; the second parsed data is DataFrame data, and the third parsed data is DataSet data.

[0066] Specifically, in the technical solution, the df.as[ds] function is a function in the Spark distributed data processing engine, which can convert DataFrame data into DataSet data. It should be noted that when converting DataFrame data into DataSet data, import spark.implicits._ can be added before conversion. spark.implicits._ is a member of the SparkSession object, and its purpose is to implicitly convert commonly used data types into specific types in DataSet. Specifically, spark.implicits._ is a class that contains common implicit conversions, which converts primitive data types (such as Int, String, etc.) to data types in Spark SQL (such as IntType, StringType, etc.). In this way, there is no need to explicitly specify the data type during subsequent data processing, and the Spark distributed data processing engine will automatically perform type inference based on implicit conversion.

[0067] Step S143: converting the third parsed data into the first parsed data based on a preset second function.

[0068] Specifically, in the technical solution, the server converts the third parsed data into the first parsed data based on a preset second function. The first parsed data is Hive table data. Hive table data can be used for large-scale data processing using SQL, and is compatible with multiple data formats and data sources, making it convenient for subsequent analysis and processing operations on Hive table data. The following is a specific way to implement step S134.

[0069] In a possible implementation, the step S134 specifically includes the following step: converting the third parsed data into the first parsed data based on a ds.write().mode(SaveMode.Overwrite).save(tablename) function; the third parsed data is DataSet data, and the first parsed data is Hive table data.

[0070] Specifically, in the technical solution, the Spark distributed data processing engine contains the API ds.write().mode(SaveMode.Overwrite).save(tablename), ds.write(): is a write API of DataSet, which is used to write DataSet to external storage, wherein ds is a DataSet object to be saved. mode(SaveMode.Overwrite): is a method for setting the save mode, wherein SaveMode is an enumeration type, indicating the mode of data saving, and there are four options: Overwrite, Append, Ignore, and ErrorIfExists. The technical solution uses Overwrite, which means that if the target file or table already exists, it will be overwritten. save(tablename): is a method for specifying the save location, wherein tablename is the target location of the data to be saved, which can be a file path or a table name.

[0071] Step S15: receiving at least one parsed data output request, the parsed data output request carrying a data output condition.

[0072] In the above step, the server receives at least one parsed data output request, and the parsed data output request carries a data output condition.

[0073] Specifically, in the technical solution, the server will receive at least one parsed data output request transmitted by a user terminal, and the user terminal includes but is not limited to mobile phones, tablets, computers, and other electronic devices. The parsed data output request carries a data output condition, that is, the parsed data output request contains a requirement for outputting the first parsed data by a certain output mode.

[0074] Step S16: outputting the first parsed data according to the output mode corresponding to the data output condition.

[0075] In the above step, the server outputs the first parsed data according to the output mode corresponding to the data output condition.

[0076] Specifically, in this technical solution, the output method corresponding to the data output condition includes but is not limited to saving the first parsed data into a database; writing it into a CSV file or Parquet file in the file system; or using a visualization tool to display the analysis results.

[0077] Saving to a database: The first parsed data can be saved to a relational database (such as MySQL, PostgreSQL, etc.) or a NoSQL database (such as MongoDB, Cassandra, etc.). This makes it easy to query and use the analysis results later.

[0078] Writing to CSV or Parquet files in the file system: You can write the first parsed data to CSV or Parquet files. CSV files are suitable for simple text formats, while Parquet files are more suitable for large-scale data storage because they can be more efficiently compressed and stored in a columnar format.

[0079] Use visualization tools to present analysis results: For visual presentation of data analysis, you can use various visualization tools, such as Matplotlib, Seaborn, Plotly, etc. These tools can draw various charts, graphs, and visual dashboards, making the analysis results more intuitive and easy to understand.

[0080] In a possible implementation, after step S13, the method further includes the following steps: receiving an analysis and processing request for parsed data, the analysis and processing request carrying a data analysis and processing condition; based on the analysis and processing condition, using SparkSQL to analyze and process the first parsed data, and / or using Hive SQL to analyze and process the first parsed data to obtain an analysis and processing result.

[0081] Specifically, in this technical solution, the server also receives a request for analyzing and processing the parsed data transmitted by the user terminal. The request carries data analysis and processing conditions, i.e., the request specifies the analysis and processing method to be used for the first parsed data. Based on the analysis and processing conditions, the server analyzes and processes the first parsed data using Spark SQL and / or Hive SQL to obtain an analysis and processing result.

[0082] Among them, Hive SQL uses Hive as a query engine, and Hive SQL can convert SQL queries into MapReduce tasks or Tez tasks for execution on a Hadoop cluster, making full use of distributed computing capabilities. Spark SQL uses Spark as an execution engine and can run on a Spark cluster, making full use of Spark's distributed computing capabilities. It can also persist query results in memory to speed up subsequent queries. If the first parsed data needs to be processed in real time, or if you want to use more flexible data processing capabilities, including support for complex data processing operations during query processing, use custom functions, and support more data sources, you can choose to use Spark SQL for analysis and processing. If the data size of the first parsed data is very large, and offline batch processing queries are the majority, you can choose to use Hive SQL for analysis and processing. It should be noted that the analysis and processing method of the first parsed data can also be specifically set according to actual conditions, and is not limited in this application.

[0083] Referring to Figure 4 It shows a structure diagram of a large-scale xml data parsing system provided by an embodiment of the present application. The system comprises an xml file set reading module, a preprocessing module, a parsing module, a processing module, and an output module; the xml file set reading module is configured to read an xml file set from a file system; the xml file set comprises a plurality of xml subfiles; the preprocessing module is configured to preprocess the xml file set to obtain a preprocessed xml file set; the preprocessed xml file set comprises a plurality of preprocessed xml subfiles; the parsing module is configured to parse the plurality of preprocessed xml subfiles based on a Spark data processing framework to obtain first parsed data; the first parsed data is structured data; the processing module is configured to receive at least one parsed data output request, the parsed data output request carrying a data output condition; and the output module is configured to output the first parsed data according to an output mode corresponding to the data output condition.

[0084] In a possible implementation, the parsing module is further configured to take the plurality of preprocessed xml subfiles as a plurality of xml nodes; the parsing module is further configured to transform the plurality of xml nodes, the plurality of initial attributes, and file content in the plurality of preprocessed xml subfiles based on the Spark data processing framework to obtain second parsed data; the parsing module is further configured to transform the second parsed data into third parsed data based on a preset first function; and the parsing module is further configured to transform the third parsed data into the first parsed data based on a preset second function.

[0085] In a possible implementation, the parsing module is further configured to determine whether the plurality of initial attributes contain nested attributes; the parsing module is further configured to, if yes, split each nested attribute into a plurality of sub-attributes, and take each initial attribute other than the nested attribute as a sub-attribute; the parsing module is further configured to, if no, take each initial attribute as a sub-attribute; the parsing module is further configured to map the ith xml node to the ith row of the DataFrame and map the jth sub-attribute to the jth column of the DataFrame based on the Spark data processing framework, where i is an integer in the range of 1 to n, j is an integer in the range of 1 to m, n and m are positive integers greater than 1; the parsing module is further configured to convert, based on the Spark data processing framework, the file content corresponding to the jth sub-attribute in the ith preprocessed xml sub-file into a preset field structure; the preset field structure includes a correspondence between the ith row and the jth column and the file content; and the parsing module is further configured to store the preset field structure in the ith row and the jth column of the DataFrame to obtain second parsed data, where the second parsed data is the DataFrame data.

[0086] In a possible implementation, the parsing module is further configured to convert the second parsed data into third parsed data based on a df.as[ds] function, where the second parsed data is the DataFrame data and the third parsed data is the DataSet data.

[0087] In a possible implementation, the parsing module is further configured to convert the third parsed data into the first parsed data based on a ds.write().mode(SaveMode.Overwrite).save(tablename) function, where the third parsed data is the DataSet data and the first parsed data is the Hive table data.

[0088] In a possible implementation, the processing module is further configured to receive an analysis and processing request of the parsed data, where the analysis and processing request carries a data analysis and processing condition; the processing module is further configured to perform analysis and processing on the first parsed data based on the analysis and processing condition by using the Spark SQL and / or the Hive SQL to obtain an analysis and processing result.

[0089] In a possible implementation, the preprocessing module is further configured to perform a data cleaning operation on each xml sub-file to remove invalid data and duplicate data in the xml sub-file to obtain a plurality of preprocessed xml sub-files.

[0090] It should be noted that the apparatus provided in the above examples is only used as an example for the division of the above functional modules in realizing its functions, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the above described functions. In addition, the apparatus and method embodiments provided in the above examples belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be described here.

[0091] The present application also discloses an electronic device. Referring to Figure 5 , Figure 5 is a structural schematic diagram of an electronic device disclosed by an embodiment of the present application. The electronic device 500 can include at least one processor 501, at least one network interface 504, a user interface 503, a memory 505, and at least one communication bus 502.

[0092] The communication bus 502 is used to realize the connection and communication between the components.

[0093] The user interface 503 can include a display screen (Display) and a camera (Camera), and the optional user interface 503 can further include a standard wired interface and a wireless interface.

[0094] The network interface 504 can optionally include a standard wired interface and a wireless interface (such as a WI-FI interface).

[0095] The processor 501 can include one or more processing cores. The processor 501 connects various parts within the server through various interfaces and lines, performs various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 505, and calling data stored in the memory 505. Alternatively, the processor 501 can be implemented in at least one of a hardware form of a digital signal processing (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor 501 can integrate a combination of one or more of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes operating systems, user interfaces, and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; and the modem is used for processing wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor 501, but can be realized by a separate chip.

[0096] The memory 505 can include a random access memory (RAM) and a read-only memory (ROM). Alternatively, the memory 505 includes a non-transitory computer-readable storage medium. The memory 505 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 505 can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area can store data involved in the above-mentioned various method embodiments, etc. The memory 505 can alternatively be at least one storage device located away from the aforementioned processor 501. Referring to Figure 5 The memory 505 as a kind of computer readable storage medium can include an operating system, a network communication module, a user interface module and an application program.

[0097] In Figure 5In the electronic device 500 shown, the user interface 503 is mainly used to provide an interface for the user to input, and obtain data input by the user; and the processor 501 can be used to invoke an application stored in the storage 505, and when executed by one or more processors 501, the electronic device 500 performs the method of one or more of the above embodiments. It should be noted that, for the above-mentioned method embodiments, in order to simply describe, they are all described as a combination of a series of actions, but those skilled in the art should know that the application is not limited to the action sequence described, because according to the application, some steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily required by the application.

[0098] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0099] In several embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic. The division of units is only a logical function division. There can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some service interface, device or unit, and can be electrical or other forms.

[0100] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0101] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically independently, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0102] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable memory. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a memory and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. The aforementioned memory includes: a U disk, a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0103] The above are only exemplary embodiments of the present disclosure, and cannot limit the scope of the present disclosure. That is, any equivalent changes and modifications made in accordance with the teachings of the present disclosure are still within the scope of the present disclosure. Other embodiments of the present disclosure will be readily apparent to those skilled in the art upon considering the specification and practicing the true principles of the present disclosure.

[0104] The present application is intended to cover any variations, uses or adaptive changes of the present disclosure that follow the general principles of the present disclosure and include common knowledge or conventional technical means in the technical field not described in the present disclosure. The specification and examples are only considered as exemplary, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A method for parsing large-scale XML data, characterized by, The method comprises: reading an xml file set from a file system; the xml file set comprises a plurality of xml subfiles; performing data cleaning preprocessing on the xml file set to obtain a preprocessed xml file set; the preprocessed xml file set comprises a plurality of preprocessed xml subfiles, and each preprocessed xml subfile comprises a plurality of initial attributes; taking the plurality of preprocessed xml subfiles as a plurality of xml nodes; parsing the plurality of xml nodes based on a Spark data processing framework to obtain first parsed data; the first parsed data is structured data; receiving at least one parsed data output request, the parsed data output request carrying a data output condition; outputting the first parsed data according to an output mode corresponding to the data output condition; wherein the parsing of the plurality of xml nodes based on the Spark data processing framework to obtain the first parsed data specifically comprises: determining whether the plurality of initial attributes contain nested attributes; if yes, splitting each nested attribute into a plurality of subattributes, and taking each initial attribute other than the nested attribute as a subattribute; if no, taking each initial attribute as a subattribute; mapping the ith xml node to the ith row of a DataFrame and mapping the jth subattribute to the jth column of the DataFrame based on the Spark data processing framework; wherein i∈[1,n], j∈[1,m], n and m are both positive integers greater than 1; converting the file content corresponding to the jth subattribute in the ith preprocessed xml subfile to a preset field structure based on the Spark data processing framework; the preset field structure comprises a correspondence between the ith row and the jth column and the file content; storing the preset field structure in the ith row and the jth column of the DataFrame to obtain second parsed data; the second parsed data is DataFrame data; transforming the second parsed data into third parsed data based on a df.as[ds] function; the second parsed data is DataFrame data, and the third parsed data is DataSet data; transforming the third parsed data into the first parsed data based on a ds.write().mode(SaveMode.Overwrite).save(tablename) function; the first parsed data is Hive table data.

2. The method of claim 1, wherein, After the parsing of the plurality of preprocessed xml subfiles based on the Spark data processing framework to obtain the first parsed data, the method further comprises: receiving an analysis processing request of the parsed data, the analysis processing request carrying a data analysis processing condition; performing analysis processing on the first parsed data based on the analysis processing condition by using Spark SQL and / or Hive SQL to obtain an analysis processing result.

3. The method of claim 1, wherein, The preprocessing of the xml file set obtains a preprocessed xml file set, and specifically includes the following steps: A data cleaning operation is performed on each xml sub-file to remove invalid data and duplicate data in the xml sub-file, thereby obtaining a plurality of preprocessed xml sub-files.

4. A system for parsing large xml data, characterized by, The system comprises an xml file set reading module (1), a preprocessing module (2), an analysis module (3), a processing module (4), and an output module (5); The xml file set reading module (1) is configured to read an xml file set from a file system; the xml file set comprises a plurality of xml sub-files; The preprocessing module (2) is configured to preprocess the xml file set to obtain a preprocessed xml file set; the preprocessed xml file set comprises a plurality of preprocessed xml sub-files, and each preprocessed xml sub-file comprises a plurality of initial attributes; the preprocessed xml sub-files are used as xml nodes; The analysis module (3) is configured to analyze the plurality of preprocessed xml sub-files based on a Spark data processing framework to obtain first analysis data; the first analysis data is structured data; The processing module (4) is configured to receive at least one analysis data output request; the analysis data output request carries a data output condition; The output module (5) is configured to output the first analysis data according to an output mode corresponding to the data output condition; The analysis module (3) is specifically configured to determine whether the plurality of initial attributes contain nested attributes; if yes, each nested attribute is split into a plurality of sub-attributes, and each initial attribute other than the nested attributes is used as a sub-attribute. If not, each of the initial attributes is taken as a sub-attribute; based on the Spark data processing framework, the i-th xml node is mapped to the i-th row of a DataFrame, and the j-th sub-attribute is mapped to the j-th column of the DataFrame; wherein i∈[1,n], j∈[1,m], n and m are both positive integers greater than 1; based on the Spark data processing framework, the file content corresponding to the j-th sub-attribute in the i-th preprocessed xml sub-file is converted into a preset field structure; the preset field structure includes the correspondence between the i-th row, the j-th column and the file content; the preset field structure is stored in the i-th row, the j-th column of the DataFrame, to obtain second parsed data; the second parsed data is DataFrame data; based on the df.as[ds] function, the second parsed data is converted into third parsed data; the second parsed data is DataFrame data, and the third parsed data is DataSet data; based on the ds.write().mode(SaveMode.Overwrite).save(tablename) function, the third parsed data is converted into the first parsed data; the first parsed data is Hive table data.

5. An electronic device, comprising: The electronic device (500) comprises a processor (501), a memory (505), a user interface (503) and a network interface (504), the memory (505) is used for storing instructions, the user interface (503) and the network interface (504) are used for communicating with other devices, and the processor (501) is used for executing the instructions stored in the memory (505) to enable the electronic device (500) to perform the method of any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, The computer readable storage medium stores instructions which, when executed, perform the method steps of any one of claims 1-3.

Images