Data analysis method and device, computer equipment and storage medium

Abstract

The invention belongs to the technical field of data processing, and relates to a data analysis method, which is applied to the field of financial science and technology, and comprises the following steps: receiving a data analysis request triggered by a user; extracting a character string and a field level string list from the data analysis request; judging whether a specified analysis object matched with the character string exists in a preset analysis object cache or not; if the specified analysis object matched with the character string does not exist, calling a preset target generation function; performing analysis processing on the character string and the field level string list based on the target generation function to obtain a corresponding analysis result; and returning an analysis result to the user. The invention further provides a data analysis device, computer equipment and a storage medium. In addition, the invention also relates to a block chain technology, and the analysis result can be stored in a block chain. According to the method, the json character string can be efficiently and accurately analyzed based on the use of the target generation function, and the corresponding analysis result is generated, so that the processing efficiency of character string analysis is effectively improved.

Description

Technical Field

[0001] The present application relates to the fields of data processing technology and financial technology, and in particular to data analysis methods, devices, computer equipment and storage media. Background Art

[0002] With the rapid development of information technology and the continuous innovation of corporate business models, the diversification of business systems has become a prominent feature of modern financial technology companies. This trend has led to the diversification of data storage needs, especially the widespread application of non-relational databases, which have shown unique advantages in processing large-scale, high-concurrency and complex data structures with their flexible data models, scalability and high performance. Among them, json (JavaScript Object Notation), as a lightweight data exchange format, has become one of the most common storage formats in NoSQL databases because it is easy for people to read and write, and supports the ability to nest complex objects.

[0003] However, with the advent of the big data era, how to efficiently integrate JSON format data stored in NoSQL databases into big data analysis platforms, especially distributed data warehouse systems based on Hadoop such as Hive, has become an urgent problem to be solved. As a powerful tool for processing big data, Hive has a series of built-in functions for data conversion and processing, but it has the problem of low parsing efficiency when processing complex JSON strings. For complex JSON strings with multiple nested layers and containing multiple data types, Hive's built-in JSON parsing functions often cannot efficiently traverse and extract the required information, resulting in slow data processing and affecting the overall analysis efficiency. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a data parsing method, apparatus, computer device and storage medium to solve the technical problem of low parsing efficiency when processing complex JSON strings.

[0005] In order to solve the above technical problems, the present application provides a data analysis method, which adopts the following technical solutions:

[0006] Receive a data parsing request triggered by a user; wherein the data parsing request carries a character string and a field-level string list, and the character string includes json data;

[0007] Extracting the string and the field-level string list from the data parsing request;

[0008] Determine whether there is a specified parsing object matching the string in a preset parsing object cache;

[0009] If there is no specified parsing object matching the string, then call the preset target generation function;

[0010] Parsing the character string and the field-level string list based on the target generation function to obtain a corresponding parsing result;

[0011] The parsing result is returned to the user.

[0012] Furthermore, the step of parsing the character string and the field-level string list based on the target generation function to obtain a corresponding parsing result specifically includes:

[0013] Execute the target generation function to initialize the string to obtain a corresponding specified object;

[0014] Building a corresponding parse tree based on the field level string list and the specified object;

[0015] Recursively traverse the parse tree to generate a corresponding initial result;

[0016] The parsing result is generated based on the initial result.

[0017] Furthermore, the step of initializing the string to obtain the corresponding designated object specifically includes:

[0018] Call the preset processing library;

[0019] Performing object mapping processing on the string based on the serializer in the processing library to obtain a corresponding generated object;

[0020] The generated object is used as the designated object.

[0021] Furthermore, the step of recursively traversing the parse tree to generate a corresponding initial result specifically includes:

[0022] Obtaining a preset recursive algorithm, and traversing the parse tree based on the recursive algorithm;

[0023] For a basic type value node received during the traversal process, obtaining a basic type value corresponding to the basic type value node, and generating a corresponding first result based on the basic type value;

[0024] For an object node received during the traversal process, obtaining a key-value pair corresponding to the object node, and generating a corresponding second result based on the key-value pair;

[0025] For array nodes received during the traversal process, generate a Cartesian product process on the array nodes to obtain a corresponding processing result, and generate a corresponding third result based on the processing result;

[0026] The first result, the second result and the third result are integrated to obtain the initial result.

[0027] Furthermore, the step of generating the analysis result based on the initial result specifically includes:

[0028] Call the preset deduplication statement and sorting statement;

[0029] Performing deduplication processing on the initial result based on the deduplication statement to obtain a corresponding first processing result;

[0030] Sorting the first processing result based on the sorting statement to obtain a corresponding second processing result;

[0031] The second processing result is used as the parsing result.

[0032] Furthermore, before the step of calling the preset target generation function, the method further includes:

[0033] Obtaining a target code corresponding to the target generation function;

[0034] Get preset processing tools;

[0035] Compile and package the target code based on the processing tool to obtain a corresponding target file package;

[0036] Deploy the target file package to a specified environment.

[0037] Furthermore, before the step of calling the preset target generation function, the method further includes:

[0038] Obtaining a preset test strategy; wherein the test strategy includes a unit test strategy and an integrated test strategy;

[0039] Performing unit testing on the target generation function based on the unit testing strategy;

[0040] If the target generation function passes the unit test, then performing an integration test on the target generation function based on the integration test strategy;

[0041] If the target generation function passes the integration test, the target generation function is stored.

[0042] In order to solve the above technical problems, the embodiment of the present application further provides a data analysis device, which adopts the following technical solution:

[0043] A receiving module, configured to receive a data parsing request triggered by a user; wherein the data parsing request carries a character string and a field level string list, and the character string includes json data;

[0044] An extraction module, used to extract the character string and the field level string list from the data parsing request;

[0045] A determination module, used to determine whether there is a specified parsing object matching the string in a preset parsing object cache;

[0046] A calling module, used for calling a preset target generation function if there is no specified parsing object matching the string;

[0047] A parsing module, used for parsing the string and the field-level string list based on the target generation function to obtain a corresponding parsing result;

[0048] The return module is used to return the analysis result to the user.

[0049] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0050] Receive a data parsing request triggered by a user; wherein the data parsing request carries a character string and a field-level string list, and the character string includes json data;

[0051] Extracting the string and the field-level string list from the data parsing request;

[0052] Determine whether there is a specified parsing object matching the string in a preset parsing object cache;

[0053] If there is no specified parsing object matching the string, then call the preset target generation function;

[0054] Parsing the character string and the field-level string list based on the target generation function to obtain a corresponding parsing result;

[0055] The parsing result is returned to the user.

[0056] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0057] Receive a data parsing request triggered by a user; wherein the data parsing request carries a character string and a field-level string list, and the character string includes json data;

[0058] Extracting the string and the field-level string list from the data parsing request;

[0059] Determine whether there is a specified parsing object matching the string in a preset parsing object cache;

[0060] If there is no specified parsing object matching the string, then call the preset target generation function;

[0061] Parsing the character string and the field-level string list based on the target generation function to obtain a corresponding parsing result;

[0062] The parsing result is returned to the user.

[0063] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0064] The present application first receives a data parsing request triggered by a user; wherein the data parsing request carries a string and a field-level string list, and the string includes JSON data; then the string and the field-level string list are extracted from the data parsing request; then it is determined whether there is a specified parsing object matching the string in the preset parsing object cache; if there is no specified parsing object matching the string, a preset target generation function is called; subsequently, the string and the field-level string list are parsed based on the target generation function to obtain a corresponding parsing result; finally, the parsing result is returned to the user. After receiving a data parsing request triggered by a user, the present application extracts the string and the field-level string list from the data parsing request, and when it is detected that there is no specified parsing object matching the string in the preset parsing object cache, the preset target generation function is intelligently called to parse the string and the field-level string list, thereby achieving efficient and accurate parsing of complex JSON strings and generating corresponding parsing results, effectively improving the processing efficiency of string parsing, which is conducive to optimizing the data processing efficiency and flexibility in hive. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0066] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0067] Figure 2 A flowchart of an embodiment of a data parsing method according to the present application;

[0068] Figure 3 is a structural schematic diagram of an embodiment of a data analysis device according to the present application;

[0069] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0070] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0071] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0072] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0073] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0074] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0075] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to the laptop computer 1011, tablet computer 1012 or mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0076] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0077] It should be noted that the data analysis method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the data analysis device is generally arranged in the server / terminal device.

[0078] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0079] Continue to refer Figure 2 , shows a flow chart of an embodiment of the data analysis method according to the present application. According to different requirements, the order of the steps in the flow chart can be changed, and some steps can be omitted. The data analysis method provided in the embodiment of the present application can be applied to any scenario where data analysis is required, and the data analysis method can be applied to products in these scenarios, for example, data analysis in the financial insurance field. The data analysis method comprises the following steps:

[0080] Step S201, receiving a data parsing request triggered by a user; wherein the data parsing request carries a character string and a field-level string list, and the character string includes json data.

[0081] In this embodiment, the electronic device (eg Figure 1The server / terminal device shown in the figure) can obtain the data parsing request through a wired connection or a wireless connection. It should be noted that the above-mentioned wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future. The executor of the present application may be a data parsing system, or simply referred to as a system. Among them, the data parsing system may specifically be a hive database. The hive database is a data warehouse tool based on Hadoop, which is used for data extraction, transformation, and loading. This is a mechanism for storing, querying and analyzing large-scale data stored in Hadoop. In addition, the above-mentioned string is a json string containing complex json data. The above-mentioned data parsing request is a request triggered by a user for parsing a string. The above-mentioned string refers to the json string to be parsed. The above-mentioned field hierarchy string list contains a list of field hierarchy paths, representing multiple object attributes that want to be parsed. For example, in the field of financial insurance, the above-mentioned object attributes may include family member customer information, annual premium, industry annual premium, aggregated annual total premium, the insured amount information of the main insurance in each policy, and the insured amount information of the supplementary insurance of the main insurance in each policy, and other attribute information.

[0082] Step S202: extract the character string and the field-level string list from the data parsing request.

[0083] In this embodiment, the character string and the field-level string list carried in the data parsing request may be extracted by parsing the request body of the data parsing request.

[0084] Step S203, determining whether there is a specified parsing object matching the character string in the preset parsing object cache.

[0085] In this embodiment, the above-mentioned parsing object cache is a pre-constructed cache medium (such as memory) for storing and processing the parsing objects corresponding to the json string that has undergone the object parsing process. If there is a specified parsing object matching the string in the above-mentioned parsing object cache, the specified parsing object can be directly called from the parsing object cache later, thereby avoiding multiple object parsing processes for the same json string, thereby effectively improving the intelligence of the object parsing process for the json string. In addition, if there is no specified parsing object matching the string in the above-mentioned parsing object cache, the string can be subjected to object parsing, and the obtained parsing object can be stored in the above-mentioned cache medium later for direct calling later.

[0086] Step S204: If there is no designated parsing object matching the character string, a preset target generation function is called.

[0087] In this embodiment, the target generation function is a pre-built custom UDTF (User-DefinedTable-Generating Function) function. The function input of the target generation function includes: a json string, a field level string list. The function output of the target generation function includes: a result set of multiple rows and columns, each column corresponding to the parsed result of a field level string. If a field does not exist in json, the corresponding column outputs a null value.

[0088] Step S205 , parsing the character string and the field-level string list based on the target generation function to obtain a corresponding parsing result.

[0089] In this embodiment, the above-mentioned specific implementation process of parsing the string and the field-level string list based on the target generation function to obtain the corresponding parsing result will be further described in detail in subsequent specific embodiments of the present application and will not be elaborated here.

[0090] Step S206: Return the analysis result to the user.

[0091] In this embodiment, the analysis result can be returned to the user by sending an email, sending a text message, or displaying it on an interface.

[0092] The present application first receives a data parsing request triggered by a user; wherein the data parsing request carries a string and a field-level string list, and the string includes JSON data; then the string and the field-level string list are extracted from the data parsing request; then it is determined whether there is a specified parsing object matching the string in the preset parsing object cache; if there is no specified parsing object matching the string, a preset target generation function is called; subsequently, the string and the field-level string list are parsed based on the target generation function to obtain a corresponding parsing result; finally, the parsing result is returned to the user. After receiving a data parsing request triggered by a user, the present application extracts the string and the field-level string list from the data parsing request, and when it is detected that there is no specified parsing object matching the string in the preset parsing object cache, the preset target generation function is intelligently called to parse the string and the field-level string list, thereby achieving efficient and accurate parsing of complex JSON strings and generating corresponding parsing results, effectively improving the processing efficiency of string parsing, which is conducive to optimizing the data processing efficiency and flexibility in hive.

[0093] In some optional implementations, step S205 includes the following steps:

[0094] The target generation function is executed to initialize the character string to obtain a corresponding designated object.

[0095] In this embodiment, the specific implementation process of initializing the string to obtain the corresponding designated object will be described in further detail in subsequent specific embodiments of the present application, and will not be elaborated on here.

[0096] A corresponding parse tree is constructed based on the field level string list and the specified object.

[0097] In this embodiment, the multiple fields that need to be parsed contained in the above-mentioned field hierarchy string list will traverse the above-mentioned specified objects (josn objects) in turn, and jointly construct a parse tree, and only retain the information of the fields that need to be parsed, and mark the array nodes and leaf nodes. In the process of constructing the parse tree, the array nodes will be traversed to obtain the array members, the number of branches will be clarified, and the values ​​will be recorded in the leaf nodes of the new tree. Among them, the array nodes need special processing, and their indexes and possible number of elements are recorded for subsequent Cartesian product operations. In addition, for complex objects that are not leaf nodes, you can choose to keep them as json strings or further parse them into nodes of subtrees. Exemplarily, for complex objects that are not leaf nodes in the original josn (such as $[].policy_detail_datas), this child node will be output as a josn string.

[0098] The parse tree is recursively traversed to generate a corresponding initial result.

[0099] In this embodiment, the specific implementation process of recursively traversing the parse tree to generate the corresponding initial result will be further described in detail in subsequent specific embodiments of the present application and will not be elaborated on here.

[0100] The parsing result is generated based on the initial result.

[0101] In this embodiment, the specific implementation process of generating the analysis result based on the initial result will be further described in detail in subsequent specific embodiments of the present application, and will not be elaborated in detail here.

[0102] The present application executes the target generation function to initialize the string to obtain the corresponding specified object; then builds a corresponding parse tree based on the field level string list and the specified object; then recursively traverses the parse tree to generate a corresponding initial result; and subsequently generates the parse result based on the initial result. The present application executes the target generation function to initialize the string to obtain the corresponding specified object; then builds a corresponding parse tree based on the field level string list and the specified object; then recursively traverses the parse tree to generate an initial result; and then generates the corresponding parse result based on the initial result, thereby achieving efficient parsing of complex json strings and generating corresponding parse results, effectively improving the processing efficiency of string parsing, which is beneficial to optimizing data processing efficiency and flexibility in Hive.

[0103] In some optional implementations of this embodiment, the initialization processing of the string to obtain the corresponding designated object includes the following steps:

[0104] Recall a preset processing library.

[0105] In this embodiment, the processing library may specifically adopt a database having a function of converting a JSON string into a JSON object, for example, a Jackson library may be adopted.

[0106] The string is subjected to object mapping processing based on the serializer in the processing library to obtain a corresponding generated object.

[0107] In this embodiment, the serializer may refer to a custom serializer / deserializer in a processing library. Based on streaming processing, the string is subjected to object mapping processing using a serializer based on the processing library to obtain a corresponding generated object. Specifically, the json string may be parsed step by step through a streaming API, rather than loading the entire json string into memory at one time. This can significantly reduce memory consumption.

[0108] The generated object is used as the designated object.

[0109] In this embodiment, the above initialization process is only performed once at the beginning of the target generation function to avoid parsing the same large json string multiple times. In addition, if the json structure contains a large amount of optional or conditional data, a lazy loading mechanism can be used, that is, these data parts are loaded or parsed only when needed.

[0110] The present application calls a preset processing library; then performs object mapping processing on the string based on the serializer in the processing library to obtain a corresponding generated object; and subsequently uses the generated object as the designated object. The present application calls a processing library, and then performs object mapping processing on the string based on the use of the serializer in the processing library, so that the initialization processing of the string can be completed quickly and accurately and the corresponding designated object can be obtained, thereby improving the processing efficiency of the initialization processing of the string and ensuring the data accuracy of the obtained designated object.

[0111] In some optional implementations, recursively traversing the parse tree to generate a corresponding initial result includes the following steps:

[0112] A preset recursive algorithm is obtained, and the parse tree is traversed based on the recursive algorithm.

[0113] In this embodiment, the above recursive algorithm can specifically adopt a breadth-first search algorithm. The current node is received as a parameter, and different operations are performed according to the type of the node (basic type value, object, array). Specifically, when the leaf node is traversed, it is returned to form a row of data. In the recursive backtracking process, the same fields in the array nodes will be traversed independently. When there are array nodes in branches belonging to different fields (that is, there are multiple rows of records), a Cartesian product needs to be performed. For example, in $[0].safeguard_info_st[1], $[0].safeguard_info_st[1].child_list[] is an array branch with 5 data, then the three attribute values ​​under $[0].safeguard_info_st[1] need to be repeatedly filled into these 5 records. If $[0].safeguard_info_st[1] also has other array branches besides child_list, and there are 3 rows of data during the traversal process, then the Cartesian product will eventually generate 15 rows of data.

[0114] For a basic type value node received during the traversal process, a basic type value corresponding to the basic type value node is obtained, and a corresponding first result is generated based on the basic type value.

[0115] In this embodiment, if the current node is a basic type value (such as a string, a number, a Boolean value), it is added to the result of the current row (which may be a list or a custom data structure) to obtain the corresponding first result.

[0116] For an object node received during the traversal process, a key-value pair corresponding to the object node is obtained, and a corresponding second result is generated based on the key-value pair.

[0117] In this embodiment, if the current node is an object, its key-value pairs are traversed. For each key-value pair, its key can be used as part of the result, and the value node is processed recursively to obtain the corresponding second result.

[0118] For array nodes received during the traversal process, Cartesian product processing is performed on the array nodes to obtain corresponding processing results, and a corresponding third result is generated based on the processing results.

[0119] In the present embodiment, the array node is the key to generate the Cartesian product. For each array element, the child nodes under it are processed recursively, and the intermediate results are saved. Wherein, a list of lists (or similar data structures) can be used to store the result set after each array element is processed. In addition, when traversing the array, for each array element, an intermediate result set (which may be a list in a list of lists) is recursively generated. And after all related arrays are traversed, nested loops (or recursive algorithms) are used to combine these intermediate result sets to generate the final Cartesian product to obtain the above-mentioned third result.

[0120] The first result, the second result and the third result are integrated to obtain the initial result.

[0121] In this embodiment, the first result, the second result and the third result are integrated to obtain integrated data, and the integrated data is used as the initial result. In addition, the default map field in the initial result is assigned a null value. Since the attributes that do not exist in the json object will not generate branches during the traversal, the map is often less than 22 elements.

[0122] The present application obtains a preset recursive algorithm, and traverses the parse tree based on the recursive algorithm; then for the basic type value node received during the traversal process, obtain the basic type value corresponding to the basic type value node, and generate the corresponding first result based on the basic type value; and for the object node received during the traversal process, obtain the key-value pair corresponding to the object node, and generate the corresponding second result based on the key-value pair; and for the array node received during the traversal process, generate the Cartesian product of the array node to obtain the corresponding processing result, and generate the corresponding third result based on the processing result; subsequently integrate the first result, the second result and the third result to obtain the initial result. The present application traverses the parse tree based on the use of a recursive algorithm, and then performs different operations according to the type of the node (basic type value, object, array) to obtain different results respectively, and then integrates all the obtained results, so as to accurately generate the final initial result, and ensure the accuracy of the obtained initial result. The subsequent generation of the corresponding parsing result based on the initial result can realize the efficient parsing of complex json strings, and generate corresponding parsing results, effectively improving the processing efficiency of string parsing.

[0123] In some optional implementations, generating the parsing result based on the initial result includes the following steps:

[0124] Call the preset deduplication statement and sorting statement.

[0125] In this embodiment, the above deduplication statement can specifically use the GROUP BY clause to remove duplicate rows from the result. The above sorting statement can specifically use the ORDER BY clause. In Hive, the query results can be sorted by using the ORDER BY clause. This clause allows one or more columns to be specified as the basis for sorting, and ascending (ASC) or descending (DESC) sorting can be specified.

[0126] The initial result is deduplicated based on the deduplication statement to obtain a corresponding first processing result.

[0127] In this embodiment, the above deduplication statement may be executed to perform deduplication processing on the initial result, thereby obtaining a corresponding first processing result.

[0128] The first processing result is sorted based on the sorting statement to obtain a corresponding second processing result.

[0129] In this embodiment, the first processing results may be sorted by executing the above-mentioned sorting statement to obtain the corresponding second processing results.

[0130] The second processing result is used as the parsing result.

[0131] The present application calls preset deduplication statements and sorting statements; then performs deduplication processing on the initial result based on the deduplication statement to obtain a corresponding first processing result; then sorts the first processing result based on the sorting statement to obtain a corresponding second processing result; and subsequently uses the second processing result as the parsing result. The present application calls preset deduplication statements and sorting statements to perform deduplication processing and sorting processing on the initial result, thereby automatically and intelligently completing the optimization processing of the initial result, improving the accuracy and standardization of the generated parsing results, and being conducive to meeting specific query requirements.

[0132] In some optional implementations of this embodiment, before step S204, the electronic device may further perform the following steps:

[0133] Obtain a target code corresponding to the target generation function.

[0134] In this embodiment, the target code refers to a pre-written Java code corresponding to the target generation function. The target code complies with the interface specification of Hive UDF / UDTF.

[0135] Get preset processing tools.

[0136] In this embodiment, the processing tool includes using a Java compiler and a packaging tool.

[0137] The target code is compiled and packaged based on the processing tool to obtain a corresponding target file package.

[0138] In this embodiment, the target code can be compiled and packaged by using a Java compiler and a packaging tool to be packaged into a JAR file package, that is, the above-mentioned target file package.

[0139] Deploy the target file package to a specified environment.

[0140] In this embodiment, the compiled and packaged target file package can be uploaded to the hive server or all nodes running Hive queries to ensure that the hive server can access the target file package.

[0141] The present application obtains the target code corresponding to the target generation function; then obtains the preset processing tool; then compiles and packages the target code based on the processing tool to obtain the corresponding target file package; and subsequently deploys the target file package to the specified environment. After completing the construction of the target generation function, the present application will also intelligently obtain the target code corresponding to the target generation function, and then compile and package the target code based on the use of the processing tool to obtain the corresponding target file package, and then deploy the target file package to the specified environment, so as to realize the automatic deployment processing of the target file corresponding to the target generation function, improve the processing efficiency of file deployment, and facilitate the subsequent guarantee that the target generation function can be smoothly called in the specified environment to automatically and quickly perform string parsing processing and generate accurate parsing results.

[0142] In some optional implementations of this embodiment, before step S204, the electronic device may further perform the following steps:

[0143] Obtain a preset test strategy; wherein the test strategy includes a unit test strategy and an integrated test strategy.

[0144] In this embodiment, the strategy content of the above-mentioned unit test strategy includes: 1) Test json parsing: write test cases to verify that the json string can be correctly parsed into the json object, and check various boundary conditions (such as empty string, json with wrong format, json containing special characters, etc.). 2) Perform test data conversion: ensure that the json object can be correctly converted back to the json string, and the converted string is consistent with the original string (or meets the expected difference). 3) Perform functional logic testing: write test cases for the specific logic of the function to ensure that the function can correctly handle various inputs and return the expected results. 4) Perform boundary case testing: pay special attention to boundary conditions, such as empty objects, empty arrays, extreme values ​​and abnormal conditions. The strategy content of the above-mentioned integration test strategy includes: 1) Perform system-level testing. Integrate the function into a larger system and test its performance in the entire system environment. Specifically include interaction with other components, correctness of data flow and system performance. 2) Perform end-to-end testing. Perform complete testing from user input to system output by simulating real user scenarios. This helps to discover problems that may occur during the integration process. 3) Perform performance testing. Use performance testing tools to simulate high concurrent requests and measure the system's response time, throughput and other performance indicators to conduct stability testing. 4) Run the system for a long time and monitor its stability and resource usage to ensure that the system can continue to run stably and recover correctly when abnormal situations occur.

[0145] The target generation function is unit tested based on the unit testing strategy.

[0146] In this embodiment, the unit test process for the target generation function can be performed according to the policy content of the above unit test policy, and a corresponding unit test result can be generated, wherein the unit test result includes passing the unit test or failing the unit test.

[0147] If the target generation function passes the unit test, an integration test is performed on the target generation function based on the integration test strategy.

[0148] In this embodiment, the integration test process for the target generation function can be performed according to the policy content of the above integration test policy, and a corresponding integration test result can be generated, wherein the integration test result includes passing the integration test or failing the integration test.

[0149] If the target generation function passes the integration test, the target generation function is stored.

[0150] In this embodiment, if it is detected that the target generation function has passed both the unit test and the integration test, it is determined that the target generation function has been verified by use and can correctly handle various boundary conditions and complex scenarios. The target generation function is then stored. There is no specific limitation on the storage method of the target generation function, which can be selected according to actual storage requirements, such as blockchain storage, local database storage, cloud server storage, etc.

[0151] The present application obtains a preset test strategy; wherein the test strategy includes a unit test strategy and an integrated test strategy; then the target generation function is unit tested based on the unit test strategy; if the target generation function passes the unit test, the target generation function is integrated tested based on the integrated test strategy; if the target generation function passes the integrated test, the target generation function is stored. After completing the construction of the target generation function, the present application will also intelligently perform unit testing and integration testing on the target generation function based on the combined use of the unit test strategy and the integrated test strategy to ensure that the target generation function can correctly handle various boundary conditions and complex scenarios, thereby ensuring the accuracy of the generated target generation function. In addition, after detecting that the target generation function has passed both the unit test and the integrated test, the target generation function will also be intelligently stored to ensure the data security of the target generation function.

[0152] In some optional implementations, the user information obtained is subject to the user's consent and complies with relevant laws and policies.

[0153] In addition, the non-Company software tools or components that appear in the embodiments of the present application are merely examples and do not represent actual use.

[0154] In addition, the main optimization point of the data parsing method proposed in this application during the execution process is to reduce the process of initializing large josn into josn objects. Each call in the two native functions will execute the initialization of the josn object once, which has a significant optimization effect on multi-field and multi-level parsing with large josn strings. And for the SQL writing process, more concise and clear code can reduce the learning and use costs of data developers. This application increases execution efficiency and saves cluster resources by optimizing the execution process of large json, multi-field, and multi-level parsing. And simplify the SQL code of josn parsing, increase ease of use and readability, and reduce usage and reading costs.

[0155] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0156] It should be emphasized that in order to further ensure the privacy and security of the above analysis results, the above analysis results can also be stored in a blockchain node.

[0157] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0158] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0159] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0160] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0161] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0162] Further references Figure 3 , as a response to the above Figure 2 In order to realize the method shown in the figure, the present application provides an embodiment of a data analysis device, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0163] like Figure 3 As shown, the data analysis device 300 described in this embodiment includes: a receiving module 301, an extraction module 302, a judgment module 303, a calling module 304, a parsing module 305 and a return module 306. Among them:

[0164] The receiving module 301 is used to receive a data parsing request triggered by a user; wherein the data parsing request carries a character string and a field level string list, and the character string includes json data;

[0165] An extraction module 302, configured to extract the character string and the field level string list from the data parsing request;

[0166] The judgment module 303 is used to judge whether there is a specified parsing object matching the string in the preset parsing object cache;

[0167] A calling module 304 is used to call a preset target generation function if there is no specified parsing object matching the string;

[0168] A parsing module 305 is used to parse the string and the field level string list based on the target generation function to obtain a corresponding parsing result;

[0169] The return module 306 is used to return the analysis result to the user.

[0170] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the data analysis method in the aforementioned implementation mode, and are not described in detail here.

[0171] In some optional implementations of this embodiment, the parsing module 305 includes:

[0172] An initialization submodule, used to execute the target generation function, and initialize the string to obtain a corresponding specified object;

[0173] A construction submodule, used for constructing a corresponding parse tree based on the field level string list and the specified object;

[0174] A first generating submodule, used for recursively traversing the parse tree to generate a corresponding initial result;

[0175] The second generating submodule is used to generate the parsing result based on the initial result.

[0176] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the data analysis method in the aforementioned implementation mode, and are not described in detail here.

[0177] In some optional implementations of this embodiment, the initialization submodule includes:

[0178] A first calling unit, used for calling a preset processing library;

[0179] A mapping unit, configured to perform object mapping processing on the string based on the serializer in the processing library to obtain a corresponding generated object;

[0180] The first determining unit is configured to use the generated object as the designated object.

[0181] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the data analysis method in the aforementioned implementation mode, and are not described in detail here.

[0182] In some optional implementations of this embodiment, the first generating submodule includes:

[0183] A traversal unit, used to obtain a preset recursive algorithm and traverse the parse tree based on the recursive algorithm;

[0184] A first processing unit is configured to obtain, for a basic type value node received during the traversal process, a basic type value corresponding to the basic type value node, and generate a corresponding first result based on the basic type value;

[0185] A second processing unit is used to obtain a key-value pair corresponding to an object node received during the traversal process, and generate a corresponding second result based on the key-value pair;

[0186] A third processing unit is used to generate a Cartesian product process on the array nodes received during the traversal process to obtain a corresponding processing result, and generate a corresponding third result based on the processing result;

[0187] An integration unit is used to integrate the first result, the second result and the third result to obtain the initial result.

[0188] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the data analysis method in the aforementioned implementation mode, and are not described in detail here.

[0189] In some optional implementations of this embodiment, the second generating submodule includes:

[0190] A second calling unit, used for calling preset deduplication statements and sorting statements;

[0191] A sorting unit, configured to perform deduplication processing on the initial result based on the deduplication statement to obtain a corresponding first processing result;

[0192] a deduplication unit, configured to sort the first processing result based on the sorting statement to obtain a corresponding second processing result;

[0193] The second determining unit is configured to use the second processing result as the parsing result.

[0194] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the data analysis method in the aforementioned implementation mode, and are not described in detail here.

[0195] In some optional implementations of this embodiment, the data parsing device further includes:

[0196] A first acquisition module, used for acquiring a target code corresponding to the target generation function;

[0197] A second acquisition module, used to acquire a preset processing tool;

[0198] A processing module, used for compiling and packaging the target code based on the processing tool to obtain a corresponding target file package;

[0199] A deployment module is used to deploy the target file package to a specified environment.

[0200] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the data analysis method in the aforementioned implementation mode, and are not described in detail here.

[0201] In some optional implementations of this embodiment, the data parsing device further includes:

[0202] A third acquisition module is used to acquire a preset test strategy; wherein the test strategy includes a unit test strategy and an integrated test strategy;

[0203] A first testing module, configured to perform unit testing on the target generation function based on the unit testing strategy;

[0204] A second testing module, configured to perform an integration test on the target generation function based on the integration testing strategy if the target generation function passes the unit test;

[0205] The storage module is used to store the target generation function if the target generation function passes the integration test.

[0206] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the data analysis method in the aforementioned implementation mode, and are not described in detail here.

[0207] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0208] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components 41-43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0209] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0210] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk equipped on the computer device 4, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of the data parsing method, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0211] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run computer-readable instructions stored in the memory 41 or process data, such as computer-readable instructions for running the data parsing method.

[0212] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0213] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0214] In an embodiment of the present application, after receiving a data parsing request triggered by a user, the string and the field-level string list are extracted from the data parsing request, and when it is detected that there is no specified parsing object matching the string in the preset parsing object cache, the preset target generation function is intelligently called to parse the string and the field-level string list, thereby achieving efficient and accurate parsing of complex JSON strings and generating corresponding parsing results, effectively improving the processing efficiency of string parsing, which is beneficial to optimizing the data processing efficiency and flexibility in hive.

[0215] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the data parsing method as described above.

[0216] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0217] In an embodiment of the present application, after receiving a data parsing request triggered by a user, the string and the field-level string list are extracted from the data parsing request, and when it is detected that there is no specified parsing object matching the string in the preset parsing object cache, the preset target generation function is intelligently called to parse the string and the field-level string list, thereby achieving efficient and accurate parsing of complex JSON strings and generating corresponding parsing results, effectively improving the processing efficiency of string parsing, which is beneficial to optimizing the data processing efficiency and flexibility in hive.

[0218] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0219] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A data analysis method, characterized in that: The steps include: Receive a data parsing request triggered by a user; wherein the data parsing request carries a character string and a field-level string list, and the character string includes json data; Extracting the string and the field-level string list from the data parsing request; Determine whether there is a specified parsing object matching the string in a preset parsing object cache; If there is no specified parsing object matching the string, then call the preset target generation function; Parsing the character string and the field-level string list based on the target generation function to obtain a corresponding parsing result; The parsing result is returned to the user.

2. The data analysis method according to claim 1, characterized in that: The step of parsing the character string and the field-level string list based on the target generation function to obtain a corresponding parsing result specifically includes: Execute the target generation function to initialize the string to obtain a corresponding specified object; Building a corresponding parse tree based on the field level string list and the specified object; Recursively traverse the parse tree to generate a corresponding initial result; The parsing result is generated based on the initial result.

3. The data analysis method according to claim 2, characterized in that: The step of initializing the string to obtain the corresponding designated object specifically includes: Call the preset processing library; Performing object mapping processing on the string based on the serializer in the processing library to obtain a corresponding generated object; The generated object is used as the designated object.

4. The data analysis method according to claim 2, characterized in that: The step of recursively traversing the parse tree to generate a corresponding initial result specifically includes: Obtaining a preset recursive algorithm, and traversing the parse tree based on the recursive algorithm; For a basic type value node received during the traversal process, obtaining a basic type value corresponding to the basic type value node, and generating a corresponding first result based on the basic type value; For an object node received during the traversal process, obtaining a key-value pair corresponding to the object node, and generating a corresponding second result based on the key-value pair; For array nodes received during the traversal process, generate a Cartesian product process on the array nodes to obtain a corresponding processing result, and generate a corresponding third result based on the processing result; The first result, the second result and the third result are integrated to obtain the initial result.

5. The data analysis method according to claim 2, characterized in that: The step of generating the analysis result based on the initial result specifically includes: Call the preset deduplication statement and sorting statement; Performing deduplication processing on the initial result based on the deduplication statement to obtain a corresponding first processing result; Sorting the first processing result based on the sorting statement to obtain a corresponding second processing result; The second processing result is used as the parsing result.

6. The data analysis method according to claim 1, characterized in that: Before the step of calling the preset target generation function, the method further includes: Obtaining a target code corresponding to the target generation function; Get preset processing tools; Compile and package the target code based on the processing tool to obtain a corresponding target file package; Deploy the target file package to a specified environment.

7. The data analysis method according to claim 1, characterized in that: Before the step of calling the preset target generation function, the method further includes: Obtaining a preset test strategy; wherein the test strategy includes a unit test strategy and an integrated test strategy; Performing unit testing on the target generation function based on the unit testing strategy; If the target generation function passes the unit test, then performing an integration test on the target generation function based on the integration test strategy; If the target generation function passes the integration test, the target generation function is stored.

8. A data analysis device, characterized in that: include: A receiving module, configured to receive a data parsing request triggered by a user; wherein the data parsing request carries a character string and a field level string list, and the character string includes json data; An extraction module, used to extract the character string and the field level string list from the data parsing request; A determination module, used to determine whether there is a specified parsing object matching the string in a preset parsing object cache; A calling module, used for calling a preset target generation function if there is no specified parsing object matching the string; A parsing module, used for parsing the string and the field-level string list based on the target generation function to obtain a corresponding parsing result; The return module is used to return the analysis result to the user.

9. A computer device, characterized in that: It comprises a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the data parsing method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the data parsing method according to any one of claims 1 to 7 are implemented.

Images