Data analysis method, device and equipment and computer readable storage medium

Abstract

The embodiment of the invention relates to the technical field of computers, and discloses a data analysis method, device and equipment and a computer readable storage medium, and the method comprises the steps: receiving to-be-analyzed data and an analysis instruction of the to-be-analyzed data; utilizing the data analysis model to predict the data structure type of the to-be-analyzed data, and matching to obtain a target analyzer; wherein the target analyzer is obtained by optimizing a parameter adjustment strategy applied by the initial data analysis model by using preset training data and training target parameters of the initial data analysis model by using the optimized parameter adjustment strategy and the preset training data; the preset training data is matched with the data structure type of the to-be-analyzed data; according to the parsing instruction of the to-be-parsed data, the target parser is used for executing corresponding parsing operation on the to-be-parsed data, and a corresponding parsing result is obtained. By means of the technical scheme, data of different data structure types can be analyzed in a self-adaptive mode, and the accuracy and efficiency of data analysis are improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of computer technology, and in particular to a data analysis method, device, equipment, and computer-readable storage medium. Background Art

[0002] With the rapid development of information technology, JSON (JavaScript Object Notation) format has become an important standard for network data exchange due to its lightweight, easy to read and write characteristics.

[0003] Traditional JSON data parsing methods often use regular expressions or state machine-based parsers, but they are not suitable for complex structure types or non-standard JSON strings. For example, when faced with JSON data with complex nesting and different structure types, the performance of the parser will drop sharply, making it difficult to adapt to the ever-changing data structure, with low data parsing accuracy and lack of flexibility and scalability.

[0004] At the same time, with the development of big data and artificial intelligence technology, the demand for JSON data processing is growing. Traditional JSON data parsing methods often consume a lot of memory and CPU resources when processing large-scale data, and the parsing speed is slow and the data parsing efficiency is low. Summary of the invention

[0005] In view of the above problems, embodiments of the present invention provide a data parsing method, apparatus, device and computer-readable storage medium for adaptively parsing data of different data structure types to improve the accuracy and efficiency of data parsing.

[0006] According to one aspect of an embodiment of the present invention, there is provided a data parsing method, the method comprising: receiving data to be parsed and a parsing instruction for the data to be parsed;

[0007] Using a data parsing model, predicting the data structure type of the data to be parsed;

[0008] According to the data structure type of the data to be parsed, a target parser is obtained by matching;

[0009] The target parser is obtained by optimizing the parameter adjustment strategy applied to the initial data parsing model using the preset training data, and training the target parameters of the initial data parsing model using the optimized parameter adjustment strategy and the preset training data; the data structure type of the preset training data matches the data structure type of the data to be parsed;

[0010] According to the parsing instruction of the data to be parsed, the target parser is used to perform corresponding parsing operations on the data to be parsed to obtain corresponding parsing results.

[0011] A data parsing method provided by the present invention utilizes a data parsing model to efficiently and accurately parse and predict the data structure type of the data to be parsed. The present invention utilizes training data of different data structure types to obtain different parsers, and matches the corresponding parsers by the data structure type of the data to be parsed, and the matched parsers are obtained based on the training data that matches the data structure type of the data to be parsed. The present invention matches the operation of the corresponding parser for data of different data structure types, and can adaptively process data of different data structure types, thereby improving the flexibility and accuracy of data parsing. At the same time, the present invention performs parsing and processing on data of different data structure types in a targeted manner, thereby improving the efficiency of data parsing.

[0012] According to another aspect of an embodiment of the present invention, there is provided a data analysis device, comprising: a receiving module, a prediction module, a matching module and an analysis module;

[0013] A receiving module, used for receiving data to be parsed and a parsing instruction for the data to be parsed;

[0014] A prediction module, used to predict the data structure type of the data to be parsed by using a data parsing model;

[0015] A matching module, used for matching and obtaining a target parser according to the data structure type of the data to be parsed;

[0016] The target parser is obtained by optimizing the parameter adjustment strategy applied to the initial data parsing model using the preset training data, and training the target parameters of the initial data parsing model using the optimized parameter adjustment strategy and the preset training data; the data structure type of the preset training data matches the data structure type of the data to be parsed;

[0017] The parsing module is used to perform corresponding parsing operations on the data to be parsed using the target parser according to the parsing instructions of the data to be parsed, so as to obtain corresponding parsing results.

[0018] According to another aspect of an embodiment of the present invention, there is provided a data analysis device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation of the data analysis method as described in any one of the above items.

[0019] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the storage medium stores at least one executable instruction, wherein the executable instruction enables a data analysis device / apparatus to perform an operation of any one of the data analysis methods described above.

[0020] The above description is only an overview of the technical solution of the embodiment of the present invention. In order to more clearly understand the technical means of the embodiment of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiment of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present invention. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings. In the accompanying drawings:

[0022] Figure 1 A schematic diagram showing a flow chart of an embodiment of a data analysis method provided by the present invention;

[0023] Figure 2 A schematic diagram showing a flow chart of another embodiment of the data analysis method provided by the present invention;

[0024] Figure 3 A schematic diagram showing the structure of an embodiment of a data analysis device provided by the present invention;

[0025] Figure 4 A schematic structural diagram of an embodiment of a data analysis device provided by the present invention is shown. DETAILED DESCRIPTION

[0026] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0027] Existing technologies often use traditional parsing libraries, stream-based parsing (SAX-like) methods, tree-based parsing (DOM-like) methods, built-in JSON methods using scripting languages, using Extensible Markup Language (XML) as a data exchange format method, using a data exchange format method (such as YAML language, YAML Ain't Markup Language), using custom parsing algorithms or using third-party optimization libraries to implement data parsing. Many existing data parsing methods are selected based on the performance requirements and resource limitations of specific application scenarios.

[0028] Among them, traditional parsing libraries, such as Jackson or Gson, usually use recursive descent parsing or regular expression-based parsing methods, which are fast when processing simple and standardized JSON data, but may not be suitable for complex or non-standard JSON strings.

[0029] Streaming parsers, such as the StAX (Streaming API for XML) parser, are suitable for processing large JSON files without loading the entire document into memory at once. Stream-based parsing methods are very efficient in memory usage and suitable for large amounts of data, but may have limitations in programming complexity and processing speed.

[0030] Tree model parsers, such as JSON-DOM, load the entire JSON document into memory and build it into a tree-like data structure. Tree-based parsing methods allow developers to easily traverse and manipulate data, but may consume a lot of memory, especially when processing large JSON files.

[0031] Many modern programming languages ​​such as JavaScript and Python provide built-in support for JSON parsing. Using the built-in JSON methods of scripting languages ​​is easy to use and has good performance, but may lack flexibility and support for complex data structures.

[0032] XML is a more complex and feature-rich data exchange format that supports advanced features such as comments, attributes, namespaces, etc. However, XML files are usually larger than JSON, are relatively slow to parse, and are not as easy to read and write as JSON.

[0033] Other data serialization formats such as YAML provide a syntax that is closer to natural language than JSON, support comments and more complex data structures. However, formats such as YAML may not be as mature as JSON in terms of parsing speed and library support.

[0034] Developers can design their own parsing algorithms to handle specific JSON data structures. Using a third-party optimized library approach provides the most flexibility, but also requires more development work and possible maintenance costs. Third-party libraries such as ujson or simplejson provide additional performance optimization and customization options. These libraries may provide better performance than the standard library, but increase the project's dependencies.

[0035] Figure 1 FIG. 1 is a flow chart showing an embodiment of a data analysis method provided by the present invention, wherein the method is executed by a data analysis device. Figure 1 As shown, the method comprises the following steps:

[0036] Step 110: receiving data to be parsed and a parsing instruction of the data to be parsed.

[0037] In some embodiments of the present invention, the data to be parsed is JSON data, and the format of the data to be parsed is preferably a JSON string.

[0038] Step 120: Utilize the data parsing model to predict the data structure type of the data to be parsed.

[0039] In some embodiments of the present invention, before predicting the data structure type of the data to be parsed using the data parsing model, the following steps are included:

[0040] Performing standardization on the data to be analyzed to obtain standardized data;

[0041] Segmenting the standardized data to obtain a plurality of target elements;

[0042] Performing semantic analysis on each of the target elements to extract a number of target features;

[0043] Convert each of the target features to obtain a numerical vector of each of the target features;

[0044] The numerical vector of each target feature is used as an input vector of the data analysis model.

[0045] Specifically, the device performs standardization processing on the JSON string of the input data parsing model. For example, the standardization processing may be a unified format processing method such as removing redundant spaces and line breaks in the JSON string. The device divides the standardized JSON string into a number of tokens, such as key-value pairs, brackets, commas and other elements.

[0046] The device uses a tokenizer to perform semantic analysis on tokens and extract key features. The device uses word embedding technology to convert key features into corresponding numerical vectors, and uses the numerical vectors of key features as input vectors of the data parsing model. Exemplarily, the word embedding technology can be to convert key features into corresponding numerical vectors through a semantic representation model such as BERT, or through a model for generating word vectors such as the Word2Vec model.

[0047] In some embodiments of the present invention, the data analysis model is a neural network model, including an input layer, a hidden layer and an output layer connected in sequence; the hidden layer includes a number of neurons and a nonlinear activation function; the data analysis model is obtained by training an initial data analysis model using preset data;

[0048] Using the data parsing model, predicting the data structure type of the data to be parsed includes:

[0049] Receiving an input vector of the data parsing model using the input layer;

[0050] Using each of the neurons in the hidden layer to extract pattern data and feature data of the input vector respectively;

[0051] The pattern data and feature data extracted by each neuron are weightedly summed and used as input of a nonlinear activation function to predict the structural features of the data to be analyzed, and the data structure type of the data to be analyzed is output through an output layer.

[0052] In some embodiments of the present invention, the data parsing model is preferably a deep learning model, such as a recursive neural network (RNN) or a Transformer model, which is used to learn the structural features of input data and can adapt to the data structure of different types of JSON data. Through continuous learning, the data parsing model can continuously learn new feature representations to better capture complex patterns and structures in the data.

[0053] Specifically, the data parsing model uses the input layer to receive the numerical vector of the key features of the input, and inputs the numerical vector of the key features into the hidden layer. The hidden layer includes a number of neurons, each of which is a processing unit for extracting complex patterns and features in the input data. After the hidden layer performs weighted summation on the output of each neuron, the result is input into the nonlinear activation function. Among them, the weight and bias of each neuron are continuously updated through the back propagation algorithm during the training process of the data parsing model to minimize the difference between the predicted output and the actual output. The nonlinear activation function can be a linear rectifier function (ReLU), a Sigmoid function, or a Tanh function, etc., which is applied to the output of the neurons in the hidden layer to introduce nonlinear characteristics and enable the neural network to learn the complex data structure in the input data. The output layer combines the output information of the hidden layer and outputs a vector representing the JSON structure characteristics. The output vector of the data parsing model encodes the structural information of the input vector, such as the nesting level and data type distribution.

[0054] The data parsing model is obtained by training the initial data parsing model using preset data, wherein the preset data is JSON data of different data structure types.

[0055] During the data parsing model training process, the device compares the actual output of the data parsing model with the predicted output, which is the actual data structure characteristics of the JSON data. The device uses a loss function, such as the cross entropy loss function or the mean square error function, to calculate the difference between the predicted output and the actual output of the data parsing model. The gradient of the loss function is back-propagated through the neural network, and the weights and biases in the data parsing model are updated using optimization algorithms, such as the gradient descent algorithm, the Adam algorithm, or the RMSprop algorithm, to reduce the difference between the predicted output and the actual output of the data parsing model and optimize the performance of the data parsing model.

[0056] Step 130: According to the data structure type of the data to be parsed, a target parser is matched;

[0057] Among them, the target parser is obtained by optimizing the parameter adjustment strategy applied by the initial data parsing model using preset training data, and training the target parameters of the initial data parsing model using the optimized parameter adjustment strategy and the preset training data; the data structure type of the preset training data matches the data structure type of the data to be parsed.

[0058] In some embodiments of the present invention, before matching and obtaining a target parser according to the data structure type of the data to be parsed, the following steps are included:

[0059] Construct preset training data according to the preset data structure type;

[0060] For the parameter adjustment strategies applied to the initial data parsing model, determine at least two combinations of parameter adjustment strategies;

[0061] Optimizing the parameter adjustment strategy applied by the initial data analysis model using the preset training data, determining an optimal parameter adjustment strategy combination, and applying the optimal parameter adjustment strategy combination to the initial data analysis model;

[0062] The target parameters of the initial data parsing model are trained using the optimal parameter adjustment strategy combination and the preset training data to obtain a target parser.

[0063] Specifically, for the parameter adjustment strategy applied by the initial data analysis model, at least two combinations of parameter adjustment strategies are determined, including: combining strategies that can be matched from several preset parameter adjustment strategies to determine at least two combinations of parameter adjustment strategies.

[0064] Specifically, the initial data parsing model is preferably a deep learning model, for example, a neural network model. The present invention uses training data of different data structure types to train the initial data parsing model. The device dynamically adjusts the parameter adjustment strategy combination according to the prediction of the model, selects an optimal parameter adjustment strategy combination, and optimizes the parameters of the initial data parsing model using the optimal parameter adjustment strategy combination to obtain a target parser, so that the difference between the predicted output and the actual output of the target parser is minimized. Among them, the parameter adjustment strategy combination can be expanded by introducing new parsing rules or algorithms to adapt to new data structures or parsing requirements.

[0065] In some embodiments of the present invention, the device optimizes the parameters of the initial data parsing model using various parameter adjustment strategy combinations to obtain different parsers. The training data is predicted using each parser, and the parser with the smallest difference between the predicted output and the actual output is selected as the target parser.

[0066] In some embodiments of the present invention, the initial data parsing model is an extensible neural network architecture that allows network layers to be added or modified at different levels to adapt to changing data characteristics and parsing requirements. Optionally, the device implements the initial data parsing model by introducing a modular design so that different parts of the model can be updated and optimized independently without affecting the normal operation of other parts.

[0067] In some embodiments of the present invention, the preset parameter adjustment strategies include: strategies for adjusting the learning rate, strategies for adjusting the model structure, strategies for adjusting the regularization parameters, strategies for adaptive optimization algorithms, strategies for adjusting the activation function of the network layer, strategies for adjusting the connection method of the network layer, early stopping strategies, or strategies for adjusting the batch size (BatchSize), etc.

[0068] Among them, the learning rate is a key hyperparameter that controls the parameter update step size of the model during the training process. By selecting an appropriate learning rate, the device can speed up the convergence of the model, avoid falling into a local optimal solution, or reduce oscillations during the training process. Exemplarily, the strategy for adjusting the learning rate can be a learning rate decay strategy. The device uses the learning rate decay strategy to dynamically adjust the learning rate of the initial data parsing model, and gradually reduces the learning rate of the model as the training progresses. The strategy of the adaptive optimization algorithm is an optimization algorithm that automatically adjusts the learning rate. For example, during the model training process, optimization algorithms such as the AdaGrad algorithm, the RMSprop algorithm, and the Adam algorithm are used to dynamically adjust the learning rate of each parameter according to the update history of the parameters, which helps the model converge more effectively during the training process.

[0069] Exemplarily, the strategy for adjusting the model structure can adaptively adjust the structure of the initial data parsing model according to the characteristics of the input data. Exemplarily, for the initial data parsing model, the number of neurons or layers in the neural network model is dynamically increased or decreased according to the complexity of the data during the model training process.

[0070] Exemplarily, the strategy of adjusting the regularization parameters is to prevent the model from overfitting, introduce L1 or L2 regularization terms, and dynamically adjust their coefficients to control the complexity of the model during model training and improve the generalization ability of the model.

[0071] Exemplarily, the strategy for adjusting the activation function of the network layer is to dynamically switch the activation function between different layers of the network according to the requirements of the parsing task to adapt to different data characteristics.

[0072] Exemplarily, a strategy for adjusting the connection mode of network layers is to introduce a gating mechanism, such as a gated recurrent unit (GRU) or a long short-term memory (LSTM) unit. The neural network model can decide whether to pass, update or forget information based on the characteristics of the input data, thereby dynamically adjusting the connection mode of the network.

[0073] For example, the strategy of early stopping is that during the model training process, if the performance on the validation set does not improve significantly within multiple consecutive training rounds (epochs), the training can be stopped early to prevent overfitting.

[0074] For example, the batch size has an important impact on the training stability and convergence speed of the model. The strategy of adjusting the batch size is to dynamically adjust the batch size according to the performance of the model during training to optimize the training process.

[0075] In some embodiments of the present invention, after training the target parameters of the initial data parsing model using the optimal parameter adjustment strategy combination and the preset training data, the method further includes:

[0076] Performing a pruning operation on the trained initial data parsing model according to a preset pruning strategy, and performing a quantization operation on the trained initial data parsing model according to a preset quantization strategy to obtain a target parser;

[0077] The preset pruning strategy is obtained by optimizing the pruning strategy applied to the trained initial data parsing model using the preset training data;

[0078] The preset quantization strategy is obtained by optimizing the quantization strategy applied by the trained initial data analysis model using preset training data.

[0079] Exemplarily, the pruning operation on the trained initial data parsing model includes:

[0080] Select an unstructured pruning method, such as weight pruning, or a structured pruning strategy, such as channel pruning or convolution kernel pruning, based on your needs. Determine the pruning granularity based on the pruning method. The pruning granularity can be fine-grained, such as pruning a single weight or neuron connection, vector level, kernel level, or filter level.

[0081] According to the selected pruning method and pruning granularity, the importance of each branch is calculated in combination with preset criteria such as L1 norm, weight magnitude, gradient magnitude or mixed criteria. The unimportant branches are determined using the threshold method, and the weights of unimportant branches are set to zero or removed to complete the model pruning. Depending on the model performance, it may be necessary to iterate the pruning process, gradually increasing the pruning ratio until the desired model size and performance balance is achieved.

[0082] Retrain or fine-tune the pruned model to recover the performance loss that may have been caused by pruning.

[0083] Exemplarily, the quantization operation on the trained initial data parsing model includes:

[0084] Determine the quantization method to be used according to requirements, such as dynamic quantization, static quantization, or quantization-aware training (QAT), and set the quantization configuration for the initial data parsing model. For example, select the appropriate quantization strategy and data type, such as int8.

[0085] Depending on the quantization configuration, perform corresponding operations on the trained initial data parsing model. For example, perform simulated quantization operations so that the model can adapt to the accuracy changes brought by quantization during training; or use a small amount of calibration data to determine the quantization scale factor and zero point; or convert the model's weights and activations from floating point numbers to low-precision integer representations.

[0086] Test the performance of the quantized model on the validation set to ensure that the quantized model meets the predetermined accuracy requirements. Further optimize the quantized model, such as adjusting the quantization strategy or fine-tuning the model parameters.

[0087] The device prunes and quantizes the trained initial data parsing model, which can reduce the model size while maintaining the model's performance as much as possible, reduce the model's resource consumption during runtime, and improve the model's inference speed. After the device prunes and quantizes the trained initial data parsing model, it saves it as the target parser.

[0088] By optimizing the parameter adjustment strategy applied to the initial data parsing model based on training data of different data structure types, different optimal parameter adjustment strategy combinations can be obtained. By using the respective optimal parameter adjustment strategy combinations and the training data of the respective data structure types to train the target parameters of the initial data parsing model, and pruning and quantizing the trained initial data parsing model, the parsers corresponding to the training data of different data structure types can be obtained.

[0089] The parsers corresponding to the training data of different data structure types are saved. When the data to be parsed of different data structure types are received, the corresponding parsers can be matched and called, and then the data of different data structure types can be processed adaptively, which improves the flexibility and accuracy of data parsing.

[0090] Step 140: According to the parsing instruction of the data to be parsed, the target parser is used to perform a corresponding parsing operation on the data to be parsed to obtain a corresponding parsing result.

[0091] In some embodiments of the present invention, the target parser integrates one or more parsing modules including: a data format conversion module, a search module, and a query module.

[0092] Specifically, the parsing instructions for the data to be parsed include: one or more of: conversion instructions for a preset data format, preset information search instructions, and preset information query instructions. According to the parsing instructions for the data to be parsed, the corresponding parsing module is called to perform the corresponding parsing operation and output the corresponding parsing results respectively.

[0093] The data analysis method provided by the present invention has the following beneficial effects:

[0094] The present invention uses a deep learning model to learn the structural features of JSON data, and trains different parsers based on different data structure types. The deep learning model can be continuously optimized through continuous learning, and is easy to adapt to new data formats and structures, which is convenient for system maintenance and function expansion. At the same time, it can process more complex data structures, such as nested arrays and objects, and improves data processing capabilities. When processing large-scale data, compared with traditional parsing methods, the present invention can obtain a parser by training data with different data structure types, and use the generalization ability of the deep learning model in the parser to adapt to data of different data structure types, and perform parsing processing separately, which can significantly improve the parsing speed and accuracy, significantly improve efficiency, and reduce parsing errors. At the same time, the present invention integrates complex query and search functions in the parser, further improving the flexibility and efficiency of processing and analyzing JSON data.

[0095] Figure 2FIG. 1 is a flow chart showing another embodiment of the data analysis method provided by the present invention, wherein the method is executed by a data analysis device. Figure 2 As shown, the method comprises the following steps:

[0096] Step 210: According to the parsing instructions of the data to be parsed, call one or more corresponding parsing modules to perform corresponding parsing operations on the data to be parsed to obtain corresponding parsing results; wherein, when calling multiple parsing modules to perform corresponding parsing operations, each of the parsing modules concurrently executes the corresponding parsing operations in a multi-threaded manner, or asynchronously executes the corresponding parsing operations.

[0097] Specifically, the target parser integration includes: one or more parsing modules of a data format conversion module, a search module, and a query module;

[0098] The query module is used to implement query functions, including:

[0099] The device extracts keywords and phrases from the query information of the data to be parsed, explores their synonyms or contextual meanings, or rewrites the original query information using synonym replacement and semantic understanding, or adds relevant keywords or phrases based on the original query information to determine the query intent of the data to be parsed and cover a wider range of information needs.

[0100] The device breaks down the query information of the data to be parsed into simpler and more specific sub-queries through query segmentation, adopts precise query strategies for each sub-query, and returns accurate query results.

[0101] Specifically, the search module is used to implement the search function, including:

[0102] The device uses natural language processing (NLP) technology to understand the implicit meaning in the search information of the data to be parsed in order to return accurate search results.

[0103] During the search process, the device uses deep learning models such as pre-trained language models to generate sentence embeddings for calculating sentence similarity to extract relevant information from the search information of the data to be parsed; it combines whitening to reduce the dimension of the embeddings to improve the efficiency of calculating sentence similarity. Among them, the device uses JSON files to manage the hyperparameters of the deep learning model, and manages the hyperparameters through command line tools such as the argparse module to improve the flexibility of deep learning model training.

[0104] During the search process, the device divides the search process into three stages: recall stage, fine sorting stage, and re-ranking stage. In the recall stage, the device uses a deep learning model to filter out candidate sets from massive texts, in the fine sorting stage, the candidate sets are semantically sorted, and in the re-ranking stage, the multi-dimensional features of the semantically sorted candidate sets are considered to obtain search results.

[0105] Specifically, when calling multiple parsing modules to perform corresponding parsing operations, each of the parsing modules concurrently executes the corresponding parsing operations in multiple threads, or asynchronously executes the corresponding parsing operations to improve the concurrent performance of the program, speed up the processing speed, and reduce memory consumption.

[0106] Exemplarily, the device uses asynchronous processing to process multiple parsing tasks simultaneously. For example, multiple parsing tasks are processed simultaneously using Async-JSON-Library, which is a powerful asynchronous JSON parser and generator suitable for processing JSON formatted data in various scenarios. The device can process data without blocking other parts of the program through asynchronous processing of multiple task programs, thereby improving the overall processing speed.

[0107] Exemplarily, the device uses a thread pool to create and manage asynchronous tasks. The device uses ExecutorService to manage and schedule parsing tasks in the thread pool. ExecutorService is an interface in Java for managing thread pools and concurrent task execution, which can create and manage asynchronous tasks. For example, the device creates a thread pool of a fixed size, submits an asynchronous task using the submit() method, and obtains the parsing task result through the returned Future object.

[0108] The device uses CompletableFuture to represent the results of asynchronous calculations and provides a rich API to handle asynchronous tasks. This approach can be combined with other asynchronous operations to simplify the code structure and provide an exception handling mechanism. For example, you can use CompletableFuture.supplyAsync() to asynchronously obtain task data, and then use thenApply() or thenCompose() to process the task data.

[0109] For example, for large-scale JSON data, the device uses streaming processing to process multiple parsing tasks. For example, the ijson library is a Python-based library that is specifically used for streaming JSON data. It implements an event-based JSON parser that can read task data one by one through an iterator instead of loading the entire file into memory at once, which can save a lot of memory when processing large data sets.

[0110] When calling multiple parsing modules to perform corresponding parsing operations, it also includes: resource management, performance optimization management, thread safety management, error handling and exception capture.

[0111] Exemplarily, the device uses the RAII (Resource Acquisition Is Initialization) pattern to manage resources, manages resources through local objects, realizes automatic resource recycling, and avoids problems such as memory leaks caused by forgetting to release resources.

[0112] Exemplarily, performance optimization management includes: using performance optimization techniques, such as caching or batch processing, to further improve parsing processing speed and reduce memory consumption. Preferably, the present invention develops an efficient resource management algorithm to adapt to different operating environments and resource constraints; at the same time, through continuous learning, the designed resource management model can optimize its resource consumption to adapt to new hardware platforms or performance requirements.

[0113] For example, ensuring the thread safety of the parser in a multi-threaded environment can be achieved through synchronization mechanisms or ensuring immutability. For example, use the synchronized keyword to synchronize the methods of the Gson instance, or create immutable objects to ensure thread safety.

[0114] For example, in an asynchronous parsing operation, the device needs to properly handle asynchronous callbacks and exceptions, such as catching the IncompleteJSONError exception to handle incomplete or illegal JSON data to avoid program crashes during parsing.

[0115] Step 220: During the process of the target parser performing the corresponding parsing operation on the data to be parsed, if the parsing operation generates an error, the parsing operation is paused, the position of the error is located, and the error cause information is analyzed to obtain.

[0116] Step 230: After correcting the error according to the error cause information, continue the parsing operation;

[0117] Alternatively, according to the error cause information, the parsing operation is terminated and the target parser is initialized.

[0118] Specifically, in the process of the target parser performing the corresponding parsing operation on the data to be parsed, the target parsed data is detected in real time, and the parsing operation produces abnormal situations, including: detecting whether there are abnormal values ​​in the target parsed data, such as NaN (unrepresentable value) or infinite value.

[0119] Alternatively, the device uses statistical analysis methods, such as calculating the mean and standard deviation of the output of each layer of the parser, to check whether the numerical error exceeds a preset threshold.

[0120] Alternatively, the device determines whether the parser is learning properly by comparing the parser's predicted output with the expected output to identify whether there is a significant difference.

[0121] Specifically, locating the location of the anomaly includes: using dynamic analysis technology to track the level and stage where the error occurs, such as whether it is during forward propagation or backward propagation; and analyzing the historical trend of the value of the erroneous data propagated between layers to identify the fault and locate the source of the fault.

[0122] Exemplarily, according to the error cause information, error correction includes: if the error is caused by improper data preprocessing, such as incorrect data range or the presence of NaN values, the data can be re-preprocessed and normalized.

[0123] Alternatively, if the error is due to model structure problems, such as improper selection of activation function or loss function, the model structure or hyperparameters can be adjusted.

[0124] Alternatively, use algorithms to perform root cause analysis of errors, such as by capturing numerical errors and monitoring the model training process to find the layers or parameters related to the errors and make adjustments.

[0125] Further, after the error is corrected, the parsing process continues while keeping monitoring of subsequent steps to ensure the accuracy and robustness of the parsing.

[0126] Alternatively, based on the error cause information, the parsing operation is terminated and the target parser is initialized. For example, when an error is detected, the training is terminated and the model is reinitialized, or it is rolled back to the previous correct state; when a network error is detected, an attempt is made to reload the data source, or when a media error occurs, an attempt is made to switch the playback speed.

[0127] In some embodiments of the present invention, different severity levels are set for different types of errors, and corresponding recovery strategies are set. For example, when certain errors with higher severity levels are detected, the HLS instance is reset and the data source is reloaded.

[0128] In some embodiments of the present invention, the device implements an error detection and recovery framework to handle various parsing errors. Through continuous learning, the designed error detection and recovery model can continuously improve the accuracy of error detection and learn new recovery strategies to deal with new abnormal situations.

[0129] In some embodiments of the present invention, the device provides a user interface for the user to view the corrected JSON data through the user interface after the error is detected and corrected, and can manually edit the JSON data to ensure the correctness of the data.

[0130] In some embodiments of the present invention, according to the parsing instruction of the data to be parsed, the target parser performs the corresponding parsing operation on the data to be parsed, and after obtaining the corresponding parsing result, it also includes: providing an extensible user interface for users to access and operate the parsed data, and allowing users to customize and expand the interface functions according to their own needs. This can be achieved by providing plug-ins or extension points, so that users can add new functions or modify existing functions according to their own needs.

[0131] In some embodiments of the present invention, the present invention designs a system architecture that is easy to maintain and expand, so that functions such as data parsing models, parsers, adaptive adjustment mechanisms for parameter strategies, error detection and recovery mechanisms, resource management mechanisms and interfaces are integrated into existing systems, including providing APIs and tools so that developers can easily add new functions or modify existing functions, adapt to the ever-changing technical environment and user needs, ensure long-term availability and relevance, and improve the flexibility and adaptability of the system.

[0132] The data analysis method provided by the present invention has the following beneficial effects:

[0133] The error detection and recovery mechanism implemented in the present invention can prevent security problems caused by parsing errors, improve the security of the system, and at the same time can cope with noise and anomalies in the data, maintain a high parsing success rate even when the data is damaged or incomplete, thereby improving the stability of the parsing.

[0134] The present invention optimizes the model by pruning and quantizing the parser, and implements a resource management strategy during the parsing process of the model, which can reduce the demand for computing resources while ensuring the performance of the parser, making the parsing process more energy-efficient. The present invention provides a data interface so that users can easily access and operate the parsed data, thereby improving data processing efficiency.

[0135] Figure 3 FIG. 2 is a schematic diagram showing the structure of an embodiment of a data analysis device provided by the present invention. Figure 3 As shown, the device 300 includes: a receiving module 310, a prediction module 320, a matching module 330 and a parsing module 340;

[0136] A receiving module 310, configured to receive data to be parsed and a parsing instruction for the data to be parsed;

[0137] A prediction module 320, used to predict the data structure type of the data to be parsed by using a data parsing model;

[0138] A matching module 330, configured to match and obtain a target parser according to the data structure type of the data to be parsed;

[0139] The target parser is obtained by optimizing the parameter adjustment strategy applied to the initial data parsing model using the preset training data, and training the target parameters of the initial data parsing model using the optimized parameter adjustment strategy and the preset training data; the data structure type of the preset training data matches the data structure type of the data to be parsed;

[0140] The parsing module 340 is used to perform corresponding parsing operations on the data to be parsed using the target parser according to the parsing instructions of the data to be parsed, so as to obtain corresponding parsing results.

[0141] In an optional manner, the device 300 further includes: a model training module and a model optimization module;

[0142] A model training module is used to construct preset training data according to a preset data structure type;

[0143] For the parameter adjustment strategies applied to the initial data parsing model, determine at least two combinations of parameter adjustment strategies;

[0144] Optimizing the parameter adjustment strategy applied by the initial data analysis model using the preset training data, determining an optimal parameter adjustment strategy combination, and applying the optimal parameter adjustment strategy combination to the initial data analysis model;

[0145] The target parameters of the initial data parsing model are trained using the optimal parameter adjustment strategy combination and the preset training data to obtain a target parser.

[0146] A model optimization module, used to perform a pruning operation on the trained initial data parsing model according to a preset pruning strategy, and to perform a quantization operation on the trained initial data parsing model according to a preset quantization strategy to obtain a target parser;

[0147] The preset pruning strategy is obtained by optimizing the pruning strategy applied to the trained initial data parsing model using the preset training data;

[0148] The preset quantization strategy is obtained by optimizing the quantization strategy applied by the trained initial data analysis model using preset training data.

[0149] In an optional manner, the prediction module 320 includes: a pre-processing unit and a processing unit;

[0150] A preprocessing unit, used for performing standardization processing on the data to be analyzed to obtain standardized data;

[0151] Segmenting the standardized data to obtain a plurality of target elements;

[0152] Performing semantic analysis on each of the target elements to extract a number of target features;

[0153] Convert each of the target features to obtain a numerical vector of each of the target features;

[0154] The numerical vector of each target feature is used as an input vector of the data analysis model.

[0155] In an optional manner, the data analysis model is a neural network model, including an input layer, a hidden layer and an output layer connected in sequence; the hidden layer includes a plurality of neurons and a nonlinear activation function; the data analysis model is obtained by training an initial data analysis model using preset data;

[0156] a processing unit, configured to receive an input vector of the data parsing model using the input layer;

[0157] Using each of the neurons in the hidden layer to extract pattern data and feature data of the input vector respectively;

[0158] The pattern data and feature data extracted by each neuron are weightedly summed and used as input of a nonlinear activation function to predict the structural features of the data to be analyzed, and the data structure type of the data to be analyzed is output through an output layer.

[0159] In an optional manner, the target parser integration includes: one or more parsing modules including a data format conversion module, a search module and a query module.

[0160] In an optional manner, the parsing module 340 includes: a parsing unit and an error detection unit;

[0161] A parsing unit, configured to call one or more corresponding parsing modules to perform corresponding parsing operations on the data to be parsed according to the parsing instructions of the data to be parsed, and obtain corresponding parsing results;

[0162] When multiple parsing modules are called to perform corresponding parsing operations, each of the parsing modules performs the corresponding parsing operations concurrently in multiple threads, or performs the corresponding parsing operations asynchronously.

[0163] an error detection unit, configured to, during the process in which the target parser performs a corresponding parsing operation on the data to be parsed, if the parsing operation generates an error, suspend the parsing operation, locate the position of the error, and analyze to obtain information on the cause of the error;

[0164] After correcting the error according to the error cause information, continue the parsing operation;

[0165] Alternatively, according to the error cause information, the parsing operation is terminated and the target parser is initialized.

[0166] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0167] Figure 4 The schematic diagram of the structure of an embodiment of the data analysis device provided by the present invention is shown. The specific embodiment of the present invention does not limit the specific implementation of the data analysis device.

[0168] like Figure 4 As shown, the data analysis device may include: a processor (processor) 402, a communication interface (Communications Interface) 404, a memory (memory) 406, and a communication bus 408.

[0169] The processor 402, the communication interface 404, and the memory 406 communicate with each other via the communication bus 408. The communication interface 404 is used to communicate with other devices such as a client or other server network elements. The processor 402 is used to execute the program 410, which can specifically execute the relevant steps in the above-mentioned data analysis method embodiment.

[0170] Specifically, the program 410 may include program code including computer executable instructions.

[0171] The processor 402 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the data analysis device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0172] The memory 406 is used to store the program 410. The memory 406 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0173] The program 410 may be specifically called by the processor 402 to enable the data analysis device to execute the data analysis method in any of the above method embodiments.

[0174] An embodiment of the present invention provides a computer-readable storage medium, which stores at least one executable instruction. When the executable instruction is executed on a data analysis device / apparatus, the data analysis device / apparatus executes the data analysis method in any of the above method embodiments.

[0175] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system or other device. In addition, the embodiments of the present invention are not directed to any particular programming language.

[0176] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. Similarly, in order to simplify the present invention and help understand one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, the various features of the embodiments of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. Wherein, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself is a separate embodiment of the present invention.

[0177] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and further may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive.

[0178] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be understood as limitations on the order of execution.

Claims

1. A data analysis method, characterized in that: The method comprises: Receiving data to be parsed and a parsing instruction for the data to be parsed; Using a data parsing model, predicting the data structure type of the data to be parsed; According to the data structure type of the data to be parsed, a target parser is obtained by matching; The target parser is obtained by optimizing the parameter adjustment strategy applied to the initial data parsing model using the preset training data, and training the target parameters of the initial data parsing model using the optimized parameter adjustment strategy and the preset training data; the data structure type of the preset training data matches the data structure type of the data to be parsed; According to the parsing instruction of the data to be parsed, the target parser is used to perform corresponding parsing operations on the data to be parsed to obtain corresponding parsing results.

2. The data analysis method according to claim 1, characterized in that: According to the data structure type of the data to be parsed, before matching and obtaining the target parser, the following steps are performed: Construct preset training data according to the preset data structure type; For the parameter adjustment strategies applied to the initial data parsing model, determine at least two combinations of parameter adjustment strategies; Optimizing the parameter adjustment strategy applied by the initial data analysis model using the preset training data, determining an optimal parameter adjustment strategy combination, and applying the optimal parameter adjustment strategy combination to the initial data analysis model; The target parameters of the initial data parsing model are trained using the optimal parameter adjustment strategy combination and the preset training data to obtain a target parser.

3. The data analysis method according to claim 2, characterized in that: After training the target parameters of the initial data parsing model using the optimal parameter adjustment strategy combination and the preset training data, the method further includes: Performing a pruning operation on the trained initial data parsing model according to a preset pruning strategy, and performing a quantization operation on the trained initial data parsing model according to a preset quantization strategy to obtain a target parser; The preset pruning strategy is obtained by optimizing the pruning strategy applied to the trained initial data parsing model using the preset training data; The preset quantization strategy is obtained by optimizing the quantization strategy applied by the trained initial data analysis model using preset training data.

4. The data analysis method according to claim 1, characterized in that: Before predicting the data structure type of the data to be parsed by using the data parsing model, the following steps are included: Performing standardization on the data to be analyzed to obtain standardized data; Segmenting the standardized data to obtain a plurality of target elements; Performing semantic analysis on each of the target elements to extract a number of target features; Convert each of the target features to obtain a numerical vector of each of the target features; The numerical vector of each target feature is used as an input vector of the data analysis model.

5. The data analysis method according to claim 4, characterized in that: The data analysis model is a neural network model, including an input layer, a hidden layer and an output layer connected in sequence; the hidden layer includes a number of neurons and a nonlinear activation function; the data analysis model is obtained by training the initial data analysis model using preset data; Using the data parsing model, predicting the data structure type of the data to be parsed includes: Utilizing the input layer to receive an input vector of the data parsing model; Using each of the neurons in the hidden layer to extract pattern data and feature data of the input vector respectively; The pattern data and feature data extracted by each neuron are weightedly summed and used as input of a nonlinear activation function to predict the structural features of the data to be analyzed, and the data structure type of the data to be analyzed is output through an output layer.

6. The data analysis method according to claim 1, characterized in that: The target parser integration includes: one or more parsing modules of a data format conversion module, a search module and a query module; According to the parsing instruction of the data to be parsed, the target parser is used to perform a corresponding parsing operation on the data to be parsed to obtain a corresponding parsing result, including: According to the parsing instruction of the data to be parsed, calling the corresponding one or more parsing modules to perform corresponding parsing operations on the data to be parsed, and obtaining corresponding parsing results; When multiple parsing modules are called to perform corresponding parsing operations, each of the parsing modules performs the corresponding parsing operations concurrently in multiple threads, or performs the corresponding parsing operations asynchronously.

7. The data analysis method according to claim 1, characterized in that: According to the parsing instruction of the data to be parsed, using the target parser to perform a corresponding parsing operation on the data to be parsed to obtain a corresponding parsing result, the method further includes: During the process of the target parser performing the corresponding parsing operation on the data to be parsed, if the parsing operation generates an error, the parsing operation is paused, the position of the error is located, and the error cause information is obtained through analysis; After correcting the error according to the error cause information, continue the parsing operation; Alternatively, according to the error cause information, the parsing operation is terminated and the target parser is initialized.

8. A data analysis device, characterized in that: The device comprises: a receiving module, a prediction module, a matching module and a parsing module; A receiving module, used for receiving data to be parsed and a parsing instruction for the data to be parsed; A prediction module, used to predict the data structure type of the data to be parsed by using a data parsing model; A matching module, used for matching and obtaining a target parser according to the data structure type of the data to be parsed; The target parser is obtained by optimizing the parameter adjustment strategy applied to the initial data parsing model using the preset training data, and training the target parameters of the initial data parsing model using the optimized parameter adjustment strategy and the preset training data; the data structure type of the preset training data matches the data structure type of the data to be parsed; The parsing module is used to perform corresponding parsing operations on the data to be parsed using the target parser according to the parsing instructions of the data to be parsed, so as to obtain corresponding parsing results.

9. A data analysis device, characterized in that: include: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform the operation of the data parsing method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The storage medium stores at least one executable instruction. When the executable instruction is executed on the data analysis device / apparatus, the data analysis device / apparatus performs the operation of the data analysis method according to any one of claims 1 to 7.

Images