Data processing method and device

Abstract

The invention provides a data processing method and device, and relates to the technical field of big data, and the method comprises the steps: taking target data as an input sequence, and inputting the input sequence into a large model, so as to recognize a missing position in the sequence; for a missing position, learning an internal structure between position data in the input sequence through data reconstruction, estimating the missing position according to the internal structure between the position data so as to output a missing value, and identifying an abnormal value in the data after the first preprocessing according to a reconstruction error and a preset threshold value; according to the type of the abnormal value, processing the abnormal value to obtain preprocessed data; and inputting a pre-established prediction model to obtain a data prediction result. Through the above scheme, the technical problem of low prediction data accuracy caused by low efficiency, insufficient accuracy and low flexibility in existing data preprocessing is solved, and the technical effect of effectively improving the prediction data accuracy is achieved.

Description

Technical Field

[0001] The present application belongs to the field of big data technology, and in particular, relates to a data processing method and device. Background Art

[0002] With the continuous development of information technology, enterprises have an increasing demand for data processing and decision support systems. Especially in the fields of finance, retail, manufacturing, etc., enterprise decision support systems have become an important tool to promote enterprise development. However, the rapid growth of enterprise data and the increase in complexity have brought great challenges to data preprocessing. Data preprocessing is an important part of the decision support system, which not only involves steps such as data cleaning, conversion, standardization and feature extraction, but also plays an important role in the accuracy of subsequent model training and analysis.

[0003] At present, data preprocessing methods mainly include: manual cleaning, conversion and feature extraction. Although these data preprocessing methods can meet the needs to a certain extent, they have problems such as low efficiency and insufficient accuracy. Different decision-making problems may require different data preprocessing methods and parameter settings. Existing data preprocessing methods often lack flexibility and cannot adapt to different decision-making needs, which limits their application scope in enterprise decision support systems.

[0004] There is no effective solution to the technical problems of low accuracy of prediction data caused by low efficiency, insufficient accuracy and low flexibility of existing data preprocessing methods. Summary of the invention

[0005] The purpose of this application is to provide a data processing method and device, which can achieve efficient and accurate preprocessing of data, thereby improving the accuracy of predicted data.

[0006] The present application provides a data processing method and device which are implemented as follows:

[0007] A data processing method, comprising:

[0008] Collect target data from multiple data sources;

[0009] The target data is used as a first input sequence and input into a large model to identify the missing position in the sequence;

[0010] For the missing positions, learn the intrinsic structure between the data at each position in the first input sequence through data reconstruction, estimate the missing positions according to the intrinsic structure between the data at each position, output the missing values, and fill the input missing values ​​into the first input sequence as the first preprocessed data;

[0011] Using the first preprocessed data as a second input sequence, inputting into the large model, and identifying outliers in the first preprocessed data according to a reconstruction error and a preset threshold;

[0012] Processing the outliers according to the type of the outliers to obtain second preprocessed data;

[0013] Converting the second preprocessed data into a data format that conforms to the prediction model to obtain converted data;

[0014] The converted data is input into a pre-established prediction model to obtain a data prediction result, wherein the pre-established prediction model is a neural network model processed by an evaluation process and a tuning mechanism.

[0015] In one embodiment, target data is collected from multiple data sources, including:

[0016] Establish connections with multiple data sources;

[0017] Determine a data structure type of each data source among the multiple data sources;

[0018] When the data source is an unstructured data source, target data is extracted from the unstructured data source by using data crawling technology;

[0019] When the data source is a structured data source, the target data is extracted from the structured data source through an API interface.

[0020] In one embodiment, after learning the intrinsic structure between the data at each position in the first input sequence through data reconstruction for the missing position, estimating the missing position according to the intrinsic structure between the data at each position to output the missing value, the method further includes:

[0021] The weight value is adjusted to reduce the influence of the missing value position on other positions in the first input sequence.

[0022] In one embodiment, after the first preprocessed data is input as a second input sequence into the large model, and outliers in the first preprocessed data are identified according to the reconstruction error and a preset threshold, the method further includes:

[0023] Adjust the weight value to reduce the influence of the outlier position on other positions in the second input sequence;

[0024] Accordingly, according to the type of outliers, the processing of outliers includes:

[0025] Depending on the type of outlier, perform one of the following actions on the outlier: remove, replace, and keep marked.

[0026] In one embodiment, converting the second preprocessed data into a data format that conforms to the prediction model to obtain converted data includes:

[0027] In the case where the data after the second preprocessing is text data, the data is converted into a numerical value between 0 and 1 by using an equal frequency binning method;

[0028] When the data after the second preprocessing is numerical data, the data is converted into distribution data with a mean of 0 and a standard deviation of 1.

[0029] In one embodiment, a prediction model is pre-established, including:

[0030] Get training data;

[0031] Constructing the training data into a time series;

[0032] The marker items in the time series are used as features and the target revenue is used as the target variable to train the original neural network to obtain a prediction model, wherein the marker items include at least one of the following: inventory data items, marketing expense data items, cash flow data items, accounts receivable and payable data items, and project quantity data items.

[0033] In one embodiment, the original neural network is established as follows, including:

[0034] In the Sequential model, each layer sets a unique input tensor and output tensor, and the initial model is formed by stacking multiple layers;

[0035] Add a SimpleRNN layer to the constructed initial model and set the number of units and input shape;

[0036] Add a Dense layer as the output layer with the number of output units as 1 to get the original neural network.

[0037] In one embodiment, after the original neural network is trained to obtain a prediction model, the method further includes:

[0038] Get test data;

[0039] Evaluating the prediction model using test data to determine the degree of match between the predicted revenue and the actual revenue, wherein the evaluation index includes at least one of the following: accuracy, recall, and F1 score;

[0040] If the evaluation fails, the number of units and the learning rate of the RNN layer in the prediction model are adjusted until the prediction model passes the evaluation.

[0041] A data processing device, comprising:

[0042] A collection module, used for collecting target data from multiple data sources;

[0043] A first preprocessing module is used to input the target data as a first input sequence into a large model to identify missing positions in the sequence; for the missing positions, learn the intrinsic structure between the data at each position in the first input sequence through data reconstruction, estimate the missing positions according to the intrinsic structure between the data at each position, output missing values, and fill the input missing values ​​into the first input sequence as first preprocessed data;

[0044] A second preprocessing module is used to input the first preprocessed data as a second input sequence into the large model, and identify outliers in the first preprocessed data according to a reconstruction error and a preset threshold; and process the outliers according to the type of the outliers to obtain second preprocessed data;

[0045] A conversion module, used for converting the second preprocessed data into a data form that conforms to the prediction model to obtain converted data;

[0046] The prediction module is used to input the converted data into a pre-established prediction model to obtain a data prediction result, wherein the pre-established prediction model is a neural network model processed by an evaluation process and a tuning mechanism.

[0047] An electronic device comprises a processor and a memory for storing instructions executable by the processor, wherein the steps of the above method are implemented when the processor executes the instructions.

[0048] A computer-readable storage medium stores a computer program / instruction, which implements the steps of the above method when executed by a processor.

[0049] A computer program product comprises a computer program / instruction, which implements the steps of the above method when executed by a processor.

[0050] The data processing method provided by the present application collects target data from multiple data sources, and then processes the missing values ​​and outliers in the collected target data through a large model to obtain preprocessed data, and further converts the preprocessed data into a data form that conforms to the prediction model, and inputs the pre-established prediction model to obtain data prediction results, wherein the pre-established prediction model is a neural network model processed by an evaluation processing and tuning mechanism. Because data preprocessing is performed through a large model, the technical problems of low accuracy of predicted data caused by low efficiency, insufficient accuracy and low flexibility in existing data preprocessing can be solved, and the technical effect of effectively improving the accuracy of predicted data can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0052] Figure 1 is a method flow chart of an embodiment of the data processing method provided by the present application;

[0053] Figure 2 is a method flow chart of an embodiment of missing value processing provided by the present application;

[0054] Figure 3 is a method flow chart of an embodiment of outlier processing provided by the present application;

[0055] Figure 4 It is a hardware structure block diagram of an electronic device of a data processing method provided by the present application;

[0056] Figure 5 It is a schematic diagram of the module structure of an embodiment of the data processing device provided by the present application. DETAILED DESCRIPTION

[0057] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0058] At present, the analysis and preprocessing of data generally requires data analysts to determine the rules for data cleaning, deduplication logic, format conversion, etc. to ensure the quality and consistency of the data. Although this method is more accurate, it consumes a lot of manpower when facing multiple system data sources and massive database tables. On the other hand, it is impossible to manually sort out tens of thousands of data items. Especially in the information decision-making process of large enterprises, the amount of data is huge and cannot be completed by manual labor alone.

[0059] In order to solve the technical problems of low accuracy of prediction data caused by inefficiency, lack of accuracy and low flexibility in existing data preprocessing, in this case, it is considered that big model technology can be introduced in the data preprocessing stage to automatically complete tasks such as data cleaning, conversion and standardization through its deep learning capabilities, thereby providing enterprises with high-quality data decision support.

[0060] Figure 1 It is a method flow chart of an embodiment of the data processing method provided by the present application. Although the present application provides method operation steps or device structures as shown in the following embodiments or drawings, more or fewer operation steps or module units may be included in the method or device based on routine or no creative labor. In the steps or structures where there is no necessary causal relationship logically, the execution order of these steps or the module structure of the device is not limited to the execution order or module structure described in the embodiments of the present application and shown in the drawings. When the method or module structure is applied to an actual device or terminal product, it can be connected according to the method or module structure shown in the embodiment or drawings for sequential execution or parallel execution (for example, a parallel processor or a multi-threaded processing environment, or even a distributed processing environment).

[0061] Specifically, Figure 1 As shown, the above data processing method may include the following steps 101 to 104:

[0062] Step 101: Collect target data from multiple data sources;

[0063] Specifically, because there are multiple data sources, the data structure of each data source is different. Therefore, different data structures can be processed in different ways. For example, collecting target data from multiple data sources can include: establishing connections with multiple data sources; determining the data structure type of each of the multiple data sources; if the data source is an unstructured data source, extracting the target data from the unstructured data source through data crawling technology; if the data source is a structured data source, extracting the target data from the structured data source through an API interface.

[0064] In order to collect data in real time, a real-time data stream processing platform can be deployed and configured; the multiple data sources are connected to the real-time data stream processing platform to ensure that the data can be transmitted in real time; and the data of the multiple data sources are collected in real time through the real-time data stream processing platform.

[0065] In order to improve the efficiency of data collection, data processing can be performed in parallel. Specifically, the data in the multiple data sources can be divided into multiple data partitions according to data type and data generation time; a parallel processing framework is used to collect data for multiple data partitions; after the data collection is completed, the collected data is merged to obtain the target data. That is, the integrity of the data can be guaranteed by merging the data.

[0066] Step 102: Process missing values ​​and outliers in the collected target data through the large model to obtain preprocessed data;

[0067] The missing values ​​and outliers are explained as follows:

[0068] 1) For missing values, the large model can be used as follows Figure 2 The method shown in the figure includes the following steps:

[0069] Step 201: input the target data as an input sequence into the large model to identify the missing position in the sequence;

[0070] Step 202: for the missing position, learn the intrinsic structure between the data at each position in the input sequence through data reconstruction, estimate the missing position according to the intrinsic structure between the data at each position, and output the missing value;

[0071] Step 203: Adjust the weight value to reduce the influence of the missing value position on other positions in the input sequence;

[0072] Step 204: Fill the input missing values ​​into the input sequence.

[0073] 2) For outliers, the large model can be used as follows Figure 3 The method shown in the figure includes the following steps:

[0074] Step 301: input the target data as an input sequence into the large model, and identify abnormal values ​​in the collected target data according to the reconstruction error and a preset threshold;

[0075] Step 302: Adjust the weight value to reduce the influence of the outlier position on other positions in the input sequence;

[0076] Step 303: According to the type of the outlier, perform one of the following operations on the outlier: delete, replace, and keep the mark.

[0077] For example, for enterprise data, a lot of data comes from process processing and manual reporting by enterprise employees, such as financial data, budget data, internal enterprise resolution data, inventory management data, etc. Because of the large amount of manual maintenance, there are many missing values ​​and outliers. In order not to lose data features when processing large amounts of data, in this example, a large model (especially a deep neural network model) is used to clean the original data (i.e., preprocessing). The neural network model identifies and processes outliers, missing values, and duplicate values ​​in the data through self-learning. For example, for missing values, the model can automatically fill in according to data distribution and contextual information; for outliers, the model can identify and correct them through algorithms such as clustering and anomaly detection. It can not only process massive amounts of data, but also automatically learn data features and retain the implicit information in the data.

[0078] Specifically, in the process of data cleaning using a deep neural network model, autoencoders and determination of sequence weight values ​​are used to handle missing values ​​and outliers in the data. When dealing with missing values ​​and outliers, determination of sequence weight values ​​can help the model focus more on key information. The collected data is obtained, converted into a data sequence, and then the relationship between different positions in the sequence is automatically learned. Specifically, the interaction between the data at each position in the sequence and the data at all other positions in the sequence is calculated to dynamically adjust the degree of influence of the data at each position on the data at other positions, so that the deep neural network module can capture long-distance data dependencies, that is, determine the relationship between data that are far apart in the sequence.

[0079] Step 103: converting the preprocessed data into a data format that conforms to the prediction model to obtain converted data;

[0080] Considering that in the data conversion stage, appropriate data conversion methods can be automatically selected and applied according to the input requirements and data characteristics of the big model. Among them, data conversion methods can include: normalization, standardization, discretization and other conversion methods, so as to convert the original data into a form suitable for model processing. Through the deep learning capabilities of the big model, the conversion method can be selected and applied more accurately to improve the accuracy and efficiency of data processing.

[0081] When processing the decision-making data of an enterprise, different types of business data use different data conversion methods, so that data of different types and magnitudes can be used as data input after preprocessing, as a training set for the enterprise data decision model. Enterprise data is usually for a certain business unit (for example: company, project, etc.), and the data generated around these businesses include but are not limited to: transaction amounts such as cost and income, characteristics of financial products such as interest rates and terms, ratios such as capital turnover rate and debt-to-asset ratio, as well as text data such as project level, confidentiality level, and business nature;

[0082] For text data, discretization can be used to increase comparability. For example, equal frequency binning can be used to convert it into a value between 0 and 1. For numerical data, standardization can be used to scale the feature to a distribution with a mean of 0 and a standard deviation of 1. Specifically, standardization can be performed according to the following formula:

[0083]

[0084] Among them, μ represents the mean of feature x and σ represents the standard deviation.

[0085] For investment and financing information such as interest rates and terms, the characteristics of each investment and financing can be measured by cash flow deduction and normalization. The normalization formula can be expressed as:

[0086]

[0087] Among them, x min represents the minimum value of feature x, x max Represents the maximum value of feature x.

[0088] In one embodiment, converting the preprocessed data into a data form that conforms to the prediction model to obtain the converted data can include: when the preprocessed data is text data, using equal frequency binning to convert the data into a numerical value between 0 and 1; when the preprocessed data is numerical data, converting the data into distribution data with a mean of 0 and a standard deviation of 1.

[0089] Step 104: input the converted data into a pre-established prediction model to obtain a data prediction result, wherein the pre-established prediction model is a neural network model processed by an evaluation process and a tuning mechanism.

[0090] The above-mentioned prediction model can be established in the following manner: obtain training data; construct the training data into a time series; use the marker items in the time series as features and the target revenue as the target variable, and train the original neural network to obtain the prediction model, wherein the marker items may include but are not limited to at least one of the following: inventory data items, marketing expense data items, cash flow data items, accounts receivable and payable data items, and project quantity data items.

[0091] Among them, the above-mentioned original neural network can be established as follows: in the Sequential model, each layer sets a unique input tensor and output tensor, and forms an initial model by stacking multiple layers; add a SimpleRNN layer to the constructed initial model, and set the number of units and input shape; add a Dense layer as the output layer, and the number of output units is 1 to obtain the original neural network.

[0092] In order to improve the accuracy of the prediction model, after the original neural network is trained to obtain the prediction model, test data can also be obtained; the prediction model is evaluated through the test data to determine the degree of match between the predicted revenue and the actual revenue, wherein the evaluation indicators may include but are not limited to at least one of the following: accuracy, recall rate and F1 score; if the evaluation fails, the number of units and the learning rate of the RNN layer in the prediction model are adjusted until the prediction model passes the evaluation.

[0093] In the above example, target data is collected from multiple data sources, and then the missing values ​​and outliers in the collected target data are processed through a large model to obtain preprocessed data, and further, the preprocessed data is converted into a data form that conforms to the prediction model and input into the pre-established prediction model to obtain data prediction results, wherein the pre-established prediction model is a neural network model processed by an evaluation and tuning mechanism. Because data preprocessing is performed through a large model, the technical problems of low accuracy of prediction data caused by low efficiency, insufficient accuracy and low flexibility in existing data preprocessing can be solved, achieving the technical effect of effectively improving the accuracy of prediction data.

[0094] The above method is described below in conjunction with a specific embodiment. However, it is worth noting that this specific embodiment is only for better illustrating the present application and does not constitute an improper limitation on the present application.

[0095] In view of the technical problems of low efficiency, low accuracy and poor flexibility in existing data preprocessing, a method for data preprocessing in an enterprise decision support system based on a large model is provided in this example. The method of this example can automatically and efficiently process large-scale and complex enterprise data, accurately perform data cleaning, conversion and feature extraction on these data, while maintaining the flexibility and scalability of the data preprocessing process. That is, by using the deep learning ability of the large model, valuable features are extracted from the original data, and processing steps such as data cleaning, conversion and standardization are performed to improve the efficiency and quality of data preprocessing, thereby providing better decision support for enterprises.

[0096] Data preprocessing includes data collection, cleaning, conversion, feature extraction, and model training. In these steps, there are mainly the following problems:

[0097] 1) Inefficiency: Relying on manual operations or simple automated scripts, these methods are inefficient when processing large-scale data. As the amount of enterprise data grows rapidly, this inefficient data processing method cannot meet enterprise needs.

[0098] 2) Insufficient accuracy: Due to the existence of noise, outliers and missing values ​​in the data, existing data cleaning and conversion methods are often unable to fully handle these problems. Such identification will affect the accuracy of subsequent model training and analysis, thereby affecting the final decision quality.

[0099] 3) Insufficient feature extraction: Feature extraction is a key step in data preprocessing. The existing method relies on manual feature selection, which cannot fully utilize the hidden information in the data and limits the performance of the model.

[0100] 4) Lack of flexibility: Different decision-making problems may require different data preprocessing methods and parameter settings. Existing data processing methods are inflexible and therefore cannot adapt to different decision-making needs, which limits the application scope of data preprocessing in enterprise decision support systems.

[0101] In response to the above problems, this example proposes a data preprocessing method in an enterprise decision support system based on a big model, introduces big model technology, and realizes automated data processing through the deep learning ability of big model technology, including: data collection, data preprocessing, feature extraction, model selection and training, model evaluation and tuning, etc. This automated data processing process can improve the efficiency and accuracy of data processing.

[0102] Specifically, in this example, a data processing method is provided, which may include the following steps:

[0103] Step 1: Data collection.

[0104] Data collection is the first step in data preprocessing. In this example, relevant data is collected from channels such as the enterprise's internal system and external data sources by defining data interfaces and setting collection rules.

[0105] When implementing, comprehensive collection and integration of enterprise treasury data can be achieved from multiple data sources. For example, a combination of API interface and data crawling technology can be used to achieve comprehensive collection of enterprise treasury data. Among them, API interface is used to obtain structured data, such as bank account information, transaction records, etc.; data crawling technology is used to extract useful information from unstructured data sources, such as news reports, market analysis reports, etc. The combination of these two technologies can ensure the comprehensiveness and real-time nature of data acquisition.

[0106] In order to ensure the accuracy and consistency of data, ETL (Extract-Transform-Load) tools can be used to extract, transform and load data during data collection. ETL is used to describe the process of extracting, transforming and loading data from the source to the destination.

[0107] In order to improve the real-time performance of data collection, the data collection mode can be changed from batch processing mode to real-time stream processing mode, that is, the data is collected and processed immediately after it is generated, thereby improving the real-time performance of data collection. Specifically, it can be achieved through the following steps:

[0108] S1: Deploy and configure a real-time data stream processing platform;

[0109] S2: Connect multiple data sources to be collected to the real-time data stream processing platform to ensure that data can be transmitted in real time;

[0110] S3: Through this real-time data stream processing platform, data from multiple data sources is collected instantly.

[0111] Furthermore, in order to improve the efficiency of data collection, a large amount of data can be divided into multiple small blocks for processing through data partitioning and parallel processing, thereby shortening the time of data collection. For example, each data source can be partitioned according to the data type or data generation time according to the preset rules, and then the parallel processing framework is used for data collection. After the collection is completed, the data is merged to ensure the integrity of the data.

[0112] Step 2: Data preprocessing.

[0113] Data preprocessing can include: data cleaning, data conversion and data standardization. In this example, a large model is introduced to process each step in data preprocessing to improve the efficiency and accuracy of data processing. The specific instructions are as follows:

[0114] 1) Data cleaning:

[0115] For enterprise data, a lot of data comes from process processing and manual reporting by enterprise employees, such as financial data, budget data, internal enterprise resolution data, inventory management data, etc. Because of the large amount of manual maintenance, there are many missing values ​​and outliers. In order to avoid losing data features when processing large amounts of data, a large model (especially a deep neural network model) is used to clean the original data in this example. The neural network model identifies and processes outliers, missing values, and duplicate values ​​in the data through self-learning. For example, for missing values, the model can automatically fill in according to data distribution and contextual information; for outliers, the model can identify and correct them through algorithms such as clustering and anomaly detection. It can not only process massive amounts of data, but also automatically learn data features and retain the implicit information in the data.

[0116] Specifically, in the process of using a deep neural network model for data cleaning, autoencoders and determination of sequence weight values ​​are used to handle missing values ​​and outliers in the data. When dealing with missing values ​​and outliers, determination of sequence weight values ​​can help the model focus more on key information.

[0117] Acquire the collected data, convert the collected data into a data sequence, and then automatically learn the relationship between different positions in the sequence. Specifically, calculate the interaction between the data at each position in the sequence and the data at all other positions in the sequence to dynamically adjust the degree of influence of the data at each position on the data at other positions, so that the deep neural network module can capture long-distance data dependencies, that is, determine the relationship between data that are far apart in the sequence.

[0118] When determining the degree of influence of the relationship between data at different positions in a sequence, the degree of influence can be indicated by a weight value, where the weight value can be calculated according to the following formula:

[0119]

[0120] Where W represents the weight value, Q (Query), K (Key), and V (Value) represent the matrices after the linear transformation of the target data sequence, corresponding to the query, key, and value of the input target data sequence respectively; k represents the dimension of the key vector K, Represents the scaling factor, which is used to prevent kWhen is very large, the dot product result is too large, causing the gradient vanishing problem; the softmax function is used to normalize each element of the dot product result to between 0 and 1, and ensure that the sum of all elements in the same row is 1, thereby obtaining the weight value.

[0121] When implementing, data processing can also be performed by combining weight values ​​and autoencoders. Specifically, in the encoder, in addition to the encoding process, weight value determination can also be added to enhance the processing capability of input data. Specifically, a single weight value or multiple weight values ​​can be used to capture key information in the input sequence.

[0122] Among them, a single weight value is to weight each element by calculating the score (i.e., weight value) between each element and all other elements in the sequence, that is, allowing the model to consider the contextual information of the entire sequence during the encoding process. Multiple weight values ​​are to divide the weight values ​​into multiple ones, each weight value represents the relationship within the data sequence in a different subspace. The multiple weight values ​​can enhance the representation ability of the model and capture information from multiple angles.

[0123] Assume that the weight value part of the encoder output is H attn , then the calculation of the encoder part can be expressed as:

[0124] H attn =A(QW Q ,KW K ,VW V )

[0125] Among them, W Q , W K and W are weight matrices of linear transformation, which are used to generate Q, K, and V respectively; then, H attn After being processed by other layers (e.g., fully connected layer, activation layer, etc.), the final encoding representation h is obtained.

[0126] In the decoder, weight values ​​are introduced to assist in generating the output sequence. Especially when dealing with sequence-to-sequence tasks (e.g., machine translation), weight values ​​can be used to enable the decoder to dynamically focus on different parts of the input sequence when generating each output word.

[0127] During decoding, the encoder output can be used as the key and value, and the decoder's current hidden state can be used as the query to calculate the attention weights and generate a context vector. The context vector contains: the information in the encoder output that is most relevant to the current decoding step. The decoder can also use the relevant information in the encoder output when processing the output sequence. The calculation method is similar to the weight value calculation, but this time Q comes from the decoder, and K and V come from the encoder output.

[0128] During the training process of the above-mentioned deep neural network, the objective function may be used to optimize the parameters of the encoder and decoder by minimizing the reconstruction error (e.g., mean square error MSE) during the training process. At the same time, the distribution of the above-mentioned weight values ​​may be added to the objective function as a regularization term to avoid overfitting, thereby improving the generalization ability of the model. When optimizing the above-mentioned neural network model, an optimization algorithm such as gradient descent may be used to update the parameters of the model. During the training process, the gradient may be calculated by the back propagation algorithm, and the weights and bias terms of the encoder and decoder may be updated according to the gradient.

[0129] Missing values ​​and outliers can be handled as follows:

[0130] Handling missing values: During the encoding process, the encoder can automatically focus on the key information in the input sequence of the neural network model and make reasonable estimates of the missing values ​​based on this key information. Through the reconstruction process, the encoder can learn the intrinsic structure of the data and generate outputs that include estimates of missing values.

[0131] For missing values, when the model learns that the input information of a certain position is missing, the weight value can be adjusted to reduce the attention of that position to other positions, or the missing value of that position can be estimated through the information of other positions.

[0132] Handling outliers: During the decoding process, outliers that do not match the normal data pattern can be automatically identified by the neural network model. By comparing the reconstruction error with a preset threshold, outliers can be detected and processed accordingly, where the processing of outliers can include: deleting, replacing, or retaining and marking.

[0133] For outliers, if the outliers appear out of place in the input sequence, the weight value can be adjusted to reduce attention to the outliers, or the impact of the outliers can be minimized through the reconstruction process.

[0134] 2) Data conversion:

[0135] In the data conversion stage, appropriate data conversion methods can be automatically selected and applied according to the input requirements and data characteristics of the big model. Among them, data conversion methods may include: normalization, standardization, discretization and other conversion methods to convert the original data into a form suitable for model processing. Through the deep learning capabilities of the big model, conversion methods can be selected and applied more accurately to improve the accuracy and efficiency of data processing.

[0136] When processing the decision-making data of an enterprise, different types of business data use different data conversion methods, so that data of different types and magnitudes can be used as data input after preprocessing, as a training set for the enterprise data decision model. Enterprise data is usually for a certain business unit (for example: company, project, etc.), and the data generated around these businesses include but are not limited to: transaction amounts such as cost and income, characteristics of financial products such as interest rates and terms, ratios such as capital turnover rate and debt-to-asset ratio, as well as text data such as project level, confidentiality level, and business nature;

[0137] For text data, discretization can be used to increase comparability. For example, equal frequency binning can be used to convert it into a value between 0 and 1. For numerical data, standardization can be used to scale the feature to a distribution with a mean of 0 and a standard deviation of 1. Specifically, standardization can be performed according to the following formula:

[0138]

[0139] Among them, μ represents the mean of feature x, and σ represents the standard deviation.

[0140] For investment and financing information such as interest rates and terms, the characteristics of each investment and financing can be measured by cash flow deduction and normalization. The normalization formula can be expressed as:

[0141]

[0142] Among them, x min represents the minimum value of feature x, x max Represents the maximum value of feature x.

[0143] That is, by introducing a big model to automate data preprocessing, the efficiency and accuracy of data processing are improved. Compared with the existing manual or rule-based data preprocessing methods, the big model in this example can automatically learn and identify data patterns, and can perform tasks such as data cleaning, conversion, and standardization more accurately. This not only reduces the need for manual intervention, but also reduces the possibility of errors, and can process larger data sets to meet the needs of large data set processing.

[0144] Furthermore, by utilizing the deep learning capabilities of the large model in feature extraction, valuable features can be automatically extracted from the preprocessed data. This feature extraction method based on the large model can be more efficient and accurate than the existing manual feature extraction because it can capture complex patterns and associations in the data. Through the automatically extracted features, the model can better understand the inherent laws of the data, thereby improving the accuracy and generalization of predictions.

[0145] Step 3: Model training and tuning.

[0146] Taking the model training and tuning for revenue forecasting as an example, after feature extraction is completed, it is necessary to select a suitable model for training. For example, a recurrent neural network (RNN) can be used for model training.

[0147] Specifically, the preprocessed data can be first constructed into a time series, and then the revenue forecasting model can be trained by using data such as inventory, advertising and marketing expenses, investment and financing cash flow, accounts receivable and payable, and number of projects as features (x) and next month's revenue as the target variable (y).

[0148] During the training process, in the Sequential model, each layer has a unique input tensor and output tensor, and a neural network can be constructed by stacking different layers. Add the SimpleRNN layer to the constructed neural network and set the appropriate number of units and input shape. Then, add the Dense layer as the output layer with 1 output unit. For the compiled model, select the mean square error MSE as the loss function and Adam as the optimizer. Divide the preprocessed data into a training set and a test set, and use the training set data to train the RNN model. After the training is completed, use the test set to evaluate the model. The evaluation indicators may include: accuracy, recall, F1 score, etc., to determine the degree of match between the predicted revenue and the actual revenue.

[0149] Then, use the trained RNN model to predict new revenue data and analyze the prediction results to evaluate the accuracy and reliability of the model. If the model evaluation results do not meet expectations, the number of units and learning rate of the RNN layer can be adjusted as needed to improve the prediction performance.

[0150] That is, through flexible model selection and training mechanisms and automated model evaluation and tuning processes, models can be quickly iterated and optimized to adapt to different application scenarios and needs. This mechanism not only improves model development efficiency, but also ensures that the performance and accuracy of the model are always in the best state.

[0151] In the above example, in the feature extraction stage, the deep learning ability of the large model is used to automatically extract valuable features from the preprocessed data, and feature selection and dimensionality reduction are performed. This feature extraction method based on the large model can more accurately capture the key information in the data and improve the prediction performance of the model. Through the flexible model selection and training mechanism, the appropriate model and algorithm can be selected for training according to the characteristics and needs of the problem. This mechanism can ensure the performance and accuracy of the model and adapt to the needs of different problems. After the model training is completed, the performance of the model is evaluated through the automated model evaluation and tuning mechanism, and the model is tuned according to the evaluation results. This automated mechanism can accelerate the iteration and optimization process of the model and improve the final performance of the model. That is, in this example, data collection, data preprocessing, feature extraction, model selection and training, model evaluation and tuning, etc. are integrated into a complete technical solution, and by optimizing the processes and algorithms of each link, efficient and accurate enterprise decision support is achieved.

[0152] In the above example, by introducing large model technology into the enterprise information system, the data preprocessing and feature extraction process is automated, which significantly improves the efficiency, accuracy and availability of data preprocessing. At the same time, through flexible model selection and training, model evaluation and tuning mechanisms, the prediction performance and accuracy of the model are ensured, thus providing more accurate data support for enterprise decision-making.

[0153] The method embodiments provided in the above embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on an electronic device as an example, Figure 4 This is a hardware structure block diagram of an electronic device of a data processing method provided by this application. Figure 4 As shown, the electronic device 10 may include one or more (only one is shown in the figure) processors 02 (the processor 02 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 04 for storing data, and a transmission module 06 for communication functions. It can be understood by those skilled in the art that Figure 4 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 4 More or fewer components as shown, or with Figure 4 Different configurations are shown.

[0154] The memory 04 can be used to store software programs and modules of application software, such as program instructions / modules corresponding to the data processing method in the embodiment of the present application. The processor 02 executes various functional applications and data processing by running the software programs and modules stored in the memory 04, that is, the data processing method of the above-mentioned application is realized. The memory 04 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 04 may further include a memory remotely arranged relative to the processor 02, and these remote memories may be connected to the electronic device 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0155] The transmission module 06 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the electronic device 10. In one example, the transmission module 06 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 06 can be a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet wirelessly.

[0156] At the software level, the data processing device can be Figure 5 As shown, including:

[0157] The acquisition module 501 is used to acquire target data from multiple data sources;

[0158] A first preprocessing module 502 is used to input the target data as a first input sequence into a large model to identify missing positions in the sequence; for the missing positions, learn the intrinsic structure between the data at each position in the first input sequence through data reconstruction, estimate the missing positions according to the intrinsic structure between the data at each position, output missing values, and fill the input missing values ​​into the first input sequence as first preprocessed data;

[0159] A second preprocessing module 503 is used to input the first preprocessed data as a second input sequence into the large model, and identify outliers in the first preprocessed data according to the reconstruction error and a preset threshold; and process the outliers according to the type of the outliers to obtain second preprocessed data;

[0160] A conversion module 504, configured to convert the second preprocessed data into a data format that conforms to a prediction model to obtain converted data;

[0161] The prediction module 505 is used to input the converted data into a pre-established prediction model to obtain a data prediction result, wherein the pre-established prediction model is a neural network model processed by an evaluation process and a tuning mechanism.

[0162] In one embodiment, the acquisition module 501 can specifically establish connections with multiple data sources; determine the data structure type of each of the multiple data sources; when the data source is an unstructured data source, extract target data from the unstructured data source through data crawling technology; when the data source is a structured data source, extract target data from the structured data source through an API interface.

[0163] In one embodiment, the acquisition module 501 can specifically deploy and configure a real-time data stream processing platform; connect the multiple data sources to the real-time data stream processing platform to ensure that data can be transmitted in real time; and collect data from the multiple data sources in real time through the real-time data stream processing platform.

[0164] In one embodiment, the acquisition module 501 can specifically divide the data in the multiple data sources into multiple data partitions according to data type and data generation time; use a parallel processing framework to collect data for the multiple data partitions; after the data collection is completed, merge the collected data to obtain the target data.

[0165] In one embodiment, the second preprocessing module 503 can specifically input the target data as an input sequence into the large model, and identify outliers in the collected target data based on the reconstruction error and a preset threshold; adjust the weight value to reduce the influence of the outlier position on other positions in the input sequence; and perform one of the following operations on the outlier according to the type of the outlier: delete, replace, and retain the mark.

[0166] In one embodiment, the conversion module 504 can specifically convert the data into a numerical value between 0 and 1 by using an equal-frequency binning method when the preprocessed data is text data; when the preprocessed data is numerical data, the data can be converted into distribution data with a mean of 0 and a standard deviation of 1.

[0167] In one embodiment, pre-establishing a prediction model may include: obtaining training data; forming the training data into a time series; using marker items in the time series as features and target revenue as target variables, and training the original neural network to obtain a prediction model, wherein the marker items may include but are not limited to at least one of the following: inventory data items, marketing expense data items, cash flow data items, accounts receivable and payable data items, and project quantity data items.

[0168] In one embodiment, the original neural network can be established in the following manner, including: in the Sequential model, each layer sets a unique input tensor and output tensor, and forms an initial model by stacking multiple layers; adding a SimpleRNN layer to the constructed initial model, and setting the number of units and input shape; adding a Dense layer as the output layer, and the number of output units is 1, to obtain the original neural network.

[0169] In one embodiment, after the original neural network is trained to obtain a prediction model, test data can be obtained; the prediction model is evaluated using the test data to determine the degree of match between the predicted revenue and the actual revenue, wherein the evaluation indicators include at least one of the following: accuracy, recall rate, and F1 score; if the evaluation fails, the number of units and the learning rate of the RNN layer in the prediction model are adjusted until the prediction model passes the evaluation.

[0170] The embodiments of the present application also provide a specific implementation of an electronic device capable of implementing all the steps in the data processing method in the above embodiments, wherein the electronic device specifically includes the following contents: a processor, a memory, a communications interface, and a bus; wherein the processor, the memory, and the communications interface communicate with each other through the bus; the processor is used to call a computer program in the memory, and when the processor executes the computer program, all the steps in the data processing method in the above embodiments are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0171] Step 1: Collect target data from multiple data sources;

[0172] Step 2: Input the target data as an input sequence into the large model to identify the missing position in the sequence;

[0173] Step 3: For the missing positions, learn the intrinsic structure between the data at each position in the input sequence through data reconstruction, estimate the missing positions according to the intrinsic structure between the data at each position, output the missing values, and fill the input missing values ​​into the input sequence as the first preprocessed data;

[0174] Step 4: Processing the outliers in the collected target data through the large model to obtain the second preprocessed data;

[0175] Step 5: converting the second preprocessed data into a data format that conforms to the prediction model to obtain converted data;

[0176] Step 6: Input the converted data into a pre-established prediction model to obtain a data prediction result, wherein the pre-established prediction model is a neural network model processed by an evaluation process and a tuning mechanism.

[0177] The embodiments of the present application also provide a computer-readable storage medium capable of implementing all the steps of the data processing method in the above embodiments. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, all the steps of the data processing method in the above embodiments are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0178] Step 1: Collect target data from multiple data sources;

[0179] Step 2: Input the target data as an input sequence into the large model to identify the missing position in the sequence;

[0180] Step 3: For the missing positions, learn the intrinsic structure between the data at each position in the input sequence through data reconstruction, estimate the missing positions according to the intrinsic structure between the data at each position, output the missing values, and fill the input missing values ​​into the input sequence as the first preprocessed data;

[0181] Step 4: Processing the outliers in the collected target data through the large model to obtain the second preprocessed data;

[0182] Step 5: converting the second preprocessed data into a data form that conforms to the prediction model to obtain converted data;

[0183] Step 6: Input the converted data into a pre-established prediction model to obtain a data prediction result, wherein the pre-established prediction model is a neural network model processed by an evaluation process and a tuning mechanism.

[0184] From the above description, it can be seen that the embodiment of the present application collects target data from multiple data sources, and then processes the missing values ​​and abnormal values ​​in the collected target data through a large model to obtain pre-processed data, and further converts the pre-processed data into a data form that conforms to the prediction model, inputs the pre-established prediction model, and obtains data prediction results, wherein the pre-established prediction model is a neural network model processed by an evaluation process and a tuning mechanism. Because data preprocessing is performed through a large model, the technical problems of low accuracy of predicted data caused by low efficiency, insufficient accuracy and low flexibility in existing data preprocessing can be solved, thereby achieving the technical effect of effectively improving the accuracy of predicted data.

[0185] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0186] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the hardware + program embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0187] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0188] Although the present application provides method operation steps as described in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-creative labor. The order of steps listed in the embodiments is only one way of executing the order of many steps and does not represent the only execution order. When the actual device or client product is executed, it can be executed in the order of the method shown in the embodiments or the drawings or in parallel (for example, in a parallel processor or multi-threaded processing environment).

[0189] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0190] Although the present specification embodiment provides the method operation steps as described in the embodiment or flow chart, more or less operation steps may be included based on conventional or non-creative means. The order of steps listed in the embodiment is only one way in the order of execution of many steps, and does not represent a unique execution order. When the device or terminal product in practice is executed, it can be executed in sequence or in parallel (such as a parallel processor or a multi-threaded processing environment, or even a distributed data processing environment) according to the method shown in the embodiment or the accompanying drawings. The term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, product or equipment including a series of elements not only includes those elements, but also includes other elements not clearly listed, or also includes elements inherent to such process, method, product or equipment. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or equipment including the elements.

[0191] For the convenience of description, the above devices are described in various modules according to their functions. Of course, when implementing the embodiments of this specification, the functions of each module can be implemented in the same or more software and / or hardware, or the module implementing the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0192] Those skilled in the art also know that, in addition to implementing the controller in a purely computer-readable program code, the controller can be made to implement the same function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered as a hardware component, and the devices for implementing various functions included therein can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules for implementing the method and structures within the hardware component.

[0193] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0194] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0195] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0196] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0197] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0198] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0199] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, the embodiments of this specification may take the form of complete hardware embodiments, complete software embodiments or embodiments combining software and hardware. Moreover, the embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0200] The present specification embodiments may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present specification embodiments may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0201] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. In the description of this specification, the description of the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of this specification. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, in the absence of contradiction, a person skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0202] The above is only an example of the embodiment of the present specification and is not intended to limit the embodiment of the present specification. For those skilled in the art, the embodiment of the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the embodiment of the present specification shall be included in the scope of the claims of the embodiment of the present specification.

Claims

1. A data processing method, characterized in that: include: Collect target data from multiple data sources; The target data is used as a first input sequence and input into a large model to identify the missing position in the sequence; For the missing positions, learn the intrinsic structure between the data at each position in the first input sequence through data reconstruction, estimate the missing positions according to the intrinsic structure between the data at each position, output the missing values, and fill the input missing values ​​into the first input sequence as the first preprocessed data; Using the first preprocessed data as a second input sequence, inputting into the large model, and identifying outliers in the first preprocessed data according to a reconstruction error and a preset threshold; Processing the outliers according to the type of the outliers to obtain second preprocessed data; Converting the second preprocessed data into a data format that conforms to the prediction model to obtain converted data; The converted data is input into a pre-established prediction model to obtain a data prediction result, wherein the pre-established prediction model is a neural network model processed by an evaluation process and a tuning mechanism.

2. The method according to claim 1, characterized in that Collect target data from multiple data sources, including: Establish connections with multiple data sources; Determine a data structure type of each data source among the multiple data sources; When the data source is an unstructured data source, target data is extracted from the unstructured data source by using data crawling technology; When the data source is a structured data source, the target data is extracted from the structured data source through an API interface.

3. The method according to claim 1, characterized in that After learning the intrinsic structure between the data at each position in the first input sequence through data reconstruction for the missing position, estimating the missing position according to the intrinsic structure between the data at each position to output the missing value, the method further includes: The weight value is adjusted to reduce the influence of the missing value position on other positions in the first input sequence.

4. The method according to claim 1, characterized in that: After the first preprocessed data is used as a second input sequence and input into the large model, and outliers in the first preprocessed data are identified according to a reconstruction error and a preset threshold, the method further includes: Adjust the weight value to reduce the influence of the outlier position on other positions in the second input sequence; Accordingly, according to the type of outliers, the processing of outliers includes: Depending on the type of outlier, perform one of the following actions on the outlier: remove, replace, and keep marked.

5. The method according to claim 1, characterized in that Converting the second preprocessed data into a data format that conforms to the prediction model to obtain converted data includes: In the case where the data after the second preprocessing is text data, the data is converted into a numerical value between 0 and 1 by using an equal frequency binning method; When the data after the second preprocessing is numerical data, the data is converted into distribution data with a mean of 0 and a standard deviation of 1.

6. The method according to claim 1, characterized in that Pre-built predictive models, including: Get training data; Constructing the training data into a time series; The marker items in the time series are used as features and the target revenue is used as the target variable to train the original neural network to obtain a prediction model, wherein the marker items include at least one of the following: inventory data items, marketing expense data items, cash flow data items, accounts receivable and payable data items, and project quantity data items.

7. The method according to claim 6, characterized in that The original neural network is established as follows, including: In the Sequential model, each layer sets a unique input tensor and output tensor, and the initial model is formed by stacking multiple layers; Add a SimpleRNN layer to the constructed initial model and set the number of units and input shape; Add a Dense layer as the output layer with the number of output units as 1 to get the original neural network.

8. The method according to claim 1, characterized in that After training the original neural network to obtain the prediction model, it also includes: Get test data; Evaluating the prediction model using test data to determine the degree of match between the predicted revenue and the actual revenue, wherein the evaluation index includes at least one of the following: accuracy, recall, and F1 score; If the evaluation fails, the number of units and the learning rate of the RNN layer in the prediction model are adjusted until the prediction model passes the evaluation.

9. A data processing device, characterized in that: include: A collection module, used for collecting target data from multiple data sources; A first preprocessing module is used to input the target data as a first input sequence into a large model to identify missing positions in the sequence; for the missing positions, learn the intrinsic structure between the data at each position in the first input sequence through data reconstruction, estimate the missing positions according to the intrinsic structure between the data at each position, output missing values, and fill the input missing values ​​into the first input sequence as first preprocessed data; A second preprocessing module, used to input the first preprocessed data as a second input sequence into the large model, and identify abnormal values ​​in the first preprocessed data according to a reconstruction error and a preset threshold; and processing the outliers according to the types of the outliers to obtain the second preprocessed data; A conversion module, used for converting the second preprocessed data into a data form that conforms to the prediction model to obtain converted data; The prediction module is used to input the converted data into a pre-established prediction model to obtain a data prediction result, wherein the pre-established prediction model is a neural network model processed by an evaluation process and a tuning mechanism.

10. An electronic device comprising a processor and a memory for storing instructions executable by the processor, characterized in that: When the processor executes the instructions, the steps of the method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

12. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images