Data storage method, electronic device, non-volatile storage medium and program product

Abstract

The invention discloses a data storage method, electronic equipment, a nonvolatile storage medium and a program product, and is applied to the technical field of storage. The method comprises the following steps: constructing a data cleaning model by taking a target neural network model with a classification function as a model structure in advance and taking a similarity degree between data of different types of data sources as a constraint condition; obtaining each piece of to-be-stored original data from the multi-class data source; and inputting each piece of to-be-stored original data into the data cleaning model, performing classification processing on each piece of to-be-stored original data by the data cleaning model under constraint conditions, and selecting target data from each piece of to-be-stored original data. And obtaining to-be-stored data in which the target data in the to-be-stored original data is removed, and storing the to-be-stored data. According to the invention, the problem of low data storage efficiency in related technologies can be solved, and efficient storage of multi-source heterogeneous data is realized.

Description

Technical Field

[0001] The present invention relates to the field of storage technology, and in particular to a data storage method, electronic equipment, non-volatile storage medium and computer program product. Background Art

[0002] With the development of big data, cloud technology, and artificial intelligence technology, the sources of data to be stored have become diversified, which has led to the formation of multi-source heterogeneous data.

[0003] When storing multi-source heterogeneous data, related technologies convert the multi-source heterogeneous data into distributed data for uploading. However, since the data attributes of multi-source heterogeneous data are multimodal, there will be a complex nonlinear separable space composed of noise, which leads to a complex conversion process and low data storage efficiency.

[0004] In view of this, realizing efficient storage of multi-source heterogeneous data is a technical problem that needs to be solved by those skilled in the art. It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the invention

[0005] The present invention provides a data storage method, an electronic device, a non-volatile storage medium and a computer program product, which can efficiently and securely store multi-source heterogeneous data.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: In one aspect, the present invention provides a data storage method, comprising: A data cleaning model is constructed in advance based on a target neural network model with a classification function as a model structure and according to the similarity between data of different types of data sources as a constraint condition; each original data to be stored is obtained from multiple types of data sources; each original data to be stored is input into the data cleaning model, and the data cleaning model classifies each original data to be stored under the constraint condition, and selects target data from each original data to be stored; the data to be stored is obtained by removing the target data from each original data to be stored, and the data to be stored is stored.

[0007] In a first exemplary implementation, the step of obtaining the data to be stored without the target data in each original data to be stored includes: Extract the data features of the original data to be stored without the target data; group the data features according to the type of data source to which they belong, and fuse the grouped data features to obtain the data to be stored.

[0008] In a second exemplary implementation, the data features are grouped according to the data source type to which they belong, and the grouped data features are fused, including: A data fusion model is trained in advance based on a support vector machine; each data feature is input into the data fusion model; the data fusion model maps the nonlinear vector space of each data feature into a linear separable space to obtain an optimal classification hyperplane; each data feature is grouped based on the optimal classification hyperplane, and data features in the same group belong to the same data source; and each data feature is fused based on a decision function determined by an optimal solution of each group of data features.

[0009] In a third exemplary embodiment, the step of extracting data features of the original data to be stored from which the target data is removed includes: Obtain a storage mean of each original data to be stored with the target data removed; based on the storage mean, perform standardization processing on each original data to be stored with the target data removed to obtain each standard data to be stored; respectively extract the feature vector of each standard data to be stored under each attribute as a data feature.

[0010] In a fourth exemplary implementation, the obtaining of each type of original data to be stored from multiple types of data sources includes: Obtain the data source type and data source location corresponding to each raw data to be stored; determine the data capture method according to the data source type, and obtain the corresponding raw data to be stored from the location of the corresponding data source according to the data capture method.

[0011] In a fifth exemplary implementation, storing the data to be stored includes: Obtain a preset configuration file and its corresponding file parsing toolkit, and use the file parsing toolkit to parse the preset configuration file to obtain configuration information; parse the received remote call service request based on the distributed service protocol data in the configuration information, and determine the fusion storage parameter information that matches the remote call service request; perform distributed fusion storage on the data to be stored according to the fusion storage parameter information.

[0012] In a sixth exemplary implementation, the configuration information includes a distributed service protocol, a routing rule, a call mode, and a data fusion strategy, the fusion storage parameter information includes a target routing rule, a target call mode, and a target fusion strategy, and distributed fusion storage is performed on the data to be stored according to the fusion storage parameter information, including: According to the target routing rule and the target calling mode, the target data to be stored corresponding to the remote calling service request is determined; according to the target fusion strategy, the target data to be stored is fused and processed, and the fused data is sent to the original requester to complete the distributed fusion storage task.

[0013] In a seventh exemplary embodiment, the data cleaning model includes an input layer, a plurality of hidden layers and an output layer; each hidden layer adopts an unsupervised Boltzmann machine to select target data during back propagation.

[0014] In an eighth exemplary implementation, the selecting target data from each of the original data to be stored based on the constraint condition includes: Each original data to be stored is used as the node data of the data cleaning model, and the data with abnormal node energy and discrete distribution are identified based on the constraint conditions to serve as the target data.

[0015] In a ninth exemplary embodiment, the training process of the data cleaning model includes: According to the number of nodes in the input layer and the number of nodes in the hidden layer, the node weight information between the layers is constructed; based on the node weight information and the bias coefficient of each node, the node energy function is determined, and the data distribution probability between the nodes is determined according to the node energy function; the node constraint conditions of each node of the hidden layer are constructed according to the data distribution probability and the node weight information, and the nodes of the hidden layer are activated based on the activation function determined by the node constraint conditions; the stored sample data of the multi-source heterogeneous data training sample set are input, and the node weight information and the bias coefficient of each node are continuously adjusted through the back propagation algorithm until the preset model training stop condition is reached.

[0016] In a tenth exemplary implementation, the activation function determined based on the node constraint condition activates each node of the hidden layer, including: Based on the data state coefficients of the nodes of the hidden layer, the nodes of the input layer and the node weight information, a binary activation function is determined; the nodes of the hidden layer are activated using the binary activation function, and the states of the nodes of the hidden layer are within a preset node state value range.

[0017] In an eleventh exemplary implementation, based on the node weight information and the bias coefficient of each node, a node energy function is determined, and according to the node energy function, a data distribution probability between each node is determined, including: According to the first input node of the input layer and its bias coefficient, the first hidden node of the hidden layer and its bias coefficient, and the node weight between the first input node and the first hidden node, the node energy information between the first input node and the first hidden node is determined; according to the node energy information between each node of the input layer and each node of the hidden layer, a node energy function is generated; according to the distribution factor and the node energy function, a normalization coefficient of the node energy function is determined, and according to the normalization coefficient and the node energy function, a data distribution probability between each node of the input layer and each node of the hidden layer is determined.

[0018] The present invention also provides an electronic device, comprising a processor, wherein the processor is used to implement the steps of the data storage method as described in any of the preceding items when executing a computer program stored in a memory.

[0019] The present invention also provides a non-volatile storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the data storage method as described in any of the preceding items are implemented.

[0020] Finally, the present invention also provides a computer program product, comprising a computer program / instruction, which implements the steps of the data storage method as described in any of the preceding items when executed by a processor.

[0021] The advantage of the technical solution provided by the present invention is that, for the original data to be stored obtained from multiple heterogeneous data sources, the correlation of the internal structure of the multi-source heterogeneous original data to be stored is first realized through the similarity constraint relationship of the multi-source heterogeneous data pre-set by the data cleaning model, which can not only improve the speed of the data cleaning operation inside the data cleaning model, but also improve the working efficiency of the data cleaning model. Under the constraint of the above correlation, the data cleaning model integrates and classifies the multi-source heterogeneous data to be stored, obtains more accurate multi-source heterogeneous data relationships, removes useless data relationships, thereby completing data cleaning, effectively improving the data quality of the data to be stored, and finally realizing efficient and secure storage of multi-source heterogeneous data.

[0022] In addition, the present invention also provides corresponding electronic devices, non-volatile storage media and computer program products for implementing the data storage method, which further makes the method more practical. The electronic devices, non-volatile storage media and computer program products have corresponding advantages.

[0023] The technical features mentioned above, the technical features to be mentioned below, and the technical features shown separately in the drawings can be arbitrarily combined with each other, as long as the combined technical features are not contradictory. All feasible feature combinations are technical contents clearly recorded in this article. Any of the multiple sub-features contained in the same sentence can be applied independently, and does not have to be applied together with other sub-features. It should be understood that the above general description and the detailed description below are only exemplary and cannot limit the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 A schematic diagram of a data storage method provided by the present invention; Figure 2 A schematic diagram of an exemplary model structure of the data cleaning model provided by the present invention; Figure 3 A schematic diagram of a framework of an exemplary application scenario provided by the present invention; Figure 4 A structural diagram of a specific implementation of the data storage device provided by the present invention; Figure 5 A structural diagram of a specific implementation of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Among them, the terms "first", "second", "third", "fourth", etc. in the specification and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations of the two are intended to cover non-exclusive inclusions. The term "exemplary" means "used as an example, embodiment or illustrative". Any embodiment described here as "exemplary" is not necessarily interpreted as being superior or better than other embodiments.

[0027] As more and more sources of electronic data are generated, the structure and types of data are also increasing, such as traditional structured data from relational databases, semi-structured data such as CSV (Comma-Separated Values) and JSON (JavaScript Object Notation) files, and unstructured data such as text, images, and videos, which form multi-source heterogeneous data. For enterprises, data may come from the company's internal systems, such as ERP (Enterprise Resource Planning) systems and CRM (Customer Relationship Management), or from external data sources, such as public data sets and customer systems. These data sources vary in data structure, storage format, and access method. In order to facilitate rapid access and analysis when needed, these data are usually stored through various storage systems and storage technologies, which not only increases the difficulty of data processing. At the same time, enterprises need to store these data safely, reliably, and efficiently so that they can be quickly accessed and analyzed when needed. To this end, enterprises usually adopt various storage systems and technologies, such as DAS (Direct Attached Storage), SAN (Storage Area Network), NAS (Network Attached Storage), and cloud storage services to meet data storage needs.

[0028] When storing multi-source heterogeneous data, related technologies usually upload all the data to be stored from multiple external data sources to the cloud at one time. In order to avoid the reduction of efficiency due to the simultaneous processing of a large amount of multi-source heterogeneous data, the multi-source heterogeneous data will be converted into distributed data for uploading. However, since the data attributes of multi-source heterogeneous data are multimodal, there is a complex nonlinear separable space composed of noise, which makes the conversion process complicated. In order to reduce the complexity, multi-source heterogeneous data from different data sources can be fused, but the data is not cleaned during the fusion process. The quality of multi-source heterogeneous data is poor, resulting in poor data fusion effect, and it is still impossible to achieve efficient storage of multi-source heterogeneous data.

[0029] In view of this, the present invention obtains the original data to be stored from multiple heterogeneous data sources, improves the data quality through a series of automated cleaning steps through a pre-built data cleaning model, and finally stores the cleaned data efficiently. After introducing the technical solution of the present invention, various non-limiting embodiments of the present invention are described in detail below. In order to better illustrate the present invention, many specific details are given in the specific embodiments below. Those skilled in the art should understand that the present invention can also be implemented without these specific details. In other examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present invention.

[0030] First see Figure 1 , Figure 1 A flow chart of a data storage method provided in this embodiment may include the following contents: S101: Preliminarily build a data cleaning model based on a target neural network model with classification function as a model structure and using the similarity between data from different types of data sources as a constraint condition.

[0031] In this embodiment, the target neural network model can be any deep learning network model with classification function, including but not limited to back propagation network, multi-layer perceptron, radial basis function network, which does not affect the implementation of the present invention. The model structure of the data cleaning model is the model structure of the target neural network model. In order to realize the functions of re-examining and verifying the input data, deleting duplicate information, correcting errors in the data, etc., constraints are also introduced, that is, the constraints are used to identify data with high similarity and erroneous data. This embodiment builds the network model of the data cleaning model according to the model structure of the target neural network model, and then obtains a large amount of training sample data and inputs it into the model to train it, and introduces constraints during the training process until the training stop condition is met, and the data cleaning model is obtained.

[0032] S102: Acquire original data to be stored from multiple types of data sources.

[0033] Among them, the raw data to be stored is the data that needs to be stored. These raw data to be stored are collected from at least two different types of data sources. The data sources include but are not limited to: relational databases such as MySQL (database name) and Oracle (database name), non-relational databases such as MongoDB (database name) and Redis (database name), files such as CSV, Excel (spreadsheet software), JSON, XML (Extensible Markup Language), API (Application Programming Interface) interface, Web (network) page. The raw data to be stored include but are not limited to log files, database data, and message queues. In order to further improve the accuracy of subsequent processing of the stored data, pre-processing, such as denoising, can be performed after the data is collected to remove interference information as much as possible.

[0034] S103: Input each original data to be stored into the data cleaning model. The data cleaning model classifies each original data to be stored under the constraint conditions and selects target data from each original data to be stored.

[0035] Among them, the data cleaning model realizes the correlation of the internal structure of the data based on the constraint conditions. Under the constraint of the above correlation, the input raw data to be stored are integrated and classified with each other through the classification function of the target neural network model to obtain more accurate multi-source heterogeneous data relationships, identify useless data relationships, and eliminate useless data relationships, thereby completing data cleaning. The target data can be redundant data and / or erroneous data and / or outliers, or the type of data that needs to be cleaned set by the user according to the actual application scenario. Correspondingly, it can be set accordingly in the constraint conditions.

[0036] This embodiment uses a data cleaning model to clean multi-source heterogeneous data. Compared with manual cleaning, it can not only ensure the same or higher accuracy, but also greatly increase the speed, especially for large-scale data sets. Compared with full-machine cleaning, the so-called full-machine cleaning is to write a cleaning program based on a specific cleaning scheme to automatically clean the data. This method is complicated to implement and difficult to maintain later. Both do not require manual labor, but this step is simple to implement through the data cleaning model, and the later maintenance is also simple. Compared with the human-machine synchronous cleaning method, human-machine synchronous cleaning is for cleaning certain special scenes, which can only be achieved through a data cleaning program. The human-computer interaction interface is designed, and manual and machine-assisted methods are adopted. When the cleaning program cannot process the data, it is processed through manual intervention and other methods, which can reduce the complexity of writing programs and reduce a large number of manual operations, but still requires real-time human participation, while the data cleaning model does not require real-time human participation, effectively improving data cleaning efficiency. In addition, compared with asynchronous cleaning between humans and machines, that is, in the process of cleaning data, when encountering problems that the program cannot handle, there is no need for direct human participation. It is only necessary to record the abnormal situation and generate a corresponding report, and then continue the cleaning work. Humans only need to perform corresponding processing according to the report in the later stage. Although it can save manpower and improve the cleaning effect, it is not applicable to scenarios with high real-time requirements. This step can efficiently identify redundant data or abnormal data through the data cleaning model, and efficiently output the cleaning results to meet the real-time needs of users.

[0037] S104: Acquire the data to be stored by removing the target data from the original data to be stored, and store the data to be stored.

[0038] In this embodiment, the data to be stored is defined as data to be stored, and the data to be stored is data obtained after at least the original data to be stored is cleaned. The so-called data cleaning includes but is not limited to removing duplicate data, erroneous data, outliers, etc. The cleaning operation can be directly performed by the data cleaning model, that is, the data cleaning model directly outputs the data after removing redundant data and / or erroneous data and / or outliers, and can also mark the target data and then perform a deletion operation again. This does not affect the implementation of the present invention. Finally, the cleaned data is stored in the enterprise data warehouse or distributed storage system.

[0039] In the technical solution provided in this embodiment, for the original data to be stored obtained from multiple heterogeneous data sources, the correlation of the internal structure of the multi-source heterogeneous original data to be stored is first realized through the similarity constraint relationship of the multi-source heterogeneous data pre-set by the data cleaning model, which can not only improve the speed of the data cleaning operation inside the data cleaning model, but also improve the working efficiency of the data cleaning model. Under the constraints of the above correlation, the data cleaning model integrates and classifies the multi-source heterogeneous data to be stored, obtains more accurate multi-source heterogeneous data relationships, removes useless data relationships, thereby completing data cleaning, effectively improving the data quality of the data to be stored, and finally realizing efficient and secure storage of multi-source heterogeneous data.

[0040] In the above embodiment, there is no limitation on how to construct the data cleaning model. Based on the above embodiment, the present invention also provides an exemplary construction method of the data cleaning model, which may include the following contents: In this embodiment, if Figure 2 As shown, the data cleaning model may include an input layer, multiple hidden layers and an output layer; the total number of input layers, hidden layers and output layers can be preset, and each layer includes multiple nodes. The input layer is used to input data, that is, input each original data to be stored, and the output layer is used to output the original data to be stored without the target data, that is, the data cleaning result. Each hidden layer uses an unsupervised Boltzmann machine, that is, RBM (restricted Boltzmann machine). RBM is a simple neural network that is unsupervised and reconstructs the original data. It converts the input into a series of outputs that can represent the input data, and these outputs can reversely reconstruct these inputs. The unsupervised Boltzmann machine is a structure that plays a role in the back propagation process of the data cleaning model, that is, it is used to select the target data during back propagation and remove it as redundant data or erroneous data.

[0041] After determining the network structure of the data cleaning model, it is also necessary to establish similarity relationship constraints for multi-source heterogeneous data. For example, if the original data to be stored uploaded by multiple data sources is in the form of a data table, the Manhattan distance algorithm can be used to obtain the similarity between the multi-source heterogeneous data, and the similarity between the multi-source heterogeneous data can be used as the constraint condition of the data cleaning model. Of course, those skilled in the art can also construct constraint conditions according to other methods, such as Euclidean distance, Minkowski distance, Hamming distance, and angle cosine.

[0042] For example, as an efficient constraint condition construction method, the similarity calculation relationship can be pre-stored locally, and the similarity calculation relationship can be expressed as: ; Among them, each raw data to be stored exists in the form of a data table, the total amount of data is P, and the data table set B can be expressed as Considering that the original data to be stored has its own characteristics or properties, such as numeric type, text type and date type, when cleaning and sorting data, corresponding processing methods can be applied to different data attributes. Accordingly, data attribute sets of each original data to be stored can also be constructed. It can be expressed as , Represents the g-th data attribute in the data table. represents the similarity between the i-th original data to be stored and the j-th original data to be stored based on data attributes, represents the attribute characteristics of the i-th original data to be stored, represents the attribute feature form of the jth raw data to be stored, and S represents the data dimension. Based on the above similarity calculation formula, it can be seen that the greater the distance between the raw data to be stored, the smaller the similarity. The correlation coefficient can be pre-set Threshold value with data parameter , as the constraint of the data cleaning model: , then there is no i-th original data to be stored and the jth original data to be stored without attributes There is no correlation between , then the i-th original data to be stored without attributes and the jth original data to be stored without attributes Based on this constraint, the above data cleaning model is constructed, which can more accurately identify redundant data and abnormal values ​​in the original data to be stored and improve data storage efficiency.

[0043] Exemplarily, the target neural network model can be, for example, a BP (back propagation) neural network, which is a multi-layer feedforward neural network trained according to the error back propagation algorithm. The calculation process of the BP neural network consists of a forward calculation process and a reverse calculation process. In the forward propagation process, the input pattern is processed layer by layer from the input layer through the hidden layer and turned to the output layer. The state of each layer of neurons only affects the state of the next layer of neurons. If the expected output cannot be obtained in the output layer, it will turn to back propagation. In the back propagation process, an unsupervised Boltzmann machine is used to return the error signal along the original connection path. By modifying the weights of each neuron, the error signal is minimized, and the target data is selected for removal. The process of removing a large amount of redundant data and erroneous data uploaded by multi-source heterogeneous data sources based on the BP neural network can be: first, the similarity constraint relationship of multi-source heterogeneous data, that is, the constraint condition realizes the correlation of the internal structure of the data, which can not only improve the speed of the later BP neural network internal data cleaning operation, but also improve the efficiency of data cleaning work. Under the constraints of the above-mentioned correlation, the BP neural network integrates and classifies the results of multi-source heterogeneous data to obtain more accurate multi-source heterogeneous data relationships and eliminate useless data relationships, thereby completing data cleaning. Correspondingly, selecting target data from each original data to be stored based on the constraints is: using each original data to be stored as the node data of the data cleaning model, and identifying node energy anomalies and discrete distribution data based on the constraints as target data.

[0044] The above embodiment does not limit how to train the data cleaning model. The present invention also provides an exemplary training method for the data cleaning model, which may include the following contents: According to the number of nodes in the input layer and the number of nodes in the hidden layer, the node weight information between the layers is constructed; based on the node weight information and the bias coefficient of each node, the node energy function is determined, and the data distribution probability between the nodes is determined according to the node energy function; the node constraint conditions of each node in the hidden layer are constructed according to the data distribution probability and the node weight information, and each node in the hidden layer is activated based on the activation function determined by the node constraint conditions; the stored sample data of the multi-source heterogeneous data training sample set are input, and the node weight information and the bias coefficient of each node are continuously adjusted through the back propagation algorithm until the preset model training stop condition is reached.

[0045] In this embodiment, node weight information is generated according to the number of nodes in the input layer and the hidden layer. The node weight information may include the weight values ​​between each node in the input layer and each node in the hidden layer, which may be expressed in matrix form, that is, a node weight matrix is ​​constructed according to the number of nodes in the input layer and the hidden layer, and a node energy function is determined based on the weight matrix, the bias coefficient and the data state. The data distribution probability is determined according to the node energy function, which includes the state probability of the input layer and the hidden layer nodes, and the constraints of the hidden layer nodes are set based on the data distribution probability to ensure that the activity of the node variables meets specific requirements. The hidden layer nodes in the model are activated using an activation function, and the back propagation algorithm is executed to adjust the weight matrix and the bias coefficient to complete the training of the model. The model training stop condition may be, for example, the number of iterations reaching a preset iteration threshold, or reaching a convergence condition, or the model accuracy reaching a preset accuracy threshold, which does not affect the implementation of the present invention.

[0046] Exemplarily, the activation function may adopt the Sigmoid function (name of the activation function), and accordingly, the activation process of each hidden layer node may be: based on the data state coefficient of the nodes of the hidden layer, the nodes of the input layer and the node weight information, determine the binary activation function; use the binary activation function to activate each node of the hidden layer, and make the state of each node of the hidden layer within a preset node state value range.

[0047] Exemplarily, the calculation process of the data distribution probability may include: determining the node energy information between the first input node and the first hidden node based on the first input node of the input layer and its bias coefficient, the first hidden node of the hidden layer and its bias coefficient, and the node weight between the first input node and the first hidden node; generating a node energy function based on the node energy information between each node of the input layer and each node of the hidden layer; determining the normalization coefficient of the node energy function based on the distribution factor and the node energy function, and determining the data distribution probability between each node of the input layer and each node of the hidden layer based on the normalization coefficient and the node energy function.

[0048] In order to further improve the data storage efficiency, the present invention also pre-stores the node energy function relationship and the data distribution probability relationship. The node energy function relationship can be expressed as: ; In the formula, Represents the i-th node data of the input layer and the jth node data of the hidden layer The node energy function between Represents the i-th node data of the input layer The bias coefficient of Represents the jth node data of the hidden layer The bias coefficient of The i-th node data of the input layer and the jth node data of the hidden layer The weight value between , which can be obtained from the node weight information. Exemplarily, the node weight information can be represented by a node weight matrix, the node weight matrix It can be expressed as: ; Among them, there are visible layer and hidden layer data in the RBM structure of each layer of the data cleaning model, and the node set C of the input layer can be expressed as , c represents the total number of input layer nodes, represents the cth node in the input layer. The node set D of the hidden layer can be represented as , d represents the total number of hidden layer nodes, represents the dth node in the hidden layer, Represents the weight value between the cth node in the input layer and the dth node in the hidden layer.

[0049] Among them, the data distribution probability relationship can be expressed as: ; Represented as the i-th node data of the input layer and the jth node data of the hidden layer The probability of data distribution between is represented as the normalization parameter, represents the distribution factor.

[0050] Based on the above data distribution probability, the activity of model node variables can be effectively obtained to optimize the data cleaning model. When the training sample data is input into the data cleaning model, the states of the data nodes in the middle layer of the model, that is, the hidden layer, are independent of each other. If the nodes in the hidden layer When is 1, the node constraint condition of the hidden layer set based on the above data distribution probability relationship can be expressed as , accordingly, the activation function can be expressed as: ; In the formula, represents the binary Sigmoid function, Represents the data state coefficient. When the Sigmoid function independent variable takes a value interval of [0, 1], the probability distribution function of the data will present a single continuous change state, which can effectively obtain the activity level of the model node variables, thereby effectively improving the recognition accuracy of redundant data and abnormal data, and is conducive to improving data storage efficiency.

[0051] As can be seen from the above, this embodiment uses a neural network and back propagation algorithm to train a data cleaning model, which can accurately and efficiently identify and remove outliers and redundant data in the data. By constructing a weight matrix and calculating an energy function, it can automatically adapt to data sets with different characteristics, without the need for a large amount of customized development for different data sources, and can efficiently process data from different sources and formats, effectively improving versatility and adaptability.

[0052] In the above embodiment, there is no limitation on how to obtain multi-source heterogeneous data. Based on the above embodiment, the present invention also provides an exemplary data collection method, which may include the following contents: Obtain the data source type and data source location corresponding to each raw data to be stored; determine the data capture method according to the data source type, and obtain the corresponding raw data to be stored from the location of the corresponding data source according to the data capture method.

[0053] In this embodiment, obtaining each raw data to be stored requires clarifying the type and location of the data source. The data source may be, for example, a relational database, a non-relational database, a file, an API interface, or a Web page. The location of the data source may be obtained in advance by communicating with the data administrator, or may be obtained from a designated address storing the corresponding location information. Different types of data sources obtain data in different ways. For example, for a database, data may be directly extracted from the database through database query statements, such as SQL (Structured Query Language) statements. For files, file parsing tools or programming languages, such as Python (programming language name) and Java (programming language name) may be used to read file contents. For an API interface, an external API interface may be called to obtain data, such as using an HTTP (Hypertext Transfer Protocol) request. For a Web page, data may be captured from the Web page by using a web crawler.

[0054] From the above, it can be seen that this embodiment acquires data through the data source location and data source type, and can efficiently process data from different sources and formats. It can not only improve the flexibility and efficiency of data processing, but also help to improve the efficiency of multi-source heterogeneous data acquisition, and thus help to improve the efficiency of multi-source heterogeneous data storage.

[0055] In order to further improve the efficiency of multi-source heterogeneous data storage, based on the above embodiment, the present invention also provides an exemplary storage method, which may include the following contents: Extract the data features of the original data to be stored without the target data; group the data features according to the type of data source to which they belong, and fuse the grouped data features to obtain the data to be stored.

[0056] After storing the data, this embodiment can extract and process the features of the multi-source heterogeneous data to be stored based on the above-mentioned data cleaning results, and perform fusion processing. In other words, the original data to be stored is obtained through data cleaning operations and data fusion operations. The storage complexity is reduced through fusion processing, and the data storage efficiency is improved.

[0057] SVM (Support Vector Machine) determines a minimum boundary function through training to maximize the shortest distance between each data. It has the principle of structural risk minimization and good generalization ability. It is a convex optimization problem. As long as the local optimal solution is determined, it must be the global optimal solution, avoiding the problem of falling into the local optimal solution. As a simple, efficient and effective fusion method, this embodiment also uses the SVM algorithm to obtain the optimal classification hyperplane of the feature, build a data fusion model, and finally realize the fusion of multi-source heterogeneous data through the established data fusion model, which may include the following contents: The data fusion model is trained in advance based on the support vector machine; each data feature is input into the data fusion model; the data fusion model maps the nonlinear vector space of each data feature into the linear separable space to obtain the optimal classification hyperplane; each data feature is grouped based on the optimal classification hyperplane, and the data features in the same group belong to the same data source; and each data feature is fused based on the decision function determined by the optimal solution of each group of data features.

[0058] In this embodiment, the SVM algorithm is used to accelerate the convergence speed through the kernel function, control the dimension of the space, and map the data feature nonlinear vector space to the linear separable space to obtain the optimal classification hyperplane. In other words, according to the input data features, kernel function , feature vector set y, output vector The corresponding optimal hyperplane can be obtained. Data features can be in vector form It means that the optimal hyperplane It can be expressed as: ; in, represents the weighting coefficient, represents the bias term, m represents the total number of vectors contained in the feature vector set, and i represents a constant.

[0059] This embodiment uses the extracted feature vectors of multi-source heterogeneous data as input vectors and constructs an SVM data fusion model. At the same time, the multi-source heterogeneous data are divided into several groups according to the optimal hyperplane, and each group of data comes from a different data source; the actual value u, training output U and generalization parameter The data fusion model based on SVM is established, and the data fusion model J can be expressed as: ; In the formula, represents the mean of the optimal solution for data i, represents the optimal solution for data i, represents the mean of the optimal solution for data j, represents the optimal solution for data j, g represents the number of sensors, Represents a constraint.

[0060] According to the above data fusion model, the optimal solution, the mean of the optimal solution and the constraints of the data can be determined, and then the decision function of the data can be established according to the optimal solution. The decision function f(x) can be expressed as: ; In the formula, represents the mean value of the data from data source g, Represents the optimal solution of the data source g. The decision function is applied to the data fusion model, and the constructed data fusion model is verified according to the kernel function, loss function and generalization parameter, and finally the fusion result is output.

[0061] In addition, when fusing multi-source heterogeneous data, data from related databases can be encapsulated, stored, and queried at the same time, so that business processing, retrieval, query, and online optimization can be performed at any time. The multi-source heterogeneous data that has completed the above processing can be classified according to the characteristics of the data, effectively reducing duplicate records of multi-source heterogeneous data, quickly processing missing records, effectively improving data quality, achieving efficient data cleaning, and realizing effective fusion of multi-source heterogeneous data.

[0062] Furthermore, in order to improve the data fusion effect, based on the above embodiment, the present invention may also process the data, which may include the following contents: Obtain the storage mean of each original data to be stored after removing the target data; based on the storage mean, standardize each original data to be stored after removing the target data to obtain each standard data to be stored; respectively extract the feature vector of each standard data to be stored under each attribute as the data feature.

[0063] After the original data to be stored is cleaned by the data cleaning model, the data set corresponding to the original data to be stored is updated , the total number of data in the data set is M, and the data state is t. The data in the data set can be standardized using the mean normalization method through the following relationship: ; In the formula, The data in the data set is obtained by standardization. Represents the data mean of the data set, i and j represent constants. Based on the standardized data, feature vectors are extracted for each attribute. The feature vector of the jth attribute of the i-th data after standardization is It can be expressed as: ; In the formula, Indicates the data attribute recording period.

[0064] As can be seen from the above, this embodiment performs fusion processing on high-quality multi-source heterogeneous data obtained through a series of automated cleaning processes, which is conducive to achieving efficient and secure storage of data in a distributed storage system.

[0065] When the data fusion process is completed, the present invention also provides an implementation method for storing the data to be stored in the distributed system using a structured method, which may include the following contents: Obtain a preset configuration file and its corresponding file parsing toolkit, and use the file parsing toolkit to parse the preset configuration file to obtain configuration information; parse the received remote call service request based on the distributed service protocol data in the configuration information, and determine the fusion storage parameter information that matches the remote call service request; perform distributed fusion storage on the data to be stored according to the fusion storage parameter information.

[0066] In this embodiment, the preset configuration file configures a variety of distributed service protocols, routing rules, call modes and data fusion strategies, which is conducive to improving the flexible configuration and high efficiency of the system. The preset configuration file is parsed by the file parsing toolkit, and the configuration information may at least include distributed service protocols, routing rules, call modes and data fusion strategies, ensuring that the various strategies and rules defined in the configuration file can be accurately understood and executed. When entering the actual business processing stage, the remote call service request from the target service is captured, and the remote call service request can be intelligently parsed based on the configured distributed service protocol data, and the corresponding target routing rules, call modes and data fusion strategies can be accurately matched according to the request characteristics and configuration information, that is, the fusion storage parameter information includes target routing rules, target call modes and target fusion strategies, which can not only improve the accuracy of request processing, but also significantly enhance the flexibility and response speed of the system. After determining the target routing rules, call modes and fusion strategies, the relevant data in the remote call service request is efficiently obtained, and deep fusion processing is performed according to the established fusion strategy. That is, the target data to be stored corresponding to the remote call service request can be determined according to the target routing rules and the target calling mode; the target data to be stored can be fused according to the target fusion strategy. After the processing is completed, the fused data can be sent to the original requester to complete the distributed fusion storage task, efficiently completing the entire process from data reception to fusion processing to result return.

[0067] It should be noted that there is no strict order of execution between the steps in the present invention. As long as they comply with the logical order, these steps can be executed simultaneously or in a certain preset order. Figure 1 This is just a schematic and does not mean that this is the only execution order.

[0068] Finally, based on the technical solution of the present invention, Figure 3 Some possible application scenarios involved in the technical solution of the present invention are introduced by way of examples. Figure 3 A schematic diagram of a hardware composition framework applicable to a data storage method provided by the present invention may include the following contents: The hardware composition framework may include a first electronic device 31 and a second electronic device 32, which are connected via a network 33. The first electronic device 31 is deployed with a processor for executing the data storage method described in any of the above embodiments, and the second electronic device 32 is deployed with a user terminal for providing a human-computer interaction interface, which is used to send a remote call service request to the first electronic device 31. The first electronic device 31 completes the corresponding task through all or part of the steps in the data storage described in any of the above embodiments.

[0069] It should be noted that the above application scenarios are only shown to facilitate understanding of the concept and principle of the present invention, and the embodiments of the present invention are not limited in this respect. On the contrary, the embodiments of the present invention can be applied to any applicable scenario.

[0070] The present invention also provides a corresponding device for the data storage method, which further makes the method more practical. Among them, the device can be described from the perspective of functional modules and hardware. The data storage device provided by the present invention is introduced below. The device is used to implement the data storage method provided by the present invention. In this embodiment, the data storage device may include or be divided into one or more program modules, and the one or more program modules are stored in a storage medium and executed by one or more processors to complete the data storage method disclosed in Example 1. The program module referred to in this embodiment refers to a series of computer program instruction segments that can perform specific functions, which is more suitable for describing the execution process of the data storage device in the storage medium than the program itself. The following description will specifically introduce the functions of each program module in this embodiment. The data storage device described below and the data storage method described above can be referenced to each other.

[0071] From the perspective of functional modules, see Figure 4 , Figure 4 A structural diagram of a data storage device provided in this embodiment under a specific implementation mode, the device may include: The model building module 401 is used to build a data cleaning model in advance based on a target neural network model with classification function as a model structure and according to the similarity between data of different types of data sources as a constraint condition.

[0072] The data acquisition module 402 is used to obtain the original data to be stored from multiple types of data sources.

[0073] The data cleaning module 403 is used to input each raw data to be stored into the data cleaning model. The data cleaning model classifies each raw data to be stored under the constraint conditions and selects target data from each raw data to be stored.

[0074] The data storage module 404 is used to obtain the data to be stored after removing the target data from the original data to be stored, and store the data to be stored.

[0075] Exemplarily, in some implementations of this embodiment, the data storage module 404 is also used to: extract data features of the original data to be stored after removing the target data; group each data feature according to the type of data source to which it belongs, and fuse the grouped data features to obtain the data to be stored.

[0076] As an exemplary implementation of the above embodiment, the above data storage module 404 can also be further used for: pre-training a data fusion model based on a support vector machine; inputting each data feature into the data fusion model; the data fusion model maps the nonlinear vector space of each data feature into a linear separable space to obtain an optimal classification hyperplane; grouping each data feature based on the optimal classification hyperplane, and the data features in the same group belong to the same data source; and fusing each data feature based on a decision function determined by the optimal solution of each group of data features.

[0077] As another exemplary implementation of the above embodiment, the above data storage module 404 can be further used to: obtain the storage mean of each original data to be stored after removing the target data; based on the storage mean, standardize each original data to be stored after removing the target data to obtain each standard data to be stored; respectively extract the feature vectors of each standard data to be stored under each attribute as data features.

[0078] Exemplarily, in some other implementations of this embodiment, the above-mentioned data acquisition module 402 can also be used to: obtain the data source type and the location of the data source corresponding to each original data to be stored; determine the data capture method according to the data source type, and obtain the corresponding original data to be stored from the location of the corresponding data source according to the data capture method.

[0079] Exemplarily, in some other implementations of the present embodiment, the data storage module 404 may also be used to: obtain a preset configuration file and its corresponding file parsing toolkit, and use the file parsing toolkit to parse the preset configuration file to obtain configuration information; parse the received remote call service request based on the distributed service protocol data in the configuration information, and determine the fusion storage parameter information that matches the remote call service request; and perform distributed fusion storage on the data to be stored according to the fusion storage parameter information.

[0080] As an exemplary implementation of the above embodiment, the above data storage module 404 can also be further used for: configuration information includes distributed service protocols, routing rules, call modes and data fusion strategies, and fusion storage parameter information includes target routing rules, target call modes and target fusion strategies; according to the target routing rules and target call modes, determine the target data to be stored corresponding to the remote call service request; according to the target fusion strategy, fuse the target data to be stored, and send the fused data to the original requester to complete the distributed fusion storage task.

[0081] Exemplarily, in some implementations of this embodiment, the data cleaning model includes an input layer, multiple hidden layers and an output layer; each hidden layer uses an unsupervised Boltzmann machine to select target data during back propagation and remove it as target data.

[0082] As an exemplary implementation of the above embodiment, the above data cleaning module 403 can be further used to: use each original data to be stored as node data of the data cleaning model, identify node energy anomalies and discrete distribution data based on constraint conditions as target data.

[0083] Exemplarily, in some other implementations of this embodiment, the above-mentioned model construction module 401 is also used to: construct node weight information between layers according to the number of nodes in the input layer and the number of nodes in the hidden layer; determine the node energy function based on the node weight information and the bias coefficient of each node, and determine the data distribution probability between each node according to the node energy function; construct node constraints of each node in the hidden layer according to each data distribution probability and the node weight information, and activate each node in the hidden layer based on the activation function determined by the node constraints; input each stored sample data of the multi-source heterogeneous data training sample set, and continuously adjust the node weight information and the bias coefficient of each node through the back propagation algorithm until the preset model training stop condition is reached.

[0084] As an exemplary implementation of the above embodiment, the above model construction module 401 can also be further used to: determine the binary activation function based on the data state coefficients of the nodes of the hidden layer, the nodes of the input layer and the node weight information; use the binary activation function to activate the nodes of the hidden layer, and make the states of the nodes of the hidden layer within a preset node state value range.

[0085] As another exemplary implementation of the above embodiment, the above model building module 401 can also be further used to: determine the node energy information between the first input node and the first hidden node according to the first input node of the input layer and its bias coefficient, the first hidden node of the hidden layer and its bias coefficient, and the node weight between the first input node and the first hidden node; generate a node energy function according to the node energy information between each node of the input layer and each node of the hidden layer; determine the normalization coefficient of the node energy function according to the distribution factor and the node energy function, and determine the data distribution probability between each node of the input layer and each node of the hidden layer according to the normalization coefficient and the node energy function.

[0086] The data storage device mentioned above is described from the perspective of functional modules. Furthermore, the present invention also provides an electronic device, which is described from the perspective of hardware. Figure 5 The following is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention in one implementation manner. Figure 5 As shown, the electronic device includes a memory 50 for storing a computer program; and a processor 51 for implementing the steps of the data storage method mentioned in any of the above embodiments when executing the computer program.

[0087] Among them, the processor 51 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the processor 51 may also be a controller, a microcontroller, a microprocessor or other data processing chip. The processor 51 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 51 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 51 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 51 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0088] The memory 50 may include one or more computer non-volatile storage media, which may be non-transitory. The memory 50 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the memory 50 may be an internal storage unit of an electronic device, such as a hard disk of a server. In other embodiments, the memory 50 may also be an external storage device of an electronic device, such as a plug-in hard disk equipped on a server, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Further, the memory 50 may also include both an internal storage unit and an external storage device of the electronic device. The memory 50 may not only be used to store application software and various types of data installed in the electronic device, such as: the code of the program in the process of executing the data storage method, etc., but also be used to temporarily store data that has been output or is to be output. In this embodiment, the memory 50 is at least used to store the following computer program 501, wherein, after the computer program is loaded and executed by the processor 51, the relevant steps of the data storage method disclosed in any of the aforementioned embodiments can be implemented. In addition, the resources stored in the memory 50 may also include an operating system 502 and data 503, etc., and the storage method may be temporary storage or permanent storage. The operating system 502 may include Windows, Unix, Linux, etc. The data 503 may include but is not limited to data corresponding to the data storage result, etc.

[0089] In some embodiments, the electronic device may further include a display screen 52, an input / output interface 53, a communication interface 54 or a network interface, a power supply 55 and a communication bus 56. Among them, the display screen 52 and the input / output interface 53, such as a keyboard, belong to the user interface, and exemplary user interfaces may also include standard wired interfaces, wireless interfaces, etc. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device, etc. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface. The communication interface 54 may exemplarily include a wired interface and / or a wireless interface, such as a WI-FI interface, a Bluetooth interface, etc., which are generally used to establish a communication connection between the electronic device and other electronic devices. The communication bus 56 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0090] Those skilled in the art will understand that Figure 5 The structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than those shown in the figure, for example, it may also include a sensor 57 for realizing various functions.

[0091] It is understandable that if the data storage method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes but is not limited to: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM, register, hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), magnetic memory, removable disk, CD-ROM, magnetic disk or optical disk, etc. Various media that can store program codes. Based on this, the present invention also provides a non-volatile storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the data storage method described in any of the above embodiments are performed.

[0092] It is understandable that if the data storage method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, the computer software product does not need to be stored in a physical storage medium. For example, it can be directly transmitted to a computer or other device with information processing capabilities through a wired network or a wireless network to perform all or part of the steps of the methods of each embodiment of the present invention. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product. Based on this, the present invention also provides a computer program product that stores a computer program. When the computer program is executed by a processor, the steps of the data storage method described in any of the above embodiments are performed.

[0093] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the hardware disclosed in the embodiment, including electronic devices, non-volatile storage media and computer program products, since they correspond to the methods disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0094] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0095] The above is a detailed introduction to a data storage method, an electronic device, a non-volatile storage medium and a computer program product provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that based on the embodiments of the present invention, for ordinary technicians in this technical field, all other embodiments obtained without creative work are within the scope of protection of the present invention. Without departing from the principles of the present invention, the present invention can also be improved and modified in a number of ways, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A data storage method, characterized in that: include: A data cleaning model is constructed in advance based on a target neural network model with classification function as the model structure and according to the similarity between data from different types of data sources as a constraint condition; Acquire each type of original data to be stored from multiple types of data sources; Inputting each raw data to be stored into the data cleaning model, the data cleaning model classifies each raw data to be stored under the constraint conditions, and selects target data from each raw data to be stored; The data to be stored are obtained by removing the target data from each original data to be stored, and the data to be stored are stored.

2. The data storage method according to claim 1, characterized in that: Acquiring data to be stored by removing target data from each original data to be stored, including: Extracting data features of the original data to be stored by removing the target data; Each data feature is grouped according to the type of data source to which it belongs, and the grouped data features are fused to obtain the data to be stored.

3. The data storage method according to claim 2, characterized in that: Each data feature is grouped according to the type of data source to which it belongs, and the grouped data features are fused, including: Pre-training data fusion model based on support vector machine; Inputting each data feature into the data fusion model; The data fusion model is used to map the nonlinear vector space of each data feature into a linear separable space to obtain an optimal classification hyperplane; based on the optimal classification hyperplane, each data feature is grouped, and the data features in the same group belong to the same data source; based on the decision function determined by the optimal solution of each group of data features, each data feature is fused.

4. The data storage method according to claim 2, characterized in that: Extract the data features of the original data to be stored and remove the target data, including: Obtaining the storage mean of each original data to be stored after removing the target data; Based on the stored mean, standardizing the original data to be stored after removing the target data to obtain standard data to be stored; The characteristic vectors of each standard data to be stored under each attribute are extracted respectively as data features.

5. The data storage method according to claim 1, characterized in that: Obtain the original data to be stored from various data sources, including: Obtain the data source type and location of each raw data to be stored; A data capture method is determined according to the data source type, and corresponding raw data to be stored is acquired from the location of the corresponding data source according to the data capture method.

6. The data storage method according to claim 1, characterized in that: Storing the data to be stored includes: Obtaining a preset configuration file and its corresponding file parsing toolkit, and using the file parsing toolkit to parse the preset configuration file to obtain configuration information; Parsing the received remote call service request based on the distributed service protocol data in the configuration information, and determining fusion storage parameter information matching the remote call service request; The data to be stored is distributedly fused and stored according to the fusion storage parameter information.

7. The data storage method according to claim 6, characterized in that: The configuration information includes a distributed service protocol, routing rules, a call mode, and a data fusion strategy, the fusion storage parameter information includes a target routing rule, a target call mode, and a target fusion strategy, and the data to be stored is distributedly fused and stored according to the fusion storage parameter information, including: Determine the target data to be stored corresponding to the remote call service request according to the target routing rule and the target call mode; The target data to be stored is fused according to the target fusion strategy, and the fused data is sent to the original requester to complete the distributed fusion storage task.

8. The data storage method according to claim 1, characterized in that: The data cleaning model includes an input layer, a plurality of hidden layers and an output layer; Each hidden layer uses an unsupervised Boltzmann machine to select target data during back propagation.

9. The data storage method according to claim 8, characterized in that: Selecting target data from each of the original data to be stored based on the constraint condition includes: Each original data to be stored is used as the node data of the data cleaning model, and the data with abnormal node energy and discrete distribution are identified based on the constraint conditions to serve as the target data.

10. The data storage method according to any one of claims 1 to 9, characterized in that: The training process of the data cleaning model includes: According to the number of nodes in the input layer and the number of nodes in the hidden layer, the node weight information between the layers is constructed; Determining a node energy function based on the node weight information and the bias coefficient of each node, and determining a data distribution probability between each node according to the node energy function; Constructing node constraints of each node of the hidden layer according to the data distribution probability and the node weight information, and activating each node of the hidden layer based on the activation function determined by the node constraints; Each stored sample data of the multi-source heterogeneous data training sample set is input, and the node weight information and the bias coefficient of each node are continuously adjusted through the back propagation algorithm until the preset model training stop condition is reached.

11. The data storage method according to claim 10, characterized in that: Activating each node of the hidden layer based on the activation function determined by the node constraint condition, comprising: Determine a binary activation function based on the data state coefficients of the nodes of the hidden layer, the nodes of the input layer and the node weight information; The binary activation function is used to activate each node of the hidden layer, and each node state of the hidden layer is made within a preset node state value range.

12. The data storage method according to claim 10, characterized in that: Based on the node weight information and the bias coefficient of each node, a node energy function is determined, and according to the node energy function, a data distribution probability between each node is determined, including: Determine node energy information between the first input node and the first hidden node according to the first input node of the input layer and its bias coefficient, the first hidden node of the hidden layer and its bias coefficient, and the node weight between the first input node and the first hidden node; Generate a node energy function according to node energy information between each node of the input layer and each node of the hidden layer; A normalization coefficient of the node energy function is determined according to the distribution factor and the node energy function, and a data distribution probability between each node of the input layer and each node of the hidden layer is determined according to the normalization coefficient and the node energy function.

13. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the processor is used to implement the steps of the data storage method according to any one of claims 1 to 12 when executing a computer program stored in the memory.

14. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data storage method according to any one of claims 1 to 12 are implemented.

15. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the data storage method according to any one of claims 1 to 12 are implemented.

Images