Method and device for processing data
By analyzing the spatio-temporal correlation of supply chain nodes and building distribution models, and extracting abnormal features, the problems of large granularity and spatial correlation of data processing in the existing technology are solved, and the accuracy and accuracy of data analysis are improved.
Patent Information
- Application Number
- CN202310659845.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-06
- Publication Date
- 2024-12-06
AI Technical Summary
The prior art has problems such as large granularity of data processing and spatial and temporal correlation in supply chain data analysis, resulting in low accuracy of data analysis.
By obtaining the operating characteristic data of supply chain nodes, analyzing the spatial and temporal correlation, building a spatial and temporal correlation distribution model, calculating reference business data, extracting abnormal characteristics, and using density clustering models to determine abnormal data points, and then processing abnormal data points.
Improves the accuracy and accuracy of data analysis, and enables more efficient identification and processing of abnormal data points in the supply chain.
Smart Images

Figure CN119090110A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of smart supply chain technology, and in particular to a method and device for processing data. Background Art
[0002] With the advancement of the digitalization of the supply chain, mining the operating data of each node in the supply chain (such as commodity sales stores, commodity wholesalers and other item providers) from massive data and analyzing the operating conditions of each node in the supply chain (such as screening out nodes with abnormal operating fluctuations) will have an important impact on the quality of supply chain integration construction.
[0003] At present, the analysis of the operating data of each node in the supply chain mainly uses a combination of feature engineering and clustering methods to process and cluster data features based on the operating data of different nodes in the supply chain, thereby screening out abnormal fluctuation data that appear to be outliers.
[0004] On the one hand, the existing methods have the problem of large granularity in processing data due to the clustering of the overall data (that is, both related and unrelated data are involved in the clustering). On the other hand, the existing methods do not take into account the impact of the spatiotemporal correlation factors between multiple nodes in the supply chain on the data, resulting in low data analysis accuracy. Summary of the invention
[0005] In view of this, an embodiment of the present invention provides a method and device for processing data, which can analyze the spatiotemporal correlation between the operating characteristic data of each node on the supply chain based on the operating characteristic data of multiple nodes; construct a spatiotemporal correlation distribution model for nodes with spatiotemporal correlation to calculate the reference operating data of each node with spatiotemporal correlation; and extract abnormal operating characteristics; use a preset density clustering model to determine abnormal data points for each node with spatiotemporal correlation; and process the corresponding nodes based on the abnormal data points. The embodiment of the present invention improves the precision and accuracy of processing operating data.
[0006] To achieve the above-mentioned purpose, according to one aspect of an embodiment of the present invention, a method for processing data is provided, which is characterized in that it includes: obtaining business characteristic data of multiple nodes in a set time and space range on a supply chain, and analyzing the time and space correlation between the business characteristic data of each of the nodes; when it is analyzed that the business characteristic data of at least two of the nodes have time and space correlation, constructing a time and space correlation distribution model for at least two of the nodes with time and space correlation, and calculating the reference business data of each of the nodes with time and space correlation through the time and space correlation distribution model; extracting business abnormality characteristics based on the reference business data; determining abnormal data points for each of the nodes with time and space correlation by using a preset density clustering model, the business characteristic data of each of the nodes with time and space correlation, and the business abnormality characteristics; providing the abnormal data points to the devices associated with the nodes corresponding to the abnormal data points on the supply chain, so as to process the corresponding nodes based on the abnormal data points.
[0007] Optionally, the analysis of the spatiotemporal correlation between the operating characteristic data of each of the nodes includes: converting the operating characteristic data of each of the nodes into a corresponding characteristic matrix, calculating the probability value of the spatiotemporal correlation between the characteristic matrices converted from each of the nodes using a likelihood ratio test model; and determining whether the operating characteristic data of at least two of the nodes have spatiotemporal correlation based on a comparison result between the probability value and a set probability threshold.
[0008] Optionally, the analysis of the spatiotemporal correlation between the operating characteristic data of each of the nodes further includes: constructing a distribution model for the operating characteristic data of at least two of the nodes that have spatiotemporal correlation; performing a preset hypothesis test on the constructed distribution model; and determining that the operating characteristic data of at least two of the nodes have spatiotemporal correlation when the result of the preset hypothesis test indicates that the distribution model conforms to one or more preset distribution models.
[0009] Optionally, constructing a spatiotemporal correlation distribution model for at least two of the nodes having spatiotemporal correlation includes: for each of the nodes, performing the following operations: calculating a plurality of business characteristic factors corresponding to the node; and constructing a spatiotemporal correlation distribution model corresponding to the node based on a distribution model constructed based on the plurality of business characteristic factors and the business characteristic data of the node.
[0010] Optionally, extracting the abnormal operation characteristics based on the reference operation data includes: extracting a standard score from a distribution model formed by the reference operation data; and determining the abnormal operation characteristics according to the extracted standard score.
[0011] Optionally, the method for processing data includes using a stacked autoencoder network model to process the operating feature data of the plurality of nodes into two-dimensional feature data; and / or processing the operating abnormality features into two-dimensional feature data.
[0012] Optionally, the method for processing data includes determining the number of layers of the middle layer of the stacked autoencoder network model according to the time interval contained in the set space-time range; setting the number of neurons in the middle layer of the stacked autoencoder network model to two dimensions; to obtain two-dimensional feature data; constructing the stacked autoencoder network model based on the middle layer of the number of layers; using the stacked autoencoder network model to process the operating feature data of multiple nodes into two-dimensional feature data, and / or processing the operating abnormality features into two-dimensional feature data, further includes: looping step N1 until the target two-dimensional feature data is determined: N1: using the stacked autoencoder network model to process the operating feature data of multiple nodes into two-dimensional feature data, and / or processing the operating abnormality features into two-dimensional feature data, use the activation function to evaluate the processing error, and when it is determined that the processing error meets the set error condition, the determined two-dimensional feature data is the target two-dimensional feature data.
[0013] Optionally, the method for processing data further includes: performing clustering operations multiple times using the original density clustering model, the business characteristic data of each of the item providers, and the two-dimensional feature data corresponding to the business abnormality characteristics; evaluating the clustering effect of each time using the silhouette coefficient, and adjusting the parameters of the original density clustering model accordingly to obtain the preset density clustering model.
[0014] To achieve the above object, according to a second aspect of an embodiment of the present invention, a device for processing data is provided, characterized in that it comprises: a correlation analysis module, a feature determination module and an abnormal data determination module; wherein,
[0015] The correlation analysis module is used to obtain the operating characteristic data of multiple nodes in the supply chain that are within a set time and space range, and analyze the time and space correlation between the operating characteristic data of each of the nodes;
[0016] The feature determination module is used to construct a spatiotemporal correlation distribution model for the at least two nodes with spatiotemporal correlation when analyzing the business feature data of each node and finding that the business feature data of at least two nodes have spatiotemporal correlation, and calculate the reference business data of each node with spatiotemporal correlation through the spatiotemporal correlation distribution model;
[0017] The abnormal data determination module is used to extract abnormal business characteristics based on reference business data; use a preset density clustering model, the business characteristic data of each node with spatiotemporal correlation and the abnormal business characteristics to determine abnormal data points for each node with spatiotemporal correlation; provide the abnormal data points to the devices associated with the nodes corresponding to the abnormal data points on the supply chain, so as to process the corresponding nodes based on the abnormal data points.
[0018] Optionally, the data processing device is used to analyze the spatiotemporal correlation between the operating characteristic data of each of the nodes, including: converting the operating characteristic data of each of the nodes into a corresponding characteristic matrix, using a likelihood ratio test model to calculate the probability value of the spatiotemporal correlation between the characteristic matrices converted from each of the nodes; and determining whether the operating characteristic data of at least two of the nodes have spatiotemporal correlation based on a comparison result between the probability value and a set probability threshold.
[0019] Optionally, the data processing device is used to analyze the spatiotemporal correlation between the business characteristic data of each of the nodes, and further includes: constructing a distribution model for the business characteristic data of at least two of the nodes that have spatiotemporal correlation; performing a preset hypothesis test on the constructed distribution model; and determining that the business characteristic data of at least two of the nodes have spatiotemporal correlation when the result of the preset hypothesis test indicates that the distribution model conforms to one or more preset distribution models.
[0020] Optionally, the data processing device is used to construct a spatiotemporal correlation distribution model for at least two of the nodes that have spatiotemporal correlation, including: for each of the nodes, performing the following operations: calculating multiple business characteristic factors corresponding to the node; and constructing a spatiotemporal correlation distribution model corresponding to the node based on a distribution model constructed based on the multiple business characteristic factors and the business characteristic data of the node.
[0021] Optionally, the data processing device is used to extract abnormal operation characteristics based on reference operation data, including: extracting standard scores from a distribution model composed of the reference operation data; and determining the abnormal operation characteristics according to the extracted standard scores.
[0022] Optionally, the data processing device is used to process the operating feature data of multiple nodes into two-dimensional feature data using a stacked autoencoder network model; and / or process the operating abnormality features into two-dimensional feature data.
[0023] Optionally, the data processing device is used to determine the number of layers of the middle layer of the stacked autoencoder network model according to the time interval contained in the set space-time range; set the number of neurons in the middle layer of the stacked autoencoder network model to two dimensions; to obtain two-dimensional feature data; construct the stacked autoencoder network model based on the middle layer of the number of layers; the use of the stacked autoencoder network model to process the operating feature data of multiple nodes into two-dimensional feature data, and / or the operating abnormality features into two-dimensional feature data, further includes: looping step N1 until the target two-dimensional feature data is determined: N1: using the stacked autoencoder network model, the operating feature data of multiple nodes are processed into two-dimensional feature data, and / or the operating abnormality features are processed into two-dimensional feature data, and then the activation function is used to evaluate the processing error. When it is determined that the processing error meets the set error condition, the determined two-dimensional feature data is the target two-dimensional feature data.
[0024] Optionally, the data processing device is further used to perform clustering operations multiple times using the original density clustering model, the business characteristic data of each of the item providers, and the two-dimensional feature data corresponding to the business abnormality characteristics; use the silhouette coefficient to evaluate the clustering effect of each time, and adjust the parameters of the original density clustering model accordingly to obtain the preset density clustering model.
[0025] To achieve the above-mentioned purpose, according to the third aspect of an embodiment of the present invention, there is provided an electronic device for processing data, characterized in that it comprises: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement any of the methods described in the above-mentioned methods for processing data.
[0026] To achieve the above-mentioned purpose, according to a fourth aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, characterized in that when the program is executed by a processor, a method as described in any of the above-mentioned methods for processing data is implemented.
[0027] An embodiment of the above invention has the following advantages or beneficial effects: it can analyze the spatiotemporal correlation between the operating characteristic data of each node on the supply chain based on the operating characteristic data of multiple nodes; construct a spatiotemporal correlation distribution model for nodes with spatiotemporal correlation to calculate the reference operating data of each node with spatiotemporal correlation; and extract abnormal operating characteristics; use a preset density clustering model to determine abnormal data points for each node with spatiotemporal correlation; and process the corresponding nodes based on the abnormal data points. The embodiment of the present invention improves the precision and accuracy of processing and analyzing operating data.
[0028] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with specific implementation examples. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.
[0030] Figure 1 is a flowchart of a method for processing data provided by an embodiment of the present invention;
[0031] Figure 2 It is a schematic diagram of a data processing flow provided by an embodiment of the present invention;
[0032] Figure 3 is a visualization diagram of processed data provided by an embodiment of the present invention;
[0033] Figure 4 is a schematic diagram of the structure of a device for processing data provided by an embodiment of the present invention;
[0034] Figure 5 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;
[0035] Figure 6 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION
[0036] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.
[0037] It should be noted that in the technical solution of this disclosure, the collection, collection, update, analysis, processing, use, transmission, storage and other aspects of user personal information involved are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good customs. Take necessary measures for user personal information to prevent illegal access to user personal information data, and maintain user personal information security, network security and national security. In order to count the usage of our products / services, we will summarize, analyze and use the technically processed user data, and share the processed statistical information with third parties. We will use secure encryption technical processing methods and other methods to ensure that the recipient of the information cannot re-identify a specific individual. After collecting supply chain information, we will de-identify the data through technical means. When displaying supply chain node information, we will desensitize your information by de-identification or anonymization to protect information security.
[0038] In recent years, with the advancement of the digitalization of the supply chain, it has become particularly important to mine the information of supply chain nodes (such as the information of outlier nodes) from massive data. As one or more nodes of the supply chain, the accurate prediction of the operating data of nodes (such as stores, supplier sites, etc.) can have an important impact on the quality of supply chain integration construction. The existing prediction method for node operating conditions in the integrated supply chain construction is to combine algorithm model prediction with operator prediction, and make predictions based on the historical data of node operating conditions and their corresponding characteristics. In actual business scenarios, it is found that due to the influence of external environment, other nodes and other factors, some nodes have abnormal fluctuations in operation. When comparing sales data between different stores, these nodes with abnormal fluctuations will appear outliers. If the abnormal fluctuations cannot be effectively identified and processed through outlier analysis, the subsequent prediction will be affected by it and produce poor prediction results. Therefore, it is very important to perform outlier detection (i.e., operating data analysis) on the operating status of the node operating data. In addition, analyzing the outlier nodes on the supply chain can directly provide data for the subsequent evaluation of the operating status of the supply chain.
[0039] The current method has the problem of large data granularity due to the clustering of the entire data (i.e. both related and unrelated data are involved in the clustering). On the other hand, the existing method does not take into account the impact of the spatiotemporal correlation factors between multiple nodes in the supply chain on the data, resulting in low data analysis accuracy.
[0040] In view of this, if Figure 1 As shown, an embodiment of the present invention provides a method for processing data, which may include the following steps:
[0041] Step S101: acquiring the operating characteristic data of a plurality of nodes in the supply chain within a set time and space range, and analyzing the time and space correlation between the operating characteristic data of each of the nodes.
[0042] Specifically, the operating data of each node in the supply chain (such as commodity sales stores, commodity wholesalers and other item providers) is obtained, wherein the operating data can be any type of data reflecting the operating status of the node, such as: commodity sales, commodity sales, store turnover, store profit, etc.; further, setting the time and space range represents that multiple nodes have one or more associations in the spatial dimension and the time dimension, for example: multiple nodes are all stores of a certain enterprise in the supply chain of Province A, or multiple nodes are multiple commodity sales stores, commodity wholesalers, etc. in a certain business district; the operating characteristic data of multiple nodes can be the operating data of all stores of a certain enterprise in Province A within a period of time (for example: one month, one quarter, half a year, etc.). It can be understood that by analyzing the operating characteristic data of multiple nodes with a set time and space range, the validity and accuracy of the analysis data are improved compared with the case of an uncertain range, and computing resources are saved.
[0043] Furthermore, an embodiment of the present invention is to eliminate the influence of the spatiotemporal correlation between the operating characteristic data of the nodes on the results of data analysis; therefore, the operating characteristic data of multiple nodes on the supply chain within the set spatiotemporal range are obtained, and the spatiotemporal correlation between the operating characteristic data of each of the nodes is analyzed.
[0044] Specifically, the method for analyzing and judging the spatiotemporal correlation between the business characteristic data of each of the nodes is as follows: the business characteristic data of each node are encoded into a matrix format, and the correlation coefficient matrix between the matrices is calculated, and then the likelihood ratio test model is used to calculate the probability value of the spatiotemporal correlation between the characteristic matrices converted from each of the nodes according to a prior assumption (for example, the prior assumption is that there is no spatiotemporal correlation between the node business characteristic data); for example, the probability value is 0.01; for example, the value of the probability threshold p is set to 0.05; the calculated probability value is compared with the set probability threshold, for example, the probability value is less than the set probability threshold; then the false positive that there is no spatiotemporal correlation between the node business characteristic data is true. Assume a premise, and a comparison result, and determine whether the business characteristic data of the two nodes have a spatiotemporal correlation; that is, the comparison result of the probability value and the set probability threshold indicates that the probability that there is no spatiotemporal correlation between the business characteristic data of the nodes is lower than the set probability threshold used to judge the hypothesis test parameter, that is, the original hypothesis is rejected, and the conclusion that the business characteristic data of the nodes have a spatiotemporal correlation is obtained; that is, the business characteristic data of each node is converted into a corresponding feature matrix, and the probability value of the spatiotemporal correlation between the feature matrices converted from each node is calculated using the likelihood ratio test model; according to the comparison result of the probability value and the set probability threshold, determine whether the business characteristic data of at least two nodes have a spatiotemporal correlation. The premise of the assumption of the embodiment of the present invention can be: the existence of spatiotemporal correlation, or the absence of spatiotemporal correlation, and the size of the set probability threshold is set correspondingly under different assumption premises.
[0045] Furthermore, when it is found that the business characteristic data of at least two of the nodes have spatiotemporal correlation among the business characteristic data of each node, the analysis is performed according to the distribution of the business characteristic data of the node. The analysis method can use a hypothesis test model to test (i.e., perform a preset hypothesis test) whether the distribution of the business characteristic data conforms to one or more preset distribution models (such as normal distribution, lognormal distribution, etc.); the steps of using the hypothesis test model to test the distribution of the business characteristic data are, for example: the null hypothesis of the first hypothesis test is that the sales data obeys the normal distribution, and the null hypothesis of the second hypothesis test is that the sales data obeys the lognormal distribution, and the test result parameter (p) value of the two tests is set to 0.05. If the first hypothesis test is not rejected, the subsequent analysis process is directly entered; if the first hypothesis test is rejected and the second one is not rejected, the business characteristic data is logarithmically transformed and then the subsequent analysis process is entered. If both tests reject the null hypothesis, it means that the distribution of the operating characteristic data does not conform to one or more preset distribution models; it can be understood that, after testing whether the distribution of the operating characteristic data conforms to one or more preset distribution models, it is further determined that the operating characteristic data of at least two of the nodes have spatiotemporal correlation; that is, a distribution model is constructed for the operating characteristic data of at least two nodes that have spatiotemporal correlation; a preset hypothesis test is performed on the constructed distribution model; when the result of the preset hypothesis test indicates that the distribution model conforms to one or more preset distribution models, it is determined that the operating characteristic data of at least two of the nodes have spatiotemporal correlation.
[0046] Step S102: When it is found that the business characteristic data of at least two of the nodes have spatiotemporal correlation among the business characteristic data of each of the nodes, a spatiotemporal correlation distribution model is constructed for the at least two nodes with spatiotemporal correlation, and the reference business data of each of the nodes with spatiotemporal correlation is calculated through the spatiotemporal correlation distribution model.
[0047] Specifically, when it is found that the business characteristic data of at least two of the nodes have spatiotemporal correlation among the business characteristic data of the nodes, a spatiotemporal correlation distribution model is constructed for the at least two nodes having the spatiotemporal correlation;
[0048] In the embodiment of the present invention, the spatiotemporal correlation distribution model is constructed for at least two nodes having spatiotemporal correlation as shown in formula (1):
[0049] y ij ~N(u i +t j ,σ) (1)
[0050] In formula (1), it is assumed that there are n stores in a similar space, and the number of observation periods of operating characteristic data is m periods, where the operating characteristic data (such as sales, etc.) of the i-th store in the j-th period is y ij ;u i u represents the sales ability of a store, tj represents the difficulty of selling a certain product in a certain period, and σ represents the fluctuation of the store's sales. i ,t j , σ, etc. are the multi-operation characteristic factors corresponding to a store (i.e., node). i +t j ,σ) represents the distribution model constructed by the business characteristic data (for example: the distribution model is a normal distribution, etc.); that is, the spatiotemporal correlation distribution model is constructed for at least two of the nodes with spatiotemporal correlation, including: for each of the nodes, performing the following operations: calculating the multiple business characteristic factors corresponding to the node; based on the distribution model constructed by the multiple business characteristic factors and the business characteristic data of the node, constructing a spatiotemporal correlation distribution model corresponding to the node.
[0051] Furthermore, the reference business data of each node with spatiotemporal correlation is calculated by the spatiotemporal correlation distribution model; Formula (1) is still used as an example of the spatiotemporal correlation distribution model, where y ij Specifically, the MCMC algorithm can be used to obtain the parameters u corresponding to each operating characteristic factor in formula (1) through simulation and iterative calculation methods. i ,t j , σ, and y ij Corresponding data; MCMC (Markov Chain Monte Carlo, Markov Chain Monte Carlo method, referred to as MCMC) is a Monte Carlo method (Monte Carlo) simulated by computer under the framework of Bayesian theory. This method introduces the Markov process into the Monte Carlo simulation to achieve dynamic simulation in which the sampling distribution changes as the simulation proceeds; MCMC is a simple and effective calculation method that is widely used in many fields, such as probability distribution estimation, Bayesian problems, etc. It can be seen that by using the MCMC algorithm, the data corresponding to each parameter in formula (1) is determined by the technical means of dynamic simulation, that is, by using the spatiotemporal correlation distribution model, the reference business data of each node with spatiotemporal correlation is calculated, which improves the accuracy of judging whether the business characteristic data of the node has abnormal fluctuations.
[0052] Step S103: extracting abnormal business characteristics based on reference business data; using a preset density clustering model, the business characteristic data and the abnormal business characteristics of each of the nodes with spatiotemporal correlation, determining abnormal data points for each of the nodes with spatiotemporal correlation.
[0053] Specifically, based on the distribution model (e.g., a normal distribution model) formed by the reference business data corresponding to the calculated nodes, the business abnormality feature is extracted; the specific method of extracting the business abnormality feature can be: extracting the standard score (e.g., the model residual standard score zscore) as the business abnormality feature, that is, determining the business abnormality feature through the extracted standard score, the standard score zscore is a quantity with equal units. It is the quotient obtained by dividing the difference between the original score and the mean of the group by the standard deviation, and measures how many standard deviations above the original score from its mean, or how many standard deviations below the mean, the original score is in units of standard deviation.
[0054] Through the data processing of step S101-step S102 and the construction of the spatiotemporal correlation distribution model, the spatiotemporal similarity is eliminated by the residual standard score of the model, so the abnormal operation characteristics determined according to the extracted standard score can reflect the operation status of the store itself. That is, the abnormal operation characteristics are extracted based on the reference operation data, including: extracting the standard score from the distribution model composed of the reference operation data; determining the abnormal operation characteristics according to the extracted standard score.
[0055] Furthermore, before performing density clustering operations, the business characteristic data and business abnormality characteristics are processed into two-dimensional data.
[0056] Preferably, in one embodiment of the present invention, the stacked autoencoder network model is used to process the business feature data of the plurality of nodes into two-dimensional feature data; and / or the business abnormality feature is processed into two-dimensional feature data. It is understandable that the two-dimensional feature data can be directly visualized to improve the user experience of managing business data.
[0057] Furthermore, before using the stacked autoencoder network model, the stacked autoencoder network model is constructed; specifically, the number of middle layers of the stacked autoencoder network model is determined according to the time interval contained in the set space-time range; for example: the time range of the business characteristic data of the analyzed node is one month, and one month is divided into four time intervals, then the number of middle layers of the stacked autoencoder network model can be set to 4; by setting the number of middle layers of the stacked autoencoder network model to be consistent with the time interval, the accuracy and consistency of processed data are improved.
[0058] Furthermore, the number of neurons in the middle layer of the stacked autoencoder network model is set to two dimensions; by inputting operating characteristic data and / or operating abnormality characteristics into the stacked autoencoder network model, two-dimensional feature data output by the model is obtained; by setting the number of neurons in the middle layer to two dimensions, the consumption of computing resources is saved and the efficiency of data processing is improved.
[0059] Further, the stacked autoencoder network model is constructed based on the middle layer of the number of layers; and in the process of performing dimensionality reduction on the feature data using the stacked autoencoder network model, multiple iterations are performed until the optimized target two-dimensional feature data is obtained; that is, the stacked autoencoder network model is used to process the business feature data of multiple nodes into two-dimensional feature data, and / or the business abnormality feature is processed into two-dimensional feature data, further comprising: looping step N1 until the target two-dimensional feature data is determined: N1: After the business feature data of multiple nodes are processed into two-dimensional feature data, and / or the business abnormality feature is processed into two-dimensional feature data using the stacked autoencoder network model, the processing error is evaluated using an activation function, and when it is determined that the processing error meets the set error condition, the determined two-dimensional feature data is the target two-dimensional feature data. Specifically, the processing (e.g., dimensionality reduction, output of two-dimensional feature data, etc.) error is evaluated using an activation function, and when it is determined that the training error meets the set error condition, the optimized target two-dimensional feature data is determined. Among them, the processing error can be evaluated using an activation function (e.g., tanh function, etc.), and the set error condition can be associated with the decay rate of the error value. During the training process, the error will continue to decrease, and the error can be plotted into a graph. For example, the error inflection point of the graph can be selected as the set error condition, that is, when it is determined that the training error meets the set error condition, the stacked autoencoder network model is used to determine the optimized target two-dimensional feature data. By evaluating the processing error, the optimal target two-dimensional feature data is obtained through the stacked autoencoder network model, further improving the accuracy of the processed data.
[0060] Furthermore, clustering is performed based on density clustering according to the obtained two-dimensional feature data; that is, using a preset density clustering model, the business feature data and the business abnormality features of each of the nodes with spatiotemporal correlation, abnormal data points are determined for each of the nodes with spatiotemporal correlation.
[0061] Preferably, in one embodiment of the present invention, the density clustering model is optimized to obtain a preset density clustering model; the specific optimization method is to perform iterative clustering operations based on the original density clustering model, that is, to perform clustering operations multiple times using the original density clustering model, the business characteristic data of each of the item providers, and the two-dimensional characteristic data corresponding to the business abnormality characteristics; to evaluate the clustering effect each time using the silhouette coefficient, and to adjust the parameters of the original density clustering model accordingly to obtain the preset density clustering model. Among them, the silhouette coefficient is an indicator for evaluating the quality of clustering effect. It combines the two factors of cohesion and separation and can be used to evaluate the impact of different algorithms or different operation modes of algorithms on clustering results based on the same original data; according to the clustering effect evaluated by the silhouette coefficient, the minimum radius and the minimum number of samples in the radius in the parameter space are adjusted accordingly, so as to optimize one or more parameters to obtain the preset density clustering model; and to determine abnormal data points for each of the nodes with spatiotemporal correlation using the preset density clustering model.
[0062] Step S104: providing the abnormal data point to a device associated with a node corresponding to the abnormal data point on the supply chain, so as to process the corresponding node based on the abnormal data point.
[0063] Specifically, the method of providing the abnormal data point to the device associated with the node corresponding to the abnormal data point on the supply chain may be: outputting one or more visualization pages displaying the abnormal data point to the device associated with the node corresponding to the abnormal data point, so as to display the information corresponding to the abnormal data point through the visualization page. The device associated with the node corresponding to the abnormal data point may be a client, which may belong to the node, the management party of the node, the superior management party of the node on the supply chain, etc.
[0064] Figure 3 Two schematic diagrams of visualized abnormal data points are shown, which intuitively display the abnormal data points in the business data in the form of a two-dimensional coordinate system, data points in the coordinate system, or broken lines, so that the supply chain management party can process the corresponding nodes based on the abnormal data points, such as further checking the nodes that generate the abnormal data points and analyzing the specific causes. The present invention does not limit the specific form and style of the visualized abnormal data points.
[0065] like Figure 2 As shown, an embodiment of the present invention provides a process of a method for processing data, which may include the following steps:
[0066] Step S201: Acquire the operating characteristic data of multiple nodes in the supply chain that are within a set time and space range, and analyze the time and space correlation between the operating characteristic data of each of the nodes.
[0067] Step S202: Determine whether there is a spatiotemporal correlation between the business characteristic data of each of the nodes. If yes, execute step S202; otherwise, execute step S205.
[0068] Step S203: constructing a spatiotemporal correlation distribution model for at least two of the nodes having spatiotemporal correlation, and calculating reference operating data of each of the nodes having spatiotemporal correlation through the spatiotemporal correlation distribution model.
[0069] Step S204: extracting abnormal operation features based on reference operation data.
[0070] Step S205: using a stacked autoencoder network model, processing the operating characteristic data of the plurality of nodes into two-dimensional data; and / or processing the operating abnormality characteristics into two-dimensional data.
[0071] Step S206: using a preset density clustering model, the business characteristic data and the business abnormality characteristics of each of the nodes with spatiotemporal correlation, to determine abnormal data points for each of the nodes with spatiotemporal correlation.
[0072] Steps S201 to S206 describe different processes executed in two cases of analyzing whether there is a corresponding spatiotemporal correlation between the business feature data of each node. It is understandable that, when the analysis result indicates that there is no spatiotemporal correlation between the business feature data, the stacked autoencoder network model included in the embodiment of the present invention can be directly used to process the business feature data of multiple nodes into two-dimensional data, and use the preset density clustering model, the business feature data of each node with spatiotemporal correlation, and the business abnormality characteristics to determine abnormal data points for each node with spatiotemporal correlation.
[0073] The specific description of processing the business characteristic data is consistent with the specific description of step S101 to step S103, and will not be repeated here.
[0074] like Figure 4 As shown, an embodiment of the present invention provides a device 400 for processing data, including: a correlation analysis module 401, a feature determination module 402 and an abnormal data determination module 403; wherein,
[0075] The correlation analysis module 401 is used to obtain the operating characteristic data of multiple nodes in the supply chain that are within a set time and space range, and analyze the time and space correlation between the operating characteristic data of each of the nodes;
[0076] The feature determination module 402 is used to construct a spatiotemporal correlation distribution model for the at least two nodes with spatiotemporal correlation when analyzing the business feature data of each node and finding that the business feature data of at least two nodes have spatiotemporal correlation, and calculate reference business data of each node with spatiotemporal correlation through the spatiotemporal correlation distribution model;
[0077] The abnormal data determination module 403 is used to extract abnormal business characteristics based on reference business data; use a preset density clustering model, the business characteristic data of each node with spatiotemporal correlation and the abnormal business characteristics to determine abnormal data points for each node with spatiotemporal correlation; provide the abnormal data points to the devices associated with the nodes corresponding to the abnormal data points on the supply chain, so as to process the corresponding nodes based on the abnormal data points.
[0078] An embodiment of the present invention also provides an electronic device for processing data, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in any of the above embodiments.
[0079] An embodiment of the present invention further provides a computer-readable medium on which a computer program is stored. When the program is executed by a processor, the method provided in any of the above embodiments is implemented.
[0080] Figure 5 An exemplary system architecture 500 is shown to which the method for processing data or the apparatus for processing data according to the embodiment of the present invention can be applied.
[0081] like Figure 5 As shown, system architecture 500 may include terminal devices 501, 502, 503, a network 504 and a server 505. Network 504 is used to provide a medium for communication links between terminal devices 501, 502, 503 and server 505. Network 504 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.
[0082] Users can use terminal devices 501, 502, 503 to interact with server 505 through network 504 to receive or send messages, etc. Various client applications can be installed on terminal devices 501, 502, 503, such as e-commerce client applications, web browser applications, search applications, instant messaging tools, and email clients.
[0083] The terminal devices 501 , 502 , and 503 may be various electronic devices having a display screen and supporting various client applications, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.
[0084] The server 505 may be a server that provides various services, such as a background management server that provides support for client applications used by users using the terminal devices 501, 502, and 503. The background management server may process the collected business characteristic data of multiple nodes and feed back abnormal data points obtained after processing to the terminal device.
[0085] It should be noted that the method for processing data provided in the embodiment of the present invention is generally executed by the server 505 , and accordingly, the device for processing data is generally disposed in the server 505 .
[0086] It should be understood that Figure 5 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.
[0087] Reference below Figure 6 , which shows a schematic diagram of the structure of a computer system 600 of a terminal device suitable for implementing an embodiment of the present invention. Figure 6 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.
[0088] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0089] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.
[0090] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present invention are executed.
[0091] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.
[0092] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0093] The modules and / or units involved in the embodiments of the present invention may be implemented in software or hardware. The modules and / or units described may also be provided in a processor, for example, they may be described as: a processor including a correlation analysis module, a feature determination module, and an abnormal data determination module; wherein the names of these modules do not constitute a limitation on the modules themselves in certain circumstances, for example, the correlation analysis module may also be described as a "module for analyzing the spatiotemporal correlation between the business feature data of each of the nodes".
[0094] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently and not assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: obtaining the business characteristic data of multiple nodes in the supply chain that belong to the set time and space range, and analyzing the time and space correlation between the business characteristic data of each of the nodes; when it is analyzed that the business characteristic data of at least two of the nodes in the business characteristic data of each of the nodes have time and space correlation, construct a time and space correlation distribution model for at least two of the nodes with time and space correlation, and calculate the reference business data of each of the nodes with time and space correlation through the time and space correlation distribution model; extract business abnormality characteristics based on the reference business data; use the preset density clustering model, the business characteristic data of each of the nodes with time and space correlation and the business abnormality characteristics to determine abnormal data points for each of the nodes with time and space correlation; provide the abnormal data points to the device associated with the node corresponding to the abnormal data point on the supply chain, so as to process the corresponding node based on the abnormal data points.
[0095] The embodiments of the present invention can analyze the spatiotemporal correlation between the operating characteristic data of each node on the supply chain based on the operating characteristic data of multiple nodes; construct a spatiotemporal correlation distribution model for the nodes with spatiotemporal correlation to calculate the reference operating data of each node with spatiotemporal correlation; and extract abnormal operating characteristics; use a preset density clustering model to determine abnormal data points for each node with spatiotemporal correlation; and process the corresponding nodes based on the abnormal data points. The embodiments of the present invention improve the precision and accuracy of processing and analyzing operating data.
[0096] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A method of processing data, It is characterized in that include: Acquire the operating characteristic data of multiple nodes in the supply chain that fall within a set time and space range, and analyze the time and space correlation between the operating characteristic data of each of the nodes; When it is found that the business characteristic data of at least two of the nodes have spatiotemporal correlation among the business characteristic data of the nodes, a spatiotemporal correlation distribution model is constructed for the at least two nodes having spatiotemporal correlation, and reference business data of each of the nodes having spatiotemporal correlation is calculated by using the spatiotemporal correlation distribution model; Extract abnormal operating characteristics based on reference operating data; Determine an abnormal data point for each of the nodes with spatiotemporal correlation by using a preset density clustering model, the business characteristic data of each of the nodes with spatiotemporal correlation, and the business abnormality characteristics; The abnormal data point is provided to a device associated with a node corresponding to the abnormal data point on the supply chain, so that the corresponding node is processed based on the abnormal data point.
2. The method according to claim 1, It is characterized in that The analyzing the spatiotemporal correlation between the business characteristic data of each of the nodes includes: The operating characteristic data of each node is converted into a corresponding characteristic matrix, and the probability value of the spatiotemporal correlation between the characteristic matrices converted from each node is calculated using a likelihood ratio test model; According to the comparison result between the probability value and the set probability threshold, it is determined whether the business characteristic data of at least two of the nodes have a temporal and spatial correlation.
3. The method according to claim 2, It is characterized in that The analyzing the spatiotemporal correlation between the business characteristic data of each of the nodes further includes: Constructing a distribution model for the business characteristic data of at least two of the nodes having spatiotemporal correlation; Performing a preset hypothesis test on the constructed distribution model; When the result of the preset hypothesis test indicates that the distribution model conforms to one or more preset distribution models, it is determined that the business characteristic data of at least two of the nodes have spatiotemporal correlation.
4. The method according to claim 1, It is characterized in that The step of constructing a spatiotemporal correlation distribution model for at least two nodes having spatiotemporal correlation comprises: For each of the nodes, perform the following operations: Calculating the multiple business characteristic factor corresponding to the node; Based on the distribution model constructed based on the multiple business characteristic factors and the business characteristic data of the node, a spatiotemporal correlation distribution model corresponding to the node is constructed.
5. The method according to claim 1, It is characterized in that The extracting of abnormal operation characteristics based on reference operation data includes: extracting a standard score from the distribution model formed by the reference operating data; Determine the abnormal operating characteristics based on the extracted standard scores.
6. The method according to claim 1 or 5, It is characterized in that The method further includes: utilizing a stacked autoencoder network model to process the operating feature data of the plurality of nodes into two-dimensional feature data, and / or to process the operating abnormality features into two-dimensional feature data.
7. The method according to claim 6, It is characterized in that Also includes: Determining the number of intermediate layers of the stacked autoencoder network model according to the time interval contained in the set spatiotemporal range; The number of neurons in the middle layer of the stacked autoencoder network model is set to two dimensions; To obtain two-dimensional feature data; construct the stacked autoencoder network model based on the middle layer of the number of layers; The method of using the stacked autoencoder network model to process the operating feature data of the plurality of nodes into two-dimensional feature data, and / or processing the operating abnormality features into two-dimensional feature data, further includes: Step N1 is executed repeatedly until the target two-dimensional feature data is determined: N1: After using the stacked autoencoder network model to process the operating feature data of the multiple nodes into two-dimensional feature data, and / or processing the operating abnormality features into two-dimensional feature numbers, an activation function is used to evaluate the processing error. When it is determined that the processing error meets the set error condition, the two-dimensional feature data is determined to be the target two-dimensional feature data.
8. The method according to claim 1, It is characterized in that Further including: Performing clustering operations multiple times using the original density clustering model, the business characteristic data of each of the item providers, and the two-dimensional characteristic data corresponding to the business abnormality characteristics; The silhouette coefficient is used to evaluate the clustering effect each time, and the parameters of the original density clustering model are adjusted accordingly to obtain the preset density clustering model.
9. A device for processing data, It is characterized in that include: Analyze correlation module, determine feature module and determine abnormal data module; among them, The correlation analysis module is used to obtain the operating characteristic data of multiple nodes in the supply chain that are within a set time and space range, and analyze the time and space correlation between the operating characteristic data of each of the nodes; The feature determination module is used to construct a spatiotemporal correlation distribution model for the at least two nodes with spatiotemporal correlation when analyzing the business feature data of each node and finding that the business feature data of at least two nodes have spatiotemporal correlation, and calculate the reference business data of each node with spatiotemporal correlation through the spatiotemporal correlation distribution model; The abnormal data determination module is used to extract abnormal business characteristics based on reference business data; use a preset density clustering model, the business characteristic data of each node with spatiotemporal correlation and the abnormal business characteristics to determine abnormal data points for each node with spatiotemporal correlation; provide the abnormal data points to the devices associated with the nodes corresponding to the abnormal data points on the supply chain, so as to process the corresponding nodes based on the abnormal data points.
10. An electronic device, It is characterized in that include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 8.
11. A computer readable medium having a computer program stored thereon, It is characterized in that When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.
Citation Information
Cited By
Intelligent supply chain management system and method based on artificial intelligence and big data
CN121581965A
