Abnormal data cleaning and elimination method and system for big data analysis
By clustering and eigenvalue analysis of multi-channel medical sales data, combined with CRITIC algorithm, the abnormal data change factor is obtained, which solves the problem that existing methods cannot accurately detect abnormal data, and improves the effectiveness of medical sales data cleaning.
Patent Information
- Application Number
- CN202411261903.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-10
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-09-10
AI Technical Summary
When existing methods deal with multi-channel pharmaceutical sales data, they cannot accurately detect abnormal data, resulting in a reduced cleaning effect of pharmaceutical sales data and the inability to effectively handle the mutual influence between multiple sales channels.
By obtaining the sales data sequence of the sales channels to be tested, performing cluster analysis, obtaining cluster clusters and feature time periods, calculating the time series discrete eigenvalues and data change eigenvalues, combining with the objective CRITIC empowerment algorithm, obtaining the data abnormal change factor, and performing Abnormal detection and cleaning.
It improves the accuracy of abnormal sales data detection and data cleaning effect, eliminates the interference of chain influencing factors on abnormal data detection, and enhances the accuracy of data cleaning.
Smart Images

Figure CN118797243B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data cleaning, and in particular to a method and system for cleaning and eliminating abnormal data for big data analysis. Background Art
[0002] By analyzing the sales data of different types of medicines through the big data analysis platform, it is possible to timely discover the changing trends of sales data of different types of medicines and make targeted adjustments to the production of different pharmaceutical products. However, in the process of acquiring, transmitting and storing medicine sales data, some data will inevitably be biased. Therefore, it is necessary to clean the medicine sales data of the big data analysis platform to ensure the analysis accuracy of the platform.
[0003] In the related technology, anomaly detection algorithms such as Local Outlier Factor (LOF) or Isolation Forest (iForest) are usually used to detect anomalies in pharmaceutical sales data and clean the abnormal data. However, since various types of medicines are usually sold through multiple channels, the pharmaceutical sales data under multiple channels affect each other. Changes in the pharmaceutical sales data of a certain channel will affect the changes in the pharmaceutical sales data of other channels, resulting in the inability of existing methods to accurately detect abnormal sales data, thereby reducing the effect of cleaning pharmaceutical sales data. Summary of the invention
[0004] In order to solve the technical problem that the pharmaceutical sales data from multiple channels influence each other, resulting in the inability of existing methods to accurately detect abnormal sales data, thereby reducing the effect of cleaning pharmaceutical sales data, the purpose of the present invention is to provide a method and system for cleaning and eliminating abnormal data for big data analysis. The technical solution adopted is as follows:
[0005] The present invention proposes a method for cleaning and eliminating abnormal data for big data analysis, the method comprising:
[0006] Obtaining a sales data sequence of the sales channel to be tested at each time point within a preset time period, wherein the sales data sequence includes sales data of different types of medicines;
[0007] Clustering the sales data sequences at all time points in the sales channel to be tested to obtain different clustering clusters and characteristic time periods of each clustering cluster; taking any clustering cluster as a target clustering cluster, and taking the sales data sequence of the characteristic time period belonging to the target clustering cluster in the sales channel to be tested as a reference sequence of the target clustering cluster; obtaining the time series discrete characteristic value of the target clustering cluster according to the difference in quantity between the reference sequence and the sales data sequence in the target clustering cluster; obtaining the data change characteristic value of each type of medicine in the target clustering cluster according to the difference in the change of sales data of the same type of medicine between each reference sequence and each sales data sequence in the target clustering cluster;
[0008] Obtaining the data abnormal change factor of each medicine type in the target cluster according to the difference in the data change characteristic value of the same medicine type between the target cluster and other clusters except the target cluster, and the difference in the time series discrete characteristic value;
[0009] Anomaly detection is performed on the sales data of the same medicine type in the target cluster, and the sales data sequence in the target cluster is cleaned in combination with the data abnormal change factor.
[0010] Furthermore, obtaining the data change characteristic value of each medicine type in the target cluster includes:
[0011] Analyze the discrete degree of sales data of each type of medicine in the sales data series at all time points in the target cluster, and the correlation between the sales data of each type of medicine, to obtain a first data change parameter of each type of medicine in the target cluster;
[0012] Analyze the degree of dispersion of the sales data of each type of medicine in the reference sequence at all time points, and the correlation between the sales data of each type of medicine, to obtain a second data change parameter of each type of medicine in the characteristic time period of the target cluster in the sales channel to be tested;
[0013] According to the difference between the first data change parameter and the second data change parameter of the same medicine type, a data change characteristic value of each medicine type in the target cluster is obtained.
[0014] Furthermore, obtaining the first data change parameter of each medicine type in the target cluster includes:
[0015] Constructing a first state matrix, wherein the rows of the first state matrix represent time points, the columns of the first state matrix represent drug types, and the row elements of the first state matrix are sales data sequences at each time point in the target cluster;
[0016] The first state matrix is input into the CRITIC objective weighting algorithm, and the objective weight value of each column output is used as the first data change parameter of each medicine type in the target cluster.
[0017] Furthermore, the step of obtaining the second data change parameter of each medicine type of the sales channel to be tested in the characteristic time period of the target cluster includes:
[0018] Constructing a second state matrix, wherein the rows of the second state matrix represent time points, the columns of the second state matrix represent drug types, and the row elements of the second state matrix are reference sequences at each time point;
[0019] The second state matrix is input into the CRITIC objective weighting algorithm, and the objective weight value of each column of the output is used as the second data change parameter of each medicine type in the characteristic time period of the sales channel to be tested in the target cluster.
[0020] Furthermore, the step of obtaining the data abnormal change factor of each medicine type in the target cluster includes:
[0021] Other clusters except the target cluster are used as reference clusters;
[0022] According to the difference of the time series discrete characteristic value between the target clustering cluster and each reference clustering cluster, obtaining the first characteristic value difference between the target clustering cluster and each reference clustering cluster;
[0023] According to the difference of the data change characteristic value of the same medicine type between the target cluster and each reference cluster, obtaining the second characteristic value difference of each medicine type between the target cluster and each reference cluster;
[0024] The first eigenvalue difference and the second eigenvalue difference are integrated to obtain the integrated eigenvalue difference of each medicine type between the target cluster and each reference cluster;
[0025] After analyzing the overall level of the comprehensive characteristic value differences of each medicine type between the target cluster and all reference clusters and performing normalization processing, the data abnormal change factor of each medicine type in the target cluster is obtained, wherein the sum of the data abnormal change factors of all medicine types in the target cluster is equal to the value 1.
[0026] Furthermore, the anomaly detection of the sales data of the same medicine type in the target cluster and the cleaning of the sales data sequence in the target cluster in combination with the data anomaly change factor include:
[0027] Input the sales data of the same medicine type in all sales data sequences in the target cluster into the anomaly detection algorithm to obtain the anomaly evaluation value of the sales data of each medicine type in each sales data sequence in the target cluster;
[0028] Using the data abnormal change factor of each medicine type in the target cluster, weighted sum and normalization are performed on the abnormal evaluation value of the sales data of each medicine type in each sales data sequence in the target cluster to obtain a cleaning judgment value of each sales data sequence in the target cluster;
[0029] Based on the cleaning judgment value, the sales data sequence in the target cluster is cleaned.
[0030] Furthermore, the cleaning of the sales data sequence in the target cluster based on the cleaning judgment value includes:
[0031] In the target cluster, the sales data series whose cleaning judgment value is greater than the preset cleaning threshold are deleted.
[0032] Furthermore, clustering the sales data series at all time points in the sales channel to be tested to obtain different clusters and the characteristic time period of each cluster includes:
[0033] Based on the differences in the pharmaceutical data of the same pharmaceutical type in each sales data sequence in the sales channel to be tested, all sales data sequences are clustered to obtain different clusters;
[0034] In each cluster, the time period between the minimum and maximum values of the time points corresponding to all sales data series is used as the characteristic time period of each cluster.
[0035] Furthermore, the step of obtaining the time series discrete feature value of the target cluster according to the difference in quantity between the reference sequence and the sales data sequence in the target cluster includes:
[0036] The number of all reference sequences is used as the numerator, the number of all sales data sequences in the target cluster is used as the denominator, and the ratio is used as the time series discrete feature value of the target cluster.
[0037] The present invention also proposes an abnormal data cleaning and elimination system for big data analysis, the system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor implementing any step of the abnormal data cleaning and elimination method for big data analysis when executing the computer program.
[0038] The present invention has the following beneficial effects:
[0039] The present invention takes into account that the pharmaceutical sales data under various sales channels influence each other, resulting in the inability of existing methods to accurately detect abnormal sales data, thereby reducing the effect of cleaning pharmaceutical sales data. Therefore, the present invention first obtains the sales data sequence of the sales channel to be tested at each time point within a preset time period. Taking into account that abnormal factors and factors affecting each other between different sales channels will cause changes in sales data, the present invention first clusters the pharmaceutical sales data sequence to obtain multiple cluster clusters and a characteristic time period of each cluster cluster, and uses the sales data sequence of the characteristic time period of the target cluster cluster in the sales channel to be tested as a reference sequence. When the target cluster cluster contains a sales data sequence affected by these two factors, the reference sequence and the sales data sequence in the target cluster cluster are compared. There is a large difference in quantity between the data sequences, and there is also a large difference in the changes in the sales data of the same type of medicine between the two. Therefore, the possibility of the affected sales data sequence in the target cluster can be reflected by the time series discrete eigenvalues and the data change eigenvalues. Considering that when there is abnormal data in the target cluster, the difference in the time series discrete eigenvalues between the target cluster and other clusters and the difference in the data change eigenvalues of the same type of medicine will increase, the data abnormal change factor can be used to reflect the possibility of abnormal sales data of each type of medicine in the target cluster. Then, combined with the data abnormal change factor, the sales data sequence in the target cluster is cleaned to improve the accuracy of abnormal sales data detection and the effect of sales data cleaning. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0041] Figure 1 A flow chart of a method for cleaning and eliminating abnormal data for big data analysis provided by one embodiment of the present invention;
[0042] Figure 2 A flow chart of a method for obtaining data change characteristic values of each medicine type in a target cluster provided by an embodiment of the present invention;
[0043] Figure 3 A flow chart of a method for obtaining an abnormal change factor of data for each type of medicine in a target cluster provided by an embodiment of the present invention;
[0044] Figure 4 A flow chart of a method for cleaning a sales data sequence in a target cluster provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0045] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of the abnormal data cleaning and elimination method and system for big data analysis proposed by the present invention, its specific implementation method, structure, features and effects, in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.
[0046] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.
[0047] The specific scheme of the abnormal data cleaning and elimination method and system for big data analysis provided by the present invention is described in detail below with reference to the accompanying drawings.
[0048] See also Figure 1 , which shows a flow chart of a method for cleaning and eliminating abnormal data for big data analysis provided by an embodiment of the present invention, the method comprising:
[0049] Step S1: obtaining a sales data sequence of the sales channel to be tested at each time point within a preset time period, wherein the sales data sequence includes sales data of different types of medicines.
[0050] By analyzing the sales data of different types of medicines through the big data analysis platform, it is possible to timely discover the changing trends of the sales data of different types of medicines, and make targeted adjustments to the production of different pharmaceutical products. However, in the process of obtaining, transmitting and storing the medicine sales data, some data will inevitably be biased. Therefore, it is necessary to clean the medicine sales data of the big data analysis platform to ensure the analysis accuracy of the platform. In related technologies, anomaly detection algorithms such as local anomaly anomaly factor algorithm or isolated forest algorithm are usually used to detect anomalies in medicine sales data and clean the abnormal data. However, since various types of medicines are usually sold through multiple channels, the medicine sales data under multiple channels affect each other. Changes in the medicine sales data of a certain channel will affect the changes in the medicine sales data of other channels. The changes in sales data caused by this factor are normal changes, which will affect the detection of abnormal data, resulting in the inability of existing methods to accurately detect abnormal sales data, thereby reducing the effect of cleaning medicine sales data. Therefore, an abnormal data cleaning and elimination method for big data analysis is proposed in an embodiment of the present invention to solve this problem.
[0051] Since medicines are sold through a variety of channels during the sales process, such as Internet e-commerce sales channels, DTP pharmacy sales channels, and traditional pharmacy sales channels, etc., the embodiment of the present invention selects the Internet e-commerce sales channel for analysis, and takes the Internet e-commerce sales channel as the sales channel to be tested. First, the sales data sequence of the sales channel to be tested at each time point within a preset time period is obtained in the big data analysis platform, wherein the data acquisition interval ranges from an integer of 1 to 3, and the unit is day. In one embodiment of the present invention, the data acquisition interval is set to 1 day, that is, every other day is a time point, and the preset time period ranges from an integer of 1 to 5, and the unit is year. In one embodiment of the present invention, the preset time period is set to 1 year. The specific values of the preset time period and the data acquisition interval can also be set by the implementer according to the specific implementation scenario, which is not limited here.
[0052] For each sales data sequence, it contains sales data of different types of medicines. The number of medicine types needs to be greater than 1, which can be defined by the implementer and is not limited here. For example, medicine types can be divided into two types: Chinese medicine and Western medicine. In this case, the sales data sequence of a certain day can be, for example, , indicating that the number of Chinese and Western medicines sold through the sales channel to be tested on that day was 50 and 60 respectively, or the types of medicines are divided into multiple types according to the therapeutic effects: cold medicine, antipyretic medicine, headache medicine and sleeping pills, etc. At this time, the sales data series of a certain day can be, for example, , indicating that the quantities of cold medicines, antipyretics, headache medicines, and sleeping pills sold through the sales channel to be tested on that day were 10, 5, 0, and 20, respectively.
[0053] It should be noted that the sales data at the same position in different sales data sequences correspond to the same type of medicine, which is convenient for subsequent calculation and analysis.
[0054] Step S2: Cluster the sales data sequences of all time points in the sales channel to be tested to obtain different clustering clusters and the characteristic time period of each clustering cluster; take any clustering cluster as the target clustering cluster, and take the sales data sequence of the characteristic time period of the target clustering cluster in the sales channel to be tested as the reference sequence of the target clustering cluster; obtain the time series discrete characteristic value of the target clustering cluster according to the difference in quantity between the reference sequence and the sales data sequence in the target clustering cluster; obtain the data change characteristic value of each type of medicine in the target clustering cluster according to the difference in the change of sales data of the same type of medicine between each reference sequence and each sales data sequence in the target clustering cluster.
[0055] Since medicines are sold through multiple channels during the sales process, the sales data of medicines between various sales channels are mutually influential. Changes in the sales data of medicines in a certain channel will affect changes in the sales data of medicines in other channels. Such factors that influence each other among multiple channels involve multiple levels such as market, region, consumer demand and disease susceptibility, and are objectively existing influencing factors. In the subsequent steps, such influencing factors are referred to as chain influencing factors. Changes in sales data caused by chain influencing factors are normal phenomena, while changes in sales data caused by abnormalities in the acquisition, transmission and storage of sales data are abnormal phenomena. Both chain influencing factors and abnormal factors can cause changes in sales data. Therefore, the embodiment of the present invention first clusters the sales data sequences at all time points in the sales channel to be tested to obtain different clustering clusters and characteristic time periods of each clustering cluster. Subsequently, based on the difference in the number of sales data sequences in the clustering cluster and the sales data sequences in the characteristic time period of the clustering cluster in the sales channel to be tested, as well as the difference in the change of sales data of the same type of medicine between the two, the possibility of the existence of sales data sequences affected by chain influencing factors and abnormal factors in each clustering cluster can be analyzed.
[0056] Preferably, in one embodiment of the present invention, the method for acquiring different clusters and characteristic time periods of each cluster specifically includes:
[0057] Based on the differences in the pharmaceutical data of the same pharmaceutical type in each sales data sequence in the sales channel to be tested, all sales data sequences are clustered to obtain different clustering clusters.
[0058] In an embodiment of the present invention, the Euclidean distance or Manhattan distance between each sales data series can be used as a distance metric, and all sales data series can be clustered using existing clustering algorithms such as an agglomerative hierarchical clustering algorithm or a k-means clustering algorithm, thereby obtaining multiple cluster clusters, which are not limited here.
[0059] In each cluster, the time period between the minimum and maximum values of the time points corresponding to all sales data sequences is taken as the characteristic time period of each cluster. For example, if a cluster contains sales data sequences on the 1st, 2nd, 5th and 9th days, the time period from the 1st to the 9th day is taken as the characteristic time period of the cluster.
[0060] Considering that each cluster may have sales data sequences affected by both chain effects and abnormal factors, it is necessary to analyze any cluster in the subsequent process. First, any cluster is taken as the target cluster. When the target cluster has sales data sequences affected by both chain effects and abnormal factors, the distribution of the sales data sequences in the target cluster in the characteristic time period has a discrete characteristic in time series, which is specifically manifested in that the sales data sequences in the target cluster and the sales data sequences in the characteristic time period of the target cluster in the sales channel to be tested have a large difference in quantity. At the same time, the distribution of sales data of the same type of medicine in each sales data sequence in the target cluster also has a discrete characteristic, which is specifically manifested in that the sales data sequences in the target cluster and There are large differences in the changes in sales data of the same type of medicine between the sales data sequences in the characteristic time period belonging to the target cluster in the sales channel to be tested. Therefore, the sales data sequence in the characteristic time period belonging to the target cluster in the sales channel to be tested can be used as the reference sequence of the target cluster. For example, if the characteristic time period of the target cluster is from the 1st day to the 9th day, the 9 sales data sequences belonging to the 1st day to the 9th day in the sales channel to be tested will be used as the reference sequence. Subsequently, based on the difference in the number of sales data sequences between the reference sequence and the target cluster and the difference in the changes in sales data of the same type of medicine, the possibility of the existence of sales data sequences in the target cluster affected by these two factors can be analyzed, thereby eliminating the influence of chain influencing factors on abnormal data detection and improving the effect of data cleaning.
[0061] From the above analysis, it can be seen that when there are sales data sequences affected by both chain influence factors and abnormal factors in the target clustering cluster, the quantity difference between the sales data sequences in the target clustering cluster and the sales data sequences in the characteristic time period of the target clustering cluster in the sales channel to be tested is large. Therefore, the difference in quantity between the reference sequence and the sales data sequences in the target clustering cluster can be analyzed. The obtained time series discrete eigenvalue reflects the discreteness of the time series distribution of the sales data sequences in the target clustering cluster. The larger the time series discrete eigenvalue, the more discrete the time series distribution of the sales data sequences in the target clustering cluster, and further indicates that the target clustering cluster is more likely to have sales data sequences affected by both chain influence factors and abnormal factors. Subsequently, the difference in time series discrete eigenvalues between the target clustering cluster and other clustering clusters can be further analyzed to eliminate the influence of chain influence factors on abnormal data detection.
[0062] Preferably, in one embodiment of the present invention, the method for obtaining the time series discrete feature value of the target cluster specifically includes:
[0063] The number of all reference sequences is used as the numerator, the number of all sales data sequences in the target cluster is used as the denominator, and the ratio is used as the time series discrete feature value of the target cluster.
[0064] It should be noted that, since the distribution characteristics of the sales data series in the target cluster may be discrete in time series, the number of reference sequences is always greater than or equal to the number of sales data series in the target cluster.
[0065] The time series discrete feature value of each cluster can be obtained by the same method as above.
[0066] From the above analysis, it can be seen that when there are sales data sequences affected by both chain influence factors and abnormal factors in the target clustering cluster, there is a large difference in the change of sales data of the same type of medicine between the sales data sequence in the target clustering cluster and the sales data sequence in the characteristic time period belonging to the target clustering cluster in the sales channel to be tested. Therefore, the difference in the change of sales data of the same type of medicine between each reference sequence and each sales data sequence in the target clustering cluster can be analyzed. The obtained data change characteristic value reflects the discreteness of the sales data of each type of medicine in the target clustering cluster. The larger the data change characteristic value, the more discrete the distribution of sales data of each type of medicine in the target clustering cluster, and the greater the degree of influence of these two factors on the sales data of each type of medicine in the target clustering cluster, which further indicates that the target clustering cluster is more likely to have sales data sequences affected by both chain influence factors and abnormal factors. Subsequently, the difference in data change characteristic values between the target clustering cluster and other clustering clusters can be further analyzed to eliminate the influence of chain influence factors on abnormal data detection.
[0067] Preferably, in one embodiment of the present invention, the method for obtaining the data change characteristic value of each medicine type in the target cluster specifically includes:
[0068] See also Figure 2 , which shows a flow chart of a method for obtaining data change characteristic values of each medicine type in a target cluster provided by an embodiment of the present invention.
[0069] Step S201: Analyze the discrete degree of sales data of each medicine type in the sales data sequence of all time points in the target cluster, and the correlation between the sales data of each medicine type, to obtain the first data change parameter of each medicine type in the target cluster.
[0070] First, it is necessary to analyze the changes in the sales data of each type of medicine in all sales data sequences in the target clustering cluster. Considering that not only the sales data of the same type of medicine has discrete characteristics in time series, but also different types of medicine have certain correlations, the embodiment of the present invention analyzes the discreteness of the sales data of each type of medicine in the sales data sequence of all time points in the target clustering cluster, as well as the correlation between the sales data of each type of medicine. The first data change parameter obtained reflects the change characteristics of the sales data of each type of medicine in the target clustering cluster. The larger the first data change parameter, the more significant the change in the sales data of each type of medicine in the target clustering cluster relative to other types of medicine, thereby providing a data basis for the subsequent calculation of the data change characteristic value of each type of medicine in the target clustering cluster.
[0071] Preferably, in one embodiment of the present invention, the method for obtaining the first data change parameter of each medicine type in the target cluster specifically includes:
[0072] Construct the first state matrix. The rows of the first state matrix represent time points, the columns of the first state matrix represent drug types, and the row elements of the first state matrix are the sales data sequences at each time point in the target cluster. For example, if there are three types of drugs, the sales data sequences on the first, second, and fifth days in the target cluster are , and , then the first state matrix is:
[0073] ,
[0074] Since the CRITIC objective weighting algorithm can comprehensively consider the data discreteness of each indicator and the correlation between the indicators, and analyze the data change of each indicator, the first state matrix can be input into the CRITIC objective weighting algorithm, and the objective weight value of each column of the output can be used as the first data change parameter of each type of medicine in the target cluster. The CRITIC objective weighting algorithm is a technical means well known to technicians in this field and will not be elaborated here.
[0075] Step S202: Analyze the degree of dispersion of the sales data of each type of medicine in the reference sequence of all time points, and the correlation between the sales data of each type of medicine, to obtain the second data change parameter of each type of medicine in the characteristic time period of the target cluster in the sales channel to be tested.
[0076] Similarly, it is necessary to analyze the changes in sales data of each type of medicine in all reference sequences, and reflect the change characteristics of the sales data of each type of medicine in the reference sequence by obtaining the second data change parameter. The larger the second data change parameter, the more significant the change in sales data of each type of medicine in the reference sequence relative to other types of medicine. Subsequently, based on the difference between the first data change parameter and the second data change parameter of the same type of medicine, the data change characteristic value of each type of medicine in the target cluster can be accurately calculated.
[0077] Preferably, in one embodiment of the present invention, the method for acquiring the second data change parameter of each medicine type in the sales channel to be tested during the characteristic time period of the target cluster specifically includes:
[0078] Similar to the calculation of the first data change parameter of each medicine type described above, a second state matrix is first constructed, wherein the rows of the second state matrix represent time points, the columns of the second state matrix represent medicine types, and the row elements of the second state matrix are reference sequences for each time point.
[0079] Then, the second state matrix is input into the CRITIC objective weighting algorithm, and the objective weight value of each column of the output is used as the second data change parameter of each medicine type in the characteristic time period of the sales channel to be tested in the target cluster.
[0080] Step S203: obtaining a data change characteristic value of each medicine type in the target cluster according to the difference between the first data change parameter and the second data change parameter of the same medicine type.
[0081] The greater the difference between the first data change parameter and the second data change parameter of the same medicine type, the greater the difference in the change in sales data of the same medicine type between the sales data sequence in the target cluster and the sales data sequence in the characteristic time period belonging to the target cluster in the sales channel to be tested, which further indicates that the target cluster is more likely to contain sales data sequences affected by both chain influence factors and abnormal factors. Therefore, the difference between the first data change parameter and the second data change parameter of the same medicine type can be used to obtain the data change characteristic value of each medicine type in the target cluster.
[0082] In one embodiment of the present invention, the absolute value of the difference between the first data change parameter of each type of medicine in the target cluster and the second data change parameter of each type of medicine in the sales channel to be tested during the characteristic time period of the target cluster can be used as the data change characteristic value of each type of medicine in the target cluster to analyze the difference between the two.
[0083] The expression of the data change characteristic value of each medicine type in the target cluster can be specifically, for example, as follows:
[0084] ,
[0085] in, Indicates the first The data change characteristic values of the medicine types; Indicates the first A first data change parameter of a medicine type; Indicates the sales channel to be tested in the characteristic time period of the target cluster. A second data change parameter of a medicine type.
[0086] The data change characteristic value of each medicine type in each cluster can be obtained by the same method as above.
[0087] Step S3: According to the difference in data change characteristic values of the same medicine type between the target cluster and other clusters except the target cluster, and the difference in time series discrete characteristic values, the data abnormal change factor of each medicine type in the target cluster is obtained.
[0088] The time series discrete eigenvalues and data change eigenvalues obtained through the above process can reflect the possibility that there are sales data sequences in the target cluster cluster that are affected by chain influencing factors and abnormal factors. However, the change in sales data caused by chain influencing factors is a normal phenomenon, and the change that occurs when the sales data is affected by abnormal factors during the acquisition, transmission and storage process is an abnormal phenomenon. Therefore, if the sales data sequence in the target cluster cluster is only detected for anomalies based on the time series discrete eigenvalues and data change eigenvalues obtained above, normal sales data will be mistaken for abnormal data, reducing the effect of data cleaning.
[0089] Considering that the influence range of chain influencing factors is large, they can cause changes in sales data in all clusters, while the influence range of abnormal factors is small, which can only cause changes in sales data in some clusters. That is to say, in the ideal case when only chain influencing factors exist, the difference in data change characteristic values of the same medicine type between the target cluster and other clusters is small, and the difference in time series discrete characteristic values between the target cluster and other clusters is also small. When abnormal factors cause sales data changes, these two differences will increase. Therefore, the difference in data change characteristic values of the same medicine type between the target cluster and other clusters except the target cluster, as well as the difference in time series discrete characteristic values can be analyzed. The data abnormal change factor obtained reflects the possibility of abnormal changes in the sales data of each medicine type in the target cluster due to the influence of abnormal factors. Subsequently, based on the data abnormal change factor, the interference of chain influencing factors on abnormal data detection can be eliminated to improve the effect of data cleaning.
[0090] Preferably, in one embodiment of the present invention, the method for obtaining the data abnormal change factor of each medicine type in the target cluster specifically includes:
[0091] See also Figure 3 , which shows a flow chart of a method for obtaining the data abnormal change factor of each medicine type in a target cluster provided by an embodiment of the present invention.
[0092] Step S301: Take clusters other than the target cluster as reference clusters, and obtain the first eigenvalue difference between the target cluster and each reference cluster according to the difference in the time series discrete eigenvalues between the target cluster and each reference cluster; obtain the second eigenvalue difference of each medicine type between the target cluster and each reference cluster according to the difference in the data change eigenvalue of the same medicine type between the target cluster and each reference cluster.
[0093] From the above analysis, it can be seen that when there are abnormal factors causing sales data changes, the time series discrete eigenvalues between the target cluster and other clusters will be greatly different, and the data change eigenvalues of the same medicine type between the target cluster and other clusters will also be greatly different. Therefore, other clusters except the target cluster can be used as reference clusters first, and the first eigenvalue difference between the target cluster and each reference cluster is obtained according to the difference in the time series discrete eigenvalue between the target cluster and each reference cluster. The larger the difference in the first eigenvalue, the greater the difference in the time series discrete eigenvalue between the target cluster and each reference cluster, which further indicates that The greater the degree to which the sales data in the target cluster is affected by abnormal factors, the second eigenvalue difference of each medicine type between the target cluster and each reference cluster is obtained based on the difference in data change characteristic values of the same medicine type between the target cluster and each reference cluster. The greater the difference in the second eigenvalue, the greater the difference in the data change characteristic value of the same medicine type between the target cluster and each reference cluster, which further indicates that the sales data of each medicine type in the target cluster is more affected by abnormal factors. Subsequently, the first eigenvalue difference and the second eigenvalue difference can be combined to accurately calculate the data abnormal change factor of each medicine type in the target cluster.
[0094] In an embodiment of the present invention, the absolute value of the difference between the time series discrete eigenvalues of the target clustering cluster and each reference clustering cluster or the square value of the difference can be used as the first eigenvalue difference between the target clustering cluster and each reference clustering cluster to analyze the difference between the two, which is not limited here.
[0095] In an embodiment of the present invention, the absolute value or the square value of the difference in data change characteristic values of the same medicine type between the target cluster and each reference cluster can be used as the second characteristic value difference of each medicine type between the target cluster and each reference cluster to analyze the difference between the two, which is not limited here.
[0096] Step S302: synthesizing the first eigenvalue difference and the second eigenvalue difference to obtain a comprehensive eigenvalue difference of each medicine type between the target cluster and each reference cluster.
[0097] Then, by combining the first eigenvalue difference and the second eigenvalue difference, the comprehensive eigenvalue difference of each medicine type between the target cluster and each reference cluster is obtained. The larger the comprehensive eigenvalue difference, the greater the possibility that the sales data of each medicine type in the target cluster is abnormal.
[0098] In an embodiment of the present invention, the product value or sum value of the first eigenvalue difference and the second eigenvalue difference can be used as the comprehensive eigenvalue difference of each medicine type between the target cluster and each reference cluster to achieve the integration of the two, which is not limited here.
[0099] Step S303: After analyzing the overall level of the comprehensive characteristic value difference of each medicine type between the target cluster and all reference clusters and performing normalization processing, the data abnormal change factor of each medicine type in the target cluster is obtained.
[0100] The greater the difference in the comprehensive characteristic value of each type of medicine between the target cluster and each reference cluster, the greater the degree to which the sales data of each type of medicine in the target cluster is affected by abnormal factors. Therefore, the overall level of the difference in the comprehensive characteristic value of each type of medicine between the target cluster and all reference clusters can be analyzed and normalized to obtain the data abnormal change factor of each type of medicine in the target cluster. The larger the data abnormal change factor, the greater the possibility that the sales data of each type of medicine in the target cluster is affected by abnormal factors and undergoes abnormal changes.
[0101] In an embodiment of the present invention, the overall level of analysis of the difference in comprehensive characteristic values of each medicine type between the target cluster and all reference clusters can be achieved by calculating the average or median of the difference in comprehensive characteristic values of each medicine type between the target cluster and all reference clusters, which is not limited here.
[0102] Since the data abnormal change factor is calculated and analyzed as a weight in the subsequent steps, after normalization processing, it is necessary to ensure that the sum of the data abnormal change factors of all medicine types in the target cluster is equal to the value 1. Therefore, the embodiment of the present invention first uses the results of the overall level analysis of the comprehensive characteristic value differences of each medicine type between the target cluster and all reference clusters as the initial abnormal change parameter of each medicine type in the target cluster, and uses the accumulated value of the initial abnormal change parameters of all medicine types in the target cluster as the denominator, the initial abnormal change parameter of each medicine type in the target cluster as the numerator, and the ratio as the data abnormal change factor of each medicine type in the target cluster, thereby realizing normalization processing.
[0103] The expression of the data abnormal change factor of each medicine type in the target cluster can be specifically, for example, as follows:
[0104] ,
[0105] ,
[0106] in, Indicates the first Abnormal change factors of data of different medicine types; Indicates the first Initial abnormal change parameters of the medicine type; Indicates the first Initial abnormal change parameters of the medicine type; Indicates the number of medicine types; Represents the target cluster and the The first eigenvalue difference between the reference clusters; Represents the target cluster and the The first The difference in the second eigenvalues of the three types of medicines; Represents the target cluster and the The first Differences in comprehensive characteristic values of the types of medicines; Indicates the number of reference clusters.
[0107] Step S4: Perform anomaly detection on the sales data of the same medicine type in the target cluster, and clean the sales data sequence in the target cluster in combination with the data abnormal change factor.
[0108] Traditional data cleaning methods usually use anomaly detection algorithms such as local anomaly anomaly factor algorithm or isolation forest algorithm to detect anomalies in pharmaceutical sales data and clean the abnormal data. However, traditional methods do not take into account the normal changes in sales data caused by chain influencing factors, and are prone to misdetecting normally changed data as abnormal data. The data abnormal change factor obtained through the above process can accurately reflect the possibility of abnormal changes in the sales data of each type of medicine in the target cluster due to abnormal factors, eliminating the interference of chain influencing factors. Therefore, the sales data of the same type of medicine in the target cluster can be detected for anomalies, and combined with the data abnormal change factor, the sales data sequence in the target cluster can be cleaned, thereby improving the accuracy of detecting abnormal sales data sequences in the target cluster, thereby improving the effect of data cleaning.
[0109] Preferably, in one embodiment of the present invention, the method for cleaning the sales data sequence in the target cluster specifically includes:
[0110] See also Figure 4 , which shows a flow chart of a method for cleaning a sales data sequence in a target cluster provided by an embodiment of the present invention.
[0111] Step S401: input the sales data of the same medicine type in all sales data sequences in the target cluster into the anomaly detection algorithm respectively, and obtain the anomaly evaluation value of the sales data of each medicine type in each sales data sequence in the target cluster.
[0112] Since each anomaly detection algorithm provides a parameter for evaluating data anomalies, for example, the local anomaly factor algorithm uses the local anomaly factor to evaluate data anomalies, and the isolation forest algorithm uses the anomaly score to evaluate data anomalies, the sales data of the same medicine type in all sales data sequences in the target cluster can be input into the anomaly detection algorithm respectively to obtain the anomaly evaluation value of the sales data of each medicine type in each sales data sequence in the target cluster. Subsequently, the anomaly evaluation value can be weighted adjusted using the data anomaly change factor of each medicine type in the target cluster to improve the accuracy of detecting abnormal sales data sequences.
[0113] In one embodiment of the present invention, a local anomaly factor algorithm is selected as the anomaly detection algorithm. In other embodiments of the present invention, for example, an isolation forest algorithm may also be selected, which is not limited here.
[0114] Step S402: using the data abnormal change factor of each medicine type in the target cluster, perform weighted summation and normalization on the abnormal evaluation values of the sales data of each medicine type in each sales data sequence in the target cluster to obtain the cleaning judgment value of each sales data sequence in the target cluster.
[0115] Since the larger the data abnormal change factor is, the more likely it is that the sales data of each type of medicine in the target clustering cluster will have abnormal changes, the data abnormal change factor of each type of medicine in the target clustering cluster can be used to perform weighted summation and normalization on the abnormal evaluation values of the sales data of each type of medicine in each sales data sequence in the target clustering cluster to obtain the cleaning judgment value of each sales data sequence in the target clustering cluster. The larger the cleaning judgment value is, the greater the possibility that each sales data sequence in the target clustering cluster has abnormal sales data, which further indicates that the sales data sequence needs to be cleaned.
[0116] In one embodiment of the present invention, the normalization process may specifically be, for example, maximum and minimum value normalization process. In other embodiments of the present invention, other normalization methods may be selected according to a specific range of values, which will not be described in detail.
[0117] The expression of the cleaning judgment value of each sales data series in the target cluster can be specifically, for example, as follows:
[0118] ,
[0119] in, Indicates the first The cleaning judgment value of a sales data series; Indicates the first Abnormal change factors of data of different medicine types; Indicates the first The first of the sales data series The abnormal evaluation value of sales data of a certain type of medicine; Indicates the number of medicine types; Represents the normalization function.
[0120] Step S403: based on the cleaning judgment value, the sales data sequence in the target cluster is cleaned.
[0121] The larger the cleaning judgment value is, the more it is necessary to clean the sales data sequence. Therefore, based on the cleaning judgment value, the sales data sequence in the target cluster is cleaned.
[0122] Preferably, in one embodiment of the present invention, the method for cleaning the sales data sequence in the target cluster further comprises:
[0123] In the target cluster, the sales data series whose cleaning judgment value is greater than the preset cleaning threshold are deleted. The preset cleaning threshold is set to 0.7. The specific value of the preset cleaning threshold can also be set by the implementer according to the specific implementation scenario and is not limited here.
[0124] The same method as above can be used to clean the sales data series in each cluster.
[0125] An embodiment of the present invention provides an abnormal data cleaning and elimination system for big data analysis, the system comprising a memory, a processor and a computer program, wherein the memory is used to store the corresponding computer program, the processor is used to run the corresponding computer program, and the computer program can implement the method described in steps S1 to S4 when running in the processor.
[0126] In summary, the embodiment of the present invention first obtains the sales data sequence of the sales channel to be tested at each time point within a preset time period, and the sales data sequence includes sales data of different types of medicines; clusters the sales data sequences at all time points in the sales channel to be tested to obtain different clustering clusters and characteristic time periods of each clustering cluster; takes any clustering cluster as the target clustering cluster, and takes the sales data sequence of the characteristic time period of the target clustering cluster in the sales channel to be tested as the reference sequence of the target clustering cluster; obtains the time series of the target clustering cluster according to the difference in quantity between the reference sequence and the sales data sequence in the target clustering cluster. Discrete eigenvalues; according to the differences in the changes in the sales data of the same medicine type between each reference sequence and each sales data sequence in the target cluster, obtain the data change eigenvalue of each medicine type in the target cluster; according to the differences in the data change eigenvalues of the same medicine type between the target cluster and other clusters except the target cluster, as well as the differences in the time series discrete eigenvalues, obtain the data abnormal change factor of each medicine type in the target cluster; perform anomaly detection on the sales data of the same medicine type in the target cluster, and clean the sales data sequence in the target cluster in combination with the data abnormal change factor.
[0127] It should be noted that the sequence of the above embodiments of the present invention is only for description and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0128] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.
Claims
1. A method for cleaning and eliminating abnormal data for big data analysis, characterized in that: The method comprises: Obtaining a sales data sequence of the sales channel to be tested at each time point within a preset time period, wherein the sales data sequence includes sales data of different types of medicines; Clustering the sales data sequences at all time points in the sales channel to be tested to obtain different clustering clusters and characteristic time periods of each clustering cluster; taking any clustering cluster as a target clustering cluster, and taking the sales data sequence of the characteristic time period belonging to the target clustering cluster in the sales channel to be tested as a reference sequence of the target clustering cluster; obtaining the time series discrete characteristic value of the target clustering cluster according to the difference in quantity between the reference sequence and the sales data sequence in the target clustering cluster; obtaining the data change characteristic value of each type of medicine in the target clustering cluster according to the difference in the change of sales data of the same type of medicine between each reference sequence and each sales data sequence in the target clustering cluster; Obtaining the data abnormal change factor of each medicine type in the target cluster according to the difference in the data change characteristic value of the same medicine type between the target cluster and other clusters except the target cluster, and the difference in the time series discrete characteristic value; Perform anomaly detection on the sales data of the same medicine type in the target cluster, and clean the sales data sequence in the target cluster in combination with the data abnormal change factor; The method of obtaining the abnormal change factor of the data of each medicine type in the target cluster includes: Other clusters except the target cluster are used as reference clusters; According to the difference of the time series discrete characteristic value between the target clustering cluster and each reference clustering cluster, obtaining the first characteristic value difference between the target clustering cluster and each reference clustering cluster; According to the difference of the data change characteristic value of the same medicine type between the target cluster and each reference cluster, obtaining the second characteristic value difference of each medicine type between the target cluster and each reference cluster; The first eigenvalue difference and the second eigenvalue difference are integrated to obtain the integrated eigenvalue difference of each medicine type between the target cluster and each reference cluster; After analyzing the overall level of the comprehensive characteristic value differences of each medicine type between the target cluster and all reference clusters and performing normalization processing, the data abnormal change factor of each medicine type in the target cluster is obtained, wherein the sum of the data abnormal change factors of all medicine types in the target cluster is equal to the value 1.
2. The method for cleaning and eliminating abnormal data for big data analysis according to claim 1, characterized in that: The step of obtaining the data change characteristic value of each medicine type in the target cluster includes: Analyze the discrete degree of sales data of each type of medicine in the sales data series at all time points in the target cluster, and the correlation between the sales data of each type of medicine, to obtain a first data change parameter of each type of medicine in the target cluster; Analyze the degree of dispersion of the sales data of each type of medicine in the reference sequence at all time points, and the correlation between the sales data of each type of medicine, to obtain a second data change parameter of each type of medicine in the characteristic time period of the target cluster in the sales channel to be tested; According to the difference between the first data change parameter and the second data change parameter of the same medicine type, a data change characteristic value of each medicine type in the target cluster is obtained.
3. The method for cleaning and eliminating abnormal data for big data analysis according to claim 2, characterized in that: The obtaining of the first data change parameter of each medicine type in the target cluster includes: Constructing a first state matrix, wherein the rows of the first state matrix represent time points, the columns of the first state matrix represent drug types, and the row elements of the first state matrix are sales data sequences at each time point in the target cluster; The first state matrix is input into the CRITIC objective weighting algorithm, and the objective weight value of each column output is used as the first data change parameter of each medicine type in the target cluster.
4. The method for cleaning and eliminating abnormal data for big data analysis according to claim 2, characterized in that: The step of obtaining the second data change parameter of each medicine type in the sales channel to be tested during the characteristic time period of the target cluster includes: Constructing a second state matrix, wherein the rows of the second state matrix represent time points, the columns of the second state matrix represent drug types, and the row elements of the second state matrix are reference sequences at each time point; The second state matrix is input into the CRITIC objective weighting algorithm, and the objective weight value of each column of the output is used as the second data change parameter of each medicine type in the characteristic time period of the sales channel to be tested in the target cluster.
5. The method for cleaning and eliminating abnormal data for big data analysis according to claim 1, characterized in that: The method of performing anomaly detection on the sales data of the same medicine type in the target cluster and cleaning the sales data sequence in the target cluster in combination with the data abnormal change factor includes: Input the sales data of the same medicine type in all sales data sequences in the target cluster into the anomaly detection algorithm to obtain the anomaly evaluation value of the sales data of each medicine type in each sales data sequence in the target cluster; Using the data abnormal change factor of each medicine type in the target cluster, weighted sum and normalization are performed on the abnormal evaluation value of the sales data of each medicine type in each sales data sequence in the target cluster to obtain a cleaning judgment value of each sales data sequence in the target cluster; Based on the cleaning judgment value, the sales data sequence in the target cluster is cleaned.
6. The method for cleaning and eliminating abnormal data for big data analysis according to claim 5, characterized in that: The step of cleaning the sales data sequence in the target cluster based on the cleaning judgment value includes: In the target cluster, the sales data series whose cleaning judgment value is greater than the preset cleaning threshold are deleted.
7. The method for cleaning and eliminating abnormal data for big data analysis according to claim 1, characterized in that: The step of clustering the sales data series at all time points in the sales channel to be tested to obtain different clusters and the characteristic time period of each cluster includes: Based on the differences in the pharmaceutical data of the same pharmaceutical type in each sales data sequence in the sales channel to be tested, all sales data sequences are clustered to obtain different clusters; In each cluster, the time period between the minimum and maximum values of the time points corresponding to all sales data series is used as the characteristic time period of each cluster.
8. The method for cleaning and eliminating abnormal data for big data analysis according to claim 1, characterized in that: The step of obtaining the time series discrete characteristic value of the target cluster according to the difference in quantity between the reference sequence and the sales data sequence in the target cluster comprises: The number of all reference sequences is used as the numerator, the number of all sales data sequences in the target cluster is used as the denominator, and the ratio is used as the time series discrete feature value of the target cluster.
9. An abnormal data cleaning and elimination system for big data analysis, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.
