A data cleaning method for clinical trial EDC system

Abstract

The present invention relates to the field of data processing technology, and more specifically, to a data cleaning method for a clinical trial EDC system. The method comprises collecting each data of each category, obtaining a number of local fluctuation degrees of each data in each category, obtaining an optimal search window for each data in each category according to the local fluctuation degrees, obtaining a final disconnection probability for each data in each category according to the skewness difference between each data in each category and the left and right data of its optimal search window, obtaining each breakpoint in each category according to the final disconnection probability of each data in each category, and then obtaining the number of bins for each category; constructing a histogram for each category according to the number of bins for each category, obtaining an abnormal score for each data in each category, and identifying abnormal points. The present invention improves the accuracy of abnormality detection results.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and more specifically, to a data cleaning method for a clinical trial EDC system. Background Art

[0002] Clinical trials are a key link in the research and development of new drugs, new therapies and medical devices. In order to ensure the scientificity and effectiveness of the test results, the data in the test process must be processed. The EDC system is an electronic data acquisition system, which is mainly used for online collection, storage and analysis of clinical trial data, such as: basic information of subjects, physiological indicators, treatment data, etc.; However, when the system collects blood pressure data of subjects, if an extremely high or low abnormal blood pressure value appears due to equipment failure or recording error, it affects the judgment of the treatment effect. Therefore, data cleaning becomes a key step to ensure the accuracy of clinical trial data.

[0003] The patent application document with the current authorization publication number CN116608904A discloses a real-time monitoring method and system for the safety status of a hydrogen storage tank, which collects the temperature data sequence and strain data sequence of the monitoring point in the hydrogen storage tank during the initial monitoring period; determines the compression time when the hydrogen in the hydrogen storage tank starts to be compressed based on the strain data sequence of the monitoring point in the initial monitoring period; calculates the first interval division number based on the temperature abnormality degree corresponding to the temperature data sequence after the compression moment; calculates the second interval division number based on the strain abnormality degree corresponding to the strain data sequence after the compression moment; uses the HBOS abnormality monitoring algorithm to perform abnormality monitoring on the temperature data sequence based on the first interval division number, and uses the HBOS abnormality monitoring algorithm to perform abnormality monitoring on the temperature data sequence based on the second interval division number.

[0004] Since the clinical trial EDC system has many data categories and presents different distribution patterns, especially there will be some data with relatively skewed distribution, and the traditional HBOS algorithm usually assumes that the data is evenly distributed and is very sensitive to the parameter of the number of bins, therefore, the conventional binning strategy adopted by the traditional HBOS algorithm when cleaning the collected data of each category may not be able to capture the data characteristics with skewed distribution, resulting in important intervals being too wide and unimportant intervals being too dense, which makes it easy to judge normal data as abnormal data and identify abnormal data as normal data. Summary of the invention

[0005] In order to solve the problem that the conventional binning strategy adopted by the HBOS algorithm when cleaning the data collected in each category may not capture the data characteristics with skewed distribution, resulting in important intervals being too wide and unimportant intervals being too dense, and normal data being easily judged as abnormal data, the present invention proposes a data cleaning method for a clinical trial EDC system, the method comprising the following steps:

[0006] Collect each data of each category; obtain the optimal search window for each data in each category;

[0007] Get each disconnection probability for each data in each category ; represents the h-th disconnection probability of the j-th data in the i-th category; Represents the skewness difference between the left and right sides of the h-th data set of the j-th data in the i-th category; represents the number of data in the optimal search window for the jth data in the i-th category; Represents the sum of the number of data in the hth data set on the left and the hth data set on the right of the jth data in the i-th category; norm() represents the normalization function; the minimum value of all disconnection probabilities of each data in each category is taken as the final disconnection probability of each data in each category;

[0008] According to the final disconnection probability of each data in each category, each breakpoint of each category is obtained; based on the breakpoint, the number of bins of each category is obtained; using the HBOS algorithm, according to the number of bins of each category, a histogram of each category is constructed; according to the histogram of each category, an anomaly score of each data in each category is obtained; based on the anomaly score, anomalies in each category are obtained, and the anomalies in each category are removed.

[0009] The innovation of the present invention lies in improving the strategy of fixing the number of bins in the HBOS algorithm. First, the optimal search window for each data in each category is obtained in combination with the local fluctuation degree of each data in each category, ensuring that there are points with large changes in the search window, which can effectively capture the local characteristics of the data. Then, according to the skewness of the data in the optimal search window of each data in each category, the final disconnection probability of each data in each category is obtained, and the breakpoint of each category is obtained based on the final disconnection probability, thereby obtaining the most suitable number of bins, improving the abnormal score evaluation of the data, and thus improving the accuracy of data cleaning of the clinical trial EDC system.

[0010] Preferably, it is characterized in that the obtaining of the optimal search window for each data in each category comprises:

[0011] Set the neighborhood data number variable u, and set u to [1, ], Represents the minimum value between the number of left and right data of the j-th data in the i-th category; for any u, obtain the local fluctuation degree of the j-th data in the i-th category;

[0012] Sort the local fluctuation degrees of the jth data in the i-th category in turn to obtain the local fluctuation degree sequence. Use the difference method to obtain the local maximum points in the local fluctuation degree sequence. The u value corresponding to the local maximum point with the largest value is recorded as ; Get the data before the jth data in the i-th category Data and later data as the optimal search window for the jth data in the i-th category.

[0013] Subsequently, the local fluctuation degree of each data in each category is combined to obtain the optimal search window for each data in each category, ensuring that there are points with large changes within the search window, which can effectively capture the local characteristics of the data.

[0014] Preferably, obtaining the local fluctuation degree of the j-th data in the i-th category includes:

[0015] For any u, obtain the u data before and the u data after the j-th data in the i-th category as the adjacent data of the j-th data in the i-th category;

[0016] ;

[0017] In the formula, Represents the local fluctuation degree of the jth data in the i-th category; Represents the value of the jth data in the i-th category; Represents the value of the vth adjacent data of the jth data in the i-th category; Represents the distance between the jth data in the i-th category and its vth adjacent data; represents the number of adjacent data of the jth data in the i-th category; exp() represents the exponential function with a natural constant as the base; norm() represents the normalization function; Represents the parameter that controls weight decay.

[0018] Preferably, obtaining the skewness difference of the h-th data set on the left and right sides of the j-th data in the i-th category includes:

[0019] Record all data between the jth data in the ith category and the hth data on the left side of its optimal search window as the hth data set on the left side of the jth data in the ith category; obtain the hth data set on the right side of the jth data in the ith category; obtain the skewness of the hth data set on the left side of the jth data in the ith category and the skewness of the hth data set on the right side of the jth data in the ith category; record the absolute value of the difference between the skewness of the hth data set on the left side of the jth data in the ith category and the skewness of the hth data set on the right side as the skewness difference between the hth data sets on the left and right sides of the jth data in the ith category.

[0020] This makes it easier to obtain the disconnection probability of each data in each category based on the skewness difference.

[0021] Preferably, obtaining each breakpoint of each category includes:

[0022] Preset disconnection probability threshold, if the final disconnection probability of the jth data in the i-th category is greater than the disconnection probability , the jth data in the i-th category is a breakpoint, and each breakpoint in the i-th category is obtained.

[0023] The resulting breakpoints can separate data with varying degrees of skewness.

[0024] Preferably, obtaining the number of bins for each category includes:

[0025] The number of breakpoints in the i-th category plus one is taken as the number of bins for the i-th category.

[0026] Preferably, obtaining the anomaly score of each data in each category includes:

[0027] ;

[0028] In the formula, Represents the anomaly score of the jth data in the i-th category; Represents the number of data in the box where the jth data in the i-th category is located in the histogram of the i-th category; The absolute value of the difference between the value of the jth data in the i-th category and the mean of all data in the bin in which it is located in the histogram of the i-th category; Represents the difference between the maximum and minimum values ​​of the data in the box where the jth data in the i-th category is located in the histogram of the i-th category; norm() represents the normalization function.

[0029] The anomaly score of the data is more accurate, which improves the accuracy of subsequent anomaly point identification.

[0030] Preferably, obtaining the outliers in each category includes:

[0031] Preset anomaly score threshold , if the anomaly score of the jth data in the i-th category is greater than the anomaly score threshold , the jth data in the i-th category is an outlier, and each outlier in the i-th category is obtained.

[0032] The present invention has the following beneficial effects: the purpose of the present invention is to obtain the local fluctuation degree of each data of each category, and based on the local fluctuation degree, obtain the optimal search window for each data in each category, ensure that there are points with large changes in the search window, and can effectively capture the local characteristics of the data, and then obtain the final disconnection probability of each data of each category according to the skewness of the data in the optimal search window of each data of each category, and obtain the breakpoint of each category according to the final disconnection probability, so as to obtain the most suitable number of bins, improve the abnormal score evaluation of the data, and thus improve the accuracy of data cleaning of the clinical trial EDC system. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] By reading the following detailed description with reference to the accompanying drawings, the above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0034] Figure 1 The present invention is a flowchart of a method for cleaning data in a clinical trial EDC system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0036] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0037] See also Figure 1 , which shows a flow chart of a data cleaning method for a clinical trial EDC system provided by an embodiment of the present invention, the method comprising the following steps:

[0038] S001. Collect data of each category.

[0039] In the embodiment of the present invention, various data of each category in the EDC system in the clinical trial are collected, and the categories represent the patient's blood pressure, the patient's body temperature, etc.; the various data of each category represent various blood pressure data under the category of the patient's blood pressure.

[0040] S002. Obtain several local fluctuation degrees of each data in each category, and obtain the optimal search window of each data in each category according to the several local fluctuation degrees of each data in each category.

[0041] It should be noted that since the clinical trial EDC system has many data categories and presents different distribution patterns, especially there will be some data with relatively skewed distribution, and the traditional HBOS algorithm usually assumes that the data is evenly distributed and is very sensitive to the parameter of the number of bins, therefore, the conventional binning strategy adopted by the traditional HBOS algorithm for cleaning each category of data in the clinical trial EDC system may not be able to capture the data characteristics with skewed distribution, resulting in important intervals being too wide and unimportant intervals being too dense, which can easily lead to inaccurate outlier detection and thus affect the data cleaning effect.

[0042] The present invention obtains different adjacent data for each data of each category, and obtains multiple local fluctuation degrees of each data in each category based on the different adjacent data. The number of adjacent data corresponding to the maximum point in the multiple local fluctuation degrees of any data in any category is used as the optimal search window for the data in the category. At this time, the data in the optimal search window for the data in the category can well reflect the local characteristics of the data in the category.

[0043] For each data in each category, its local fluctuation degree can be measured according to the numerical difference between each data and its adjacent data, and weights are introduced so that adjacent data in different positions contribute differently to the local fluctuation degree. The closer the distance between two data, the greater the contribution of the numerical difference to the local fluctuation degree of the data.

[0044] In the embodiment of the present invention, the minimum value of the number of left data and the number of right data of the j-th data in the i-th category is obtained, which is recorded as ;

[0045] Set the neighborhood data number variable u, and set u to [1, ], Represents the minimum value between the number of left data and the number of right data of the j-th data in the i-th category;

[0046] For any u, obtain the u data before and after the j-th data in the i-th category as the adjacent data of the j-th data in the i-th category, and according to the adjacent data of the j-th data in the i-th category, obtain the specific formula of the local fluctuation degree of the j-th data in the i-th category:

[0047] ;

[0048] In the formula, Represents the local fluctuation degree of the jth data in the i-th category; Represents the value of the jth data in the i-th category; Represents the value of the vth adjacent data of the jth data in the i-th category; Represents the distance between the jth data in the i-th category and its vth adjacent data; represents the number of adjacent data of the jth data in the i-th category; exp() represents the exponential function with a natural constant as the base; norm() represents the normalization function; The weight represents the absolute value of the difference between the jth data in the i-th category and its vth adjacent data. If the distance between the jth data in the i-th category and its vth adjacent data is closer, it means that the vth adjacent data of the jth data in the i-th category contributes more to its local fluctuation degree. represents a parameter for controlling weight decay. The larger its value, the slower the weight decays with distance, that is, the weight will not approach 0 quickly with the increase of distance. Therefore, in the embodiment of the present invention, the preset In other embodiments, the implementer may preset The value of . The larger the value of , the greater the difference between the j-th data in the i-th category and its adjacent data. At this time, the greater the local fluctuation of the j-th data in the i-th category.

[0049] According to each u value, several local fluctuation degrees of the jth data in the i-th category are obtained;

[0050] Sort the local fluctuation degrees of the jth data in the i-th category in turn to obtain a local fluctuation degree sequence, use the difference method to obtain several local maximum points in the local fluctuation degree sequence, and take the u value corresponding to the local maximum point with the largest value as the optimal number of neighborhood data for the jth data in the i-th category, recorded as ; Get the data before the jth data in the i-th category Data and later data as the optimal search window for the jth data in the i-th category.

[0051] S003. According to the skewness difference between each data in each category and the data on the left and right sides of its optimal search window, obtain the final disconnection probability of each data in each category; according to the final disconnection probability of each data in each category, obtain each breakpoint in each category, and then obtain the number of bins for each category; according to the number of bins for each category, construct a histogram for each category.

[0052] It should be noted that, within the optimal search window for each data of each category, if the difference between the skewness between any data of any category and the data on the left side of its optimal search window and the skewness between the data of the category and the data on the right side of its optimal search window is large, it means that the skewness change of the data on both sides of the data is greater, which means that the data may be the dividing point of the skewness change, and the possibility that the data of this category is a breakpoint is greater, and when the number of data on both sides of the data of this category is smaller, it means that the mutation situation at the data is relatively large, and the possibility that the data of this category is a breakpoint is greater; subsequently, by setting the threshold, the data belonging to the breakpoint can be obtained, and then the number of bins can be obtained.

[0053] In the embodiment of the present invention, all data between the jth data in the i-th category and the hth data on the left side of its optimal search window (including the jth data in the i-th category and the hth data on the left side of its optimal search window) are obtained, recorded as the hth data set on the left side of the jth data in the i-th category, and the skewness of the hth data set on the left side of the jth data in the i-th category is obtained;

[0054] Get all the data between the jth data in the i-th category and the hth data on the right side of its optimal search window (including the jth data in the i-th category and the hth data on the right side of its optimal search window), record it as the hth data set on the right side of the jth data in the i-th category, and get the skewness of the hth data set on the right side of the jth data in the i-th category;

[0055] Get the absolute value of the difference between the skewness of the hth data set on the left side of the jth data in the i-th category and the skewness of the hth data set on the right side, and record it as the skewness difference between the hth data set on the left and right sides of the jth data in the i-th category;

[0056] Get the h-th disconnection probability of the j-th data in the i-th category:

[0057] ;

[0058] In the formula, represents the h-th disconnection probability of the j-th data in the i-th category; Represents the skewness difference between the left and right sides of the h-th data set of the j-th data in the i-th category; represents the number of data in the optimal search window for the jth data in the i-th category; Represents the sum of the number of data in the hth data set on the left and the hth data set on the right of the jth data in the i-th category.

[0059] The minimum value of all disconnection probabilities of the j-th data in the i-th category is taken as the final disconnection probability of the j-th data in the i-th category.

[0060] It should be noted that the reason why the minimum value of all disconnection probabilities of the jth data in the i-th category is compared with the disconnection probability threshold is that if the minimum value of the disconnection probability is greater than the disconnection probability threshold, it means that the jth data must be a breakpoint.

[0061] Preset disconnection probability threshold In other embodiments, the implementer may preset the disconnection probability threshold according to the specific implementation situation. If the final disconnection probability of the jth data in the i-th category is greater than the disconnection probability , the jth data in the i-th category is a breakpoint, and each breakpoint in the i-th category is obtained.

[0062] The number of breakpoints in the i-th category plus one is taken as the number of bins of the i-th category, and the HBOS algorithm is used to construct a histogram of the i-th category according to the number of bins of the i-th category.

[0063] S004. According to the histogram of each category, obtain the anomaly score of each data in each category, and based on the anomaly score, obtain the abnormal point in each category.

[0064] It should be noted that according to the HBOS algorithm, the smaller the number of data in the box of any data in any category in the histogram of the category, the larger the anomaly score of the data in the category, and the greater the difference between the data in the category and all the data in the box of the histogram of the category, the larger the anomaly score of the data in the category. Subsequently, an anomaly score threshold is preset to identify abnormal data and complete data cleaning.

[0065] In this embodiment of the present invention, the anomaly score of each data in each category is obtained:

[0066] ;

[0067] In the formula, Represents the anomaly score of the jth data in the i-th category; Represents the number of data in the box where the jth data in the i-th category is located in the histogram of the i-th category; The absolute value of the difference between the value of the jth data in the i-th category and the mean of all data in the bin in which it is located in the histogram of the i-th category; Represents the difference between the maximum and minimum values ​​of the data in the box where the jth data in the i-th category is located in the histogram of the i-th category; Represents the difference between the jth data in the i-th category and all the data in the bin where it is located in the histogram of the i-th category. The larger its value is, the more abnormal the jth data in the i-th category is. The smaller the value of is, the more abnormal the j-th data in the i-th category is.

[0068] Preset anomaly score threshold In other embodiments, the implementer may preset the abnormal score threshold according to the specific implementation situation. If the anomaly score of the jth data in the i-th category is greater than the anomaly score threshold , the jth data in the i-th category is an outlier, and each outlier in the i-th category is obtained. Similarly, the outliers in each category are obtained and then removed.

[0069] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A data cleaning method for a clinical trial EDC system, characterized in that: include: Collect individual data for each category; Get the optimal search window for each data in each category; Get each disconnection probability for each data in each category ; represents the h-th disconnection probability of the j-th data in the i-th category; Represents the skewness difference between the left and right sides of the h-th data set of the j-th data in the i-th category; represents the number of data in the optimal search window for the jth data in the i-th category; Represents the number of data in the hth data set on the left and the hth data set on the right of the jth data in the i-th category; norm() represents the normalization function; The minimum value of all disconnection probabilities of each data in each category is taken as the final disconnection probability of each data in each category; According to the final disconnection probability of each data in each category, obtain the respective breakpoints of each category; Based on the breakpoints, obtain the number of bins for each category; Use the HBOS algorithm to construct a histogram for each category based on the number of bins in each category; According to the histogram of each category, get the anomaly score of each data in each category; Based on the anomaly scores, outliers in each category are obtained, and outliers in each category are removed.

2. A data cleaning method for a clinical trial EDC system according to claim 1, characterized in that: The step of obtaining the optimal search window for each data in each category includes: Set the neighborhood data number variable u, and set u to [1, ], Represents the minimum value between the number of left and right data of the j-th data in the i-th category; for any u, obtain the local fluctuation degree of the j-th data in the i-th category; Sort the local fluctuation degrees of the jth data in the i-th category in turn to obtain the local fluctuation degree sequence. Use the difference method to obtain the local maximum points in the local fluctuation degree sequence. The u value corresponding to the local maximum point with the largest value is recorded as ; Get the data before the jth data in the i-th category Data and later data as the optimal search window for the jth data in the i-th category.

3. A data cleaning method for a clinical trial EDC system according to claim 2, characterized in that: The obtaining of the local fluctuation degree of the j-th data in the i-th category includes: For any u, obtain the u data before and the u data after the j-th data in the i-th category as the adjacent data of the j-th data in the i-th category; ； In the formula, Represents the local fluctuation degree of the jth data in the i-th category; Represents the value of the jth data in the i-th category; Represents the value of the vth adjacent data of the jth data in the i-th category; Represents the distance between the jth data in the i-th category and its vth adjacent data; represents the number of adjacent data of the jth data in the i-th category; exp() represents the exponential function with a natural constant as the base; norm() represents the normalization function; Represents the parameter that controls weight decay.

4. A data cleaning method for a clinical trial EDC system according to claim 1, characterized in that: The acquisition of the skewness difference of the hth data set on the left and right sides of the jth data in the ith category includes: Record all data between the jth data in the ith category and the hth data on the left side of its optimal search window as the hth data set on the left side of the jth data in the ith category; obtain the hth data set on the right side of the jth data in the ith category; obtain the skewness of the hth data set on the left side of the jth data in the ith category and the skewness of the hth data set on the right side of the jth data in the ith category; record the absolute value of the difference between the skewness of the hth data set on the left side of the jth data in the ith category and the skewness of the hth data set on the right side as the skewness difference between the hth data sets on the left and right sides of the jth data in the ith category.

5. A data cleaning method for a clinical trial EDC system according to claim 1, characterized in that: The method of obtaining the breakpoints of each category includes: Preset disconnection probability threshold, if the final disconnection probability of the jth data in the i-th category is greater than the disconnection probability , the jth data in the i-th category is a breakpoint, and each breakpoint in the i-th category is obtained.

6. A data cleaning method for a clinical trial EDC system according to claim 1, characterized in that: The method of obtaining the number of bins for each category includes: The number of breakpoints in the i-th category plus one is taken as the number of bins for the i-th category.

7. A data cleaning method for a clinical trial EDC system according to claim 1, characterized in that: The method of obtaining the anomaly score of each data in each category includes: ； In the formula, Represents the anomaly score of the jth data in the i-th category; Represents the number of data in the box where the jth data in the i-th category is located in the histogram of the i-th category; The absolute value of the difference between the value of the jth data in the i-th category and the mean of all data in the bin in which it is located in the histogram of the i-th category; Represents the difference between the maximum and minimum values ​​of the data in the box where the jth data in the i-th category is located in the histogram of the i-th category; norm() represents the normalization function.

8. A data cleaning method for a clinical trial EDC system according to claim 1, characterized in that: The obtaining of abnormal points in each category includes: Preset anomaly score threshold , if the anomaly score of the jth data in the i-th category is greater than the anomaly score threshold , the jth data in the i-th category is an outlier, and each outlier in the i-th category is obtained.

Images