Verification machine learning data split system and method thereof

Abstract

A machine learning validation data splitting system and method are disclosed. The machine learning validation data splitting method is applied to a blood pressure data set including blood pressure data of a plurality of subjects. The method includes dividing a measurement range of systolic pressure and diastolic pressure into a plurality of first intervals and a plurality of second intervals, respectively, generating a plurality of classes based on the first and second intervals, each class including a first interval and a second interval, determining and recording a matching condition of the blood pressure data of each subject and the classes, each matching condition including a plurality of labels corresponding to the plurality of classes, each label having one of a first state and a second state, the first state representing a class to which the blood pressure data matches the label, the second state representing a class to which the blood pressure data does not match the label, and performing a distribution procedure based on the matching conditions to distribute the plurality of subjects into a plurality of sets.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to machine learning, in particular, a validation data splitting system for machine learning and a method thereof. BACKGROUND

[0002] Validation strategy is indispensable when tuning the hyperparameters of machine learning and deep learning models. For training and validation purposes, data splitting or cross-validation (CV) techniques can be used to divide the data into training, validation, and test data sets. These techniques provide the power to judge the accuracy and generalizability of the model on independent and new data sets. However, these techniques can also lead to overfitting or bias problems if the data splitting strategy is not properly designed.

[0003] Existing cross-validation methods such as Leave-One-Out, Holdout, and K-fold CV are common practices in standard machine learning problems (e.g., regression problems). However, blood pressure estimation is not a standard regression problem. For example, the data points in the blood pressure data set are not completely independent of each other. That is, these data sets usually have many segments from the same record or subject, which can contain very similar information. In addition, the blood pressure estimation problem has two targets, systolic blood pressure (SBP) and diastolic blood pressure (DBP), which is more similar to a multi-task or multi-output regression problem. Finally, the distribution of systolic and diastolic blood pressure is usually skewed, as extreme blood pressure is rarely seen, so blood pressure estimation becomes an imbalanced regression problem.

[0004] To properly split the data during cross-validation, the above differences must be considered. For example, random data splitting can result in data segments from the same subject appearing in the training, validation, and test data sets at the same time. Since data segments from the same subject carry similar information, this will lead to the collapse of independence between sets and bring overly optimistic results. In addition, due to the skewed and imbalanced distribution, random data splitting can cause problems such as different distributions between each set or data shift, and even lack of minority cases in the test data set. SUMMARY

[0005] Therefore, the present application proposes a machine learning data splitting system and method for verification to avoid the above problems. Cross-validation of blood pressure estimation tasks can keep the data of the same subject in the same set and keep the distribution of systolic and diastolic blood pressure in the original data set in different data sets as much as possible. In other words, in the multiple data sets generated by the present application, there is a high similarity between the trends of systolic and diastolic blood pressure in multiple data sets.

[0006] According to an embodiment of the present application, a machine learning data splitting method for verification is suitable for a blood pressure data set. The blood pressure data set includes multiple blood pressure data of multiple subjects. The types of the blood pressure data include systolic and diastolic blood pressure. The method includes the following steps performed by an operation device: dividing the measurement range of systolic blood pressure into multiple first intervals and dividing the measurement range of diastolic blood pressure into multiple second intervals, generating multiple categories according to the first intervals and the second intervals, each category including a first interval and a second interval, judging and recording the matching condition of the blood pressure data of each subject and the category, thereby generating multiple matching conditions corresponding to the multiple subjects, each matching condition including multiple labels corresponding to the multiple categories, each label having one of a first state and a second state, the first state representing the category corresponding to the blood pressure data matching label, and the second state representing the category corresponding to the blood pressure data non-matching label, and performing a distribution program according to the matching condition to distribute the multiple subjects into multiple sets.

[0007] Preferably, the distribution program includes: calculating the required number of subjects corresponding to the multiple sets according to the number of the multiple subjects and the number of the multiple sets; for each of the multiple categories, calculating the matching number of the category with the first state in the multiple subjects, thereby obtaining multiple matching numbers corresponding to the multiple categories; calculating the required number of matching categories according to the multiple matching numbers and the number of the multiple sets; and when one or more of the multiple subjects exist in the blood pressure data set, performing the following steps: selecting one from the multiple categories as a specified category, the matching number of the specified category being the minimum value; selecting a specified subject from the blood pressure data set, the label of the specified category of the specified subject being the first state; selecting one from the multiple sets as a specified set according to the specified category; distributing the specified subject to the specified set and removing the specified subject from the blood pressure data set; and updating the required number of matching categories and the required number of subjects after distributing the specified subject.

[0008] Preferably, wherein selecting one of the plurality of sets as the designated set comprises: finding a first maximum value in the plurality of numbers of matched categories corresponding to the plurality of sets; and when the number of the first maximum value is equal to 1, taking the set corresponding to the first maximum value as the designated set.

[0009] Preferably, further comprising: when the number of the first maximum value is greater than 1, finding a second maximum value in the plurality of numbers of subject demands corresponding to the plurality of sets; and when the number of the second maximum value is equal to 1, taking the set corresponding to the second maximum value as the designated set.

[0010] Preferably, further comprising: when the number of the second maximum value is greater than 1, randomly selecting one of the plurality of sets corresponding to the second maximum value as the designated set.

[0011] According to an embodiment of the present disclosure, a system for verifying machine learning data splitting includes a measuring device, a storage device, and a computing device. The measuring device is configured to generate a blood pressure dataset, wherein the blood pressure dataset includes a plurality of blood pressure data of a plurality of subjects, and the blood pressure data includes systolic pressure and diastolic pressure. The storage device is communicatively coupled to the measuring device to receive and store the blood pressure dataset, and to store a computer-readable recording medium. The computing device is communicatively coupled to the storage device, and the computing device is configured to execute the computer-readable recording medium to perform the following steps: dividing a measurement range of the systolic pressure into a plurality of first intervals, and dividing a measurement range of the diastolic pressure into a plurality of second intervals, generating a plurality of categories based on the first intervals and the second intervals, each category including one first interval and one second interval, determining and recording a matching condition of the blood pressure data of each subject and the categories, thereby generating a plurality of matching conditions corresponding to the plurality of subjects, each matching condition including a plurality of labels corresponding to the plurality of categories, each label having one of a first state and a second state, the first state representing a category corresponding to a matching label of the blood pressure data, the second state representing a category corresponding to a non-matching label of the blood pressure data, and executing a distribution program based on the matching conditions to distribute the plurality of subjects into a plurality of sets.

[0012] Preferably, the processor operating on the computer readable recording medium is further configured to perform the steps of: calculating a number of subjects required corresponding to the plurality of sets according to the number of subjects and the number of sets; for each of the plurality of categories, calculating a number of matches of the category having the first state among the plurality of subjects, thereby obtaining a plurality of numbers of matches corresponding to the plurality of categories; calculating a number of categories of matches according to the plurality of numbers of matches and the number of sets; and when one or more of the plurality of subjects exist in the set of blood pressure data, performing the steps of: selecting one of the plurality of categories as a designated category, the designated category having the smallest number of matches; selecting a designated subject from the set of blood pressure data, the designated subject having the first state of the designated category; selecting one of the plurality of sets as a designated set according to the designated category; assigning the designated subject to the designated set, and removing the designated subject from the set of blood pressure data.

[0013] Preferably, the processor operating on the computer readable recording medium includes the steps of: finding a maximum number of categories of matches among the plurality of numbers of categories of matches corresponding to the plurality of sets when selecting one of the plurality of sets as the designated set; and when the maximum number of categories of matches is equal to one, taking the set corresponding to the maximum number of categories of matches as the designated set.

[0014] Preferably, the processor operating on the computer readable recording medium is further configured to perform the steps of: when the maximum number of categories of matches is greater than one, finding a maximum number of subjects required among the plurality of numbers of subjects required corresponding to the plurality of sets; and when the maximum number of subjects required is equal to one, taking the set corresponding to the maximum number of subjects required as the designated set.

[0015] Preferably, the processor operating on the computer readable recording medium is further configured to perform the steps of: when the maximum number of subjects required is greater than one, randomly selecting one of the plurality of sets corresponding to the maximum number of subjects required as the designated set.

[0016] In summary, the validation machine learning data splitting system and method has the following contributions or effects: first, the proposed method saves all samples from the same subject in the same set (training data set, validation data set or test data set); second, the proposed method can achieve similar blood pressure distribution on the training data set, validation data set and test data set; third, the proposed method can maintain the blood pressure distribution of different data sets when there are multiple constraints (for example, systolic pressure and diastolic pressure, in addition to pulse rate, heartbeat and any other blood pressure related constraints that affect model training), which means that the systolic pressure distribution of the training / validation data set and the test data set is similar, and the diastolic pressure distribution of these data sets is also similar.

[0017] The above description of the present application and the following description of the embodiments are used to demonstrate and explain the spirit and principles of the present application, and provide further explanation of the claims of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is a schematic diagram of the application applied to machine learning;

[0019] Figure 2 is a block architecture diagram of a validation machine learning data splitting system according to an embodiment of the present application;

[0020] Figure 3 is a flowchart of a validation machine learning data splitting method according to an embodiment of the present application;

[0021] Figure 4 is a schematic diagram of a blood pressure data set;

[0022] Figure 5 is Figure 3 a detailed flowchart of the steps in

[0023] Figure 6 is a schematic diagram of the systolic pressure distribution and diastolic pressure distribution generated according to the traditional data splitting method; and

[0024] Figure 7 is a schematic diagram of the systolic pressure distribution and diastolic pressure distribution generated according to the validation machine learning data splitting method of the embodiment of the present application.

[0025] SYMBOL DESCRIPTION:

[0026] 100: data splitting system

[0027] 10: measuring device

[0028] 30: storage device

[0029] 50: operation device

[0030] S1-S6, S61-S69: steps

[0031] D0: blood pressure dataset

[0032] D1: training dataset

[0033] D2: validation dataset

[0034] D3: testing dataset DETAILED DESCRIPTION

[0035] The detailed features and characteristics of the present application are described in the embodiments below, which are sufficient for any person skilled in the relevant art to understand the technical content of the present application and to implement it, and any person skilled in the relevant art can easily understand the relevant ideas and characteristics of the present application according to the content, claims and drawings applied in the present specification. The following examples are further detailed to illustrate the ideas of the present application, but do not limit the scope of the present application in any way.

[0036] Figure 1 is a schematic diagram of the present application applied to machine learning. As shown in Figure 1 , after the blood pressure dataset D0 is pre-processed, the data splitting system and method proposed by the present application can be applied to split the blood pressure dataset D0 into the training dataset D1, the validation dataset D2 and the testing dataset D3. The training dataset D1 and the validation dataset D2 are used to train and validate the blood pressure prediction model. The testing dataset D3 is used to test the blood pressure prediction model. In other application scenarios, the original dataset can also be pre-processed to generate the blood pressure dataset, and then the blood pressure dataset is split. In other words, the present application does not limit the execution order of the data preprocessing.

[0037] Figure 2 is a block architecture diagram of a data splitting system for validating machine learning according to an embodiment of the present application. As shown in Figure 2 , the system 100 includes a measuring device 10, a storage device 30 and a computing device 50.

[0038] The measurement device 10 is configured to generate a blood pressure dataset, where the blood pressure dataset comprises blood pressure data of a plurality of subjects. Each subject comprises a plurality of blood pressure data, where each blood pressure data comprises systolic blood pressure (SBP) and diastolic blood pressure (DBP). In one embodiment, the measurement device 10 is, for example, a wearable device with pulse oximetry, which applies photoplethysmography (PPG) to obtain a PPG signal, and then converts the PPG signal to blood pressure data by a microprocessor built in the wearable device. In another embodiment, the measurement device 10 is, for example, a wearable device with electrodes, which applies electrocardiography (ECG) to obtain an ECG signal, and then converts the ECG signal to blood pressure data by a microprocessor built in the wearable device. In yet another embodiment, the measurement device 10 is, for example, an electronic sphygmomanometer or an arm-cuff sphygmomanometer.

[0039] The storage device 30 is communicatively coupled to the measurement device 10 to receive and store the blood pressure dataset, and to store a computer-readable recording medium. In one embodiment, the storage device 30 is, for example, volatile memory and / or non-volatile memory. The non-volatile memory includes read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), flash memory, phase-change random access memory (PRAM), magnetic RAM (MRAM), resistive RAM (RRAM), and / or ferroelectric RAM (FRAM). The volatile memory can include dynamic RAM (DRAM), static RAM (SRAM), and / or synchronous DRAM (SDRAM). In another embodiment, the storage device 30 is, for example, at least one of a hard disk drive (HDD), a solid-state drive (SSD), a compact flash (CF) card, a secure digital (SD) card, a micro-SD card, a mini-SD card, an extreme digital (xD) card, and a memory stick.

[0040] The computing device 50 is communicatively coupled to the storage device 30. The computing device 50 is configured to execute a computer-readable recording medium to implement the method for verifying machine learning data splitting according to an embodiment of the present disclosure. In an embodiment, the computing device 50 is, for example, a microprocessor such as a central processing unit (CPU), a graphic processing unit (GPU), and / or an application processor (AP); a logic chip such as a field programmable gate array (FPGA) and an application-specific IC (ASIC).

[0041] Referring to Figure 3 and Figure 4 , Figure 3 is a flowchart of the method for verifying machine learning data splitting according to an embodiment of the present disclosure, Figure 4 is a schematic diagram of a blood pressure data set. Figure 3 The method shown is applicable to a blood pressure data set. In an embodiment, the blood pressure data set includes blood pressure data of a plurality of subjects, and the types of blood pressure data include systolic pressure and diastolic pressure, but are not limited to the two.

[0042] The following is an example of actual numerical values to illustrate the data structure of the blood pressure data set: 500 people are measured for blood pressure for 60 minutes by the measuring device 10 (such as an electronic sphygmomanometer), and after the measuring device 10 obtains the raw measurement data of all the people, a pre-processing procedure such as noise removal, signal sampling, etc. can be selectively performed according to needs. Assuming that 1 blood pressure data is extracted at a sampling frequency of 2 minutes, the blood pressure data is generated according to the raw measurement data, and each person can contribute 30 blood pressure data (60 / 2), wherein each blood pressure data includes a systolic pressure value and / or a diastolic pressure value. The blood pressure data set described in the present disclosure is the collection of all blood pressure data of the 500 people, totaling 15,000 blood pressure data (500*30).

[0043] A schematic diagram of the blood pressure data set is shown in Figure 4 As can be seen from Figure 4 , both the systolic pressure and the diastolic pressure have skewed distribution characteristics, in other words, there are more data in a certain value range such as 60-70 mmHg, and there are fewer data in another value range such as 150-170 mmHg.

[0044] In step S1, the computing device 50 obtains the blood pressure data set from the storage device 30.

[0045] In step S2, the operation device 50 divides the measurement range of the systolic blood pressure into a plurality of first intervals. In step S3, the operation device 50 divides the measurement range of the diastolic blood pressure into a plurality of second intervals. The present application does not limit the order of execution of step S2 and step S3.

[0046] In one embodiment, the division of the first intervals and the second intervals is exemplified by actual numerical values. For example, the systolic blood pressure can be divided into four first intervals, which are (1) below 100 mmHg, (2) between 100 mmHg and 140 mmHg, (3) between 140 mmHg and 160 mmHg, and (4) above 160 mmHg. For example, the diastolic blood pressure can be divided into four second intervals, which are (1) below 60 mmHg, (2) between 60 mmHg and 80 mmHg, (3) between 80 mmHg and 100 mmHg, and (4) above 100 mmHg. The above numerical values are merely for exemplification and are not intended to limit the present application. In other words, the present application does not limit the size of the range of each first / second interval, the number of first / second intervals, and the like. In other embodiments, in addition to the two blood pressure constraints shown, i.e., the systolic blood pressure and the diastolic blood pressure, a third blood pressure constraint, such as the pulse pressure difference, can be added to the method described in the present application, and the operation device 50 divides the range of the third blood pressure constraint into a plurality of third intervals. Figure 3

[0047] In step S4, the operation device 50 generates a plurality of categories according to the plurality of first intervals and the plurality of second intervals. The number of categories is the number of combinations of the first intervals and the second intervals. In the above example, 16 categories (4*4) can be generated according to the four first intervals and the four second intervals, as shown in Table 1 below. Each of the 16 categories includes one of the four first intervals and one of the four second intervals. For example, category 6 represents 100≤SBP<140 and 60≤DBP<80.

[0048] Table 1, Category Example

[0049] DBP < 60 60 < SBP < 80 80 < DBP < 100 100 < DBP SBP < 100 1 2 3 4 100 < SBP < 140 5 6 7 8 140 < SBP < 160 9 10 11 12 SBP < 160 13 14 15 16

[0050] ​In step S5, the operation device 50 judges and records the matching condition of each subject's blood pressure data and each category, and further generates a plurality of matching conditions corresponding to all subjects. Each matching condition includes a plurality of marks corresponding to a plurality of categories, each mark having one of a first state and a second state, the first state representing that at least one systolic pressure data in a plurality of blood pressure data of a subject is located in a first interval included in a category corresponding to the mark, and at least one diastolic pressure data in the plurality of blood pressure data of the same subject is located in a second interval included in the category corresponding to the mark, and the second state representing that none of the blood pressure data matches the category corresponding to the mark. Table 2 below illustrates a plurality of matching conditions of a plurality of subjects in a plurality of categories by using actual values as examples.

[0051] Table 2, example of matching conditions.

[0052]

[0053]

[0054] In the example shown in Table 2, the matching conditions of nine subjects in three categories are included. In Table 2, each column represents a matching condition, and a mark of 1 in the matching condition represents the first state, and a mark of 0 represents the second state. For example, the matching condition of subject A is (1, 0, 1), which represents that at least one blood pressure data of subject A matches category 1, no blood pressure data matches category 2, and at least one blood pressure data matches category 3. In general, step S5 is used to generate a blood pressure category distribution of a plurality of subjects, and the data structure of the distribution is a 0-1 matrix composed of a plurality of matching conditions, the number of columns of the matrix is equal to the number of categories, and the number of rows of the matrix is equal to the number of subjects.

[0055] In step S6, the operation device 50 performs a distribution procedure according to a plurality of matching conditions of all subjects to distribute all subjects into a plurality of sets. In an embodiment, each set corresponds to a fold in K-fold cross-validation. The present application does not limit the number and size of the sets. The distribution procedure must take into account that each category is as evenly distributed as possible in each set, so as to maintain the SBP and DBP distributions in different sets. In addition, since the distribution procedure takes subjects as the distribution unit, it avoids the plurality of blood pressure data from the same subject being distributed into different sets, which leads to the collapse of data independence.

[0056] Please refer to Figure 5 , Figure 5 is Figure 3 the detailed flowchart of step S6 in

[0057] In step S61, the operation device 50 calculates the number of subjects required for each set according to the number of subjects and the number of sets. Assuming that the number of sets is 3, labeled as Set 1, Set 2 and Set 3, and the number of subjects is 9, as shown in the allocation example of Table 2. Therefore, the number of subjects required for Set 1 is 3, the number of subjects required for Set 2 is 3, and the number of subjects required for Set 3 is 3. In other words, the number of subjects required is the number of subjects divided by the number of sets. If there is a remainder, the remaining subjects are allocated to some of the sets according to the procedure described later.

[0058] In step S62, for each category, the operation device 50 calculates the number of matches of this category having the first state among the subjects, and then obtains the number of matches corresponding to the categories. This step S62 can be regarded as a loop procedure. The operation device 50 processes one category at a time (depending on the parallel processing capability of the operation device 50, it can also process N categories at a time), until all categories are processed. Based on the allocation example of Table 2, the operation device 50 can obtain the following Table 3 after performing step S62.

[0059] Table 3, example of the number of matches.

[0060] Subject Class 1 Class 2 Class 3 A 1 0 1 B 0 0 1 C 0 1 0 D 0 0 1 E 0 1 1 F 1 1 0 G 1 0 1 H 1 0 1 I 0 0 1 Number of matches 4 3 7

[0061] In step S63, the operation device 50 calculates the number of matches required for each category according to the number of matches and the number of sets. In an embodiment, the number of matches required for each category is the average value obtained by dividing the number of matches by the number of sets. Based on the example of the number of matches of Table 3, the operation device 50 can obtain the following Table 4 after performing step S63. For example, the number of matches for Category 1 is 4, so the number of matches required for each set for Category 1 is 1.3 (4 / 3 rounded to one decimal place).

[0062] Table 4, example of the number of matches required for each category and sets.

[0063]

[0064] In step S64, the operation device 50 determines whether there are subjects in the blood pressure data set that have not been allocated to a set. If the determination is yes, the procedure of steps S65-S69 is performed, and then the determination of step S64 is re-determined. The procedure of steps S65-S69 is repeated until all subjects are allocated to a set, and then the determination of step S64 becomes no, and the data splitting method of an embodiment of the present application ends.

[0065] In step S65, the operation device 50 selects one of the plurality of categories as a designated category. In one embodiment, the designated category is the one with the minimum number of matched category requirements. For example, in Table 4, category 2 corresponds to the minimum number of matched category requirements (2.3 > 1.3 > 1), and thus the operation device 50 selects category 2 as the designated category in the first iteration. In other embodiments, the operation device 50 randomly selects the designated category.

[0066] In step S66, the operation device 50 selects a designated subject from the blood pressure data set. The designated subject has a label of the designated category in the first state. In the foregoing example, in the first iteration, the blood pressure data set includes nine subjects A-I that have not been assigned. When the designated category is category 2, subjects C, E, and F are selected as the designated subjects because these three subjects C, E, and F have labels of 1 in category 2, which represents the first state. In addition, because three subjects C, E, and F are selected as the designated subjects, the process of steps S66-S69 is repeated three times until all of the designated subjects are assigned, and step S66 selects a new designated subject in the next iteration.

[0067] In step S67, the operation device 50 selects one of the plurality of sets as a designated set. In detail, the operation device 50 determines whether a first condition is satisfied. If the first condition is satisfied, the designated set is generated. If the first condition is not satisfied, the operation device 50 determines whether a second condition is satisfied. If the second condition is satisfied, the designated set is generated. If the second condition is not satisfied, the operation device 50 randomly selects one of the plurality of sets that satisfy a third condition as the designated set.

[0068] The first condition is that a first maximum value is found in the number of matched category requirements covered by the designated category, and the number of the first maximum value is equal to 1. If the first condition is satisfied, the set corresponding to the first maximum value is the designated set.

[0069] The second condition is that a first maximum value is found in the number of matched category requirements covered by the designated category, and the number of the first maximum value is greater than 1; and a second maximum value is found in the number of requirements of all subjects, and the number of the second maximum value is equal to 1. If the second condition is satisfied, the set corresponding to the second maximum value is the designated set.

[0070] The third condition is that a first maximum value is found in the number of matched category requirements covered by the designated category, and the number of the first maximum value is greater than 1; and a second maximum value is found in the number of requirements of all subjects, and the number of the second maximum value is greater than 1.

[0071] The following is an example of the execution of step S67, using Table 4 as an example. The specified category is category 2, which was generated in step S66. The number of matching categories covered by category 2 is (1, 1, 1). Since the maximum value is 1, and the number of such maximum values is 3, the first condition is not satisfied. The number of subjects required for set 1 through set 3 is (3, 3, 3). Since the maximum value is 3, and the number of such maximum values is 3, the second condition is not satisfied. The set satisfying the third condition includes set 1, set 2, and set 3, so the operation device 50 randomly selects one of these three sets as the specified set.

[0072] In step S68, the operation device 50 assigns the specified subject to the specified set, and removes the specified subject from the blood pressure data set. In the example above, the specified subjects generated in step S66 include subjects C, E, and F, and the specified set generated in step S67 is set 1, 2, and 3. In one embodiment, when multiple specified subjects are generated in step S66, the operation device 50 can randomly select one of them for use in step S68. For example, in step S68, the operation device assigns subject C to set 1, and then removes the blood pressure data of subject C from the blood pressure data set.

[0073] In step S69, the operation device 50 updates the number of matching categories required and the number of subjects required. In the example above, the updated results are shown in Table 5 below. Note that the number of matching categories required for set 1 in category 2 is reduced from 1 to 0 (1-1), because set 1 has already been assigned to one subject C in category 2. In addition, the number of subjects required for set 1 is also reduced from 3 to 2 (3-1), because set 1 has already been assigned to one subject C.

[0074] Table 5, example of the results of the first iteration of the assignment.

[0075]

[0076]

[0077] After step S69 is completed, the process returns to step S64. Since subjects A through B and D through I remain in the blood pressure data set, the result of the determination in step S64 is YES, and the second iteration of the process of steps S65 through S69 is performed.

[0078] In step S65, the number of matching categories corresponding to category 2 is still the minimum (note that matching categories with a number less than 1 are not considered), so in the second iteration of the process, the operation device 50 still selects category 2 as the specified category.

[0079] In step S66, the operation device 50 selects the specified subjects E and F.

[0080] In step S67, the matching category demand number of each of Set 2 and Set 3 in Category 2 is (1, 1) respectively. Since the maximum value is 1 and the number of this maximum value is 2, the first condition is not satisfied. The subject demand number of Set 1 to Set 3 is (2, 3, 3) respectively. Since the maximum value is 3 and the number of this maximum value is 2, the second condition is not satisfied. The sets satisfying the third condition include Set 2 and Set 3, and therefore the operation device 50 randomly selects one of these two sets as the designated set.

[0081] In step S68, the designated subject E is assigned to the designated Set 2, for example, and the designated subject E is removed from the blood pressure data set.

[0082] In step S69, the updated results are shown in Table 6 below. Note that the matching category demand number of Set 2 decreases according to the matching status (0, 1, 1) of the subject E.

[0083] Table 6, example of assignment results of the second iteration.

[0084]

[0085] By analogy with the above procedure, one designated subject is assigned to one designated set each time the iteration procedure of steps S65 to S69 is executed, and therefore the total number of iterations is equal to the number of subjects in the blood pressure data set. Tables 7, 8 and 9 show examples of the third iteration, the sixth iteration and the last iteration (the ninth iteration) respectively. For the sake of understanding, the examples assume that the implementation of "random selection" is in alphabetical order and Arabic numeral order.

[0086] Table 7, example of assignment results of the third iteration.

[0087]

[0088]

[0089] Table 8, example of assignment results of the sixth iteration.

[0090]

[0091]

[0092] Table 9, example of assignment results of the ninth iteration.

[0093]

[0094]

[0095] Please refer toFigure 6 and Figure 7 , Figure 6 is a schematic diagram of systolic pressure distribution and diastolic pressure distribution generated according to a conventional data splitting method, Figure 7 is a schematic diagram of systolic pressure distribution and diastolic pressure distribution generated according to a data splitting method for verifying machine learning according to an embodiment of the present application. In Figure 6 , the distributions of the training data set, the verification data set and the test data set are not consistent. Taking Figure 6 the systolic pressure distribution schematic diagram as an example, the training data set has the largest amount of data at 130 mmHg, but the verification data set has the largest amount of data at 125 mmHg. In addition, the verification data set and the test data set both contain a small amount of data at about 190 mmHg, but the training data set has almost no data at this point. This will result in the blood pressure estimation model trained being unable to be used to estimate systolic pressure values above 190 mmHg. In Figure 7 , whether it is systolic pressure distribution or diastolic pressure distribution, the training data set, the verification data set, the test data set and other three data sets all have similar data distribution trends. For example, if a data set has more data in blood pressure interval A and less data in blood pressure interval B, the other data sets will also have the same characteristics. Therefore, such a data set splitting method helps to improve the generality and accuracy of the blood pressure prediction model.

[0096] In summary, the data splitting system and method for verifying machine learning proposed by the present application has the following contributions or effects: first, the proposed method keeps all samples from the same subject in the same set (training data set or test data set); second, the proposed method can achieve similar blood pressure distribution on the training data set, the verification data set and the test data set; third, the proposed method can maintain the blood pressure distribution of different data sets in the presence of multiple constraints (such as systolic pressure and diastolic pressure, in addition to any blood pressure related constraints that affect model training such as pulse rate, heartbeat, etc.), which means that the systolic pressure distribution of the training / verification data set and the test data set is similar, and the diastolic pressure distribution of these data sets is also similar.

[0097] Although the present application has been described with reference to the foregoing embodiments, it is not used to limit the present application. Without departing from the spirit and scope of the present application, any changes and modifications are within the scope of the patent protection of the present application. For the protection scope of the present application, please refer to the appended claims.

Claims

1. A method of validating a machine learning data split, the method comprising: A method for processing a blood pressure dataset, wherein the blood pressure dataset comprises a plurality of blood pressure data of a plurality of subjects, the plurality of blood pressure data comprises systolic pressure and diastolic pressure, and the method comprises the following steps performed by a computing device: dividing a measurement range of the systolic pressure into a plurality of first intervals and dividing a measurement range of the diastolic pressure into a plurality of second intervals; generating a plurality of categories according to the plurality of first intervals and the plurality of second intervals, each of the plurality of categories comprises one of the plurality of first intervals and one of the plurality of second intervals; determining and recording a matching condition of the plurality of blood pressure data of each of the plurality of subjects with the plurality of categories, thereby generating a plurality of matching conditions corresponding to the plurality of subjects, each of the plurality of matching conditions comprises a plurality of labels corresponding to the plurality of categories, each of the plurality of labels has one of a first state and a second state, the first state represents that a systolic pressure data in the plurality of blood pressure data is in the first interval of the category corresponding to the label and a diastolic pressure data in the plurality of blood pressure data is in the second interval of the category corresponding to the label, and the second state represents that the plurality of blood pressure data does not match the category corresponding to the label; and performing a distribution procedure according to the plurality of matching conditions to distribute the plurality of blood pressure data of each of the plurality of subjects into a plurality of groups; wherein the distribution procedure comprises: calculating a number of subjects required corresponding to the plurality of groups according to a number of the plurality of subjects and a number of the plurality of groups; the number of subjects required is the number of the subjects divided by the number of the groups; for each of the plurality of categories, calculating a number of matches of the category having the first state in the plurality of subjects, thereby obtaining a plurality of numbers of matches corresponding to the plurality of categories; calculating a number of matching categories required according to the plurality of numbers of matches and the number of the plurality of groups; the number of matching categories required is an average value obtained by dividing the number of matches by the number of the groups; and when one or more of the plurality of subjects exist in the blood pressure dataset, performing the following steps: selecting one of the plurality of categories as a specified category, the number of matches of the specified category is a minimum value; selecting a specified subject from the blood pressure dataset, the label of the specified category of the specified subject is in the first state; selecting one of the plurality of groups as a specified group according to the specified category; assigning the specified subject to the specified group and removing the specified subject from the blood pressure dataset; and after assigning the specified subject, updating the number of matching categories required and the number of subjects required; wherein selecting one of the plurality of groups as the specified group comprises: finding a first maximum value in the plurality of numbers of matching categories corresponding to the plurality of groups; and when the number of the first maximum value is equal to 1, taking the group corresponding to the first maximum value as the specified group. when the number of the first maximum values is greater than 1, finding a second maximum value in the number of the subjects required corresponding to the multiple sets; and when the number of the second maximum values is equal to 1, taking the set corresponding to the second maximum value as the specified set; when the number of the second maximum values is greater than 1, randomly selecting one of the multiple sets corresponding to the second maximum values as the specified set.

2. A validation machine learning data split system, comprising: The method comprises: a measuring device for generating a blood pressure data set, wherein the blood pressure data set comprises a plurality of subjects, a plurality of blood pressure data of each of the plurality of subjects, and types of the plurality of blood pressure data comprising systolic pressure and diastolic pressure; a storage device communicatively connected to the measuring device for receiving and storing the blood pressure data set and storing a computer-readable recording medium; and an operation device communicatively connected to the storage device, wherein the operation device is configured to execute the computer-readable recording medium to perform the following steps: dividing a measuring range of the systolic pressure into a plurality of first intervals and dividing a measuring range of the diastolic pressure into a plurality of second intervals; generating a plurality of categories according to the plurality of first intervals and the plurality of second intervals, wherein each of the plurality of categories comprises one of the plurality of first intervals and one of the plurality of second intervals; judging and recording a matching condition of the plurality of blood pressure data of each of the plurality of subjects with the plurality of categories, thereby generating a plurality of matching conditions corresponding to the plurality of subjects, wherein each of the plurality of matching conditions comprises a plurality of marks corresponding to the plurality of categories, each of the plurality of marks has one of a first state and a second state, the first state represents that a systolic pressure data in the plurality of blood pressure data is located in the first interval included in the category corresponding to the mark and a diastolic pressure data in the plurality of blood pressure data is located in the second interval included in the category corresponding to the mark, and the second state represents that the plurality of blood pressure data does not match the category corresponding to the mark; and performing a distribution procedure according to the plurality of matching conditions to distribute the plurality of blood pressure data of each of the plurality of subjects into a plurality of sets; wherein the processor executing the computer-readable recording medium is further configured to perform the following steps: calculating a plurality of subject required numbers corresponding to the plurality of sets according to a number of the plurality of subjects and a number of the plurality of sets, wherein the subject required number is the number of the subjects divided by the number of the sets; for each of the plurality of categories, calculating a matching number of the category having the first state in the plurality of subjects, thereby obtaining a plurality of matching numbers corresponding to the plurality of categories; calculating a plurality of matching category required numbers according to the plurality of matching numbers and the number of the plurality of sets, wherein the matching category required number is an average value obtained by dividing the matching number by the number of the sets; and when one or more of the plurality of subjects exist in the blood pressure data set, performing the following steps: selecting one of the plurality of categories as a specified category, wherein the matching number of the specified category is the minimum value; ​ selecting a specified subject from the blood pressure data set, the marker of the specified category of the specified subject being the first state; selecting a specified set from the plurality of sets according to the specified category; assigning the specified subject to the specified set and removing the specified subject from the blood pressure data set; updating the plurality of matched category demand numbers and the plurality of subject demand numbers after assigning the specified subject; wherein the processor running the computer readable recording medium comprises: finding a maximum matched category number from the plurality of matched category numbers corresponding to the plurality of sets when selecting a specified set from the plurality of sets; and when the maximum matched category number is equal to 1, taking the set corresponding to the maximum matched category number as the specified set; wherein the processor running the computer readable recording medium is further configured to: when the maximum matched category number is greater than 1, finding a maximum subject demand number from the plurality of subject demand numbers corresponding to the plurality of sets; and when the maximum subject demand number is equal to 1, taking the set corresponding to the maximum subject demand number as the specified set; when the maximum subject demand number is greater than 1, randomly selecting a set from the sets corresponding to the maximum subject demand number as the specified set.

Images