A method, system and device for filling missing data in questionnaire

Abstract

The present invention discloses a method, system and device for filling missing data in a questionnaire. The method comprises: establishing a priori membership matrix based on first prior knowledge; performing clustering and iteration on a relaxation condition based on a physical examination data set and the priori membership matrix to obtain cluster centers; constructing habit feature prompt vectors of different groups based on the cluster centers and questionnaire information; and filling missing values ​​of physical examination data with missing questionnaire information according to the habit feature prompt vectors. The system comprises: a group feature mining module and a missing filling module. The device comprises a memory and a processor for executing the above-mentioned method for filling missing data in a questionnaire. By using the present invention, missing values ​​can be filled in for questionnaire surveys, reducing the cost waste caused by the need to re-collect missing questionnaire information. The present invention can be widely used in the field of data filling.

Description

Technical Field

[0001] The present invention relates to the field of data filling, and in particular to a method, system and device for filling missing data in a questionnaire. Background Art

[0002] At present, two types of data are generally collected when conducting environmental health risk assessments: "physical examination data" and "questionnaire information" of the population. However, due to the influence of many factors such as the cultural quality of the population being assessed, questionnaire information usually contains a lot of missing information, and it is impossible to form a complete cohort population information with the physical examination data, which significantly increases the uncertainty of subsequent health risk assessments. In the process of questionnaire data statistics, there is no basis for filling in missing values. Most of them are ignored or directly filled with the average value. The accuracy of the questionnaire survey results obtained by this filling is not high. Summary of the invention

[0003] In view of this, in order to solve the technical problem that the existing questionnaire missing data filling method does not perform indicator-based missing filling, thereby resulting in low accuracy of questionnaire statistical information, in a first aspect, the present invention proposes a questionnaire missing data filling method, the method comprising the following steps:

[0004] Establishing a priori membership matrix based on first priori knowledge;

[0005] Perform clustering and iteration on the relaxation condition based on the physical examination data set and the prior membership matrix to obtain cluster centers;

[0006] Constructing habit feature prompt vectors of different groups based on the cluster centers and questionnaire information;

[0007] According to the habit feature prompt vector, missing values ​​are filled in for the physical examination data with missing questionnaire information.

[0008] In some embodiments, the step of establishing a priori membership matrix based on the first priori knowledge specifically includes:

[0009] Obtaining medical examination data;

[0010] Introducing expert opinions to compare the physical examination data to obtain the constraint relationship between any two sample data;

[0011] According to the constraint relationship between any two sample data, an initial constraint matrix is ​​constructed;

[0012] Constructing a priori membership matrix according to the initial constraint matrix;

[0013] The constraint relationships include must-connect constraints, no-connect constraints, and no-relation constraints.

[0014] Through this preferred step, an initial constraint matrix is ​​constructed, which is subsequently used to annotate the membership of samples to clusters.

[0015] In some embodiments, the step of performing clustering and iteration on the relaxation condition based on the physical examination data set and the prior membership matrix to obtain cluster centers specifically includes:

[0016] Taking the physical examination data set and the prior membership matrix as input, clustering is performed to obtain initial cluster centers and an initial membership matrix;

[0017] Introducing relaxation conditions;

[0018] Update the constraint matrix according to preset rules;

[0019] Taking the membership matrix after adding the relaxation condition, the cluster center, and the updated constraint matrix as input, clustering is performed to obtain the updated membership matrix and cluster center;

[0020] Introduce relaxation conditions to the updated membership matrix and update the constraint matrix again;

[0021] Repeat the steps: clustering, obtaining the updated membership matrix and cluster centers, introducing relaxation conditions for the membership matrix, and updating the constraint matrix until the preset conditions are met, and the cluster center of the last iteration is the final cluster center.

[0022] In some embodiments, the updating rule of the constraint matrix is ​​specifically:

[0023] For non-zero values ​​in the initial constraint matrix, they are always retained;

[0024] For the initial matrix with a value of 0, it is updated based on the value of the result constraint matrix obtained after each iteration.

[0025] In some embodiments, the membership matrix after adding the relaxation condition satisfies the following conditions:

[0026]

[0027] Among them, a represents the relaxation value, u ij represents the degree of membership, i represents the cluster number, j represents the sample number, and c represents the number of clusters.

[0028] In some embodiments, the step of constructing habit feature prompt vectors of different groups based on the cluster centers and questionnaire information specifically includes:

[0029] Count the questionnaire information data of different groups participating in clustering to obtain the habit characteristics of different groups;

[0030] Combining the habit features of the different groups with the cluster centers to obtain physical examination feature vectors and habit feature vectors of the different groups;

[0031] According to the physical examination feature vectors and habit feature vectors of the different groups, habit feature prompt vectors of the different groups are constructed.

[0032] Through this preferred step, a filling hint vector is constructed and the data is clustered for subsequent comparison.

[0033] In some embodiments, the step of filling missing values ​​of the physical examination data with missing questionnaire information according to the habit feature prompt vector specifically includes:

[0034] The Euclidean distance between the physical examination data with missing questionnaire information and the eigenvectors of physical examination characteristics was calculated to obtain the degree of similarity;

[0035] According to the similarity, the actual information is supplemented with a habit feature prompt vector corresponding to the group.

[0036] Through this preferred step, the Euclidean distance is used to determine which cluster the instance corresponding to the missing data is similar to, and the hint vector corresponding to the cluster is used to fill in the missing item.

[0037] In a second aspect, the present invention further proposes a system for filling missing data in a questionnaire, the system comprising:

[0038] The group feature mining module establishes a priori membership matrix based on the first prior knowledge; performs clustering and iteration on the relaxation condition based on the physical examination data set and the priori membership matrix to obtain cluster centers; and constructs habit feature prompt vectors of different groups based on the cluster centers and questionnaire information;

[0039] The missing value filling module fills the missing values ​​of the physical examination data with missing questionnaire information according to the habit feature prompt vector.

[0040] In a third aspect, the present invention further proposes a device for filling missing data in a questionnaire, comprising:

[0041] at least one processor;

[0042] at least one memory for storing at least one program;

[0043] When the at least one program is executed by the at least one processor, the at least one processor implements the method for filling missing data in a questionnaire as described above.

[0044] Based on the above scheme, the present invention provides a method, system and device for filling missing data in a questionnaire. First, an initial constraint relationship is established according to the physical examination data of intact individuals and expert prior knowledge for clustering. Then, a prompt vector is constructed using the cluster center vector and the habitual feature vector of its group. The missing information is filled with the prompt vector of the cluster that is most similar to the physical examination data of the sample with missing information, thereby improving data quality and reducing cost waste caused by the need to re-collect missing questionnaire information. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a flowchart of the steps of a method for filling missing data in a questionnaire according to the present invention;

[0046] Figure 2 is a schematic diagram of fuzzy clustering according to a specific embodiment of the present invention;

[0047] Figure 3 It is a schematic diagram of a process of iterative clustering to generate cluster centers according to a specific embodiment of the present invention;

[0048] Figure 4 is a schematic diagram of a process for filling missing data in a specific embodiment of the present invention;

[0049] Figure 5 It is a structural block diagram of a system for filling missing data in a questionnaire according to the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0051] It should be noted that, for the convenience of description, only the parts related to the relevant invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other.

[0052] It should be understood that the "system", "device", "unit" and / or "module" used in this application is a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the word can be replaced by other expressions.

[0053] Unless the context clearly indicates an exception, the words "a", "an", "a kind" and / or "the" do not refer to the singular, but also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of clearly identified steps and elements, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements. The elements defined by the sentence "includes a..." do not exclude the existence of other identical elements in the process, method, commodity or device that includes the elements.

[0054] In the description of the embodiments of the present application, "plurality" means two or more than two. The following terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0055] In addition, flow charts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, the various steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes, or a certain step or several steps of operations may be removed from these processes.

[0056] Reference Figure 1 , is a flow chart of an optional example of a method for filling missing data in a questionnaire proposed by the present invention. The method can be applied to a computer device. The imaging method proposed in this embodiment may include but is not limited to the following steps:

[0057] Part I: Soft-constrained fuzzy clustering model based on pairwise constraints:

[0058] Step S1, establishing a priori membership matrix based on first priori knowledge;

[0059] Necessary data type description. When conducting environmental health risk assessment, we need to include the following types of population data: k ,N k It represents the kth item in the physical examination data and the corresponding kth living habits (questionnaire content, it should be noted that the information selection result of the questionnaire is N k refers to the result of an individual's selection of frequency options such as "none, once a week, three times a week", there is a corresponding relationship between the k-th item in the physical examination data and the corresponding k-th item of living habits (that is, the frequency of this question in the questionnaire will directly lead to a significant change in the k-th item in the physical examination data). j Refers to the number of individuals or instances involved.

[0060]

[0061] S1.1. Obtain physical examination data;

[0062] S1.2, introducing expert opinions to compare the physical examination data and obtain the constraint relationship between any two sample data;

[0063] Experts mark the constraint relationship between any two samples based on their professional knowledge. ML, CL, and NL respectively indicate whether the two instances are necessarily connected, not connected, and have no relationship. ML is marked with 1 to represent a necessarily connected relationship between the two samples, CL is marked with -1 to represent a necessarily unconnected relationship between the two samples, and NL is marked with 0. NL represents that the expert cannot accurately determine whether the two instances are ML or CL. The initial constraint pair is obtained and a constraint matrix is ​​generated.

[0064] ML=1,CL=-1,NL=0

[0065]

[0066] S1.3, constructing an initial constraint matrix according to the constraint relationship between any two sample data;

[0067] S1.4, constructing a priori membership matrix according to the initial constraint matrix;

[0068] Generate a priori membership matrix U0 according to the constraint matrix (wherein the membership values ​​refer to the constraint matrix; if there is a constraint relationship between the samples and the categories to which the samples belong are relatively certain, then the differences between the memberships of the samples are larger; if there are no constraints on the samples, then the differences between the memberships of the samples are smaller).

[0069] Step S2, performing clustering on the relaxation condition based on the physical examination data set and the prior membership matrix and iterating to obtain cluster centers;

[0070] S2.1, taking the physical examination data set and the prior membership matrix as input, clustering is performed, and after the clustering is stable, the initial cluster center and the initial membership matrix are obtained;

[0071] S2.2, introduce relaxation conditions;

[0072] S2.3, update the constraint matrix according to the preset rules;

[0073] S2.4, taking the membership matrix after adding the relaxation condition, the cluster center, and the updated constraint matrix as input, clustering is performed to obtain the updated membership matrix and cluster center;

[0074] S2.5, introduce relaxation conditions to the updated membership matrix and update the constraint matrix again;

[0075] S2.6, repeat the steps: perform clustering, obtain updated membership matrix and cluster center, introduce relaxation conditions to the membership matrix, and update the constraint matrix until the preset conditions are met, and take the cluster center of the last iteration as the final cluster center.

[0076] The dataset X (the input physical examination dataset here includes both individuals with and without missing questionnaire information) and prior membership are used as input, and clustering is performed using machine learning technology. It is iterated several times until the clustering result is stable, and further cluster centers V and membership matrix U are obtained. Then, the obtained membership U and initial constraint matrix are used as input for further algorithms, and the updated membership U and result constraint matrix after clustering are output. Compared with the traditional clustering algorithm for membership u ij Require:

[0077]

[0078] The difference is that we have added relaxation conditions on this basis. We proposed a soft constraint method on membership to solve the constraint conflict problem in the clustering process: for the first clustering, only consider the samples whose values ​​in the Con_0 matrix are not 0, that is, whether there is inconsistency between the clustering results and the constraint matrix when comparing the relationship between any two instances between Con_0 and Con_i; for clustering other than the first time, the constraint matrices Con and Con_i updated in the last iteration are compared to solve it. In addition, the constraint matrix iteration rules are as follows: There are two cases of 0 and non-0 in the initial constraint matrix determined by the expert annotation. For non-0 values, we always retain them. For 0 values, we update them based on the value of the result constraint matrix obtained in each iteration. For example:

[0079] The initial constraint matrix is ​​as follows:

[0080]

[0081] The resulting constraint matrix of clustering is as follows:

[0082]

[0083] The updated constraint matrix is ​​as follows:

[0084]

[0085] Compare the resulting constraint matrix with the constraint matrix updated in the previous iteration. If two samples χ j ,x k Violating the ML constraints, their memberships are as follows:

[0086] sample <![CDATA[v1]]> <![CDATA[v2]]> <![CDATA[v3]]> <![CDATA[χ j ]]> <![CDATA[0.73(u ij_max )]]> <![CDATA[0.14(u ij_kmax )]]> <![CDATA[0.13(u ij_else )]]> <![CDATA[χ k ]]> <![CDATA[0.26(u ik_jmax )]]> <![CDATA[0.49(u ik_max )]]> <![CDATA[0.25(u ik_else )]]>

[0087] where u ik_jmax Refers to u ij_max The χ of the same cluster k The membership value of is similar to the rest. Based on this, the membership correction between constraint conflict sample pairs and the calculation method of membership soft constraint range a are proposed as follows:

[0088] Step 1: Calculate the β value according to formula (1):

[0089]

[0090] According to formula (1), the β value of the example is calculated to be 0.41

[0091] Step 2: Update the membership according to formula (2) and formula (3):

[0092] update_u ik_jmax =u ik_jmax +β (2)

[0093] update_u ik_max =|u ik_max -β| (3)

[0094] According to formula (2) and (3), the example is calculated:

[0095] update_u ik_jmax =0.67

[0096] update_u ik_max =0.09

[0097] Step 3: Calculate the relaxation value a according to formula (4):

[0098] a=update_u ik_jmax +update_u ik_max +u ik_else -1 (4)

[0099] The a value of the example is calculated according to formula (4) to be 0.01.

[0100] Then the updated membership degree u ij satisfy:

[0101]

[0102] Thus, the optimized membership matrix U' is obtained. The iterative process refers to Figure 3 .

[0103] Things to note here:

[0104] 1. First we need to select the benchmark instance x_r for each cluster i ,x_r h For example, if there are only two clusters, then we need to ensure that the selected benchmark instance χ_r i ,χ_r h The distances to their respective clusters are the shortest, and the relations between the two benchmark instances in Con_i and Con_0 are both CL relations.

[0105] 2. For a benchmark instance χ_r i There may be conflicts with multiple samples, forming a violation ML constraint matrix C based on the benchmark instance i , each row of the matrix represents a conflict pair representation instance χ_r i , x k The ML relationship is violated, so a value a can be calculated for each row, so an appropriate screening method should be used when selecting the final soft constraint range of the cluster class.

[0106]

[0107] like Figure 2 As shown, χ_r i ,χ_r h is the benchmark instance, v i ,v h is the cluster center, and x k are other sample points, which are related to the benchmark instance χ_r i Belongs to ML relations but is classified into different classes, and x_r h Belonging to the CL relationship but being classified into the same category is a constraint violation, and these two situations are the same, so we only need to consider the benchmark instance violating the ML relationship.

[0108] Finally, when the results of the algorithm stabilize, the vector of each cluster center is output as the physical examination data feature vector of its group:

[0109]

[0110] Step S3, constructing habit feature prompt vectors of different groups based on the cluster center and questionnaire information;

[0111] S3.1. Count the questionnaire information data of different groups participating in clustering to obtain the habit characteristics of different groups;

[0112] For example, if we count the smoking habits of a healthy group of 100 people, 70 of them do not smoke, 17 smoke three times a week, and 13 smoke five times a week, we can get the habit distribution of this group by calculating the proportion of people who choose each option in the total number of people, and then take the option with the largest proportion as the habit characteristic of this group.

[0113] S3.2, combining the habit features of the different groups with the cluster centers to obtain physical examination feature vectors and habit feature vectors of the different groups;

[0114] S3.3. Construct habit feature prompt vectors of different groups based on the physical examination feature vectors and habit feature vectors of different groups.

[0115] Combining the habit features with the previous cluster centers, we can get the physical examination feature vector for each group The habit feature vector corresponding to the group Construct habit feature prompt vectors for different groups.

[0116] in,

[0117] Physical examination feature vector for each group

[0118]

[0119] The habit feature vector corresponding to the group

[0120]

[0121] Part II: Fill in the missing information in the questionnaire. Please refer to the filling process. Figure 4 .

[0122] S4. Filling missing values ​​of the physical examination data with missing questionnaire information according to the habit feature prompt vector.

[0123] S4.1, calculate the Euclidean distance between the physical examination data with missing questionnaire information and the physical examination feature vector to obtain the degree of similarity;

[0124] Obtain the physical examination data of individuals with missing information and calculate the physical examination data of each sample with missing questionnaire information and The Euclidean distance between them is used to find the most similar group category with missing information samples. The smaller the Euclidean distance, the more similar they are.

[0125]

[0126] S5. Based on the similarity, the actual information is supplemented with a habit feature prompt vector corresponding to the group.

[0127] like Figure 5 As shown, a system for filling missing data in a questionnaire includes:

[0128] The group feature mining module establishes a priori membership matrix based on the first prior knowledge; performs clustering and iteration on the relaxation condition based on the physical examination data set and the priori membership matrix to obtain cluster centers; and constructs habit feature prompt vectors of different groups based on the cluster centers and questionnaire information;

[0129] The missing value filling module fills the missing values ​​of the physical examination data with missing questionnaire information according to the habit feature prompt vector.

[0130] The contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0131] A device for filling missing data in questionnaire:

[0132] at least one processor;

[0133] at least one memory for storing at least one program;

[0134] When the at least one program is executed by the at least one processor, the at least one processor implements the method for filling missing data in a questionnaire as described above.

[0135] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0136] A storage medium stores instructions executable by a processor, wherein the instructions executable by the processor are used to implement a method for filling missing data in a questionnaire as described above when executed by the processor.

[0137] The contents of the above method embodiments are all applicable to the present storage medium embodiments. The functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0138] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A method for filling missing data in a questionnaire, characterized in that: The following steps are involved: Establishing a priori membership matrix based on first priori knowledge; Perform clustering and iteration on the relaxation condition based on the physical examination data set and the prior membership matrix to obtain cluster centers; Constructing habit feature prompt vectors of different groups based on the cluster centers and questionnaire information; Filling missing values ​​of physical examination data with missing questionnaire information according to the habit feature prompt vector; The step of establishing a priori membership matrix based on the first priori knowledge specifically includes: Obtaining medical examination data; Introducing expert opinions to compare the physical examination data to obtain the constraint relationship between any two sample data; According to the constraint relationship between any two sample data, an initial constraint matrix is ​​constructed; Constructing a priori membership matrix according to the initial constraint matrix; The constraint relationships include must-connect constraints, no-connect constraints and no-relation constraints; The step of performing clustering and iteration on the relaxation condition based on the physical examination data set and the prior membership matrix to obtain the cluster center specifically includes: Taking the physical examination data set and the prior membership matrix as input, clustering is performed, and after the clustering is stable, an initial cluster center and an initial membership matrix are obtained; Introducing relaxation conditions; Update the constraint matrix according to preset rules; Taking the membership matrix after adding the relaxation condition, the cluster center, and the updated constraint matrix as input, clustering is performed to obtain the updated membership matrix and cluster center; Introduce relaxation conditions to the updated membership matrix and update the constraint matrix again; Repeat the steps: clustering, obtaining the updated membership matrix and cluster center, introducing relaxation conditions to the membership matrix, and updating the constraint matrix until the preset conditions are met, and the cluster center of the last iteration is used as the final cluster center; The membership matrix after adding the relaxation condition satisfies the following conditions: Among them, a represents the relaxation value, u ij represents the degree of membership, i represents the cluster number, j represents the sample number, and c represents the number of clusters; The update rule of the constraint matrix is ​​specifically: For non-zero values ​​in the initial constraint matrix, they are always retained; For the initial matrix with a value of 0, the update is based on the value of the result constraint matrix obtained after each iteration; The step of constructing habit feature prompt vectors of different groups based on the cluster center and questionnaire information specifically includes: Count the questionnaire information data of different groups participating in clustering to obtain the habit characteristics of different groups; Combining the habit features of the different groups with the cluster centers to obtain physical examination feature vectors and habit feature vectors of the different groups; According to the physical examination feature vectors and habit feature vectors of the different groups, habit feature prompt vectors of the different groups are constructed.

2. A method for filling missing data in a questionnaire according to claim 1, characterized in that: The step of filling missing values ​​of the physical examination data with missing questionnaire information according to the habit feature prompt vector specifically includes: The Euclidean distance between the physical examination data with missing questionnaire information and the eigenvectors of physical examination characteristics was calculated to obtain the degree of similarity; According to the similarity, the actual information is supplemented with a habit feature prompt vector corresponding to the group.

3. A system for filling missing data in questionnaires, characterized in that: A method for filling missing data in a questionnaire according to claim 1, comprising: The group feature mining module establishes a priori membership matrix based on the first prior knowledge; performs clustering and iteration on the relaxation condition based on the physical examination data set and the priori membership matrix to obtain cluster centers; and constructs habit feature prompt vectors of different groups based on the cluster centers and questionnaire information; The missing value filling module fills the missing values ​​of the physical examination data with missing questionnaire information according to the habit feature prompt vector.

4. A device for filling missing data in a questionnaire, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements a method for filling missing data in a questionnaire as described in any one of claims 1-2.

Images