Method for determining maximum information coefficient and method for judging data relevancy

Abstract

The invention relates to the technical field of data processing, in particular to a method for determining a maximum information coefficient and a method for judging data relevancy, which comprises the following steps of: in an X-axis direction, taking two sections with a being equal to 2 to divide Q in equal depth; arranging the scatter points in a descending order in the Y-axis direction, connecting the scatter points into a scatter line, taking an intersection point of the scatter line and the division line in the X-axis direction as a division point in the Y-axis direction, and taking a corresponding value in the Y-axis direction as the division P in the Y-axis direction; solving a maximum mutual information value; calculating a feature matrix MX (D) under X-axial isometric division; interchanging horizontal and vertical coordinates, and repeating the steps to form a characteristic matrix MY (D) of Y-axis equal-depth division; and taking the maximum element in the feature matrix MX (D) and the feature matrix MY (D) as the final maximum information coefficient value of the two variables. By means of the determination method and the data relevancy judgment method, the calculation amount of the maximum information coefficient solving process can be reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method for determining a maximum information coefficient and a method for determining data relevance. Background Art

[0002] The correlation between data is an important basis for data screening. The maximum information coefficient is a method for calculating information correlation based on mutual information.

[0003] In the prior art, a Chinese invention patent document with publication number CN116881748A and publication date October 13, 2023 is proposed. The technical solution disclosed in the patent document is as follows: a partitioning and combination clustering method for big data, including obtaining the mutual information of partitioning and combination, creating a feature matrix, defining the maximum mutual information coefficient, and solving the dynamic programming algorithm for the maximum mutual information coefficient of the partitioning and combination.

[0004] The maximum mutual information coefficient can be solved by the above dynamic programming algorithm. However, in actual use, the above technical solution does not improve the grid division. Considering the size of the big data n, if the number of grid divisions is still n according to convention, 0.6 , calculate n 0.6 The amount of calculation of the information coefficient is too large. Summary of the invention

[0005] To solve the above technical problems, the present invention proposes a method for determining the maximum information coefficient and a method for judging data relevance, which can bring together targets with the same paradigm and re-divide the grid, thereby reducing the amount of calculation in the maximum information coefficient solution process.

[0006] The present invention is achieved by adopting the following technical solutions:

[0007] A method for determining a maximum information coefficient comprises the following steps:

[0008] Step S1. In the X-axis direction, take two equal-depth divisions Q with a=2;

[0009] Step S2. Arrange the scattered points in descending order along the Y axis, and connect the scattered points into a scattered point line, and use the intersection of the scattered point line and the dividing line along the X axis as the dividing point along the Y axis, and the corresponding value along the Y axis is the dividing point P along the Y axis;

[0010] Step S3. Finding the maximum mutual information value;

[0011] Step S4. Calculate the characteristic matrix M under the X-axis equal depth division X (D);

[0012] Step S5. Swap the horizontal and vertical coordinates, repeat steps S1 to S3, and calculate the feature matrix M under the Y-axis equal depth division Y (D);

[0013] Step S6. The feature matrix M X (D) and the feature matrix M Y The maximum element in (D) is taken as the maximum information coefficient value of the two variables.

[0014] Step S3 specifically refers to: dynamically approximating and solving the maximum mutual information value.

[0015] Step S3 specifically includes the following steps:

[0016] Step S 31 .Remember the division along the Y axis P = <c0=0,...,c k =n>, where c0≤c1≤…≤c k , the number of samples in the jth row is c j -c j-1 , k is the number of segments along the Y axis;

[0017] Step S 32 . Calculate the previous c r The samples are divided into two segments F(c r ,2)Value:

[0018]

[0019] Where r∈[2,k], H() is the Shannon entropy;

[0020] Step S 33 .According to F(c r ,2), iterate and calculate one by one, dynamically approximate the solution F(c r ,l), and get the maximum value F(c k ,l):

[0021]

[0022] Among them, l∈[3,t], r∈[l,k],

[0023] Where B is n 0.6 ;

[0024] Step S 34 . Calculate the maximum mutual information value I max (P; Q):

[0025] I max (P; Q) = F(c k ,l)+H(Q).

[0026] Step S 31 The number of segments k is:

[0027] k≤cB / a,

[0028] In the formula, c is a constant and B is n 0.6 .

[0029] Step S4 specifically refers to: dynamically approximating and solving the characteristic matrix.

[0030] Step S4 specifically includes the following steps:

[0031] Step S 41 . Calculate the characteristic matrix of the X-axis under a=2 equal depth divisions:

[0032]

[0033] In the formula, F(c k ,l)+H(Q) is the maximum mutual information value;

[0034] Step S 42 Repeat steps S2 to S 41 , dynamically approximate the solution of the characteristic matrix M under the equal-depth division of the X-axis a∈[3,B] segment X (D).

[0035] A method for determining data relevance uses the maximum information coefficient value determined by the above determination method to determine the data relevance.

[0036] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0037] 1. The present invention reduces the grid division, thereby reducing the amount of calculation in the entire solution process. Specifically, the present invention uses dynamic solution to find the paradigm of the target, and gathers the targets with the same paradigm together to re-divide the grid, reducing the amount of calculation in the maximum information coefficient solution process, making the maximum information coefficient solution faster and more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, wherein:

[0039] Figure 1 It is a schematic diagram of the process of the present invention;

[0040] Figure 2 It is a schematic diagram of the division of the X-axis and the Y-axis in the present invention. DETAILED DESCRIPTION

[0041] Example 1

[0042] As a basic implementation mode of the present invention, the present invention includes a method for determining a maximum information coefficient, comprising the following steps:

[0043] Step S1. In the X-axis direction, divide Q into two equal-depth segments with a=2.

[0044] Step S2. Arrange the scattered points in descending order along the Y axis, and connect the scattered points into a scatter line. The intersection of the scatter line and the dividing line along the X axis is used as the dividing point along the Y axis. The corresponding value along the Y axis is the dividing point P along the Y axis.

[0045] Step S3: Find the maximum mutual information value.

[0046] Step S4. Calculate the characteristic matrix M under the X-axis equal depth division X (D).

[0047] Step S5. Swap the horizontal and vertical coordinates, repeat steps S1 to S3, and calculate the feature matrix M divided along the Y axis with equal depth Y (D).

[0048] Step S6. The feature matrix M X (D) and the feature matrix M Y The maximum element in (D) is taken as the maximum information coefficient value of the two variables.

[0049] Example 2

[0050] As a preferred embodiment of the present invention, the present invention includes a method for determining a maximum information coefficient, comprising the following steps:

[0051] Step S1. In the X-axis direction, divide Q into two equal-depth segments with a=2.

[0052] Step S2. Arrange the scattered points in descending order along the Y axis, and connect the scattered points into a scatter line. The intersection of the scatter line and the dividing line along the X axis is used as the dividing point along the Y axis. The corresponding value along the Y axis is the dividing point P along the Y axis.

[0053] Step S3: dynamically approximate and solve the maximum mutual information value.

[0054] Step S4. Dynamically approximate the characteristic matrix M under the X-axis equal depth division. X (D).

[0055] Step S5. Swap the horizontal and vertical coordinates, repeat steps S1 to S3, and dynamically approximate the characteristic matrix M divided along the Y axis with equal depth. Y (D).

[0056] Step S6. The feature matrix M X (D) and the feature matrix M YThe maximum element in (D) is taken as the maximum information coefficient value of the two variables.

[0057] Example 3

[0058] As another preferred embodiment of the present invention, the present invention includes a method for determining a maximum information coefficient, comprising the following steps:

[0059] Step S1. In the X-axis direction, divide Q into two equal-depth segments with a=2.

[0060] Step S2. Arrange the scattered points in descending order along the Y axis, and connect the scattered points into a scatter line. The intersection of the scatter line and the dividing line along the X axis is used as the dividing point along the Y axis. The corresponding value along the Y axis is the dividing point P along the Y axis.

[0061] Step S3. Solve for the maximum mutual information value. Considering that the X-axis partition Q is fixed, the value of H(Q) is a constant. If H(P)-H(Q,P) is the maximum, then the mutual information value I(P;Q)=H(Q)+H(P)-H(Q,P) reaches the maximum. Therefore, the following steps are included:

[0062] Step S 31 .Remember the division along the Y axis P = <c0=0,...,c k =n>, where c0≤c1≤…≤c k , the number of samples in the jth row is c j -c j-1 , k is the number of segments along the Y axis.

[0063] Step S 32 . Calculate the previous c r The samples are divided into two segments F(c r ,2)Value:

[0064]

[0065] Where r∈[2,k] and H() is the Shannon entropy.

[0066] Step S 33 .According to F(c r ,2), iterate and calculate one by one, dynamically approximate the solution F(c r ,l), and get the maximum value F(c k ,l):

[0067]

[0068] Among them, l∈[3,t], r∈[l,k], B is n 0.6 .

[0069] Step S 34. Calculate the maximum mutual information value I max (P; Q):

[0070] I max (P; Q) = F(c k ,l)+H(Q).

[0071] Step S4. Calculate the characteristic matrix M under the X-axis equal depth division X (D).

[0072] Step S5. Swap the horizontal and vertical coordinates, repeat steps S1 to S3, and calculate the feature matrix M divided along the Y axis with equal depth Y (D);

[0073] Step S6. The feature matrix M X (D) and the feature matrix M Y The maximum element in (D) is taken as the maximum information coefficient value of the two variables.

[0074] Example 4

[0075] As the best embodiment of the present invention, the present invention includes a method for determining the maximum information coefficient. Since the maximum information coefficient of a specific set of two variables represents the information correlation degree of the two variables, the present invention takes into account that the change of grid division should not affect the intrinsic correlation degree of the two variables, so a method for determining the optimal grid division can be adopted by simplifying part of the grid division work. The X-axis is fixedly divided, and the Y-axis is divided differently. Among all possible grid divisions, the division with the largest amount of mutual information is the optimal division of the Y-axis; the Y-axis is fixedly divided, and the X-axis is divided differently. Among all possible grid divisions, the division with the largest amount of mutual information is the optimal division of the X-axis; through the optimal division of the optimal X and Y axes, the characteristic matrix M(D) composed of the mutual information coefficient is obtained, and the largest element is selected from it as the maximum information coefficient.

[0076] For details, please refer to the attached manual. Figure 1 , including the following steps:

[0077] Step S1. Refer to the instruction manual Figure 2 , in the X-axis direction, take two equal-depth divisions Q of a=2, so that scattered points with similar values ​​can enter the same grid segment.

[0078] Step S2. Arrange the scattered points in descending order along the Y axis, and connect the scattered points into a scatter line. The intersection of the scatter line and the dividing line along the X axis is used as the dividing point along the Y axis. The corresponding value along the Y axis is the dividing point P along the Y axis.

[0079] Step S3: dynamically approximate and solve the maximum mutual information value.

[0080] Define F(c k ,l) is the maximum value of H(P)-H(Q,P) for all scattered data in segment l. Considering that the X-axis division Q is fixed, the value of H(Q) is a constant. Therefore, when H(P)-H(Q,P) is the maximum, the mutual information value I(P;Q)=H(Q)+H(P)-H(Q,P) reaches the maximum. Therefore, this step specifically includes the following steps:

[0081] Step S 31 .Remember the division along the Y axis P = <c0=0,...,c k =n>, where c0≤c1≤…≤c k , the number of samples in the jth row is c j -c j-1 , k is the number of segments along the Y axis. To reduce complexity, set k≤cB / a. In the formula, c is a constant, usually 5. The recommended value of B is n 0.6 .

[0082] Step S 32 . Calculate the previous c r The samples are divided into two segments F(c r ,2)Value:

[0083]

[0084] Where r∈[2,k] and H() is the Shannon entropy.

[0085] Step S 33 .According to F(c r ,2), iterate and calculate one by one, and we can get the following table 1, dynamic approximate solution F(c r ,l), and get the maximum value F(c k ,l):

[0086]

[0087] Among them, l∈[3,t], r∈[l,k],

[0088] Table 1

[0089]

[0090] Step S 34 . Calculate the maximum mutual information value I max (P; Q):

[0091] I max (P; Q) = F(c k ,l)+H(Q).

[0092] Step S4. Solve the characteristic matrix using dynamic approximation. Specifically, the following steps are included:

[0093] Step S 41 . Calculate the characteristic matrix of the X-axis under the equal depth division of a=2 segments, and you can get the first row in Table 2 below.

[0094]

[0095] In the formula, F(c k ,l)+H(Q) is the maximum mutual information value.

[0096] Step S 42 Repeat steps S2 to S 41 , calculate the characteristic matrix of the equal-depth division of segment a∈[3,B]; then we can get the following Table 2, which is the dynamic approximation of the characteristic matrix M of the equal-depth division of the X-axis X (D).

[0097] Table 2

[0098]

[0099] Step S5. Swap the horizontal and vertical coordinates, repeat steps S1 to S3, and form a feature matrix M with equal depth division along the Y axis. Y (D).

[0100] Step S6. The feature matrix M X (D) and the feature matrix M Y The maximum element in (D) is taken as the maximum information coefficient value of the two variables.

[0101] This embodiment also includes a method for determining data relevance, which uses the maximum information coefficient value determined by the above determination method to determine data relevance.

[0102] In summary, after reading the present invention document, ordinary technicians in this field can make various other corresponding transformation schemes based on the technical scheme and technical concept of the present invention without creative mental labor, which all fall within the scope of protection of the present invention.

Claims

1. A method for determining a maximum information coefficient, characterized in that: The following steps are involved: Step S1. In the X-axis direction, take two equal-depth divisions Q with a=2; Step S2. Arrange the scattered points in descending order along the Y axis, and connect the scattered points into a scattered point line, and use the intersection of the scattered point line and the dividing line along the X axis as the dividing point along the Y axis, and the corresponding value along the Y axis is the dividing point P along the Y axis; Step S3. solving the maximum mutual information value; Step S4. Calculate the characteristic matrix M under the X-axis equal depth division X (D); Step S5. Swap the horizontal and vertical coordinates, repeat steps S1 to S3, and calculate the feature matrix M under the Y-axis equal depth division Y (D); Step S6. The feature matrix M X (D) and the feature matrix M Y The maximum element in (D) is taken as the maximum information coefficient value of the two variables.

2. The method for determining the maximum information coefficient according to claim 1, wherein: Step S3 specifically refers to: dynamically approximating and solving the maximum mutual information value.

3. The method for determining the maximum information coefficient according to claim 2, wherein: Step S3 specifically includes the following steps: Step S 31 .Remember the division along the Y axis P = <c0=0,...,c k =n>, where c0≤c1≤…≤c k , the number of samples in the jth row is c j -c j-1 , k is the number of segments along the Y axis; Step S 32 . Calculate the previous c r The samples are divided into two segments F(c r ,2)Value: Where r∈[2,k], H() is the Shannon entropy; Step S 33 .According to F(c r ,2), iterate and calculate one by one, dynamically approximate the solution F(c r ,l), and get the maximum value F(c k ,l): Among them, l∈[3,t], r∈[l,k], Where B is n 0.6 ; Step S 34 . Calculate the maximum mutual information value I max (P; Q): I max (P;Q)=F(c k ,l)+H(Q)。 4. The method for determining the maximum information coefficient according to claim 3, characterized in that: Step S 31 The number of segments k is: k≤cB / a, In the formula, c is a constant and B is n 0.6 .

5. The method for determining the maximum information coefficient according to claim 1, characterized in that: Step S4 specifically refers to: dynamically approximating and solving the characteristic matrix.

6. A method for determining a maximum information coefficient according to claim 5, characterized in that: Step S4 specifically includes the following steps: Step S 41 . Calculate the characteristic matrix of the X-axis under a=2 equal depth divisions: In the formula, F(c k ,l)+H(Q) is the maximum mutual information value; Step S 42 Repeat steps S2 to S 41 , dynamically approximate the solution of the characteristic matrix M under the equal-depth division of the X-axis a∈[3,B] segment X (D).

7. A method for determining data relevance, characterized in that: The data correlation is determined by using the maximum information coefficient value, wherein the maximum information coefficient value is determined according to the method for determining the maximum information coefficient according to any one of claims 1 to 6.

Images