A bidirectional fuzzy capsule parallel attribute reduction acceleration method based on particle-group collaboration
By adopting parallel partitioning and bidirectional fuzzy particle compartment model on the Spark platform, combined with the particle-group collaborative attribute reduction method, the problem of redundant attribute deletion in traditional data processing is solved, the efficiency and accuracy of large-scale data processing is improved, and the simplification of simplification of large-scale data processing is improved, and the The data analysis process was carried out.
Patent Information
- Application Number
- CN202410285400.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-13
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-03-13
AI Technical Summary
Traditional data processing technology lacks a built-in redundant attribute deletion mechanism, resulting in inefficiency in large-scale data processing and increased storage requirements, and it is difficult to efficiently extract valuable information from massive data.
The parallel partitioning technology based on the Spark platform is adopted, combined with the bidirectional fuzzy particle compartment model and the particle-group collaborative attribute reduction method, and data partitioning and aggregation between the main node and the child node are reduced to redundant attribute calculations and improve attribute reduction efficiency.
It significantly accelerates the speed of data processing, reduces redundant attribute calculations, improves the efficiency and accuracy of data processing, reduces storage requirements, and simplifies the data analysis process.
Smart Images

Figure CN118170823B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of large-scale data mining, and in particular to a bidirectional fuzzy particle compartment parallel attribute reduction acceleration method based on particle-group collaboration. Background Art
[0002] With the rapid development of Internet technology and the widespread popularity of intelligent devices, big data is showing an explosive growth and continuous accumulation trend. Efficiently extracting valuable information from massive data has become an important research topic of concern to many scholars. Traditional data processing technology lacks a built-in redundant attribute removal mechanism, which brings great troubles to data storage and processing. In addition, big data is usually diverse and complex, which limits the in-depth understanding and accurate analysis of the data.
[0003] In "Fuzzy-rough feature selection accelerator", a forward approximation acceleration mechanism is proposed. This mechanism achieves attribute reduction acceleration by gradually reducing the sample size without changing the appropriate attributes. In "Granular cabin: An efficient solution to neighborhood learning in big data", a cabin model is proposed. When calculating the neighborhood, only a few samples located in the cabin need to be considered, which greatly compresses the neighborhood search space. In "Bee: Towards a robust attribute reduction", an integrated selector framework based on the bucket model is proposed. By introducing attribute groups, candidate attributes are evaluated on each bucket model to determine the appropriate attributes. Although this type of accelerator does not need to scan the entire sample and reduces the amount of calculation of the distance between samples, there is still a situation of repeated distance calculation, which consumes a lot of time.
[0004] For large-scale data mining, it is necessary to efficiently extract valuable information from massive data, use more effective methods to reduce redundant attributes, improve the efficiency of classified information, and identify information hidden in the data. In order to further improve the efficiency and accuracy of big data processing, attribute screening becomes particularly important. By removing redundant attributes, redundant information and noise in the data set can be reduced, thereby improving the quality and availability of the data. This can simplify the data processing process, reduce storage requirements, and speed up the analysis and mining of data. Summary of the invention
[0005] In order to address the deficiencies in the above methods, the present invention provides a bidirectional fuzzy particle cabin parallel attribute reduction acceleration method based on particle-group collaboration, which solves the challenges faced by traditional data processing technology, namely the demand for a large amount of computing and memory resources, thereby improving the efficiency and feasibility of data processing.
[0006] The inventive concept of the present invention is as follows: first, on the master node of the Spark platform, the large-scale data set is evenly divided according to the number of parallel partitions; second, the bidirectional fuzzy capsule method is used on the child node to select a sample subset that is much smaller than the data set and very close to the actual fuzzy granularity, so as to achieve acceleration in granularity; the attribute group is combined with the bidirectional fuzzy capsule proposed in this paper to search for simplification using the particle-group collaborative attribute reduction method. Finally, the simplification obtained by each child node is aggregated to the master node, and the simplification results of all partitions are aggregated using the counter operator. The attributes of the sorted simplification results are evaluated again at the master node.
[0007] In order to achieve the above purpose, the present invention is implemented by the following measures: a bidirectional fuzzy capsule parallel attribute reduction acceleration method based on particle-group collaboration, comprising the following steps:
[0008] S10: On the master node, read the large-scale data set and convert it into a decision-making information system<U,AT,D> , where U represents the sample set of the data set, AT represents the non-empty finite set of conditional attributes of the data set, and D represents the non-empty finite set of decision categories of the data set. The communication between the master node Master and the slave node Slave is established, and the large-scale data set is read. The data set U is divided into m data subsets and broadcasted to the corresponding slave nodes Slaver;
[0009] S20: In the child node Slaver, a bidirectional fuzzy particle compartment of the sample is generated according to the three constructed virtual samples, one auxiliary sample and the introduced bidirectional mutual neighbor strategy;
[0010] S30: calculating the fuzzy membership degree, and calculating the fuzzy upper and lower approximation of the decision class D relative to the conditional attribute;
[0011] S40: In the child node Slaver, by integrating the concept of attribute groups, candidate attributes similar to potential attributes will be temporarily ignored in each iteration;
[0012] S50: Aggregate the reduction results of each child node to the main node, and sort them again.
[0013] The specific steps of step S20 are as follows:
[0014] Step S21: On the child node Slaver, the corresponding data subset is represents the i-th sample, and n represents the data set In order to select a sample subset that is much smaller than the data set and very close to the actual fuzzy granularity by gradually reducing the fuzzy information granularity without changing the selected attribute result, three virtual samples are constructed by the attribute value size of the given data set. and
[0015]
[0016] Among them, attribute a represents the conditional attribute of the data set, max represents the maximum value of the corresponding attribute, ave represents the summed average value of the corresponding attribute, and min represents the minimum value of the corresponding attribute;
[0017] Step S22: Given a decision information system B represents the conditional attribute subset of the data subset, and finds the corresponding attribute value closest to a(x). For each attribute, select the The corresponding attribute value close to the sample attribute is obtained to obtain an auxiliary sample point, and then the distance between the auxiliary sample point and the sample is calculated to obtain the sample Fuzzy neighborhood radius under conditional attribute subset B The definition is as follows:
[0018]
[0019] Among them, a(x) and Represent sample x and sample The conditional attributes of , Δ represents the knowledge distance between the calculated samples, Represents finding the first The corresponding attribute value;
[0020] Step S23: Three virtual samples obtained in step S21 and And step S22 obtains the blur area radius Fuzzy capsules can be constructed The structure is:
[0021]
[0022] in, and The fuzzy granular spaces corresponding to the three virtual samples are constructed. The specific results are as follows
[0023]
[0024] Step S24: Calculate samples according to step 22 and samples The corresponding fuzzy neighborhood radii are and And step S23 calculates the sample and samples The corresponding fuzzy compartments are and
[0025] Step S25: Considering the bidirectional connection between samples, the mutual information strategy is integrated into the fuzzy capsule model based on Spark parallelization. Bidirectional fuzzy capsule under conditional attribute subset B for:
[0026]
[0027] The specific steps of step S30 are as follows:
[0028] Step S31: The distance between the sample and each sample in its bidirectional fuzzy capsule is calculated using the Gaussian kernel distance, and an equivalent representation of the fuzzy relationship is given from the perspective of the bidirectional fuzzy capsule, as shown below:
[0029]
[0030] The parameters The value of Sample y belongs to a two-way fuzzy compartment A sample of
[0031] Step S32: The samples are divided into f mutually exclusive decision classes by decision attributes, namely f represents the number of decision classes, based on The bidirectional fuzzy cabin decision is defined as follows:
[0032]
[0033] in, represents the jth decision class, Representation sample In the fuzzy similarity relationship corresponding to the attribute subset B, Indicates the number of samples that meet the conditions;
[0034] Step S33: is the fuzzy cabin decision of decision class D obtained under attribute subset B, which is also A fuzzy partition on the definition of the upper and lower approximation of decision class D relative to B is:
[0035]
[0036]
[0037] Step S34: Calculate the fuzzy dependency function using the fuzzy upper and lower approximation sets. The calculation formula is as follows:
[0038]
[0039] The specific steps of step S40 are as follows:
[0040] Step S41: AT is a set of attributes that characterize the sample. Calculate the knowledge distance of similarity between two attribute sets It can be expressed as:
[0041]
[0042] in
[0043] Step S42: Divide the attributes into k groups according to the attribute correlation, and set the potential attribute set
[0044] Step S43: Calculate the fuzzy compartment related to conditional subset B. The calculation formula is as follows:
[0045]
[0046] Step S44: Add the candidate attributes to the conditional subset B in sequence, calculate the attribute importance SIG(a,B,D) of the attribute subset B∪{a}, find the maximum value in the obtained SIG(a,B,D), add the corresponding attribute a to the attribute set A, and temporarily store the redundant attributes in F. The calculation formula of the attribute importance is as follows:
[0047]
[0048] Step S45: determine if attribute set A does not satisfy the constraint condition. If not, find the attribute with the highest attribute importance and the corresponding fuzzy compartment smaller than the selected attribute in the remaining attribute groups that do not contain any selected attribute, and add it to attribute set A.
[0049] The specific steps of step S50 are as follows:
[0050] Step S51: Data subset The corresponding attribute reduction subset calculated on the m' child nodes is in represents the reduced subset calculated from the mth data subset, Represents the simplified subset calculated by the m'th data subset, transmits the simplified subset of each child node to the master node Master, and uses the counter operator to aggregate the simplified results of all partitions to obtain the large-scale data set attribute simplified result A;
[0051] Step S52: Select the attributes in the simplified result A in turn. indivual, And calculate whether it satisfies the constraints. If so, then attributes form a new reduced subset A.
[0052] Compared with the prior art, the present invention has the following beneficial effects:
[0053] 1. The bidirectional fuzzy capsule model constructed by the present invention only needs to consider a few samples located in the fuzzy capsule, without considering the entire data set, which greatly speeds up the search of the fuzzy neighborhood. At the same time, the present invention uses the bidirectional fuzzy capsule to find the upper and lower approximate sets, so that this invention can be applied to numerical continuous data sets. The particle-group collaborative attribute reduction method used by the present invention greatly reduces the number of attribute iterations during attribute reduction and shortens the attribute reduction time.
[0054] 2. The present invention utilizes Spark distributed architecture to divide and transfer data sets to each child node. Multiple node attribute reduction algorithms are parallel and do not interfere with each other. Finally, each child node is aggregated and attribute importance is re-calculated at the master node. When facing big data attribute reduction, this method stores the calculated data in memory, which speeds up data processing and improves the classification accuracy of the reduction. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] Figure 1 It is an overall framework diagram of a bidirectional fuzzy particle compartment parallel attribute reduction acceleration method based on particle-group collaboration of the present invention.
[0056] Figure 2 The figure is a detailed processing flow chart of a bidirectional fuzzy particle compartment parallel attribute reduction acceleration method based on particle-group collaboration of the present invention.
[0057] Figure 3 The present invention is a data processing flow chart under the Spark framework of a bidirectional fuzzy particle cabin parallel attribute reduction acceleration method based on particle-group collaboration. DETAILED DESCRIPTION
[0058] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. Of course, the specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.
[0059] Example 1
[0060] See also Figures 1 to 3 The technical solution provided in this embodiment is a bidirectional fuzzy capsule parallel attribute reduction acceleration method based on particle-group collaboration, comprising the following steps:
[0061] S10: On the master node, read the large-scale data set and convert it into a decision-making information system<U,AT,D> , where U represents the sample set of the data set, AT represents the non-empty finite set of conditional attributes of the data set, and D represents the non-empty finite set of decision categories of the data set. The communication between the master node Master and the slave node Slave is established, and the large-scale data set is read. The data set U is divided into m data subsets and broadcasted to the corresponding slave nodes Slaver;
[0062] Converting one of the raw data subsets into a decision information system is as follows:
[0063]
[0064] S20: In the child node Slaver, a bidirectional fuzzy particle compartment of the sample is generated according to the three constructed virtual samples, one auxiliary sample and the introduced bidirectional mutual neighbor strategy;
[0065] S30: calculating the fuzzy membership degree, and calculating the fuzzy upper and lower approximation of the decision class D relative to the conditional attribute;
[0066] S40: In the child node Slaver, by integrating the concept of attribute groups, candidate attributes similar to potential attributes will be temporarily ignored in each iteration;
[0067] S50: Aggregate the reduction results of each child node to the main node, and sort them again.
[0068] The specific steps of step S20 are as follows:
[0069] Step S21: On the child node Slaver, the corresponding data subset is represents the i-th sample, and n represents the data set In order to select a sample subset that is much smaller than the data set and very close to the actual fuzzy granularity by gradually reducing the fuzzy information granularity without changing the selected attribute result, three virtual samples are constructed by the attribute value size of the given data set. and
[0070]
[0071] Among them, attribute a represents the conditional attribute of the data set, max represents the maximum value of the corresponding attribute, ave represents the summed average value of the corresponding attribute, and min represents the minimum value of the corresponding attribute;
[0072] Finding three virtual samples of this data subset is as follows:
[0073]
[0074] Step S22: Given an information system, B represents the conditional attribute subset of the data subset, and finds the corresponding attribute value closest to a(x). For each attribute, select the The corresponding attribute value close to the sample attribute is obtained to obtain an auxiliary sample point, and then the distance between the auxiliary sample point and the sample is calculated to obtain the sample Fuzzy neighborhood radius under conditional attribute subset B The definition is as follows:
[0075]
[0076] Among them, a(x) and Represent sample x and sample The conditional attributes of , Δ represents the knowledge distance between the calculated samples, Represents finding the first The corresponding attribute value;
[0077] By sample For example, The value of is 3; it can be concluded The corresponding attribute values of the auxiliary sample points are [0.18 0.42 0.48 0.21 0.36 0.11], and the auxiliary sample to sample is calculated. The distance of the sample Neighborhood radius
[0078] Step S23: Three virtual samples obtained in step S21 and And step S22 obtains the blur area radius Fuzzy capsules can be constructed The structure is:
[0079]
[0080] in, and The fuzzy granular spaces corresponding to the three virtual samples are constructed. The specific results are as follows
[0081]
[0082] Step S24: Calculate samples according to step 22 and samples The corresponding fuzzy neighborhood radii are and And step S23 calculates the sample and samples The corresponding fuzzy compartments are and
[0083] Calculate the distance from the virtual sample to all samples:
[0084]
[0085]
[0086]
[0087]
[0088]
[0089] Then, get the sample Fuzzy Particle Cabin The sample Add to Fuzzy Particle Cabin In the above example, we scan the remaining samples and get
[0090] Step S25: Considering the bidirectional connection between samples, the mutual information strategy is integrated into the fuzzy capsule model based on Spark parallelization. Bidirectional fuzzy capsule under conditional attribute subset B for:
[0091]
[0092] Repeat the above steps to obtain because Included but Does not include samples Finally get
[0093] The specific steps of step S30 are as follows:
[0094] Step S31: The distance between the sample and each sample in its bidirectional fuzzy capsule is calculated using the Gaussian kernel distance, and an equivalent representation of the fuzzy relationship is given from the perspective of the bidirectional fuzzy capsule, as shown below:
[0095]
[0096] The parameters The value of Sample y belongs to a two-way fuzzy compartment A sample of
[0097] Step S32: The samples are divided into f mutually exclusive decision classes by decision attributes, namely f represents the number of decision classes, based on The bidirectional fuzzy cabin decision is defined as follows:
[0098]
[0099] in, represents the jth decision class, Representation sample In the fuzzy similarity relationship corresponding to the attribute subset B, Indicates the number of samples that meet the conditions;
[0100] Step S33: is the fuzzy cabin decision of decision class D obtained under attribute subset B, which is also A fuzzy partition on the definition of the upper and lower approximation of decision class D relative to B is:
[0101]
[0102]
[0103] Step S34: Calculate the fuzzy dependency function using the fuzzy upper and lower approximation sets. The calculation formula is as follows:
[0104]
[0105] The specific steps of step S40 are as follows:
[0106] Step S41: AT is a set of attributes that characterize the sample. Calculate the knowledge distance of similarity between two attribute sets It can be expressed as:
[0107]
[0108] in
[0109] Step S42: Divide the attributes into k groups according to the attribute correlation, and set the potential attribute set
[0110] Since there are fewer attributes, the attributes in this data subset are divided into two groups, and the attribute groups are
[0111] Step S43: Calculate the fuzzy compartment related to conditional subset B. The calculation formula is as follows:
[0112]
[0113] Step S44: Add the candidate attributes to the conditional subset B in sequence, calculate the attribute importance SIG(a,B,D) of the attribute subset B∪{a}, find the maximum value in the obtained SIG(a,B,D), add the corresponding attribute a to the attribute set A, and temporarily store the redundant attributes in F. The calculation formula of the attribute importance is as follows:
[0114]
[0115] Step S45: determine if attribute set A does not satisfy the constraint condition. If not, find the attribute with the highest attribute importance and the corresponding fuzzy compartment smaller than the selected attribute in the remaining attribute groups that do not contain any selected attribute, and add it to attribute set A.
[0116] The specific steps of step S50 are as follows:
[0117] Step S51: Data subset The corresponding attribute reduction subset calculated on the m' child nodes is in represents the reduced subset calculated from the mth data subset, Represents the reduced subset calculated for the m'th data subset. The reduced subset of each child node is transmitted to the master node Master. The reduced results of all partitions are aggregated using the counter operator to obtain the attribute reduction result A of the large-scale data set.
[0118] Step S52: Select the attributes in the simplified result A in turn. indivual, And calculate whether it satisfies the constraints. If so, then attributes form a new reduced subset A.
[0119] Example 2
[0120] See also Figures 1 to 3 The technical solution provided in this embodiment is a bidirectional fuzzy capsule parallel attribute reduction acceleration method based on particle-group collaboration, comprising the following steps:
[0121] S10: On the master node, read the large-scale data set and convert it into a decision-making information system<U,AT,D> , where U represents the sample set of the data set, AT represents the non-empty finite set of conditional attributes of the data set, and D represents the non-empty finite set of decision categories of the data set. The communication between the master node Master and the slave node Slave is established, and the large-scale data set is read. The data set U is divided into m data subsets and broadcasted to the corresponding slave nodes Slaver;
[0122] Converting one of the raw data subsets into a decision information system is as follows:
[0123]
[0124] S20: In the child node Slaver, a bidirectional fuzzy particle compartment of the sample is generated according to the three constructed virtual samples, one auxiliary sample and the introduced bidirectional mutual neighbor strategy;
[0125] S30: calculating the fuzzy membership degree, and calculating the fuzzy upper and lower approximation of the decision class D relative to the conditional attribute;
[0126] S40: In the child node Slaver, by integrating the concept of attribute groups, candidate attributes similar to potential attributes will be temporarily ignored in each iteration;
[0127] S50: Aggregate the reduction results of each child node to the main node, and sort them again.
[0128] The specific steps of step S20 are as follows:
[0129] Step S21: On the child node Slaver, the corresponding data subset is represents the i-th sample, and n represents the data set In order to select a sample subset that is much smaller than the data set and very close to the actual fuzzy granularity by gradually reducing the fuzzy information granularity without changing the selected attribute result, three virtual samples are constructed by the attribute value size of the given data set. and
[0130]
[0131] Among them, attribute a represents the conditional attribute of the data set, max represents the maximum value of the corresponding attribute, ave represents the summed average value of the corresponding attribute, and min represents the minimum value of the corresponding attribute;
[0132] Finding three virtual samples of this data subset is as follows:
[0133]
[0134] Step S22: Given an information system, B represents the conditional attribute subset of the data subset, and finds the corresponding attribute value closest to a(x). For each attribute, select the The corresponding attribute value close to the sample attribute is obtained to obtain an auxiliary sample point, and then the distance between the auxiliary sample point and the sample is calculated to obtain the sample Fuzzy neighborhood radius under conditional attribute subset B The definition is as follows:
[0135]
[0136] Among them, a(x) and Represent sample x and sample The conditional attributes of , Δ represents the knowledge distance between the calculated samples, Represents finding the first The corresponding attribute value;
[0137] By sample For example, The value of is 4; it can be concluded The attribute values of the corresponding auxiliary sample points are [0.10 0.18 0.53 0.04 0.04 0.03 0.02 0.35 0.19 0.30], and the auxiliary sample to sample is calculated. The distance of the sample Neighborhood radius
[0138] Step S23: Three virtual samples obtained in step S21 and And step S22 obtains the blur area radius Fuzzy capsules can be constructed The structure is:
[0139]
[0140] in, and The fuzzy granular spaces corresponding to the three virtual samples are constructed. The specific results are as follows
[0141]
[0142] Step S24: Calculate samples according to step 22 and samples The corresponding fuzzy neighborhood radii are and And step S23 calculates the sample and samples The corresponding fuzzy compartments are and
[0143] Then, get the sample Fuzzy Particle Cabin The sample Add to Fuzzy Particle Cabin In the above example, we scan the remaining samples and get
[0144] Step S25: Considering the bidirectional connection between samples, the mutual information strategy is integrated into the fuzzy capsule model based on Spark parallelization. Bidirectional fuzzy capsule under conditional attribute subset B for:
[0145]
[0146] Repeat the above steps to obtain because Included but Does not include samples Finally
[0147] The specific steps of step S30 are as follows:
[0148] Step S31: The distance between the sample and each sample in its bidirectional fuzzy capsule is calculated using the Gaussian kernel distance, and an equivalent representation of the fuzzy relationship is given from the perspective of the bidirectional fuzzy capsule, as shown below:
[0149]
[0150] The parameters The value of Sample y belongs to a two-way fuzzy compartment A sample of
[0151] Step S32: The samples are divided into f mutually exclusive decision classes by decision attributes, namely f represents the number of decision classes, based on The bidirectional fuzzy cabin decision is defined as follows:
[0152]
[0153] in, represents the jth decision class, Representation sample In the fuzzy similarity relationship corresponding to the attribute subset B, Indicates the number of samples that meet the conditions;
[0154] Step S33: is the fuzzy cabin decision of decision class D obtained under attribute subset B, which is also A fuzzy partition on the definition of the upper and lower approximation of decision class D relative to B is:
[0155]
[0156]
[0157] Step S34: Calculate the fuzzy dependency function using the fuzzy upper and lower approximation sets. The calculation formula is as follows:
[0158]
[0159] The specific steps of step S40 are as follows:
[0160] Step S41: AT is a set of attributes that characterize the sample. Calculate the knowledge distance of similarity between two attribute sets It can be expressed as:
[0161]
[0162] in
[0163] Step S42: Divide the attributes into k groups according to the attribute correlation, and set the potential attribute set
[0164] Since there are fewer attributes, the attributes in this data subset are divided into two groups, and the attribute groups are
[0165] Step S43: Calculate the fuzzy compartment related to conditional subset B. The calculation formula is as follows:
[0166]
[0167] Step S44: Add the candidate attributes to the conditional subset B in sequence, calculate the attribute importance SIG(a,B,D) of the attribute subset B∪{a}, find the maximum value in the obtained SIG(a,B,D), add the corresponding attribute a to the attribute set A, and temporarily store the redundant attributes in F. The calculation formula of the attribute importance is as follows:
[0168]
[0169] Step S45: determine if attribute set A does not satisfy the constraint condition. If not, find the attribute with the highest attribute importance and the corresponding fuzzy compartment smaller than the selected attribute in the remaining attribute groups that do not contain any selected attribute, and add it to attribute set A.
[0170] The specific steps of step S50 are as follows:
[0171] Step S51: Data subset The corresponding attribute reduction subset calculated on the m' child nodes is in represents the reduced subset calculated from the mth data subset, Represents the reduced subset calculated for the m'th data subset. The reduced subset of each child node is transmitted to the master node Master. The reduced results of all partitions are aggregated using the counter operator to obtain the attribute reduction result A of the large-scale data set.
[0172] Step S52: Select the attributes in the simplified result A in turn. indivual, And calculate whether it satisfies the constraints. If so, then attributes form a new reduced subset A.
[0173] The above description is only an exemplary embodiment of the present invention, and does not limit the scope of patent protection of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the scope of patent protection of the present invention.
Claims
1. A bidirectional fuzzy capsule parallel attribute reduction acceleration method based on particle-group collaboration, characterized in that: The following steps are involved: S10: On the master node, read the large-scale data set and convert it into a decision-making information system<U,AT,D> , where U represents the sample set of the data set, AT represents the non-empty finite set of conditional attributes of the data set, and D represents the non-empty finite set of decision categories of the data set. The communication between the master node Master and the slave node Slave is established, and the large-scale data set is read. The data set U is divided into m data subsets and broadcasted to the corresponding slave nodes Slaver; S20: In the child node Slaver, a bidirectional fuzzy particle compartment of the sample is generated according to the three constructed virtual samples, one auxiliary sample and the introduced bidirectional mutual neighbor strategy; The specific steps of step S20 are as follows: Step S21: On the child node Slaver, the corresponding data subset represents the i-th sample, and n represents the data set The number of samples is , and three virtual samples are constructed by the attribute value size of the given data set and Among them, attribute a represents the conditional attribute of the data set, max represents the maximum value of the corresponding attribute, ave represents the summed average value of the corresponding attribute, and min represents the minimum value of the corresponding attribute; Step S22: Given a decision information system<U,AT,D> , B represents the conditional attribute subset of the data subset, and finds the corresponding attribute value closest to a(x). For each attribute, select the The corresponding attribute value close to the sample attribute is obtained to obtain an auxiliary sample point, and then the distance between the auxiliary sample point and the sample is calculated to obtain the sample Fuzzy neighborhood radius under conditional attribute subset B The definition is as follows: Among them, a(x) and Represent sample x and sample The conditional attributes of , Δ represents the knowledge distance between the calculated samples, Represents finding the first corresponding attribute values; Step S23: Three virtual samples obtained in step S21 and And step S22 obtains the blur area radius Constructing a fuzzy capsule The structure is: in, and The fuzzy granular spaces corresponding to the three virtual samples are constructed, and the results are as follows Step S24: Calculate samples according to step 22 and samples The corresponding fuzzy neighborhood radii are and And step S23 calculates the sample and samples The corresponding fuzzy compartments are and Step S25: Considering the bidirectional connection between samples, the mutual information strategy is integrated into the fuzzy capsule model based on Spark parallelization. Bidirectional fuzzy capsule under conditional attribute subset B for: S30: firstly calculate the fuzzy membership, and then calculate the fuzzy upper and lower approximations of the decision class D relative to the conditional attribute; S40: In the child node Slaver, by integrating the concept of attribute groups, candidate attributes similar to potential attributes will be temporarily ignored in each iteration; S50: Aggregate the reduction results of each child node to the main node, and sort them again.
2. According to the particle-group collaboration-based bidirectional fuzzy particle cabin parallel attribute reduction acceleration method of claim 1, it is characterized in that: The specific steps of step S30 are as follows: Step S31: The distance between the sample and each sample in its bidirectional fuzzy capsule is calculated using the Gaussian kernel distance, and an equivalent representation of the fuzzy relationship is given from the perspective of the bidirectional fuzzy capsule, as shown below: The parameters The value of Sample y belongs to a two-way fuzzy compartment A sample of Step S32: The samples are divided into f mutually exclusive decision classes by decision attributes, namely f represents the number of decision classes, based on The bidirectional fuzzy cabin decision is defined as follows: in, represents the jth decision class, Representation sample In the fuzzy similarity relationship corresponding to attribute subset B, Indicates the number of samples that meet the conditions; Step S33: is the fuzzy cabin decision of decision class D obtained under attribute subset B, which is also A fuzzy partition on the definition of the upper and lower approximation of decision class D relative to B is: Step S34: Calculate the fuzzy dependency function using the fuzzy upper and lower approximation sets. The calculation formula is as follows:
3. According to the particle-group collaboration-based bidirectional fuzzy particle cabin parallel attribute reduction acceleration method of claim 1, it is characterized in that: The specific steps of step S40 are as follows: Step S41: AT is a set of attributes that characterize the sample. Calculate the knowledge distance of similarity between two attribute sets It is expressed as: in Step S42: Divide the attributes into k groups according to the attribute correlation, and set the potential attribute set Step S43: Calculate the fuzzy compartment related to conditional subset B. The calculation formula is as follows: Step S44: Add the candidate attributes to the conditional subset B in sequence, calculate the attribute importance SIG(a,B,D) of the attribute subset B∪{a}, find the maximum value in the obtained SIG(a,B,D), add the corresponding attribute a to the attribute set A, and temporarily store the redundant attributes in F. The calculation formula of the attribute importance is as follows: Step S45: determine if attribute set A does not satisfy the constraint condition. If not, find the attribute with the highest attribute importance and the corresponding fuzzy compartment smaller than the selected attribute in the remaining attribute groups that do not contain any selected attribute, and add it to attribute set A.
4. According to the particle-group collaboration-based bidirectional fuzzy particle cabin parallel attribute reduction acceleration method of claim 1, it is characterized in that: The specific steps of step S50 are as follows: Step S51: Data subset The corresponding attribute reduction subset calculated on the m' child nodes is in represents the reduced subset calculated from the mth data subset, Represents the reduced subset calculated by the m'th data subset, transmits the reduced subset of each child node to the master node Master, and uses the counter operator to aggregate the reduced results of all partitions to obtain the attribute reduced result A of the large-scale data set; Step S52: Select the attributes in the simplified result A in turn. indivual, And calculate whether it satisfies the constraints. If so, then attributes form a new reduced subset A.
