Sample matching degree calculation optimization method, device, medium and computer program product

Abstract

The application discloses a sample matching degree calculation optimization method, which comprises the following steps: obtaining each first local sample ID, and mapping a first hash value corresponding to each first local sample ID to a preset value interval to obtain each first hash mapping value; determining a global sampling order value corresponding to each second device based on a first sampling number corresponding to a first sampling hash mapping value set selected from each first hash mapping value and a second sampling number corresponding to a second sampling hash mapping value set sent by each second device, and selecting a global sampling hash mapping value corresponding to each global sampling order value from the first sampling hash mapping value set and each second sampling hash mapping value set; and calculating a sample matching degree between a first device and each second device based on each global sampling order value and each global sampling hash mapping value. The application solves the technical problem of low efficiency in calculating the sample matching degree in federated learning.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence technology in financial technology (Fintech), in particular to a sample matching degree calculation optimization method, device, medium and computer program product. BACKGROUND

[0002] With the continuous development of financial technology, especially Internet technology finance, more and more technologies (such as distributed, artificial intelligence, etc.) are applied in the financial field, but the financial industry also puts forward higher requirements for technology, such as higher requirements for the distribution of the corresponding to-do list of the financial industry.

[0003] With the continuous development of computer software and artificial intelligence, big data cloud service application, the application of federated learning technology is also more and more widely, and when performing federated learning, it is usually necessary to determine the sample matching degree between each participant of federated learning. At present, the sample matching degree between each participant is usually estimated by filter, such as Bloom filter, etc. However, the sample matching degree between each participant estimated by filter usually needs to compare the sample ID of each participant one by one, and when the number of samples of each participant is large, the calculation amount and calculation complexity of comparing the sample ID of each participant one by one will be extremely high, thereby causing the efficiency of calculating the sample matching degree in federated learning to be extremely low. SUMMARY

[0004] The main purpose of the present application is to provide a sample matching degree calculation optimization method, device, medium and computer program product, which aims to solve the technical problem of low efficiency of calculating sample matching degree in the prior art federated learning.

[0005] To achieve the above-mentioned purpose, the present application provides a sample matching degree calculation optimization method, which is applied to a first device, and the sample matching degree calculation optimization method comprises:

[0006] Obtain each first local sample ID, and map the first hash value corresponding to each first local sample ID to a preset value interval to obtain the first hash mapping value corresponding to each first local sample ID;

[0007] Select a first sampling hash mapping value set from each first hash mapping value, and receive a second sampling hash mapping value set sent by each second device, wherein the second sampling hash mapping value set is selected by the second device from the second hash mapping value corresponding to each second local sample ID;

[0008] determining a global sampling order value corresponding to each of the second devices based on the first sampling number corresponding to the first sampling hash mapping value set and the second sampling number corresponding to each of the second sampling hash mapping value sets, and selecting a global sampling hash mapping value corresponding to each of the global sampling order values from the first sampling hash mapping value set and each of the second sampling hash mapping value sets;

[0009] calculating a sample matching degree between the first device and each of the second devices based on each of the global sampling order values and each of the global sampling hash mapping values.

[0010] The present application provides a sample matching degree calculation optimization method, which is applied to a second device, and the sample matching degree calculation optimization method comprises the following steps:

[0011] obtaining each second local sample ID, and mapping a second hash value corresponding to each of the second local sample IDs to a preset value interval to obtain a second hash mapping value corresponding to each of the second local sample IDs;

[0012] sending a second sampling hash mapping value set selected from each of the second hash mapping values to a first device, so that the first device calculates a sample matching degree between the first device and each of the second devices based on each of the second sampling hash mapping value sets sent by the second devices and a first sampling hash mapping value set generated based on each first local sample ID.

[0013] The present application also provides a sample matching degree calculation optimization device, which is a virtual device and is applied to a first device, and the sample matching degree calculation optimization device comprises the following steps:

[0014] a mapping module, configured to obtain each first local sample ID, and map a first hash value corresponding to each of the first local sample IDs to a preset value interval to obtain a first hash mapping value corresponding to each of the first local sample IDs;

[0015] a receiving module, configured to select a first sampling hash mapping value set from each of the first hash mapping values, and receive a second sampling hash mapping value set sent by each second device, wherein the second sampling hash mapping value set is selected from a second hash mapping value corresponding to each second local sample ID by the second device;

[0016] The selecting module is configured to determine a global sampling order value corresponding to each of the second devices based on a first sampling number corresponding to the first sampling hash mapping value set and second sampling numbers corresponding to the second sampling hash mapping value sets, and select a global sampling hash mapping value corresponding to each of the global sampling order values from the first sampling hash mapping value set and the second sampling hash mapping value sets.

[0017] The calculating module is configured to calculate a sample matching degree between the first device and each of the second devices based on each of the global sampling order values and each of the global sampling hash mapping values.

[0018] The sample matching degree calculation optimization apparatus is a virtual apparatus, and the sample matching degree calculation optimization apparatus is applied to the second devices. The sample matching degree calculation optimization apparatus comprises:

[0019] The mapping module is configured to obtain each of the second local sample IDs, and map a second hash value corresponding to each of the second local sample IDs to a preset value interval to obtain a second hash mapping value corresponding to each of the second local sample IDs.

[0020] The sending module is configured to send a second sampling hash mapping value set selected from the second hash mapping values to the first device, so that the first device calculates a sample matching degree between the first device and each of the second devices based on the second sampling hash mapping value sets sent by each of the second devices and a first sampling hash mapping value set generated based on each of the first local sample IDs.

[0021] The sample matching degree calculation optimization apparatus is an entity apparatus. The sample matching degree calculation optimization apparatus comprises a memory, a processor, and a program of the sample matching degree calculation optimization method stored in the memory and executable on the processor. The program of the sample matching degree calculation optimization method is executable on the processor to implement the steps of the sample matching degree calculation optimization method.

[0022] The medium is a readable storage medium. The readable storage medium stores a program of a sample matching degree calculation optimization method. The program of the sample matching degree calculation optimization method is executable on the processor to implement the steps of the sample matching degree calculation optimization method.

[0023] The computer program product comprises a computer program. The computer program is executable on the processor to implement the steps of the sample matching degree calculation optimization method.

[0024] The application provides a sample matching degree calculation optimization method, device, medium and computer program product. Compared with the technical means of estimating the sample matching degree between each participant by using a filter in the prior art, the application first acquires each first local sample ID, maps a first hash value corresponding to each first local sample ID to a preset value interval to obtain a first hash mapping value corresponding to each first local sample ID, then selects a first sampling hash mapping value set from each first hash mapping value, receives a second sampling hash mapping value set sent by each second device, wherein the second sampling hash mapping value set is selected by the second device from a second hash mapping value corresponding to each second local sample ID, then determines a global sampling order value corresponding to each second device based on a first sampling number corresponding to the first sampling hash mapping value set and a second sampling number corresponding to each second sampling hash mapping value set, selects a global sampling hash mapping value corresponding to each global sampling order value from the first sampling hash mapping value set and each second sampling hash mapping value set, and then estimates the sample matching degree between the first device and each second device based on each global sampling order value and each global sampling hash mapping value, so as to achieve the purpose of calculating the sample matching degree between the first device and each second device based on part of the hash values selected from the first device and each second device, without calculating the sample matching degree by one-to-one comparison of the sample IDs of the first device and each second device, thereby reducing the calculation amount and calculation complexity of calculating the sample matching degree in federated learning. Therefore, the technical defects of high calculation amount and calculation complexity when one-to-one comparing the sample IDs of each participant when the sample number of each participant is large, and the low efficiency of calculating the sample matching degree in federated learning are overcome, and the efficiency of calculating the sample matching degree in federated learning is improved. BRIEF DESCRIPTION OF DRAWINGS

[0025] The accompanying drawings, which are incorporated into and form a part of the specification, illustrate preferred embodiments of the present application and, together with the description, serve to explain the principles of the application.

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0027] Figure 1 The flowchart of the first embodiment of the sample matching degree calculation optimization method of the present application;

[0028] Figure 2 The flowchart of the first device and the second device interacting to calculate the intersection sample number in the sample matching degree calculation optimization method of the present application;

[0029] Figure 3 A flowchart of a second embodiment of the sample matching degree calculation optimization method of the present application is shown in the figure.

[0030] Figure 4 A flowchart of a third embodiment of the sample matching degree calculation optimization method of the present application is shown in the figure.

[0031] Figure 5 A device structure diagram of the hardware running environment involved in the sample matching degree calculation optimization method of the embodiments of the present application is shown in the figure.

[0032] Figure 6 A hardware architecture diagram of federated learning involved in the embodiments of the present application is shown in the figure.

[0033] The object implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0034] It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0035] The embodiments of the present application provide a sample matching degree calculation optimization method. In a first embodiment of the sample matching degree calculation optimization method of the present application, referring to Figure 1 , the sample matching degree calculation optimization method is applied to a first device, and the sample matching degree calculation optimization method comprises:

[0036] Step S10, obtaining each first local sample ID, and mapping the first hash value corresponding to each first local sample ID to a preset value interval to obtain the first hash mapping value corresponding to each first local sample ID;

[0037] In the present embodiment, it should be noted that the sample matching degree calculation optimization method is applied to federated learning, and the first device is a task initiation participant with a federated server function in federated learning, wherein the federated server function is a function of aggregating data sent by all participants in federated learning, the first local sample ID is the identity of the sample in the first device, such as an ID card number and a mobile phone number, and the preset value interval is a specific value interval set in advance, preferably, the preset value interval can be set to 0 to 1.

[0038] obtain the first hash mapping value corresponding to each of the first local sample IDs, specifically, obtain each of the first local sample IDs, and perform hash processing on each of the first local sample IDs respectively to obtain the first hash value corresponding to each of the first local sample IDs, and then map each of the first hash values to the preset value interval based on the size of each of the first hash values to obtain the first hash mapping value corresponding to each of the first local sample IDs.

[0039] The step of mapping each of the first hash values corresponding to each of the first local sample IDs to the preset value interval to obtain the first hash mapping value corresponding to each of the first local sample IDs comprises:

[0040] Step S11, performing hash processing on each of the first local sample IDs respectively to obtain the first hash value;

[0041] In this embodiment, based on a preset first type of hash function, hash processing is performed on each of the first local sample IDs respectively to obtain the first hash value, wherein the preset first type of hash function includes but is not limited to hash functions such as sha256 and SM3.

[0042] Step S12, mapping each of the first hash values to a floating point number in the preset value interval to obtain the first hash mapping value.

[0043] In this embodiment, each of the first hash values is mapped to a floating point number in the preset value interval to obtain the first hash mapping value, specifically, each of the first hash values is taken modulo by a preset modulo function to map each of the first hash values to a floating point number in the preset value interval to obtain the first hash mapping value.

[0044] In another implementable manner, step S12 further comprises:

[0045] performing secondary hash on the first hash value by a preset second type of hash function to obtain a secondary hash value, and taking the value of the bit position with a preset number of bits in the secondary hash value as the floating point number in the preset value interval, wherein the result bit number of the preset second type of hash function is smaller than the result bit number of the preset first type of hash function, for example, the preset second type of hash function can be set as a hash function such as md5.

[0046] Step S20: selecting a first sampling hash mapping value set from the first hash mapping values, and receiving a second sampling hash mapping value set sent by each second device, wherein the second sampling hash mapping value set is selected by the second device from the second hash mapping values corresponding to the second local sample IDs.

[0047] In this embodiment, it should be noted that the second device is a task cooperation participant of federated learning, and the task cooperation participant needs to send its own data to the task initiating participant for aggregation, and the number of the second devices is at least 1.

[0048] In this embodiment, it should be noted that the second device is a task cooperation participant of federated learning, and the task cooperation participant needs to send its own data to the task initiating participant for aggregation, and the number of the second devices is at least 1.

[0049] In this embodiment, it should be noted that the second device is a task cooperation participant of federated learning, and the task cooperation participant needs to send its own data to the task initiating participant for aggregation, and the number of the second devices is at least 1.

[0050] Step S21: sorting each of the first hash mapping values to obtain a local sorting result.

[0051] In this embodiment, each of the first hash mapping values is sorted to obtain a local sorting result, and specifically, each of the first hash mapping values is sorted in descending order based on the size of each of the first hash mapping values to obtain a local sorting result.

[0052] Step S22: selecting a first sampling hash mapping value set from the first hash mapping values based on the local sorting result, and the first sampling hash mapping value set is a hash mapping value with a first sampling number of the first hash mapping values in the rear of the sorting.

[0053] In the embodiment, based on the local ranking result, a first sampling number of hash mapping values with a late ranking are selected from the first hash mapping values as the first sampling hash mapping value set. Specifically, based on the local ranking result, a first sampling number of hash mapping values with a late ranking are selected from the first hash mapping values, the first sampling hash mapping values are obtained, and the first sampling hash mapping values are collectively used as the first sampling hash mapping value set.

[0054] In step S30, based on the first sampling number corresponding to the first sampling hash mapping value set and the second sampling number corresponding to each second sampling hash mapping value set, a global sampling ranking value corresponding to each second device is determined, and a global sampling hash mapping value corresponding to each global sampling ranking value is selected from the first sampling hash mapping value set and each second sampling hash mapping value set.

[0055] In the embodiment, it should be noted that the global sampling ranking value is a global size ranking position of the global sampling hash mapping value corresponding to the global sampling ranking value in the first sampling hash mapping value set and the corresponding second sampling hash mapping value set.

[0056] Based on the first sampling number corresponding to the first sampling hash mapping value set and the second sampling number corresponding to each second sampling hash mapping value set, a global sampling ranking value corresponding to each second device is determined, and a global sampling hash mapping value corresponding to each global sampling ranking value is selected from the first sampling hash mapping value set and each second sampling hash mapping value set. Specifically, for each second hash mapping value set sent by the second device, the following steps are performed:

[0057] The first sampling number corresponding to the first sampling hash mapping value set and the second sampling number corresponding to the second sampling hash mapping value set are averaged to obtain the global sampling ranking value corresponding to the second device. Then, the first hash mapping value set and the second hash mapping value set are merged to obtain a global hash mapping value set. Then, a hash mapping value with a size ranking position of the global sampling ranking value in the global hash mapping value set is selected as the global sampling hash mapping value.

[0058] In addition, in another implementable manner, the step of determining the global sampling ranking value corresponding to each second device based on the first sampling number corresponding to the first sampling hash mapping value set and the second sampling number corresponding to each second sampling hash mapping value set further includes:

[0059] The smaller value between the first sample quantity corresponding to the first sample hash mapping value set and the second sample quantity corresponding to each second sample hash mapping value set is taken as the global sample ranking value.

[0060] The step of selecting the global sample hash mapping value corresponding to each global sample ranking value from the first sample hash mapping value set and each second sample hash mapping value set comprises:

[0061] In step S31, the first sample hash mapping value set is aggregated with each second sample hash mapping value set respectively to obtain each aggregation result.

[0062] In this embodiment, the first sample hash mapping value set is aggregated with each second sample hash mapping value set respectively to combine the first sample hash mapping value set with each second sample hash mapping value set respectively, and each aggregation result is obtained.

[0063] In step S32, the hash mapping value with the ranking position of each global sample ranking value in the size ranking is selected from each aggregation result as each global sample hash mapping value.

[0064] In this embodiment, the hash mapping value with the ranking position of each global sample ranking value in the size ranking is selected from each aggregation result as each global sample hash mapping value. Specifically, by sorting the hash mapping values in each aggregation result, the hash mapping value with the ranking position of each global sample ranking value in the size ranking is selected from each aggregation result as the global sample hash mapping value corresponding to each second device.

[0065] In step S40, based on each global sample ranking value and each global sample hash mapping value, the sample matching degree between the first device and each second device is calculated respectively.

[0066] In the embodiment, based on the global sampling ranking values and the global sampling hash mapping values, the sample matching degrees between the first device and each of the second devices are calculated respectively. Specifically, based on the global sampling ranking value and the global sampling hash mapping value corresponding to each of the second devices, the number of intersection samples between the first device and each of the second devices is estimated respectively, and then the proportion of each of the intersection sample numbers in the corresponding total sample number is calculated to obtain the sample matching degree between the first device and each of the second devices. The sample matching degree is the proportion of the overlapping samples between the first device and the second device. The total sample number can be set as the number of the first local sample IDs, or the sum of the number of the first local sample IDs and the number of the second local sample IDs in the corresponding second device.

[0067] In addition, it should be noted that, at present, the way of estimating the sample matching degree between each participant through the filter is that a malicious participant can easily deduce the local sample ID of other participants through rainbow attack, which has low security. Since the data exchanged between the first device and each of the second devices is only the mapping value of the hash value of the sample ID in the preset value interval, but not the hash value itself, the malicious participant cannot deduce the local sample ID of the participant through rainbow attack, thereby improving the security of calculating the sample matching degree in federated learning, and further improving the security of federated learning modeling and sample alignment.

[0068] Further, based on the sample matching degrees corresponding to each of the second devices, a federated learning modeling device is selected from each of the second devices, and then the first device performs federated learning modeling with each of the federated learning modeling devices to obtain a federated learning model, thereby achieving the purpose of selecting a device more matched with the first device for federated learning on the basis of quickly calculating the sample matching degree, and improving the effect and efficiency of federated learning modeling.

[0069] Further, when the federated learning model is a vertical federated learning model, based on the quickly calculated sample matching degrees between the first device and each of the second devices, a vertical federated learning device can be quickly selected from each of the second devices, and then the first device only needs to jointly perform sample prediction with each of the vertical federated learning devices, without jointly performing sample prediction with the second device removed based on the sample matching degree. Therefore, the data interaction process and data calculation process between the first device and the second device removed based on the sample matching degree are reduced, thereby achieving the purpose of quickly jointly performing sample prediction with each of the vertical federated learning devices on the basis of quickly calculating the sample matching degrees, and improving the efficiency of sample prediction in vertical federated learning. The vertical federated learning model can be a bank risk control model, thereby improving the efficiency of user loan risk prediction. The vertical federated learning model can also be a message recommendation model, thereby improving the efficiency of message recommendation.

[0070] The step of calculating the sample matching degree between the first device and each of the second devices based on the global sampling ranking value and the global sampling hash mapping value respectively comprises:

[0071] In the embodiment, the union sample number between the first device and each of the second devices is estimated based on the global sampling ranking value and the global sampling hash mapping value respectively, and specifically, for the global sampling ranking value corresponding to each of the second devices and the global sampling hash mapping value corresponding to each of the second devices, the following steps are performed:

[0072] In the embodiment, the union sample number between the first device and each of the second devices is estimated based on the global sampling ranking value and the global sampling hash mapping value respectively, and specifically, for the global sampling ranking value corresponding to each of the second devices and the global sampling hash mapping value corresponding to each of the second devices, the following steps are performed:

[0073] The difference between the global sampling ranking value and 1 is calculated, and the ratio between the difference and the global sampling hash mapping value is calculated to obtain the union sample number corresponding to the second device, wherein the formula for estimating the union sample number is as follows:

[0074]

[0075] Wherein, |M u | is the union sample number, k u is the global sampling ranking value, K u is the aggregation result, max(K u ) is the global sampling hash mapping value.

[0076] The step S42, based on each of the first local sample ID corresponding to the first sample set number, each of the second local sample ID corresponding to the second sample set number sent by each of the second device and each of the union sample number, respectively calculating the intersection sample number between the first device and each of the second device;

[0077] In the embodiment, the intersection sample number between the first device and each of the second devices is calculated based on the first sample set number corresponding to each of the first local sample ID, the second sample set number corresponding to each of the second local sample ID sent by each of the second device and each of the union sample number, and specifically, the first sample set number corresponding to each of the first local sample ID is obtained, and the second sample set number corresponding to each of the second local sample ID sent by each of the second device is received, and then for each of the second local sample ID corresponding to the second sample set number sent by each of the second device, the following steps are performed:

[0078] obtaining the second device corresponding to the intersection sample quantity, as Figure 2 The figure shows the flow diagram of the first device and the second device interacting to calculate the intersection sample quantity, wherein Guest is the first device, Host is the second device, H(id) is the first sampling hash value set and the second sampling hash value set, and global K is the global sampling order value and the global sampling hash value.

[0079] Step S43, based on each of the intersection sample quantity, calculating the sample matching degree between the first device and each of the second device.

[0080] In this embodiment, based on each of the intersection sample quantity, calculating the sample matching degree between the first device and each of the second device, specifically, calculating the ratio between each of the intersection sample quantity and the corresponding sample total quantity, obtaining the sample matching degree between the first device and each of the second device, for example, assuming that the intersection sample quantity is 100, the number of each of the first local sample ID is 1000, and then the sample matching degree is 10%.

[0081] Wherein, after the step of calculating the sample matching degree between the first device and each of the second device based on each of the global sampling order value and each of the global sampling hash value, the sample matching degree calculation optimization method further comprises:

[0082] Step A10, based on each of the sample matching degree, eliminating low matching degree devices with sample matching degree lower than a preset matching degree threshold in each of the second device, obtaining each high matching degree device; and performing longitudinal federated learning modeling with each of the high matching degree device, obtaining a longitudinal federated learning model;

[0083] In this embodiment, it should be noted that when performing longitudinal federated learning modeling, the higher the sample matching degree between each participant of longitudinal federated learning modeling, the better the effect of longitudinal federated learning modeling, the low matching degree device is the second device with sample matching degree lower than a preset matching degree threshold, and the high matching degree device is the second device with sample matching degree not lower than a preset matching degree threshold, the embodiment of the application realizes the purpose of eliminating participants with low sample matching degree before longitudinal federated learning modeling, so that the participants participating in longitudinal federated learning are all participants with high sample matching degree, and then the longitudinal federated learning model will converge faster, reducing the process of participants with low sample matching degree participating in longitudinal federated learning modeling, so as to improve the efficiency of longitudinal federated learning modeling.

[0084] Step B10, based on the sample matching degree of each of the samples, eliminating high matching degree devices with sample matching degree not lower than a preset matching degree threshold in each of the second devices, obtaining low matching degree devices; and performing transverse federated learning modeling with each of the low matching degree devices to obtain a transverse federated learning model.

[0085] In this embodiment, it should be noted that when performing transverse federated learning modeling, the lower the sample matching degree between each participant of transverse federated learning modeling, the better the effect of longitudinal federated learning modeling. The embodiment of the present application achieves the purpose of eliminating participants with high sample matching degree before transverse federated learning modeling, so that the participants of transverse federated learning are all participants with low sample matching degree, and thus the transverse federated learning model will converge faster, reducing the process of participants with high sample matching degree participating in transverse federated learning modeling. Therefore, the efficiency of longitudinal federated learning modeling is improved.

[0086] The embodiment of the present application provides a sample matching degree calculation optimization method. Compared with the technical means of estimating the sample matching degree between each participant by using a filter in the prior art, the embodiment of the present application first obtains each first local sample ID, and maps the first hash value corresponding to each of the first local sample IDs to a preset value interval to obtain the first hash mapping value corresponding to each of the first local sample IDs. Then, a first sampling hash mapping value set is selected from each of the first hash mapping values, and a second sampling hash mapping value set sent by each second device is received, wherein the second sampling hash mapping value set is selected by the second device from the second hash mapping value corresponding to each second local sample ID. Then, based on the first sampling number corresponding to the first sampling hash mapping value set and the second sampling number corresponding to each of the second sampling hash mapping value sets, the global sampling sorting value corresponding to each of the second devices is determined, and the global sampling hash mapping value corresponding to each of the global sampling sorting values is selected from the first sampling hash mapping value set and each of the second sampling hash mapping value sets. Then, based on each of the global sampling sorting values and each of the global sampling hash mapping values, the sample matching degree between the first device and each of the second devices is estimated respectively, achieving the purpose of calculating the sample matching degree between the first device and each of the second devices based on part of the hash values selected from the first device and each of the second devices. It is not necessary to calculate the sample matching degree by one-to-one comparison of the sample IDs of the first device and each of the second devices, reducing the calculation amount and calculation complexity of calculating the sample matching degree in federated learning. Therefore, the technical defect that the calculation amount and calculation complexity of one-to-one comparison of the sample IDs of each participant will be extremely high when the sample number of each participant is large, and thus the efficiency of calculating the sample matching degree in federated learning is extremely low is overcome, and the efficiency of calculating the sample matching degree in federated learning is improved.

[0087] Further, with reference to Figure 3In another embodiment of the present application based on the first embodiment, after the step of calculating the sample matching degree between the first device and each of the second devices based on each of the global sampling order values and each of the global sampling hash mapping values, the sample matching degree calculation optimization method further comprises:

[0088] In step S50, based on each of the sample matching degrees, low matching degree devices with a sample matching degree lower than a preset matching degree threshold are removed from each of the second devices to obtain high matching degree devices.

[0089] In this embodiment, it should be noted that before longitudinal federated learning is performed, sample alignment is usually performed to determine the common samples between each participant of the longitudinal federated learning. Currently, the common sample IDs between each participant are usually determined when each sample ID between each participant is compared one by one through a filter to calculate the sample matching degree. However, the calculation amount and calculation complexity of comparing the sample IDs of each participant one by one will be extremely high, which will result in low efficiency of sample alignment.

[0090] Based on each of the sample matching degrees, low matching degree devices with a sample matching degree lower than a preset matching degree threshold are removed from each of the second devices to obtain high matching degree devices. Specifically, each of the to-be-removed sample matching degrees lower than the preset matching degree threshold is selected from each of the sample matching degrees, and then low matching degree devices corresponding to each of the to-be-removed sample matching degrees are removed from each of the second devices to obtain high matching degree devices.

[0091] In step S60, sample alignment is performed with each of the high matching degree devices to obtain a sample alignment result.

[0092] In this embodiment, sample alignment is performed with each of the high matching degree devices to obtain a sample alignment result. Specifically, sample alignment is performed with each of the high matching degree devices to obtain each common sample ID between the first device and each of the high matching degree devices, and each of the common sample IDs is taken as the sample alignment result.

[0093] The sample alignment method provided in the embodiments of the present application is as follows: after the sample matching degrees of the first device and each second device are calculated, low matching degree devices with sample matching degrees lower than a preset matching degree threshold are removed from each second device based on the sample matching degrees, each high matching degree device is obtained, and then sample alignment is performed with each high matching degree device to obtain a sample alignment result, thereby achieving the purpose of selective sample alignment of the first device with each second device, and avoiding sample alignment with low matching degree devices in each second device. Since sample alignment is performed by comparing sample IDs between each participant one by one through a filter, the calculation complexity and calculation amount are much greater than the calculation complexity and calculation amount of calculating the sample matching degrees in the embodiments of the present application. Therefore, the calculation process of calculating the sample matching degrees is used to replace the calculation process of sample alignment with low matching degree devices in each second device, thereby reducing the calculation amount in the sample alignment process, and improving the efficiency of sample alignment.

[0094] Further, with reference to Figure 4 , the sample matching degree calculation optimization method is applied to a second device, and the sample matching degree calculation optimization method further includes:

[0095] In step C10, each second local sample ID is obtained, and a second hash value corresponding to each second local sample ID is mapped to a preset value interval to obtain a second hash mapping value corresponding to each second local sample ID.

[0096] In the embodiments, it should be noted that the sample matching degree calculation optimization method is applied to federated learning, the first device is a task cooperation participant in federated learning, the task cooperation participant needs to send its own data to a task initiation participant for aggregation, the number of second devices is at least 1, the second local sample ID is an identity of a sample in a second device, such as an ID card number and a mobile phone number, and the preset value interval is a specific value interval set in advance, and preferably, the preset value interval can be set as 0 to 1.

[0097] Each second local sample ID is obtained, and a second hash value corresponding to each second local sample ID is mapped to a preset value interval to obtain a second hash mapping value corresponding to each second local sample ID. Specifically, each second local sample ID is obtained, and each second local sample ID is subjected to hash processing to obtain a second hash value corresponding to each second local sample ID, and then each second hash value is mapped to a preset value interval based on the size of each second hash value to obtain a second hash mapping value corresponding to each second local sample ID.

[0098] The step of mapping each second local sample ID to a second hash value to a preset value interval to obtain a second hash mapping value corresponding to each second local sample ID comprises:

[0099] Step C11, respectively, each second local sample ID is hashed to obtain each second hash value;

[0100] In this embodiment, based on a preset first type of hash function, each second local sample ID is hashed to obtain each second hash value, wherein the preset first type of hash function includes but is not limited to sha256 and SM3 hash functions such as national secret.

[0101] Step C12, each second hash value is mapped to a floating point number in the preset value interval to obtain each second hash mapping value.

[0102] In this embodiment, each second hash value is mapped to a floating point number in the preset value interval to obtain each second hash mapping value, specifically, each second hash value is mapped to a floating point number in the preset value interval by a preset remainder function, and each second hash value is obtained. The second hash mapping value.

[0103] In another implementable way, step C12 further comprises:

[0104] By a preset second type of hash function, the second hash value is hashed twice to obtain a local secondary hash value, and the value of the preset number of bits in the local secondary hash value is taken as a floating point number in the preset value interval, wherein the result bit number of the preset second type of hash function is less than the result bit number of the preset first type of hash function, for example, the preset second type of hash function can be set to md5 hash function.

[0105] Step C20, each second hash mapping value is selected to send a second sampling hash mapping value set to the first device, so that the first device calculates the sample matching degree between the first device and each second device based on each second sampling hash mapping value set sent by the second device and the first sampling hash mapping value set generated based on each first local sample ID.

[0106] In the embodiment, the second sampling hash mapping value set selected from the second hash mapping values is sent to the first device, so that the first device calculates the sample matching degree between the first device and each of the second devices based on the second sampling hash mapping value set sent by each of the second devices and the first sampling hash mapping value set generated based on each first local sample ID. Specifically, the second sampling number of the smallest size of each hash mapping value in the second hash mapping value is selected as the second sampling hash mapping value set, and the second sampling hash mapping value set is sent to the first device. The first device determines the global sample ranking value corresponding to each of the second devices based on the first sampling number corresponding to the first sampling hash mapping value set and the second sampling number corresponding to each of the second sampling hash mapping value sets, and selects the global sample hash mapping value corresponding to each of the global sample ranking values in the first sampling hash mapping value set and each of the second sampling hash mapping value sets. Then, the first device calculates the sample matching degree between the first device and each of the second devices based on each of the global sample ranking values and each of the global sample hash mapping values. The process of calculating the sample matching degree between the first device and each of the second devices based on the first sampling hash mapping value set and the second sampling hash mapping value set by the first device can refer to the specific content in steps S30 and its refinement steps and S40 and its refinement steps, which will not be described here.

[0107] The embodiment of the present application provides a sample matching degree calculation optimization method. Compared with the technical means of estimating the sample matching degree between each participant by a filter in the prior art, the embodiment of the present application first obtains each second local sample ID, maps the second hash value corresponding to each second local sample ID to a preset value interval, obtains the second hash mapping value corresponding to each second local sample ID, and then sends a second sampling hash mapping value set selected from each second hash mapping value to a first device. The first device calculates the sample matching degree between the first device and each of the second devices based on the second sampling hash mapping value set sent by each of the second devices and the first sampling hash mapping value set generated based on each first local sample ID. The purpose of calculating the sample matching degree between the first device and each of the second devices based on part of the hash values selected in the first device and each of the second devices is achieved. The calculation amount and the calculation complexity of calculating the sample matching degree by one-to-one comparison of the sample IDs of the first device and each of the second devices are reduced. Therefore, the technical defect that the calculation amount and the calculation complexity of one-to-one comparison of the sample IDs of each participant are extremely high when the number of samples of each participant is large, and the efficiency of calculating the sample matching degree in federated learning is extremely low is overcome, and the efficiency of calculating the sample matching degree in federated learning is improved.

[0108] Referring to Figure 5 , Figure 5 is a device structure schematic diagram of a hardware running environment involved in an embodiment of the present application.

[0109] As Figure 5 shown, the sample matching degree calculation optimization device can include a processor 1001, such as a CPU, a memory 1005, and a communication bus 1002. The communication bus 1002 is used to realize the connection and communication between the processor 1001 and the memory 1005. The memory 1005 can be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 can also be an optional storage device independent of the aforementioned processor 1001.

[0110] Optionally, the sample matching degree calculation optimization device can also include a rectangular user interface, a network interface, a camera, an RF (Radio Frequency, radio frequency) circuit, a sensor, an audio circuit, a WiFi module, and the like. The rectangular user interface can include a display screen (Display) and an input sub-module such as a keyboard (Keyboard). The optional rectangular user interface can also include a standard wired interface and a wireless interface. The network interface can optionally include a standard wired interface and a wireless interface (such as a WI-FI interface).

[0111] Those skilled in the art can understand that Figure 5 the sample matching degree calculation optimization device structure shown in the above does not constitute a limitation on the sample matching degree calculation optimization device, and can include more or fewer components than those shown, or combine certain components, or different component arrangements.

[0112] Figure 5 As shown, the memory 1005, as a computer storage medium, can include an operating system, a network communication module, and a sample matching degree calculation optimization program. The operating system is a program that manages and controls the hardware and software resources of the sample matching degree calculation optimization device, supports the running of the sample matching degree calculation optimization program and other software and / or programs. The network communication module is used to realize the communication between the components in the memory 1005, and the communication between other hardware and software in the sample matching degree calculation optimization system.

[0113] Figure 5 In the sample matching degree calculation optimization device shown in the above , the processor 1001 is used to execute the sample matching degree calculation optimization program stored in the memory 1005, to realize the steps of the sample matching degree calculation optimization method described in any one of the above.

[0114] The sample matching degree calculation optimization device embodiment of the present application is basically the same as each embodiment of the above sample matching degree calculation optimization method, and will not be repeated here.

[0115] The present application also provides a sample matching degree calculation optimization device, which is applied to a first device and includes:

[0116] A mapping module is configured to obtain each first local sample ID and map a first hash value corresponding to each first local sample ID to a preset value interval to obtain a first hash mapping value corresponding to each first local sample ID.

[0117] A receiving module is configured to select a first sampling hash mapping value set from each first hash mapping value and receive a second sampling hash mapping value set sent by each second device, wherein the second sampling hash mapping value set is selected by each second device from a second hash mapping value corresponding to each second local sample ID.

[0118] A selecting module is configured to determine a global sampling order value corresponding to each second device based on a first sampling number corresponding to the first sampling hash mapping value set and a second sampling number corresponding to each second sampling hash mapping value set, and select a global sampling hash mapping value corresponding to each global sampling order value from the first sampling hash mapping value set and each second sampling hash mapping value set.

[0119] A calculating module is configured to calculate a sample matching degree between the first device and each second device based on each global sampling order value and each global sampling hash mapping value.

[0120] Optionally, the calculating module is further configured to:

[0121] estimate a union sample number between the first device and each second device based on each global sampling order value and each global sampling hash mapping value;

[0122] calculate an intersection sample number between the first device and each second device based on a first sample set number corresponding to each first local sample ID, a second sample set number corresponding to each second local sample ID sent by each second device, and each union sample number;

[0123] calculate a sample matching degree between the first device and each second device based on each intersection sample number.

[0124] Optionally, the selecting module is further configured to:

[0125] aggregate each of the first sampling hash mapping value sets with each of the second sampling hash mapping value sets respectively to obtain each of an aggregation result;

[0126] select, in each of the aggregation results, a hash mapping value with a ranking position in size ordering as each of the global sampling ranking values.

[0127] Optionally, the mapping module is further configured to:

[0128] perform hash processing on each of the first local sample IDs respectively to obtain each of a first hash value;

[0129] map each of the first hash values into a floating point number in the preset value range to obtain each of a first hash mapping value.

[0130] Optionally, the receiving module is further configured to:

[0131] perform ordering on each of the first hash mapping values to obtain a local ordering result;

[0132] select, based on the local ordering result, a first sampling number of hash mapping values with a late ranking in each of the first hash mapping values as each of the first sampling hash mapping value sets.

[0133] Optionally, the sample matching degree calculation optimization apparatus is further configured to:

[0134] based on each of the sample matching degrees, eliminate, in each of the second devices, a low matching degree device with a sample matching degree lower than a preset matching degree threshold to obtain each of a high matching degree device;

[0135] perform sample alignment with each of the high matching degree devices to obtain a sample alignment result.

[0136] Optionally, the sample matching degree calculation optimization apparatus is further configured to:

[0137] based on each of the sample matching degrees, eliminate, in each of the second devices, a low matching degree device with a sample matching degree lower than a preset matching degree threshold to obtain each of a high matching degree device; perform longitudinal federated learning modeling with each of the high matching degree devices to obtain a longitudinal federated learning model; and / or

[0138] based on each of the sample matching degrees, eliminate, in each of the second devices, a high matching degree device with a sample matching degree not lower than a preset matching degree threshold to obtain each of a low matching degree device; perform transverse federated learning modeling with each of the low matching degree devices to obtain a transverse federated learning model.

[0139] The specific embodiments of the sample matching degree calculation optimization apparatus of the present application are basically the same as those of the above-mentioned sample matching degree calculation optimization method, and will not be repeated here.

[0140] The embodiment of the present application further provides a sample matching degree calculation optimization device, which is applied to a first device and comprises:

[0141] a mapping module, configured to obtain each second local sample ID and map a second hash value corresponding to each second local sample ID to a preset value interval to obtain a second hash mapping value corresponding to each second local sample ID;

[0142] a sending module, configured to send a second sampling hash mapping value set selected from the second hash mapping values to the first device, so that the first device calculates a sample matching degree between the first device and each second device based on a first sampling hash mapping value set generated based on each first local sample ID and each second sampling hash mapping value set sent by each second device.

[0143] Optionally, the mapping module is further configured to:

[0144] perform hash processing on each second local sample ID to obtain each second hash value;

[0145] map each second hash value to a floating point number in the preset value interval to obtain each second hash mapping value.

[0146] The specific implementation of the sample matching degree calculation optimization device of the present application is basically the same as that of the above-mentioned sample matching degree calculation optimization method, and will not be repeated here.

[0147] The embodiment of the present application provides a medium, which is a readable storage medium, and the readable storage medium stores one or more programs, and the one or more programs can also be executed by one or more processors to implement the steps of the sample matching degree calculation optimization method of any one of the above-mentioned embodiments.

[0148] The specific implementation of the readable storage medium of the present application is basically the same as that of the above-mentioned sample matching degree calculation optimization method, and will not be repeated here.

[0149] The embodiment of the present application provides a computer program product, and the computer program product comprises one or more computer programs, and the one or more computer programs can also be executed by one or more processors to implement the steps of the sample matching degree calculation optimization method of any one of the above-mentioned embodiments.

[0150] The specific implementation of the computer program product of the present application is basically the same as that of the above-mentioned sample matching degree calculation optimization method, and will not be repeated here.

[0151] The above merely provides the preferred embodiments of the present application, and is not intended to limit the patent scope of the present application. Any equivalent structure or equivalent flow transformation made by the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent processing scope of the present application.

Claims

1. A sample matching degree calculation optimization method, characterized by, The sample matching degree calculation optimization method is applied to a first device, the first device is a task initiating participant with a federal server function in federated learning, the federal server function is a function of aggregating data sent by all participants in federated learning, and the sample matching degree calculation optimization method comprises: Obtain each first local sample ID, and map the first hash value corresponding to each first local sample ID to a preset value interval to obtain a first hash mapping value corresponding to each first local sample ID, the first local sample ID being an identity of a loan user in the first device; Select a first sampling hash mapping value set from each first hash mapping value, and receive a second sampling hash mapping value set sent by each second device, wherein the second sampling hash mapping value set is selected from a second hash mapping value corresponding to each second local sample ID by the second device; Based on the first sampling quantity corresponding to the first sampling hash mapping value set and the second sampling quantity corresponding to each second sampling hash mapping value set, determine a global sampling ranking value corresponding to each second device, and select a global sampling hash mapping value corresponding to each global sampling ranking value from the first sampling hash mapping value set and each second sampling hash mapping value set; Based on each global sampling ranking value and each global sampling hash mapping value, estimate the union sample quantity between the first device and each second device; Based on the first sample set quantity corresponding to each first local sample ID, the second sample set quantity corresponding to each second local sample ID sent by each second device, and each union sample quantity, calculate the intersection sample quantity between the first device and each second device; Based on each intersection sample quantity, calculate the sample matching degree between the first device and each second device to reduce the calculation amount of the first device in federated learning, while ensuring the security of the sample matching degree calculation in federated learning.

2. The method of claim 1, wherein the similarity measure is calculated by: The step of selecting a global sampling hash mapping value corresponding to each global sampling ranking value from the first sampling hash mapping value set and each second sampling hash mapping value set comprises: Aggregating each second sampling hash mapping value set with the first sampling hash mapping value set respectively to obtain each aggregation result; Selecting a hash mapping value with a ranking position corresponding to each global sampling ranking value as each global sampling hash mapping value in each aggregation result.

3. The sample matching degree calculation optimization method as described in claim 1, characterized in that, The step of mapping each first hash value corresponding to each first local sample ID to a preset value interval to obtain a first hash mapping value corresponding to each first local sample ID comprises: Hash processing each first local sample ID respectively to obtain each first hash value; Mapping each first hash value to a floating point number in the preset value interval to obtain each first hash mapping value.

4. The method of claim 1, wherein the sample matching degree is calculated by using a similarity measure. The step of selecting a first sampling hash mapping value set from each first hash mapping value comprises: Sorting each first hash mapping value to obtain a local ranking result; Based on the local sorting result, select a first sampling number of hash mapping values with late ranking in each of the first hash mapping values as the first sampling hash mapping value set.

5. The method of claim 1, wherein the sample matching degree is calculated by using a similarity measure. After the step of calculating the sample matching degree between the first device and each of the second devices based on each of the global sampling sorting values and each of the global sampling hash mapping values, the sample matching degree calculation optimization method further comprises: Based on each of the sample matching degrees, eliminate low matching degree devices with sample matching degrees lower than a preset matching degree threshold from each of the second devices to obtain high matching degree devices; Perform sample alignment with each of the high matching degree devices to obtain a sample alignment result.

6. The method of claim 1, wherein the sample matching degree is calculated by using a similarity measure. After the step of calculating the sample matching degree between the first device and each of the second devices based on each of the global sampling sorting values and each of the global sampling hash mapping values, the sample matching degree calculation optimization method further comprises: Based on each of the sample matching degrees, eliminate low matching degree devices with sample matching degrees lower than a preset matching degree threshold from each of the second devices to obtain high matching degree devices; and perform longitudinal federated learning modeling with each of the high matching degree devices to obtain a longitudinal federated learning model; and / or Based on each of the sample matching degrees, eliminate high matching degree devices with sample matching degrees not lower than a preset matching degree threshold from each of the second devices to obtain low matching degree devices; and perform transverse federated learning modeling with each of the low matching degree devices to obtain a transverse federated learning model.

7. A sample matching degree calculation optimization method characterized by comprising: The sample matching degree calculation optimization method is applied to a second device, and the sample matching degree calculation optimization method comprises: Obtain each second local sample ID, and map a second hash value corresponding to each of the second local sample IDs to a preset value interval to obtain a second hash mapping value corresponding to each of the second local sample IDs; Obtain each second local sample ID, and map a second hash value corresponding to each of the second local sample IDs to a preset value interval to obtain a second hash mapping value corresponding to each of the second local sample IDs; The second sampling hash mapping value set selected from the second hash mapping values is sent to the first device, so that the first device determines a global sampling ranking value corresponding to each second device based on a first sampling number corresponding to the first sampling hash mapping value set and a second sampling number corresponding to each second sampling hash mapping value set, selects a global sampling hash mapping value corresponding to each global sampling ranking value from the first sampling hash mapping value set and each second sampling hash mapping value set, respectively estimates a union sample number between the first device and each second device based on each global sampling ranking value and each global sampling hash mapping value, respectively calculates an intersection sample number between the first device and each second device based on a first sample set number corresponding to each first local sample ID, a second sample set number corresponding to each second local sample ID sent by each second device, and each union sample number, and calculates a sample matching degree between the first device and each second device based on each intersection sample number, wherein the first sampling hash mapping value set is obtained by the first device acquiring each first local sample ID, mapping a first hash value corresponding to each first local sample ID to a preset value interval to obtain a first hash mapping value corresponding to each first local sample ID, and selecting the first hash mapping value, and the first local sample ID is an identity of a loan user in the first device.

8. The sample matching degree calculation optimization method as described in claim 7, characterized in that, The step of mapping each second hash value corresponding to each second local sample ID to a preset value interval to obtain a second hash mapping value corresponding to each second local sample ID comprises: respectively performing hash processing on each second local sample ID to obtain each second hash value; mapping each second hash value to a floating point number in the preset value interval to obtain each second hash mapping value.

9. A sample matching degree calculation optimization device characterized by comprising: The sample matching degree calculation optimization device comprises a memory, a processor, and a program stored in the memory and used for implementing the sample matching degree calculation optimization method, The memory is configured to store the program for implementing the sample matching degree calculation optimization method. The processor is configured to execute the program for implementing the sample matching degree calculation optimization method to implement the steps of the sample matching degree calculation optimization method according to any one of claims 1 to 6 or 7 to 8.

10. A medium, the medium being a readable storage medium, characterized by The readable storage medium has the program for implementing the sample matching degree calculation optimization method stored thereon, and the program for implementing the sample matching degree calculation optimization method is executed by the processor to implement the steps of the sample matching degree calculation optimization method according to any one of claims 1 to 6 or 7 to 8.

11. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the sample matching degree calculation optimization method according to any one of claims 1 to 6 or 7 to 8.

Images