Federated learning and data processing method, device, equipment and storage medium

Abstract

The application provides a federated learning and data processing method, device, equipment and storage medium, which can be applied to privacy, security, artificial intelligence, intelligent transportation, auxiliary driving and the like. The federated learning method comprises: a first computing platform acquires first forward output data, the first forward output data being obtained by processing sample data of the first computing platform by a second bottom model in a second computing platform; a second forward output data is obtained by processing sample data of the first computing platform by a first bottom model; a first processing result is obtained by processing the first forward output data and the second forward output data by a top teacher model; a second processing result is obtained by processing the second forward data by a first top student model; and the first top student model is trained based on the first processing result and the second processing result and a label. In subsequent reasoning, the first top student model does not communicate with the second computing platform, thereby reducing communication overhead.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a federated learning and data processing method, apparatus, device and storage medium. Background Technology

[0002] Federated Learning (FL) technology, by jointly training machine learning models through interaction only with encrypted model gradients or intermediate computation results, avoids the direct transmission of raw data and provides a new solution for cross-departmental, cross-organizational, and cross-industry data collaboration.

[0003] Federated learning includes vertical federated learning. In the current inference process of vertical federated learning, the Host and Guest sides need to perform joint inference. That is, the Host side sends its forward output to the Guest side, and the Guest side concatenates the output and inputs it into the top model to obtain the inference result. However, the current federated learning inference method has high communication overhead, resulting in latency. Summary of the Invention

[0004] This application provides a federated learning and data processing method, apparatus, device, and storage medium that can save communication overhead, thereby reducing latency and improving data processing efficiency.

[0005] In a first aspect, this application provides a federated learning method applied to a first labeled computing platform, the first computing platform comprising a first bottom model, a first top student model, and a top teacher model, including:

[0006] Obtain first forward output data, which is obtained by the second bottom model in the second computing platform based on the sample data held by the second computing platform, and the second computing platform is the sample data holder;

[0007] The sample data held by the first computing platform is processed through the first bottom model to obtain the second forward output data;

[0008] The first forward output data and the second forward output data are processed using the top teacher model to obtain the first processing result;

[0009] The second forward data is processed using the first top student model to obtain the second processing result;

[0010] Based on the first processing result and the second processing result, as well as the label, the first top student model is trained.

[0011] Secondly, this application provides a data processing method applied to a first computing platform holding a tag, the method comprising:

[0012] The target data in the first computing platform is processed using the first bottom model trained in the first computing platform to obtain the intermediate calculation results corresponding to the target data.

[0013] The intermediate calculation results are processed using the first top student model trained in the first computing platform to obtain the inference results.

[0014] Thirdly, this application provides a data processing method applied to a first computing platform holding a tag, the method comprising:

[0015] The target data in the second computing platform is processed by the second bottom model trained in the second computing platform to obtain the intermediate calculation results corresponding to the target data.

[0016] The intermediate calculation results are processed using the second top student model trained in the second computing platform to obtain the inference results.

[0017] Fourthly, this application provides a data processing method applied to a first computing platform holding a tag, the method comprising:

[0018] Obtain the first intermediate calculation result sent by the second computing platform. The first intermediate calculation result is obtained by the second bottom model trained in the second computing platform based on the first data in the second computing platform.

[0019] Based on the first intermediate calculation result and the top teacher model trained in the first computing platform, the inference result is obtained.

[0020] Fifthly, this application provides a federated learning device applied to a first computing platform holding tags, the first computing platform including a first bottom model, a first top student model, and a top teacher model, comprising:

[0021] The acquisition unit is used to acquire first forward output data, which is obtained by the second bottom model in the second computing platform based on the sample data held by the second computing platform, and the second computing platform is the sample data holder;

[0022] The first processing unit is used to process the sample data held by the first computing platform through the first bottom model to obtain the second forward output data.

[0023] The second processing unit is used to process the first forward output data and the second forward output data through the top teacher model to obtain the first processing result;

[0024] The third processing unit is used to process the second forward data through the first top student model to obtain the second processing result;

[0025] The training unit is used to train the first top student model based on the first processing result, the second processing result, and the label.

[0026] Sixthly, this application provides a data processing apparatus applied to a first computing platform holding a tag, comprising:

[0027] The intermediate computing unit is used to process the target data in the first computing platform through the first bottom model trained in the first computing platform to obtain the intermediate computing result corresponding to the target data.

[0028] The inference unit is used to process the intermediate calculation results through the first top student model trained in the first computing platform to obtain the inference result;

[0029] The trained first top student model is trained based on the first processing result, the second processing result, and the labels held by the first computing platform. The first processing result is obtained by processing the first forward output data and the second forward output data by the top teacher model in the first computing platform. The second processing result is obtained by processing the second forward output data by the first top student model. The first forward output data is obtained by processing the sample data held by the second bottom model in the second computing platform. The second forward output data is obtained by processing the sample data held by the first computing platform by the first bottom model.

[0030] In a seventh aspect, this application provides a data processing apparatus applied to a second computing platform holding sample data, comprising:

[0031] The intermediate computing unit is used to process the target data in the second computing platform through the second bottom model trained in the second computing platform to obtain the intermediate computing results corresponding to the target data;

[0032] The inference unit is used to process the intermediate calculation results through the second top student model trained in the second computing platform to obtain the inference result;

[0033] The trained second top student model is trained based on the third processing result, the first processing result, and the labels held by the first computing platform. The first computing platform includes the trained first top student model, which is trained based on the first processing result, the second processing result, and the labels. The first processing result is obtained by processing the first forward output data and the second forward output data through the top teacher model in the first computing platform. The second processing result is obtained by processing the second forward output data through the first top student model. The third processing result is obtained by processing the first forward output data through the second top student model. The first forward output data is obtained by the second bottom model processing the sample data held by the second computing platform. The second forward output data is obtained by the first bottom model in the first computing platform processing the sample data held by the first computing platform.

[0034] Eighthly, this application provides a data processing apparatus applied to a first computing platform holding a tag, comprising:

[0035] The acquisition unit is used to acquire the first intermediate calculation result sent by the second computing platform. The first intermediate calculation result is obtained by the second bottom model trained in the second computing platform based on the first data processing in the second computing platform.

[0036] The inference unit is used to obtain the inference result based on the first intermediate calculation result and the top teacher model trained in the first computing platform;

[0037] The trained top teacher model is trained based on the first processing result and the labels held by the first computing platform. The first computing platform includes a trained first top student model, which is trained based on the first processing result, the second processing result, and the labels. The first processing result is obtained by processing the first forward output data and the second forward output data by the top teacher model. The second processing result is obtained by processing the second forward output data by the first top student model. The first forward output data is obtained by the second bottom model processing the sample data held by the second computing platform. The second forward output data is obtained by the first bottom model in the first computing platform processing the sample data held by the first computing platform.

[0038] In a tenth aspect, this application provides an electronic device including a processor and a memory. The memory is used to store a computer program, and the processor is used to invoke and run the computer program stored in the memory to perform the methods described in any one of the first to fourth aspects.

[0039] Eleventhly, a chip is provided for implementing the methods of any one of the first to fourth aspects or their respective implementations. Specifically, the chip includes: a processor for calling and running a computer program from a memory, causing a device equipped with the chip to perform the methods of the first to fourth aspects as described above.

[0040] In a twelfth aspect, a computer-readable storage medium is provided for storing a computer program that causes a computer to perform the methods described in the first to fourth aspects.

[0041] In a twelfth aspect, a computer program product is provided, including computer program instructions that cause a computer to perform the methods described in the first to fourth aspects.

[0042] In a thirteenth aspect, a computer program is provided that, when run on a computer, causes the computer to perform the methods described in the first to fourth aspects.

[0043] In summary, the first computing platform of this application includes a first bottom model, a first top student model, and a top teacher model, while the second computing platform includes a second bottom model. The first computing platform acquires first forward output data, which is obtained by the second bottom model in the second computing platform processing the sample data held by the second computing platform. The first computing platform processes the sample data held by the first computing platform through the first bottom model to obtain second forward output data. The first computing platform processes the first and second forward output data through the top teacher model to obtain a first processing result, and simultaneously processes the second forward data through the first top student model to obtain a second processing result. Finally, the first computing platform trains the first top student model based on the first and second processing results and labels. In this embodiment, the sample data held by the first and second computing platforms are used for joint training of the first top student model, which improves the training accuracy of the first top student model. In addition, the sample data of the second computing platform is not directly used during the specific training process of the first top student model. Thus, during subsequent inference, the first top student model performs local inference without communicating with the second computing platform, thereby reducing communication overhead. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a schematic diagram illustrating the principle of traditional vertical federated deep learning.

[0046] Figure 2 This is a schematic diagram illustrating an application scenario according to an embodiment of this application;

[0047] Figure 3 A flowchart illustrating a federated learning method provided in an embodiment of this application;

[0048] Figure 4 This is a schematic diagram of a model training method according to an embodiment of this application;

[0049] Figure 5 This is a schematic diagram of a federated learning method provided in an embodiment of this application;

[0050] Figure 6 This is a schematic diagram of a model training method according to an embodiment of this application;

[0051] Figure 7 This is a schematic diagram of a federated learning method provided in an embodiment of this application;

[0052] Figure 8 This is a schematic diagram of federated learning involved in an embodiment of this application;

[0053] Figure 9 This is a schematic flowchart of a data processing method provided in an embodiment of this application;

[0054] Figure 10 This is a schematic diagram illustrating a reasoning process according to an embodiment of this application;

[0055] Figure 11 This is a schematic flowchart of a data processing method provided in an embodiment of this application;

[0056] Figure 12 This is a schematic diagram illustrating a reasoning process according to an embodiment of this application;

[0057] Figure 13 This is a schematic flowchart of a data processing method provided in an embodiment of this application;

[0058] Figure 14 This is a schematic diagram illustrating a reasoning process according to an embodiment of this application;

[0059] Figure 15This is a schematic block diagram of a federated learning device provided in one embodiment of this application;

[0060] Figure 16 This is a schematic block diagram of a data processing apparatus provided in an embodiment of this application;

[0061] Figure 17 This is a schematic block diagram of a data processing apparatus provided in an embodiment of this application;

[0062] Figure 18 This is a schematic block diagram of a data processing apparatus provided in an embodiment of this application;

[0063] Figure 19 This is a schematic block diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0064] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0065] It should be understood that, in the embodiments of the present invention, "B corresponding to A" means that B is associated with A. In one implementation, B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information.

[0066] In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0067] Furthermore, to facilitate a clear description of the technical solutions in the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and the terms "first" and "second" are not necessarily different.

[0068] The federated learning and data processing method provided in this application can be applied to various scenarios such as privacy, security, artificial intelligence, smart transportation, and assisted driving.

[0069] In some embodiments, the methods of this application can also be applied to cloud security to protect privacy and improve the security of privacy data. Cloud security refers to the collective term for security software, hardware, users, organizations, and security cloud platforms based on cloud computing business models. Cloud security integrates emerging technologies and concepts such as parallel processing, grid computing, and unknown virus behavior detection. Through a large number of clients in a mesh network, it monitors abnormal software behavior in the network, obtains the latest information on Trojans and malware on the Internet, sends it to the server for automatic analysis and processing, and then distributes solutions for viruses and Trojans to each client.

[0070] In some embodiments, the methods of this application can also be applied to Artificial Intelligence (AI). AI is the theory, methods, technology, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine capable of reacting in a manner similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

[0071] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0072] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0073] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0074] In some embodiments, if the technical solutions of this application are applied to intelligent transportation, they can be used in conjunction with Intelligent Traffic Systems (ITS) and Intelligent Vehicle Infrastructure Cooperative Systems (IVICS). Intelligent Traffic Systems, also known as Intelligent Transportation Systems, effectively integrate advanced technologies (information technology, computer technology, data communication technology, sensor technology, electronic control technology, automatic control theory, operations research, artificial intelligence, etc.) into transportation, service control, and vehicle manufacturing, strengthening the connection between vehicles, roads, and users to form a comprehensive transportation system that ensures safety, improves efficiency, improves the environment, and saves energy. Intelligent Vehicle Infrastructure Cooperative Systems, or vehicle-road cooperative systems for short, represent a development direction of Intelligent Traffic Systems (ITS). These systems utilize advanced wireless communication and next-generation Internet technologies to implement dynamic real-time information interaction between vehicles and roads in all directions. Based on the collection and fusion of dynamic traffic information across all times and spaces, they conduct active vehicle safety control and road cooperative management, fully realizing effective coordination between people, vehicles, and roads, ensuring traffic safety, improving traffic efficiency, and thus forming a safe, efficient, and environmentally friendly road traffic system.

[0075] It should be noted that, in addition to applications in privacy, security, artificial intelligence, smart transportation, and assisted driving, the embodiments of this application can also be applied to other scenarios and fields, and the embodiments of this application do not impose any limitations on them.

[0076] This application relates to Federated Learning (FL) technology. FL trains machine learning models jointly by exchanging only encrypted model gradients or intermediate computation results, avoiding the direct transmission of raw data and providing a new solution for cross-departmental, cross-organizational, and cross-industry data collaboration. Based on the distribution of data among different participants, federated learning can be divided into Vertical Federated Learning (VFL), Horizontal Federated Learning (HFL), and Federated Transfer Learning (FTL). Vertical Federated Deep Learning achieves deep learning model training without data exchange through information exchange via split layers (also known as Cut Layers or Interactive Layers).

[0077] Vertical federated learning improves model training performance by uniting multiple participants with the same sample space but different feature space datasets, increasing the feature dimensionality involved in training. Similarly, in this scenario, the Host (feature holder) and Guest (label holder, who may also hold features) share the same sample space. To train a deep learning model, the Guest aims to collaboratively apply the Host's feature data without exposing its privacy, while ensuring its own label information is not leaked. Figure 1 As shown, the core idea of ​​traditional vertical federated deep learning is to allow each participant to train a bottom model locally using its own features. During training, the guest party fuses the forward outputs of all bottom models (either by concatenation or by calculating the mean) and inputs it into its top model. It then uses the data labels and the top model's output to calculate the loss function and backpropagate gradients, thus updating the parameters of both the top and bottom models. Whether in vertical federated learning or split learning scenarios, the original data of the host and guest parties does not leave their domains or share data; the data is usable but not visible, thereby strengthening the protection of data privacy.

[0078] Depend on Figure 1As can be seen, in the current inference process of vertical federated learning, the Host and Guest sides need to perform joint inference. That is, the Host side sends its forward output to the Guest side for concatenation before inputting it into the top model to obtain the inference result. This traditional joint inference method has poor real-time performance. For example, in modern advertising bidding systems, the number of requests per second can reach millions, while all user identification, bidding, and ad delivery processes need to be completed within 10 to 100 milliseconds. However, current inference methods cannot meet the real-time requirements. In addition, current inference methods suffer from high communication overhead. For example, due to the need for high-frequency message interaction, traditional joint inference methods result in heavy communication overhead.

[0079] In some embodiments, quantization and pruning techniques are used to quantize and prune the forward output of the bottom model to reduce communication overhead. Quantization can reduce the precision of parameters, thereby reducing the amount of communication required for data of the same dimension and accelerating inference computation. Pruning can remove less important parts of the parameters, retaining only the parameters that are important to the inference result, thereby reducing the number of parameters that need to be transmitted. However, parameter quantization and pruning can only reduce communication overhead to a certain extent and cannot significantly improve the real-time performance of the bidding system. Furthermore, strict quantization and pruning may greatly impair the availability of the system and are not suitable for widespread application in real-world systems.

[0080] To address the aforementioned technical issues, this application embodiment, in the federated learning process, firstly sets up a first bottom model, a first top student model, and a top teacher model on the Guest side (i.e., the first computing platform), and sets up a second bottom model on the Host side (i.e., the second computing platform). The Host side processes the sample data it holds using the second bottom model to obtain first forward output data, and sends this first forward output data to the Guest side. The Guest side processes the sample data it holds using the first bottom model to obtain second forward output data. The Guest side processes the first and second forward output data using the top teacher model to obtain a first processing result. Simultaneously, the Guest side processes the second forward data using the first top student model to obtain a second processing result. Finally, the Guest side trains the first top student model based on the first and second processing results and the labels it holds. In this embodiment, the sample data held by the Host and Guest parties are used to jointly train the first top student model, which improves the training accuracy of the first top student model. In addition, the sample data of the Host party is not directly used in the specific training process of the first top student model. In this way, the first top student model performs local inference in subsequent inference without communicating with the Host party, thereby reducing communication overhead.

[0081] The application scenarios of the embodiments of this application are described below.

[0082] Figure 2 This is a schematic diagram of an application scenario according to an embodiment of this application, including a first computing platform 101 and a second computing platform 102. The first computing platform 101 can be understood as the Guest side, and the second computing platform 102 can be understood as the Host side.

[0083] The host has some feature data of the sample, but does not have the sample's label data.

[0084] The Guest party possesses the tag information, and in some embodiments, the Guest may also possess another portion of the sample's feature data.

[0085] The Host and Guest do not share or transmit any sample feature or tag data. This ensures that the sample data does not leave the local machine; that is, the Host's feature data does not leave the Host, and the Guest's tag and feature data do not leave the Guest, thus guaranteeing privacy protection at the sample data level.

[0086] In some embodiments, the first computing platform 101 and / or the second computing platform 102 include at least one terminal device. The terminal device may include, but is not limited to, mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc. The device is often equipped with a display device, which may also be a monitor, display screen, touch screen, etc., and the touch screen may also be a touchscreen, touch panel, etc.

[0087] In some embodiments, the first computing platform 101 and / or the second computing platform 102 include servers. There may be one or more servers. When there are multiple servers, at least two servers provide different services, and / or at least two servers provide the same service, such as providing the same service through load balancing; this embodiment does not limit this. The servers may be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Server 102 may also be a node in a blockchain.

[0088] In this embodiment of the application, the first computing platform 101 and the second computing platform 102 can be directly or indirectly connected through wired or wireless communication, and this application does not impose any restrictions.

[0089] In the embodiments of this application, during the federated learning process, the second computing platform 102 includes a first bottom model. The second computing platform 102 inputs local sample data into the first bottom model, the first bottom model inputs first forward output data, and sends the first forward output data to the first computing platform 101. The first computing platform 101 uses the first forward output data to train at least one top student model.

[0090] In some embodiments, at least one top student model includes a first top student model, which is trained based on the first forward output data. The first computing platform 101 then uses the trained first top student model. The second computing platform 102 processes the local data using the first top student model to obtain the inference result. In other words, in this embodiment, the second computing platform 102 uses the locally stored trained first top student model to perform data inference locally, without needing to communicate with the first computing platform 101, thereby reducing communication overhead and improving data processing speed.

[0091] In some embodiments, at least one top student model includes a second top student model, which is trained based on the first forward output data and the data included in the first computing platform 101. Thus, the first computing platform 101 processes its local data using the second top student model to obtain inference results. In other words, in this embodiment, the first computing platform 101 uses the locally stored second top student model to perform data inference locally, without needing to communicate with the second computing platform 102, thereby reducing communication overhead and improving data processing speed.

[0092] It should be noted that the application scenarios of this application embodiment include, but are not limited to, those of other applications. Figure 1 As shown.

[0093] The technical solutions of the embodiments of this application will be described in detail below through some examples. The following embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0094] First, the federated learning process of this application embodiment will be introduced. The federated learning process can be understood as the process by which the first computing platform trains the model based on federated learning.

[0095] Figure 3 This is a flowchart illustrating a federated learning method provided in one embodiment of this application.

[0096] like Figure 3 As shown, the federated learning in this application embodiment includes:

[0097] S301, The first computing platform acquires the first forward output data.

[0098] Among them, the first forward output data z (A) The second bottom model in the second computing platform is based on the sample data X held by the second computing platform. (A) The data was processed, and the second computing platform is the holder of the sample data.

[0099] The first computing platform in this application embodiment can be understood as the Guest side, and the second computing platform can be understood as the Host side.

[0100] In this system, the second computing platform holds the sample data, while the first computing platform holds the tags. The second and first computing platforms do not exchange raw data. That is, the second computing platform will not send the sample data it holds to the first computing platform, and the first computing platform will not send the tags it holds to the second computing platform. This ensures that the raw data does not leave its domain, is not shared, and remains usable but not visible, thereby strengthening the protection of data privacy.

[0101] The second computing platform holds the sample data, but the model of the first computing platform needs to be trained using the sample data from the second computing platform. In this embodiment, the second computing platform does not send the sample data it holds to the first computing platform for model training. Instead, it processes the sample data and sends the processed data to the first computing platform for model training, thereby achieving accurate model training without exposing privacy information.

[0102] In this embodiment of the application, to address the problem of increased communication overhead and data processing latency caused by frequent data interactions between the first and second computing platforms during inference, resulting in poor real-time data processing, the first computing platform of this embodiment includes at least one top student model capable of local recommendation. For example, one or more of these top student models can perform data inference processing on the second computing platform without interacting with the first computing platform during the inference process. Alternatively, one or more of these top student models can perform data inference processing on the first computing platform without interacting with the second computing platform during the inference process. This allows for local data inference processing without communication with the other party, thereby reducing communication overhead, lowering data processing latency, and improving the timeliness of data processing.

[0103] In some embodiments, such as Figure 4 As shown, the first computing platform in this application embodiment includes a first top student model, which can be jointly trained to improve the training accuracy of the first top student model. In addition, the first top student model may not directly use the sample data of the second computing platform during the training process. In this way, the first top student model performs local inference during subsequent inference without communicating with the second computing platform, thereby reducing communication overhead.

[0104] Furthermore, in this embodiment, to achieve accurate training of the first top student model, the first computing platform also includes a first bottom model and a top teacher model. Meanwhile, to protect the privacy of the sample data of the second computing platform, the second computing platform includes a second bottom model. The second bottom model processes the sample data held by the second computing platform before sending it to the first computing platform, instead of directly sending the sample data held by the second computing platform to the first computing platform. This achieves accurate model training without exposing privacy information.

[0105] The federated learning process involved in the embodiments of this application will be described in detail below.

[0106] Specifically, in this embodiment of the application, the second computing platform initializes a bottom model, which, for ease of description, is referred to as the second bottom model V. (A) (·), the second computing platform uses the second bottom model V (A) (·) Process the sample data X(A) held by the second computing platform to obtain the first forward output data z. (A) For example, the second computing platform inputs its own sample data into the second bottom model V. (A) In (·), the second bottom model V (A) (·) Output the first forward output data z (A) The first forward output data z (A) This can be understood as an intermediate calculation result.

[0107] This application implements the second bottom model V (A) The specific network structure of (·) is not restricted; it can be set according to the actual scenario and needs.

[0108] In some embodiments of this application, the second computing platform uses a second bottom model V. (A) (·) Process the held sample data to obtain the first forward output data z. (A) Then, the first forward output data z (A) Send it to the first computing platform so that the first computing platform can use the first forward output data z (A) Perform model training.

[0109] In some embodiments, the second computing platform will output the first forward data z (A) The data is stored on another platform, and the first computing platform obtains the first forward output data z from the other platform. (A) .

[0110] Optionally, the first computing platform can also obtain the first forward output data z from the second computing platform through other means. (A) However, the embodiments in this application do not impose any limitations on this.

[0111] The first computing platform obtains the first forward output data z. (A) Then, perform the following steps S302.

[0112] S302. The first computing platform processes the sample data held by the first computing platform through the first bottom model to obtain the second forward output data.

[0113] In this embodiment, the first computing platform includes not only labels, a first top student model, and a top teacher model T, but also... (B) (·), also includes sample data and the first bottom model V (B) (·)

[0114] In this embodiment, if the first computing platform also includes sample data, in order to increase the training data of the model and achieve effective training of the model, the first computing platform of this application embodiment uses the intermediate calculation results of the sample data in the second computing platform, namely the first forward output result, and the sample data held by the first computing platform itself to train the first top student model, thereby improving the training accuracy of the model.

[0115] During model training, the first computing platform uses its first bottom model V. (B) (·), for the first computing platform including sample data X (B) After processing, the second forward output data z is obtained. (B) For example, the first computing platform inputs sample data into the first bottom model V. (B) (·), thus obtaining the first bottom model V (B) The output of (·) will be the first bottom model V (B) The output of (·) is denoted as the second forward output data z. (B) .

[0116] This application embodiment describes the first bottom model V. (B) The specific network structure of (·) is not restricted; it can be set according to the actual scenario and needs.

[0117] In some embodiments, the network structures of the first bottom model and the second bottom model can be the same.

[0118] S303, the first computing platform processes the first forward output data and the second forward output data through the top teacher model to obtain the first processing result.

[0119] In order to achieve accurate training of the first top student model, the embodiments of this application, such as Figure 4 As shown, the first computing platform sets up a top teacher model, which guides the training of the first top student model, thereby achieving accurate training of the first top student model.

[0120] This application embodiment does not limit the specific method by which the first computing platform processes the first forward output data and the second forward output data through the top teacher model to obtain the first processing result.

[0121] In some embodiments, the first computing platform continuously inputs the first forward output data and the second forward output data into the top teacher model to obtain the first processing result.

[0122] In some embodiments, the first computing platform simultaneously inputs the first forward output data and the second forward output data into the top teacher model to obtain the first processing result.

[0123] In some embodiments, after processing the first forward output data and the second forward output data, the first computing platform inputs the data into the top teacher model to obtain the first processing result.

[0124] For example, S303 above includes S303-A1 and S303-A2:

[0125] S303-A1: The first forward output data and the second forward output data are fused to obtain fused forward output data;

[0126] S303-A2: The fusion forward output data is processed through the top teacher model to obtain the first processing result.

[0127] In this example, the first computing platform outputs the first forward data z. (A) Second forward output data z (B) The data is fused to obtain fused forward output data. In this embodiment, the first forward output data z... (A) Second forward output data z (B) The specific method of fusion is not limited; for example, the first forward output data z... (A) Second forward output data z (B) Directly concatenating the data yields the fused forward output data. For example, the first forward output data z... (A) Second forward output data z (B) The average value is denoted as the fused forward output data.

[0128] After the first computing platform obtains the fused forward output data, it uses the top teacher model T... (B) (·), process the fused forward output data to obtain the first processing result P. T For example, the fused forward output data is input into the top teacher model T. (B) (·), to obtain the top teacher model T (B) (·) Output the first processed result P T .

[0129] Based on the above steps, the first computing platform obtains the top teacher model T. (B) (·) Output the first processed result P T Then, based on the first processing result P T Train the first top student model.

[0130] This application embodiment refers to the top teacher model T. (B)The specific network structure of (·) is not restricted; it can be set according to the actual scenario and needs.

[0131] In some embodiments, the aforementioned top teacher model T (B) (·) represents a pre-trained model. During the model training process in this embodiment, the top teacher model T is used... (B) (·) No training.

[0132] In some embodiments of this application, during the modeling process, the top teacher model T is... (B) (·) Training is required. In other words, the first computing platform, in addition to using the first processing result P, T In addition to training the first top student model, the first computing platform also uses the first processing result P T For the top teacher model T (B) (·) Training is performed. Specifically, the first computing platform uses the top teacher model T. (B) (·) In the first processed result P of the current output T And label y, determine the top teacher model T (B) The value of the loss function (·) is based on the top teacher model T. (B) The value of the loss function L (·) T Determine the top teacher model T (B) The third gradient of the (·) parameter And based on the third gradient For the top teacher model T (B) The (·) parameter is updated.

[0133] This application embodiment uses a top teacher model T for the first computing platform. (B) (·) In the first processed result P of the current output T And label y, determine the top teacher model T (B) The specific process of determining the value of the loss function (·) is not restricted.

[0134] In one example, the first computing platform determines the top teacher model T based on the following formula (1). (B) The value of the loss function L (·) T :

[0135] L T =CE(y, P) T (1)

[0136] Here, CE stands for cross-entropy operation.

[0137] S304. The first computing platform processes the second forward data through the first top student model to obtain the second processing result.

[0138] It should be noted that there is no strict order in which S304 and S303 are executed. That is, S304 can be executed before S303, after S303, or in parallel with S303. This application embodiment does not impose any restrictions on this.

[0139] In this embodiment, if the first computing platform includes sample data, that is, if the first computing platform includes sample data in addition to labels, the first computing platform initializes a first bottom model V. (B) (·) and a first top student model S (B) (·), through the first bottom model V (B) (·) Process the sample data included in the first computing platform to obtain the second forward output data z. (B) Next, the first computing platform uses the first top student model S (B) (·) Regarding the second forward output data z (B) The second processing result was obtained after processing. For example, the first computing platform will output the second forward data z. (B) Input the first top student model S (B) In (·), the first top student model S is obtained. (B) (·) The second processed result output

[0140] The embodiments of this application do not limit the specific network structure of the first top student model, but can be set according to actual needs and scenarios.

[0141] After determining the first processing result and the second processing result through the above steps, the first computing platform executes the following step S305.

[0142] S305, the first computing platform trains the first top student model based on the first processing result and the second processing result, as well as the label.

[0143] As described above, the first processing result is obtained by the top teacher model processing the first forward output data and the second forward output data corresponding to the second computing platform, while the second processing result is obtained by the first top student model processing the second forward output data corresponding to the first computing platform. Therefore, the gradient of the parameters of the first top student model can be determined through the first processing result, the second processing result, and the labels, and then the parameters of the first top student model can be updated based on this gradient.

[0144] This application embodiment does not limit the specific method by which the first computing platform trains the first top student model based on the first processing result, the second processing result, and the label.

[0145] In some embodiments, S305 above includes the following steps S305-A1 to S305-A2:

[0146] S305-A1, based on the second processing result, the first processing result, and the label, determine the loss of the first top student model;

[0147] S305-A2: Based on the loss of the first top student model, determine the first gradient of the parameters of the first top student model, and update the parameters of the first top student model based on the first gradient.

[0148] This application embodiment determines a first top student model S on a first computing platform. (B) The specific manner of loss (·) is not restricted.

[0149] In some embodiments, the first computing platform determines a first divergence value between the first processing result and the second processing result; determines a first cross-entropy between the label and the second processing result; and obtains the loss of the first top student model based on the first divergence value and the first cross-entropy.

[0150] In one example, the first computing platform determines the first top student model S using the following formula (2). (B) (·) loss

[0151]

[0152] Where α is a preset value, CE is the cross-entropy operation, and KL is the divergence operation. The first cross-entropy, This is the first divergence value.

[0153] Based on the above method, the first computing platform determines the first top student model S. (B) After calculating the loss (·), the first top student model S is determined based on this loss. (B) The first gradient of the parameter (·) is used to apply the first gradient to the first top student model S. (B) The parameters (·) are updated to achieve the first top student model S (B) An update of (·).

[0154] As can be seen from the above, the first top student model S (B) (·) During training, the first forward output data z of the first computing platform (i.e., the Host side) was not directly used. (A) Therefore, in some embodiments, the first computing platform can utilize the trained first top student model S (B)(·), data inference processing is performed independently on the first computing platform. During the inference process, there is no interaction with the second computing platform, thereby reducing communication overhead and improving data processing efficiency.

[0155] In this embodiment of the federated learning method, the first computing platform includes a first bottom model, a first top student model, and a top teacher model, while the second computing platform includes a second bottom model. The first computing platform acquires first forward output data, which is obtained by the second bottom model in the second computing platform processing the sample data held by the second computing platform. The first computing platform processes the sample data held by the first computing platform through the first bottom model to obtain second forward output data. The first computing platform processes the first forward output data and the second forward output data through the top teacher model to obtain a first processing result, and simultaneously processes the second forward data through the first top student model to obtain a second processing result. Finally, the first computing platform trains the first top student model based on the first and second processing results and labels. In this embodiment, the first top student model is jointly trained using sample data held by the first computing platform and the second computing platform, which improves the training accuracy of the first top student model. In addition, the sample data of the second computing platform is not directly used in the specific training process of the first top student model. In this way, the first top student model performs local inference in subsequent inference without communicating with the second computing platform, thereby reducing communication overhead.

[0156] Figure 5 This is a flowchart illustrating a federated learning method provided in one embodiment of this application. Figure 6 This is a schematic diagram of a training model provided in an embodiment of this application. Figure 6 As shown, this application embodiment also includes a second top student model, in which case, as Figure 5 As shown, the embodiments of this application include the following steps:

[0157] S501, the first computing platform processes the first forward output data z through the second top student model. (A) The process is performed to obtain the third result.

[0158] In this embodiment, in addition to defining a first top student model, the first computing platform also defines another top student model S. (A) (·), the top student model S (A) (·) is denoted as the second top student model S. (A) (·). The first computing platform uses the second top student model S. (A) (·), for the first forward output data z (A) The process was performed to obtain the third result. For example, the first computing platform will output the first forward data z (A) Input the second top student model S (A) In (·), we obtain the second top student model S. (A) (·) The third processing result output

[0159] S502, the first computing platform determines the loss of the second top student model based on the third processing result, the first processing result, and the label.

[0160] Next, the first computing platform, based on the third processing result... First processing result P T And the label y, determine the second top student model S (A) The value of the loss function (·)

[0161] This application embodiment determines the second top student model S on the first computing platform. (A) loss function of (·) There are no restrictions on the specific methods used.

[0162] In some embodiments, a second divergence value is determined between the first processing result and the third processing result; a second cross-entropy is determined between the label and the third processing result; and the loss of the second top student model is obtained based on the second divergence value and the second cross-entropy.

[0163] In one example, the first computing platform determines the second top student model S based on the following formula (3). (A) (·) loss

[0164]

[0165] This is the second cross-entropy.

[0166] S503. The first computing platform determines the second gradient of the parameters of the second top student model based on the loss of the second top student model, and updates the parameters of the second top student model based on the second gradient.

[0167] Based on the above method, the first computing platform determines the second top student model S. (A) The value of the loss function (·) Then, based on the value of this loss function Determine the second top student model S (A) The second gradient of the (·) parameter is used, and this second gradient is applied to the second top student model S. (A) The parameters (·) are updated to achieve the second top student model S. (A)An update of (·).

[0168] In some embodiments, the first computing platform uses the trained second top student model for local inference.

[0169] In some embodiments, as can be seen from the above, the second top student model S (A) (·) During training, the second forward output data z of the second computing platform (i.e., the Guest side) was not directly used. (B) Therefore, in some embodiments, the first computing platform is on the second top student model S. (A) (·) After training, the trained second top student model S can be... (A) (·) is sent to the second computing platform, so that the second computing platform can use the trained second top student model S. (A) (·), data inference processing is performed independently on the second computing platform. During the inference process, there is no interaction with the first computing platform, thereby reducing communication overhead and improving data processing efficiency.

[0170] In some embodiments, the first computing platform, in addition to training the first top student model S (B) (·) and the second top student model S (A) In addition to (·), the first computing platform also determines the second bottom model V in the second computing platform. (A) The backpropagation gradient corresponding to (·) is sent to the second computing platform so that the second computing platform can determine the second bottom model V based on the backpropagation gradient. (A) The gradient of the (·) parameter, and based on the second bottom model V (A) The gradient of the (·) parameter with respect to the second bottom model V (A) (·) The parameters are updated. Specifically, the embodiments of this application also include the following steps:

[0171] Step 11: The first computing platform determines the second bottom model V based on the backpropagation method. (A) The backpropagation gradient corresponding to (·);

[0172] Step 12: The first computing platform will use the second bottom model V (A) The backpropagation gradient corresponding to (·) is sent to the second computing platform, so that the second computing platform can be based on the second bottom model V. (A) The backpropagation gradient corresponding to (·) is used for the second bottom model V. (A) The (·) parameter is updated.

[0173] In this embodiment of the application, the second bottom model V (A) (·) and the top teacher model T (B)(·) is used as a whole for model training. The top teacher model T is used as the core. (B) The gradient of the (·) parameter can be determined based on the loss function, and the second bottom model V (A) The gradients of the (·) parameters are determined using backpropagation and the chain rule. In other words, the first computational platform first determines the top teacher model T. (B) The gradient of the parameters (·) is calculated, and the chain rule is used to perform the top teacher model T. (B) (·) Parameter update. Next, determine the second bottom model V. (A) The backpropagation gradient corresponding to (·) is then sent to the second computing platform, enabling the second computing platform to determine the second bottom model V using this backpropagation gradient. (A) The gradient of the (·) parameter, and based on the second bottom model V (A) The gradient of the (·) parameter with respect to the second bottom model V (A) The (·) parameter is updated.

[0174] In one example, the first computing platform uses the following formula (4) to determine the second bottom model V. (A) The backpropagation gradient ▽ corresponding to (·) (A) :

[0175]

[0176] The first computing platform determined the second bottom model V. (A) The backpropagation gradient ▽ corresponding to (·) (A) Then, the backpropagation gradient ▽ (A) This is sent to the second computing platform. The second computing platform can then use this backpropagation gradient ▽ (A) For the second bottom model V (A) The (·) parameter completes one update.

[0177] In some embodiments, the first computing platform also supports the first bottom model V. (B) (·) Training is performed. Specifically, the embodiments of this application also include the following steps:

[0178] Step 21: The first computing platform determines the first bottom model V based on the backpropagation method. (B) The backpropagation gradient corresponding to (·);

[0179] Step 22: The first computing platform is based on the first bottom model V. (B) The backpropagation gradient corresponding to (·) determines the first bottom model V. (B) The gradient of the (·) parameter, and based on the first bottom model V (B) The gradient of the (·) parameter with respect to the first bottom model V (B)The (·) parameter is updated.

[0180] In this embodiment of the application, the second bottom model V (A) (·), Top Teacher Model T (B) (·) and the first bottom model V (B) (·) is used as a whole for model training. The top teacher model T is used as the core. (B) The gradient of the (·) parameter can be determined based on the loss function, and the second bottom model V (A) (·) and the first bottom model V (B) The gradients of the (·) parameters are determined using backpropagation and the chain rule. In other words, the first computational platform first determines the top teacher model T. (B) The gradient of the parameters (·) is calculated, and the chain rule is used to perform the top teacher model T. (B) (·) Parameter update. Next, determine the second bottom model V. (A) The backpropagation gradient corresponding to (·) is then sent to the second computing platform, enabling the second computing platform to determine the second bottom model V using this backpropagation gradient. (A) The gradient of the (·) parameter, and based on the second bottom model V (A) The gradient of the (·) parameter with respect to the second bottom model V (A) The parameters (·) are updated. Simultaneously, the first computing platform determines the first bottom model V. (B) The backpropagation gradient corresponding to (·) is based on the first bottom model V. (B) The backpropagation gradient corresponding to (·) determines the first bottom model V. (B) The gradient of the (·) parameter is calculated, and the first bottom model V is used. (B) The gradient of the (·) parameter with respect to the first bottom model V (B) The (·) parameter is updated.

[0181] In one example, the first computing platform uses the following formula (5) to define the first bottom model V. (B) The backpropagation gradient g corresponding to (·) (B) :

[0182]

[0183] Among them, W (B) For the first bottom model V (B) (·)parameter.

[0184] The first computing platform determined the first bottom model V. (B) The backpropagation gradient g corresponding to (·) (B) Then, based on this backpropagation gradient g (B) Implement the first bottom model V (B)(·) is an update of the parameter.

[0185] It should be noted that the above model update steps are performed once for each sample in each epoch (iteration round), and repeated for several epochs until the model converges on the training sample set.

[0186] The federated learning method provided in this application involves a first computing platform training a second top student model based on the first forward output data during model training, and then sending the trained second top student model to the second computing platform. Since the second top student model does not directly use the sample data from the first computing platform during training, the second computing platform can use the trained second top student model for independent data inference without needing to communicate with the first computing platform, thereby reducing communication overhead and improving data processing efficiency.

[0187] The preceding text describes the federated learning process of this application's embodiments, using the first computing platform as the execution entity. The following section, in conjunction with... Figure 7 Taking the interaction process between the first computing platform and the second platform as an example, a specific embodiment of federated learning provided in this application will be introduced.

[0188] Figure 7 This is a schematic diagram of a federated learning method flow provided in an embodiment of this application, as shown below. Figure 7 As shown, it includes:

[0189] S701, The second computing platform initializes the second bottom model.

[0190] S702. The first computing platform initializes the first bottom model, the top teacher model, the second top student model, and the first top student model.

[0191] It should be noted that the specific implementation processes of S701 and S702 are not sequential; that is, S701 can be executed before S702, or after S702, or simultaneously with S702.

[0192] Figure 7 The embodiments shown can be understood as a specific embodiment of this application.

[0193] like Figure 8 As shown, the second computing platform initializes a bottom model, namely the second bottom model V. (A) (·). The first computing platform initializes a bottom model, namely the first bottom model V. (B) (·) and a top model T (B) (·). Among them, the top model T (B) (·) is considered to be the top teacher model T in the knowledge distillation process.(B) (·). Furthermore, the first computing platform needs to define a first top student model S corresponding to the first computing platform. (B) (·)

[0194] Optionally, if the first computing platform allows the second computing platform to perform unilateral reasoning, then the first computing platform further defines a second top student model S corresponding to the second computing platform. (A) (·)

[0195] For example, the parameters of all the models defined above are randomly initialized.

[0196] In some embodiments, if the first computing platform does not include sample data, i.e. feature data, the first computing platform may not have a first bottom model V. (B) (·) and the first top student model S (B) (·)

[0197] After completing the above model initialization process, the first and second computing platforms define the current training sample dataset as follows: Where i is the sample number. For example, X can be used. (A) and X (B) These refer to the sample data held by the second computing platform and the first computing platform, respectively, and y represents the label of the sample X.

[0198] S703, the second computing platform processes the sample data held by the second computing platform through the second bottom model to obtain the first forward output data.

[0199] For example, the second computing platform will hold the sample data X. (A) Input the second bottom model V (A) (·), thus obtaining the second bottom model V (A) (·) Output the first forward output data z (A) .

[0200] S704, the second computing platform sends the first forward output data to the first computing platform.

[0201] S705, the first computing platform processes the sample data held by the first computing platform through the first bottom model to obtain the second forward output data.

[0202] For example, the first computing platform will hold the sample data X. (B) Input the first bottom model V (B) (·), thus obtaining the first bottom model V (B) (·) Output the second forward output data z (B) .

[0203] S706, the first computing platform fuses the first forward output data and the second forward output data to obtain fused forward output data.

[0204] This application embodiment addresses the first forward output data z. (A) Second forward output data z (B) There are no restrictions on the specific methods of integration.

[0205] For example, the first forward output data z (A) Second forward output data z (B) By directly splicing the data, we obtain the fused forward output data.

[0206] For example, the first forward output data z (A) Second forward output data z (B) The average value is denoted as the fused forward output data.

[0207] S707, the first computing platform processes the fused forward output data through the top teacher model to obtain the first processing result.

[0208] For example, the first computing platform will integrate the forward output data into the top teacher model T. (B) (·), to obtain the top teacher model T (B) (·) Output the first processed result P T .

[0209] S708, the first computing platform uses the first processing result and label to determine the loss of the top teacher model, and based on the loss of the top teacher model, determines the third gradient of the top teacher model parameters, and updates the top teacher model parameters based on the third gradient.

[0210] The specific implementation process of S708 described above can be referred to the description of the above embodiments, and will not be repeated here.

[0211] S709, the first computing platform processes the first forward output data through the second top student model to obtain the third processing result.

[0212] For example, the first computing platform will output the first forward data z (A) Input the second top student model S (A) (·), thus obtaining the second top student model S (A) (·) The third processing result output

[0213] S710 and the first computing platform determine the loss of the second top student model based on the third processing result, the first processing result, and the label; and based on the loss of the second top student model, determine the second gradient of the parameters of the second top student model, and update the parameters of the second top student model based on the second gradient.

[0214] The specific implementation process of S710 can be referred to the description of S703 above, and will not be repeated here.

[0215] S711, the first computing platform processes the second forward output data through the first top student model to obtain the second processing result.

[0216] For example, the first computing platform will output the second forward data z (B) Input the first top student model S (B) (·), thus obtaining the first top student model S (B) (·) The second processed result output

[0217] S712, the first computing platform determines the loss of the first top student model based on the second processing result, the first processing result, and the label; based on the loss of the first top student model, it determines the first gradient of the parameters of the first top student model, and updates the parameters of the first top student model based on the first gradient.

[0218] The specific implementation process of S712 can be referred to the description of S305 above, and will not be repeated here.

[0219] S713. The first computing platform determines the backpropagation gradient corresponding to the first bottom model based on the backpropagation method; and updates the parameters of the first bottom model based on the backpropagation gradient corresponding to the first bottom model.

[0220] For example, the first computing platform determines the backpropagation gradient corresponding to the first bottom model based on the backpropagation method, determines the gradient of the parameters of the first bottom model based on the backpropagation gradient corresponding to the first bottom model, and updates the parameters of the first bottom model based on the gradient of the parameters of the first bottom model.

[0221] The specific implementation process of S713 can be referred to the description of steps 21 to 22 above, and will not be repeated here.

[0222] S714. The first computing platform determines the backpropagation gradient corresponding to the second bottom model based on the backpropagation method.

[0223] The specific implementation process of S714 can be referred to the description of step 11 above, and will not be repeated here.

[0224] S715, the first computing platform sends the backpropagation gradient corresponding to the second bottom model to the second computing platform.

[0225] S716 The second computing platform determines the gradient of the second bottom model parameters based on the backpropagation gradient corresponding to the second bottom model, and updates the second bottom model parameters based on the gradient of the second bottom model parameters.

[0226] The above model update steps are performed once for each sample in each epoch (iteration round), and repeated for several epochs until the model converges on the training sample set.

[0227] As can be seen from the above, in the embodiments of this application, the first computing platform trains at least one top student model, such as training a second top student model and a first top student model.

[0228] In some embodiments, the first computing platform uses a second top student model and / or a first top student model for data inference.

[0229] In some embodiments, the first computing platform can send the second top student model to the second computing platform, so that the second computing platform can perform inference independently using the trained second top student model. In this case, the embodiments of this application further include the following step S717.

[0230] S717, the first computing platform sends the trained second top student model to the second computing platform.

[0231] As described above, in the federated learning process of this application embodiment, the sample data of both the first and second computing platforms remain locally, satisfying the privacy requirements of vertical federated learning. Simultaneously, a top student model is introduced to enable unilateral inference between the first and second computing platforms, reducing communication costs during inference and improving data inference efficiency.

[0232] In this embodiment of the application, the first computing platform utilizes the first forward output data z sent by the second computing platform. (A) At the same time, the top teacher model T on the first computing platform side was completed. (B) (·) and the training of the second student model. It is important to note that the first top student model S on the first computing platform side... (B) (·) did not directly use the first forward output data z of the second computing platform. (A) Training is performed, therefore the first computing platform can utilize the first bottom model V. (B) (·) and the first top student model S (B) (·) is used for unilateral model reasoning.

[0233] Similarly, the second top student model S on the second computing platform side (A) (·) did not directly use the second forward output data z of the first computing platform. (B) Training is performed, therefore the first computing platform can utilize the first bottom model V. (B) (·) and the first top student model S (B) (·) is used for unilateral model reasoning.

[0234] The federated learning method provided in this application, through a first computing platform, builds upon traditional vertical federated deep learning by using the existing top model as the teacher model and defining at least one top student model for model training based on knowledge distillation. The second computing platform requires no additional design or operation during the training phase. Thus, during data inference, the Host or Guest can use the trained top student model independently for data inference without needing to communicate with each other, thereby reducing communication overhead, improving data processing efficiency, and meeting the low latency and low communication overhead requirements of scenarios such as advertising bidding systems.

[0235] The federated learning method involved in the embodiments of this application has been introduced above. The reasoning process of the embodiments of this application will be introduced below.

[0236] Figure 9 This is a schematic flowchart of a data processing method provided in an embodiment of this application, as shown below. Figure 9 As shown, embodiments of this application include:

[0237] S601. The first computing platform processes the target data in the first computing platform using the first bottom model trained in the first computing platform to obtain the intermediate calculation results corresponding to the target data.

[0238] S602. The first computing platform processes the intermediate calculation results using the first top student model trained in the first computing platform to obtain the inference results.

[0239] In this embodiment, the specific training process of the first bottom model and the first top student model refers to the specific description of the above federated learning embodiment. For example, the trained first top student model is trained based on the first processing result, the second processing result, and the labels held by the first computing platform. The first processing result is obtained by processing the first forward output data and the second forward output data by the top teacher model in the first computing platform. The second processing result is obtained by processing the second forward output data by the first top student model. The first forward output data is obtained by the second bottom model in the second computing platform based on the sample data held by the second computing platform. The second forward output data is obtained by the first bottom model processing the sample data held by the first computing platform.

[0240] In this embodiment, such as Figure 10 As shown, the first computing platform includes a pre-trained first bottom model V. (B) (·) and the trained first top student model S (B) (·). In this way, the first computing platform can use the locally trained first bottom model V (B) (·) and the trained first top student model S (B) (·) Performing unilateral data reasoning does not require communication with a second computing platform, thereby reducing communication overhead and improving data processing efficiency.

[0241] Specifically, such as Figure 9 and Figure 10 As shown, the first computing platform first inputs its target data into the trained first bottom model V. (B) The intermediate calculation results corresponding to the target data are obtained by processing the data within the parentheses (·). Then, the first computing platform inputs the intermediate calculation results corresponding to the target data into the trained first top student model S. (B) (·) Perform unilateral data reasoning to obtain the reasoning result corresponding to the target data.

[0242] It should be noted that the reasoning result in the embodiments of this application can be understood as the task processing result. For example, if the task is a prediction task, the reasoning result can be the prediction result; if the task is a classification task, the reasoning result can be the classification result, etc.

[0243] In this embodiment, the first computing platform (i.e., the Guest) can use the trained top student model to perform independent inference processing on the target data held by the first computing platform without communicating with the second computing platform (i.e., the Host), thereby reducing communication overhead, improving data processing efficiency, and meeting the needs of low latency and low communication overhead in scenarios such as advertising bidding systems.

[0244] Figure 11This is a schematic flowchart of a data processing method provided in an embodiment of this application, as shown below. Figure 11 As shown, embodiments of this application include:

[0245] S801, the second computing platform processes the target data in the second computing platform through the second bottom model trained in the second computing platform, and obtains the intermediate calculation results corresponding to the target data;

[0246] S802, the second computing platform processes the intermediate calculation results using the second top student model trained in the second computing platform to obtain the inference results.

[0247] In this embodiment, the specific training process of the second bottom model and the second top student model refers to the specific description of the above federated learning embodiment. For example, the trained second top student model is trained based on the third processing result, the first processing result, and the labels held by the first computing platform. The first computing platform includes the trained first top student model, which is trained based on the first processing result, the second processing result, and the labels. The first processing result is obtained by processing the first forward output data and the second forward output data through the top teacher model in the first computing platform. The second processing result is obtained by processing the second forward output data through the first top student model. The third processing result is obtained by processing the first forward output data through the second top student model. The first forward output data is obtained by the second bottom model processing the sample data held by the second computing platform, and the second forward output data is obtained by the first bottom model in the first computing platform processing the sample data held by the first computing platform.

[0248] In this embodiment, such as Figure 12 As shown, the second computing platform includes a trained second bottom model V. (A) (·) and the trained second top student model S (A) (·). In this way, the second computing platform can use the locally trained second bottom model and the trained second top student model S. (A) (·) Performing one-way data reasoning does not require communication with the first computing platform, thereby reducing communication overhead and improving data processing efficiency.

[0249] Specifically, such as Figure 11 and Figure 12 As shown, the second computing platform first inputs its target data into the trained second bottom model V. (A) The intermediate calculation results corresponding to the target data are obtained by processing the data in (·). Then, the second computing platform inputs the intermediate calculation results corresponding to the target data into the trained second top student model S. (A)(·) Perform unilateral data reasoning to obtain the reasoning result corresponding to the target data.

[0250] In this embodiment, the second computing platform (i.e., the Host) can use the trained top student model to perform independent inference processing on the target data held by the second computing platform without communicating with the first computing platform (i.e., the Guest), thereby reducing communication overhead, improving data processing efficiency, and meeting the needs of low latency and low communication overhead in scenarios such as advertising bidding systems.

[0251] Figure 13 This is a schematic flowchart of a data processing method provided in an embodiment of this application, as shown below. Figure 13 As shown, embodiments of this application include:

[0252] S901, The first computing platform obtains the first intermediate computing result sent by the second computing platform.

[0253] The first intermediate calculation result is the second bottom model V trained in the second computing platform. (A) (·) is obtained based on the first data processing in the second computing platform.

[0254] During the data inference process, the first computing platform and the second platform communicate and interact.

[0255] Specifically, the second computing platform inputs the first data held by the second computing platform into the trained second bottom model V. (A) (·), to obtain the first intermediate calculation result. Then, the second computing platform sends the first intermediate calculation result to the first computing platform.

[0256] S902, the first computing platform obtains inference results based on the first intermediate computing results and the top teacher model trained in the first computing platform.

[0257] In this embodiment, the specific training process of the second bottom model and the top teacher model refers to the specific description of the above federated learning embodiment. For example, the trained top teacher model is trained based on the first processing result and the labels held by the first computing platform. The first computing platform includes a trained first top student model, which is trained based on the first processing result, the second processing result, and the labels. The first processing result is obtained by processing the first forward output data and the second forward output data by the top teacher model. The second processing result is obtained by processing the second forward output data by the first top student model. The first forward output data is obtained by the second bottom model processing the sample data held by the second computing platform, and the second forward output data is obtained by the first bottom model in the first computing platform processing the sample data held by the first computing platform.

[0258] In this embodiment of the application, during the data reasoning process, the second computing platform sends the first intermediate calculation result to the first computing platform so that the first computing platform can determine the reasoning result based on the first computing platform.

[0259] The above-mentioned implementation methods of S902 include, but are not limited to, the following:

[0260] Method 1: The first computing platform uses the trained top teacher model T (B) (·), process the first intermediate calculation result to obtain the reasoning result.

[0261] For example, if the first computing platform does not include data for inference, then the first computing platform directly inputs the first intermediate calculation result into the trained top teacher model T. (B) In (·), the trained top teacher model T (B) (·) Output the reasoning result.

[0262] Method 2, using the first bottom model V trained in the first computing platform (B) (·), process the second data in the first computing platform to obtain the second intermediate computing result; fuse the first intermediate computing result and the second intermediate computing result to obtain the intermediate fused computing result; and use the trained top teacher model T (B) (·), process the intermediate fusion calculation results to obtain the inference results.

[0263] like Figure 14 As shown, if the first computing platform includes second data for inference, then the first computing platform inputs the second data from the first computing platform into the trained first bottom model V. (B) In the (·) section, the second intermediate calculation result is obtained. Next, the first computing platform fuses the first and second intermediate calculation results to obtain an intermediate fused calculation result. This can be achieved by concatenating the first and second intermediate calculation results, or by averaging the first and second intermediate calculation results. Finally, the intermediate fused calculation result is input into the trained top teacher model T. (B) (·), to obtain the trained top teacher model T (B) (·) The inference result output.

[0264] The data processing method provided in this application embodiment can realize joint reasoning of a first computing platform and a second computing platform. The top teacher mode is used in the reasoning process, which can improve the accuracy of data reasoning.

[0265] The above text combined Figures 3 to 14The method embodiments of this application are described in detail below, in conjunction with... Figures 15 to 19 The following describes in detail the device embodiments of this application.

[0266] Figure 15 This is a schematic block diagram of a federated learning device provided in one embodiment of this application. The device 10 can be applied to a first computing platform, which includes a first bottom model, a first top student model, and a top teacher model.

[0267] like Figure 15 As shown, the federated learning device 10 includes:

[0268] The acquisition unit 11 is used to acquire first forward output data, which is obtained by the second bottom model in the second computing platform based on the sample data held by the second computing platform. The second computing platform is the sample data holder.

[0269] The first processing unit 12 is used to process the sample data held by the first computing platform through the first bottom model to obtain the second forward output data.

[0270] The second processing unit 13 is used to process the first forward output data and the second forward output data through the top teacher model to obtain a first processing result;

[0271] The third processing unit 14 is used to process the second forward data through the first top student model to obtain the second processing result;

[0272] Training unit 15 is used to train the first top student model based on the first processing result, the second processing result, and the label.

[0273] In some embodiments, the training unit 15 is specifically configured to determine the loss of the first top student model based on the second processing result, the first processing result, and the label; determine the first gradient of the parameters of the first top student model based on the loss of the first top student model; and update the parameters of the first top student model based on the first gradient.

[0274] In some embodiments, the training unit 15 is specifically configured to determine a first divergence value between the first processing result and the second processing result; determine a first cross-entropy between the label and the second processing result; and obtain the loss of the first top student model based on the first divergence value and the first cross-entropy.

[0275] In some embodiments, the second processing unit 13 is specifically used to fuse the first forward output data and the second forward output data to obtain fused forward output data; and to process the fused forward output data through the top teacher model to obtain the first processing result.

[0276] In some embodiments, the training unit 15 is further configured to process the first forward output data using the second top student model to obtain a third processing result; determine the loss of the second top student model based on the third processing result, the first processing result, and the label; determine the second gradient of the parameters of the second top student model based on the loss of the second top student model, and update the parameters of the second top student model based on the second gradient.

[0277] In some embodiments, the training unit 15 is specifically configured to determine a second divergence value between the first processing result and the third processing result; determine a second cross-entropy between the label and the third processing result; and obtain the loss of the second top student model based on the second divergence value and the second cross-entropy.

[0278] In some embodiments, the training unit 15 is further configured to send the trained second top student model to the second computing platform.

[0279] In some embodiments, the training unit 15 is further configured to use the first processing result and the label to determine the loss of the top teacher model; based on the loss of the top teacher model, determine the third gradient of the parameters of the top teacher model, and update the parameters of the top teacher model based on the third gradient.

[0280] In some embodiments, the training unit 15 is further configured to determine the backpropagation gradient corresponding to the second bottom model based on the backpropagation method; and send the backpropagation gradient corresponding to the second bottom model to the second computing platform so that the second computing platform updates the parameters of the second bottom model based on the backpropagation gradient corresponding to the second bottom model.

[0281] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details will not be provided here. Specifically, Figure 15 The apparatus shown can perform the embodiments of the federated method described above, and the foregoing and other operations and / or functions of the various modules in the apparatus are respectively for implementing the embodiments of the method described above, which will not be described in detail here for the sake of brevity.

[0282] Figure 16This is a schematic block diagram of a data processing apparatus provided in one embodiment of this application. The apparatus 20 can be applied to a first computing platform.

[0283] like Figure 16 As shown, the data processing device 20 includes:

[0284] Intermediate computing unit 21 is used to process the target data in the first computing platform through the first bottom model trained in the first computing platform to obtain the intermediate computing result corresponding to the target data;

[0285] Inference unit 22 is used to process the intermediate calculation results through the first top student model trained in the first computing platform to obtain inference results;

[0286] The trained first top student model is trained based on the first processing result, the second processing result, and the labels held by the first computing platform. The first processing result is obtained by processing the first forward output data and the second forward output data by the top teacher model in the first computing platform. The second processing result is obtained by processing the second forward output data by the first top student model. The first forward output data is obtained by processing the sample data held by the second bottom model in the second computing platform. The second forward output data is obtained by processing the sample data held by the first computing platform by the first bottom model.

[0287] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details will not be provided here. Specifically, Figure 16 The apparatus shown can perform the embodiments of the above data processing method, and the foregoing and other operations and / or functions of each module in the apparatus are for implementing the above method embodiments, which will not be described in detail here for the sake of brevity.

[0288] Figure 17 This is a schematic block diagram of a data processing apparatus provided in one embodiment of this application. The apparatus 30 can be applied to a second computing platform.

[0289] like Figure 17 As shown, the data processing device 30 includes:

[0290] The intermediate computing unit 31 is used to process the target data in the second computing platform through the second bottom model trained in the second computing platform to obtain the intermediate computing result corresponding to the target data;

[0291] Inference unit 32 is used to process the intermediate calculation results through the second top student model trained in the second computing platform to obtain the inference result;

[0292] The trained second top student model is trained based on the third processing result, the first processing result, and the labels held by the first computing platform. The first computing platform includes the trained first top student model, which is trained based on the first processing result, the second processing result, and the labels. The first processing result is obtained by processing the first forward output data and the second forward output data through the top teacher model in the first computing platform. The second processing result is obtained by processing the second forward output data through the first top student model. The third processing result is obtained by processing the first forward output data through the second top student model. The first forward output data is obtained by the second bottom model processing the sample data held by the second computing platform. The second forward output data is obtained by the first bottom model in the first computing platform processing the sample data held by the first computing platform.

[0293] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details will not be provided here. Specifically, Figure 17 The apparatus shown can perform the embodiments of the above data processing method, and the foregoing and other operations and / or functions of each module in the apparatus are for implementing the above method embodiments, which will not be described in detail here for the sake of brevity.

[0294] Figure 18 This is a schematic block diagram of a data processing apparatus provided in one embodiment of this application. The apparatus 40 can be applied to a first computing platform.

[0295] like Figure 18 As shown, the data processing device 40 includes:

[0296] Acquisition unit 41 is used to acquire the first intermediate calculation result sent by the second computing platform. The first intermediate calculation result is obtained by the second bottom model trained in the second computing platform based on the first data processing in the second computing platform.

[0297] Inference unit 42 is used to obtain inference results based on the first intermediate calculation result and the top teacher model trained in the first computing platform;

[0298] The trained top teacher model is trained based on the first processing result and the labels held by the first computing platform. The first computing platform includes a trained first top student model, which is trained based on the first processing result, the second processing result, and the labels. The first processing result is obtained by processing the first forward output data and the second forward output data by the top teacher model. The second processing result is obtained by processing the second forward output data by the first top student model. The first forward output data is obtained by the second bottom model processing the sample data held by the second computing platform. The second forward output data is obtained by the first bottom model in the first computing platform processing the sample data held by the first computing platform.

[0299] In some embodiments, the inference unit 42 is specifically used to process the first intermediate calculation result through the trained top teacher model to obtain the inference result.

[0300] In some embodiments, the inference unit 42 is specifically used to process the second data in the first computing platform using a first bottom model trained in the first computing platform to obtain a second intermediate calculation result; to fuse the first intermediate calculation result and the second intermediate calculation result to obtain an intermediate fusion calculation result; and to process the intermediate fusion calculation result using the trained top teacher model to obtain the inference result.

[0301] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details will not be provided here. Specifically, Figure 18 The apparatus shown can perform the embodiments of the above data processing method, and the foregoing and other operations and / or functions of each module in the apparatus are for implementing the above method embodiments, which will not be described in detail here for the sake of brevity.

[0302] The apparatus of this application embodiment has been described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that this functional module can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the method embodiments in this application can be completed by integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the method disclosed in this application embodiment can be directly embodied as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. Optionally, the software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps in the above method embodiments.

[0303] Figure 19 This is a schematic block diagram of an electronic device provided in an embodiment of this application, which is used to perform the above-described data processing method embodiment.

[0304] like Figure 19 As shown, the electronic device 50 may include:

[0305] The system includes a memory 51 and a processor 52. The memory 51 stores a computer program 53 and transfers the program code 53 to the processor 52. In other words, the processor 52 can retrieve and run the computer program 53 from the memory 51 to implement the methods described in the embodiments of this application.

[0306] For example, the processor 52 can be used to execute the above method steps according to the instructions in the computer program 53.

[0307] In some embodiments of this application, the processor 52 may include, but is not limited to:

[0308] General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0309] In some embodiments of this application, the memory 51 includes, but is not limited to:

[0310] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0311] In some embodiments of this application, the computer program 53 may be divided into one or more modules, which are stored in the memory 51 and executed by the processor 52 to complete the page recording method provided in this application. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 53 in the electronic device.

[0312] like Figure 19 As shown, the electronic device 50 may further include:

[0313] Transceiver 54, which can be connected to processor 52 or memory 51.

[0314] The processor 52 can control the transceiver 54 to communicate with other devices; specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 54 may include a transmitter and a receiver. The transceiver 54 may further include antennas, and the number of antennas may be one or more.

[0315] It should be understood that the various components in the electronic device 50 are connected through a bus system, which includes a data bus, a power bus, a control bus, and a status signal bus.

[0316] According to one aspect of this application, a computer storage medium is provided that stores a computer program thereon, which, when executed by a computer, enables the computer to perform the methods of the above-described method embodiments.

[0317] This application also provides a computer program product containing instructions that, when executed by a computer, cause the computer to perform the methods described in the above method embodiments.

[0318] According to another aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the method described in the above-described method embodiments.

[0319] In other words, when implemented using software, it can be implemented wholly or partially in the form of a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0320] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0321] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0322] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0323] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A federated learning method, characterized in that, A first computing platform for holding tags, the first computing platform including a first bottom model, a first top student model, a second top student model, and a top teacher model, including: Obtain first forward output data, which is obtained by the second bottom model in the second computing platform based on the sample data held by the second computing platform, and the second computing platform is the sample data holder; The sample data held by the first computing platform is processed through the first bottom model to obtain the second forward output data; The first forward output data and the second forward output data are processed using the top teacher model to obtain the first processing result; Based on the first processing result and the label, the top teacher model and the first bottom model are trained; The second forward output data is processed using the first top student model to obtain the second processing result; Based on the first processing result and the second processing result, as well as the label, the first top student model is trained. The trained first top student model and the trained first bottom model are used to process the target data in the first computing platform. The first forward output data is processed using the second top student model to obtain the third processing result; Based on the third processing result, the first processing result, and the label, the second top student model is trained. The training of the second top student model is based on the first forward output data, but not on the second forward output data. The trained second top student model is sent to the second computing platform so that the second computing platform can use the trained second top student model and the trained second bottom model to process the target data in the second computing platform.

2. The method according to claim 1, characterized in that, The step of training the first top student model based on the first processing result, the second processing result, and the label includes: Based on the second processing result, the first processing result, and the label, the loss of the first top student model is determined; Based on the loss of the first top student model, the first gradient of the parameters of the first top student model is determined, and the parameters of the first top student model are updated based on the first gradient.

3. The method according to claim 2, characterized in that, The step of determining the loss of the first top student model based on the second processing result, the first processing result, and the label includes: Determine the first divergence value between the first processing result and the second processing result; Determine the first cross-entropy between the label and the second processing result; Based on the first divergence value and the first cross-entropy, the loss of the first top student model is obtained.

4. The method according to claim 1, characterized in that, The first processing result is obtained by processing the first forward output data and the second forward output data through the top teacher model, including: The first forward output data and the second forward output data are fused to obtain fused forward output data; The first processing result is obtained by processing the fused forward output data through the top teacher model.

5. The method according to any one of claims 1-4, characterized in that, The step of training the second top student model based on the third processing result, the first processing result, and the label includes: Based on the third processing result, the first processing result, and the label, the loss of the second top student model is determined; Based on the loss of the second top student model, the second gradient of the parameters of the second top student model is determined, and the parameters of the second top student model are updated based on the second gradient.

6. The method according to claim 5, characterized in that, The step of determining the loss of the second top student model based on the third processing result, the first processing result, and the label includes: Determine a second divergence value between the first processing result and the third processing result; Determine the second cross-entropy between the label and the third processing result; Based on the second divergence value and the second cross-entropy, the loss of the second top student model is obtained.

7. The method according to any one of claims 1-4, characterized in that, Based on the first processing result and the label, the top teacher model is trained, including: Using the first processing result and the label, determine the loss of the top teacher model; Based on the loss of the top teacher model, the third gradient of the parameters of the top teacher model is determined, and the parameters of the top teacher model are updated based on the third gradient.

8. The method according to any one of claims 1-4, characterized in that, The method further includes: Based on the backpropagation method, the backpropagation gradient corresponding to the second bottom model is determined; The backpropagation gradient corresponding to the second bottom model is sent to the second computing platform so that the second computing platform updates the parameters of the second bottom model based on the backpropagation gradient corresponding to the second bottom model.

9. A data processing method, characterized in that, The method, applied to a first computing platform holding a tag, includes: The target data in the first computing platform is processed using the first bottom model trained in the first computing platform to obtain the intermediate calculation results corresponding to the target data. The intermediate calculation results are processed using the first top student model trained in the first computing platform to obtain the inference results; The trained first top student model is trained based on the first processing result, the second processing result, and the labels held by the first computing platform. The first processing result is obtained by processing the first forward output data and the second forward output data by the top teacher model in the first computing platform. The second processing result is obtained by processing the second forward output data by the first top student model. The first forward output data is obtained by processing the sample data held by the second bottom model in the second computing platform based on the sample data held by the second computing platform. The second forward output data is obtained by processing the sample data held by the first computing platform by the first bottom model. The trained first bottom model and the top teacher model are trained based on the first processing result and the labels.

10. A data processing method, characterized in that, The method, applied to a second computing platform holding sample data, includes: The target data in the second computing platform is processed by the second bottom model trained in the second computing platform to obtain the intermediate calculation results corresponding to the target data. The intermediate calculation results are processed using the second top student model trained in the second computing platform to obtain the inference results; The trained second top student model is trained based on the third processing result, the first processing result, and the labels held by the first computing platform. The first computing platform includes the trained first top student model, which is trained based on the first processing result, the second processing result, and the labels. The first processing result is obtained by processing the first and second forward output data through the top teacher model in the first computing platform. The second processing result is obtained by processing the second forward output data through the first top student model. The third processing result is obtained by processing the first forward output data through the second top student model. The first forward output data is obtained by the second bottom model processing the sample data held by the second computing platform. The second forward output data is obtained by the first bottom model in the first computing platform processing the sample data held by the first computing platform. The training of the second top student model is based on the first forward output data, not the second forward output data. The top teacher model is trained based on the first processing result and the labels, and is trained as a whole with the first bottom model.

11. A data processing method, characterized in that, The method, applied to a first computing platform holding a tag, includes: Obtain the first intermediate calculation result sent by the second computing platform. The first intermediate calculation result is obtained by the second bottom model trained in the second computing platform based on the first data in the second computing platform. Based on the first intermediate calculation result and the top teacher model trained in the first computing platform, the inference result is obtained; The trained top teacher model and the trained second bottom model are trained based on the first processing result and the labels held by the first computing platform. The first computing platform includes a trained first top student model, which is trained based on the first processing result, the second processing result, and the labels. The first processing result is obtained by processing the first forward output data and the second forward output data by the top teacher model. The second processing result is obtained by processing the second forward output data by the first top student model. The first forward output data is obtained by the second bottom model based on the sample data held by the second computing platform. The second forward output data is obtained by the first bottom model in the first computing platform processing the sample data held by the first computing platform.

12. The method according to claim 11, characterized in that, The first inference result obtained based on the first intermediate calculation result and the top teacher model trained in the first computing platform includes: The trained top teacher model is used to process the first intermediate calculation result to obtain the inference result.

13. The method according to claim 11, characterized in that, The first inference result obtained based on the first intermediate calculation result and the top teacher model trained in the first computing platform includes: The second data in the first computing platform is processed using the first bottom model trained in the first computing platform to obtain the second intermediate calculation result; The first intermediate calculation result and the second intermediate calculation result are fused together to obtain the intermediate fused calculation result; The trained top teacher model is used to process the intermediate fusion calculation results to obtain the inference result.

14. A federated learning device, characterized in that, A first computing platform for holding tags, the first computing platform including a first bottom model, a first top student model, a second top student model, and a top teacher model, including: The acquisition unit is used to acquire first forward output data, which is obtained by the second bottom model in the second computing platform based on the sample data held by the second computing platform. The second computing platform is the sample data holder. The first processing unit is used to process the sample data held by the first computing platform through the first bottom model to obtain the second forward output data. The second processing unit is used to process the first forward output data and the second forward output data through the top teacher model to obtain the first processing result; The third processing unit is used to process the second forward output data through the first top student model to obtain the second processing result; The training unit is used to train the first top student model based on the first processing result and the second processing result, as well as the label. The trained first top student model and the trained first bottom model are used to process the target data in the first computing platform. The training unit is further configured to process the first forward output data using the second top student model to obtain a third processing result; train the second top student model based on the third processing result, the first processing result, and the label; and send the trained second top student model to the second computing platform so that the second computing platform can use the trained second top student model and the trained second bottom model to process the target data in the second computing platform, wherein the training of the second top student model is based on the first forward output data, but not on the second forward output data. The training unit is also used to train the top teacher model and the first bottom model based on the first processing result and the label.

15. A data processing apparatus, characterized in that, The first computing platform applied to tag-holding includes: An intermediate computing unit is used to process the target data in the first computing platform using a first bottom model trained in the first computing platform, and obtain intermediate computing results corresponding to the target data. The inference unit is used to process the intermediate calculation results through the first top student model trained in the first computing platform to obtain the inference result; The trained first top student model is trained based on the first processing result, the second processing result, and the labels held by the first computing platform. The first processing result is obtained by processing the first forward output data and the second forward output data by the top teacher model in the first computing platform. The second processing result is obtained by processing the second forward output data by the first top student model. The first forward output data is obtained by processing the sample data held by the second bottom model in the second computing platform based on the sample data held by the second computing platform. The second forward output data is obtained by processing the sample data held by the first computing platform by the first bottom model. The trained first bottom model and the top teacher model are trained based on the first processing result and the labels.

16. A data processing apparatus, characterized in that, A second computing platform for holding sample data includes: The intermediate computing unit is used to process the target data in the second computing platform through the second bottom model trained in the second computing platform to obtain the intermediate computing results corresponding to the target data; The inference unit is used to process the intermediate calculation results through the second top student model trained in the second computing platform to obtain the inference result; The trained second top student model is trained based on the third processing result, the first processing result, and the labels held by the first computing platform. The first computing platform includes the trained first top student model, which is trained based on the first processing result, the second processing result, and the labels. The first processing result is obtained by processing the first and second forward output data through the top teacher model in the first computing platform. The second processing result is obtained by processing the second forward output data through the first top student model. The third processing result is obtained by processing the first forward output data through the second top student model. The first forward output data is obtained by the second bottom model processing the sample data held by the second computing platform. The second forward output data is obtained by the first bottom model in the first computing platform processing the sample data held by the first computing platform. The training of the second top student model is based on the first forward output data, not the second forward output data. The top teacher model is trained based on the first processing result and the labels, and is trained as a whole with the first bottom model.

17. A data processing apparatus, characterized in that, The first computing platform applied to tag-holding includes: The acquisition unit is used to acquire the first intermediate calculation result sent by the second computing platform. The first intermediate calculation result is obtained by the second bottom model trained in the second computing platform based on the first data processing in the second computing platform. The inference unit is used to obtain the inference result based on the first intermediate calculation result and the top teacher model trained in the first computing platform; The trained top teacher model and the trained second bottom model are trained based on the first processing result and the labels held by the first computing platform. The first computing platform includes a trained first top student model, which is trained based on the first processing result, the second processing result, and the labels. The first processing result is obtained by processing the first forward output data and the second forward output data by the top teacher model. The second processing result is obtained by processing the second forward output data by the first top student model. The first forward output data is obtained by the second bottom model based on the sample data held by the second computing platform. The second forward output data is obtained by the first bottom model in the first computing platform processing the sample data held by the first computing platform.

18. An electronic device, characterized in that, Including processor and memory; The memory is used to store computer programs; The processor is configured to execute the computer program to implement the method as described in any one of claims 1 to 8 or 9 to 13 above.

19. A computer-readable storage medium, characterized in that, Used to store a computer program that causes a computer to perform the method as described in any one of claims 1 to 8 or 9 to 13.

Images