Data processing methods, data representation learning methods, systems, and devices

Abstract

The application provides a data processing method, a data representation learning method, a system and a device. In the method, the cloud server synthesizes the data representations of the augmented samples of the data samples sent by each client device in the data representation learning process, calculates the contrast loss and gradient of the data representation of each client device, takes the data representations of the augmented samples belonging to the same data sample as positive samples, takes the data representations of the augmented samples belonging to different client devices as negative samples, performs contrast learning, and obtains a pre-trained data representation model. In the distributed contrast learning framework, the data difference degree of each global client device is fully considered, the expression ability, the generalization ability and the performance of the data representation model are improved, and the accuracy of the data processing result is improved when the data representation model is used for a downstream data processing task.

Description

Technical Field

[0001] This application relates to computer technology, and more particularly to a data processing method, a data representation learning method, a system, and an apparatus. Background Technology

[0002] In deep learning, representation refers to the form and method by which the model's input observation samples are represented using the model's parameters. Representation learning refers to learning effective representations for observation samples. For example, graph representation learning can learn a graph representation model to transform graph data with complex structures into vector representations in a low-dimensional space that preserve the diverse attributes and structural features of the graph data; this is called a graph representation. Image representation learning can determine an image representation model to encode images into image features; this is also called an image representation. Text representation learning can determine a text representation model to encode input text information into corresponding embedding representations; this is also called a text representation.

[0003] In various practical applications such as molecular property prediction, image recognition, and natural language processing, users / organizations typically possess a limited number of samples or labels, especially labeled data, which is difficult to obtain and scarce. Taking molecular property prediction as an example, different molecular research institutions share roughly the same feature domains but differ in label domains. The number of samples is vast, but the number of labels is limited and expensive, and the data distribution among different institutions is highly unbalanced. Due to the limited number of samples or labels possessed by each user / organization, the generalization ability of data representation models generated through supervised learning methods is poor, resulting in low accuracy when applying these data representations to downstream tasks. Summary of the Invention

[0004] This application provides a data processing method, a data representation learning method, a system, and an apparatus to address the problem that data representation models generated by traditional contrastive learning methods have poor generalization ability, resulting in low accuracy in data processing when applied to downstream tasks.

[0005] Firstly, this application provides a data representation learning method applied to a cloud server, comprising:

[0006] Data representations of augmented samples extracted by multiple client devices using a local model are obtained. Each data representation includes multiple augmented sample data representations of the same data sample. For each client device, the data representations of augmented samples belonging to the same data sample are used as positive samples, and the data representations of augmented samples belonging to different client devices are used as negative samples. The gradient of the contrastive loss of the client device is calculated and sent to that client device. The gradient of the contrastive loss of the client device is used to optimize the local model of the client device to obtain a pre-trained data representation model. The data representation model is used to encode the input target data to obtain the data representation of the target data. The data representation of the target data is used for data processing to obtain the data processing result of the target data.

[0007] Secondly, this application provides a data representation learning method applied to a client device, comprising:

[0008] Multiple augmented samples of locally stored data samples are encoded using a local model to obtain multiple augmented sample data representations. These data representations are then sent to a cloud server, whereby the cloud server treats augmented sample data representations belonging to the same data sample as positive samples and augmented sample data representations belonging to different client devices as negative samples, calculating the gradient of the contrast loss for each client device. Based on the gradient sent by the cloud server, the model parameters of the local model are optimized to obtain a pre-trained data representation model. This data representation model is used to encode the input target data to obtain a data representation of the target data. This data representation of the target data is then used for data processing to obtain the data processing result of the target data.

[0009] Thirdly, this application provides a data processing method applied to a client device, comprising:

[0010] A pre-trained data representation model is obtained, which is pre-trained using the method described in any one of claims 1-9; the pre-trained data representation model is used as an encoding module to construct a data processing model; the data processing model is trained using local training data to obtain a trained data processing model; the target data to be processed is input into the encoding module of the trained data processing model for encoding to obtain a data representation of the target data; the data processing is performed according to the data representation of the target data through the decoding module of the trained data processing model to obtain a data processing result.

[0011] Fourthly, this application provides a data representation learning system, including: a first server, a second server, and multiple client devices.

[0012] The client device is configured to encode multiple augmented samples of locally stored data samples using a local model to obtain a sample representation of multiple augmented samples for each data sample, and send the sample representation of the multiple augmented samples to a first server.

[0013] The first server is configured to, for each client device, take the data representation of augmented samples belonging to the same data sample as a positive sample, and the data representation of augmented samples belonging to different client devices as a negative sample, calculate the gradient of the contrast loss of the data representation of the client device, and send it to the client device.

[0014] The client device is further configured to optimize the model parameters of the local model based on the gradient of the contrast loss represented by the data sent by the first server and send it to the second server.

[0015] The second server is used to aggregate the local model parameters sent by each of the client devices to obtain aggregated global model parameters, and then send the aggregated global model parameters to each of the client devices.

[0016] The client device is also used to update the local model based on the aggregated global model parameters sent by the second server, and to start the next round of iterative learning to obtain a pre-trained data representation model.

[0017] Fifthly, this application provides a cloud server, including: a processor and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the methods described in the first, fourth, or sixth aspects above.

[0018] In a sixth aspect, this application provides a client device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the methods described in the second, third, fifth, or seventh aspects above.

[0019] The data processing method, data representation learning method, system, and device provided in this application calculate the contrastive loss and gradient of the data representation for each client device by integrating the data representations of augmented samples sent by various client devices during the data representation learning process. Specifically, by using the data representations of augmented samples belonging to the same data sample as positive samples and the data representations of augmented samples belonging to different client devices as negative samples for contrastive learning, this approach can fully consider the imbalance in data distribution among client devices and the degree of data difference among global client devices within a distributed contrastive learning framework. This significantly improves the model's expressive power, generalization ability, and performance. At the same time, it makes the model's expressive power and the degree of difference in model evaluation metrics mutually reinforcing; the stronger the model's expressive power, the more accurate the degree of data difference represented by the output data representation, thus improving the accuracy of data processing results when used for downstream data processing. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0021] Figure 1 An example diagram of a distributed contrastive learning architecture is provided for this application;

[0022] Figure 2 A flowchart of a molecular graph representation learning method provided for an exemplary embodiment of this application;

[0023] Figure 3 A detailed flowchart of a molecular graph representation learning method provided for an exemplary embodiment of this application;

[0024] Figure 4 A distributed contrastive learning system architecture diagram provided for another exemplary embodiment of this application;

[0025] Figure 5 A general flowchart of a dual-server molecular graph representation learning system provided for an exemplary embodiment of this application;

[0026] Figure 6 A flowchart of a molecular diagram representation method provided in an exemplary embodiment of this application;

[0027] Figure 7 A flowchart of an image recognition method provided as an exemplary embodiment of this application;

[0028] Figure 8 A flowchart of a natural language processing method provided in an exemplary embodiment of this application;

[0029] Figure 9This is a schematic diagram of the structure of a cloud server provided in an example embodiment of this application;

[0030] Figure 10 This is a schematic diagram of the structure of a client device provided in an example embodiment of this application.

[0031] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0032] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0033] First, let me explain the terms used in this application:

[0034] Federated learning: A distributed machine learning technique that satisfies privacy protection by enabling collaborative training of a global model through the exchange of models or intermediate results, without requiring data to leave the domain of any party.

[0035] Self-supervised learning: a learning paradigm that does not require sample labels. It improves the model's feature extraction ability by designing auxiliary tasks to mine features from unlabeled samples and use them as supervision information.

[0036] Contrastive learning: A self-supervised learning paradigm that utilizes the mutual information between model sample data to extract unique feature information from the sample data. Mutual information is an indicator that measures the degree of correlation between random variables.

[0037] In many practical applications such as molecular property prediction, image recognition, and natural language processing, users / organizations typically possess a limited number of samples or labels, especially label data, which is difficult to obtain and scarce. Due to the limited number of samples or labels possessed by each user / organization, the representation models generated by supervised learning methods exhibit poor generalization ability, resulting in poor performance when applied to downstream tasks.

[0038] For example, in the scenario of molecular property prediction, since the number of molecular graph samples or labels owned by each user / organization is very small, the generalization ability of the molecular graph representation model generated by supervised learning methods is poor, resulting in low accuracy of molecular property prediction when applied to downstream tasks.

[0039] Taking image recognition as an example, since each user / organization has very few image samples or labels, the generalization ability of the image representation model generated by supervised learning methods is poor, resulting in low image recognition accuracy when applied to downstream tasks.

[0040] Taking natural language processing scenarios as an example, since users / organizations each have very few labels in their text corpora, the generalization ability of text representation models generated by supervised learning methods is poor, resulting in low accuracy of natural language processing when applied to downstream tasks.

[0041] To address the poor generalization ability of the aforementioned molecular graph representation models, this application provides a distributed contrastive learning architecture, such as... Figure 1 As shown, it includes a cloud server and multiple client devices, where each client device corresponds to a user / organizational structure that has its own data. Figure 1 The example shown only includes two client devices; however, the architecture can include many more client devices.

[0042] A cloud server can be a device independent of each individual client device, or it can be jointly designated by multiple client devices corresponding to users / organizational structures. Specifically, the server can be a server cluster set up in the cloud. This server stores global model parameters, can communicate with each client device, and is responsible for distributing global model parameters to each client device, calculating comparative loss and gradients for each client device, aggregating local model parameters sent by each client device to optimize global model parameters, and controlling iterative learning.

[0043] The client device locally stores a private dataset and a local model. The private dataset includes data samples and sample labels. Specifically, the client device can be a hardware device used by various users / organizations that has network communication, computing, and information display capabilities, including but not limited to desktop computers, IoT devices, and cloud-deployed clusters.

[0044] Specifically, the cloud server distributes global model parameters to each client device. Each client device updates its local model based on the global model parameters and uses the local model to extract sample representations of data samples, such as graph representations of graph data, image representations of images, and text representations of text corpora. Based on the sample representations submitted by each client device, the cloud server, for each client device, treats sample representations belonging to the same data sample as positive samples and sample representations belonging to different client devices as negative samples, calculates the gradient of the contrastive loss for that client device, and sends it to that client device. The client device receives the gradient of the contrastive loss sent by the cloud server, backpropagates based on this gradient to update its local model parameters, and sends the updated local model parameters back to the cloud server. The cloud server aggregates the local model parameters from each client device to obtain new global model parameters, distributes the new global model parameters to each client device, and begins the next round of iterative training. Until the convergence condition is met, the cloud server determines the final model parameters and distributes them to each client device. Each client device updates its local model based on the final model parameters to obtain a pre-trained representation model.

[0045] Furthermore, when applied to downstream tasks, the client device can use a pre-trained data representation model as an encoding model to construct a downstream task model. The model parameters of the downstream task model are then fine-tuned (trained) using a local dataset to obtain a downstream task model for executing a specific downstream task. This local dataset refers to a dataset specific to the downstream task, which is different from the set of data samples used in the aforementioned pre-training process. The datasets used for different downstream tasks are typically different. The target data to be processed is input into the downstream task model, the data representation of the target data is extracted through the encoding module, and the corresponding decoding processing for the downstream task is performed based on the data representation to obtain the execution result of the downstream task, i.e., the data processing result.

[0046] For example, in the field of graph data processing, such as molecular graph research, based on the distributed contrastive learning architecture described above, the cloud server distributes global model parameters to client devices of various users / institutions possessing graph data. Each client device updates its local model according to the global model parameters and uses the local model to extract sample representations of augmented graphs from the locally stored graph data. The cloud server, based on the sample representations of augmented graphs submitted by each client device, calculates the gradient of the contrastive loss for each client device, taking the sample representations of augmented graphs belonging to the same graph data as positive samples and the sample representations of augmented graphs belonging to different client devices as negative samples, and sends this gradient to the client device. The client device receives the gradient of the contrastive loss sent by the cloud server, backpropagates according to this gradient to update its local model parameters, and sends the updated local model parameters to the cloud server. The cloud server aggregates the local model parameters of each client device to obtain new global model parameters, distributes the new global model parameters to each client device, and begins the next round of iterative training. This continues until the convergence condition is met, at which point the cloud server determines the final model parameters and distributes them to each client device. Each client device updates its local model based on the final model parameters to obtain a pre-trained data representation model, which is used to encode the input graph data to obtain the corresponding graph representation.

[0047] Furthermore, when applied to downstream graph data processing tasks, client devices can use a pre-trained data representation model as an encoding model to construct a graph data processing model. The model parameters of the graph data processing model are then fine-tuned (trained) using a local labeled graph dataset to obtain the trained graph data processing model. When graph data processing is required on any target graph, the target graph can be input into the graph data processing model. The encoding module extracts the graph representation of the target graph, and decoding is performed based on this representation to obtain the graph data processing result for the target graph.

[0048] For example, when applied to downstream molecular property prediction tasks, client devices can use a pre-trained data representation model as an encoding model to construct a molecular property prediction model. The model parameters of the data representation model are then fine-tuned (trained) using a local molecular graph dataset labeled with molecular properties, resulting in a molecular property prediction model. When it is necessary to predict whether a target molecule possesses a certain property, the molecular graph of the target molecule can be input into the molecular property prediction model. The encoding module extracts the graph representation of the target molecule's molecular graph, and decoding based on this representation yields the predicted molecular property result.

[0049] For example, in the field of image recognition, based on the distributed contrastive learning architecture described above, the cloud server distributes global model parameters to each client device storing private image data. Each client device updates its local model according to the global model parameters and uses the local model to extract image representations of augmented images of locally stored sample images. The cloud server, based on the image representations of augmented images submitted by each client device, calculates the gradient of the contrastive loss for each client device, taking the image representations of augmented images belonging to the same sample image as positive samples and the image representations of augmented images belonging to different client devices as negative samples, and sends it to that client device. The client device receives the gradient of the contrastive loss sent by the cloud server, backpropagates according to the gradient to update its local model parameters, and sends the updated local model parameters to the cloud server. The cloud server aggregates the local model parameters of each client device to obtain new global model parameters, distributes the new global model parameters to each client device, and begins the next round of iterative training. Until the convergence condition is met, the cloud server determines the final model parameters and distributes them to each client device. Each client device updates its local model according to the final model parameters to obtain a pre-trained image representation model.

[0050] Furthermore, when applied to downstream image recognition tasks, the pre-trained image representation model can serve as an encoding module (backbone network) for extracting image features. Client devices can use the pre-trained image representation model as an encoding module to build an image recognition model. The model parameters are then fine-tuned (trained) using a local image dataset with labeled recognition results to obtain the image recognition model. When image recognition of any target image is required, the target image can be input into the image recognition model. The encoding module extracts the image representation of the target image, and decoding based on this representation yields the image recognition result.

[0051] Image recognition models can be applied to a variety of specific application scenarios, such as face recognition, intelligent video analysis (e.g., analyzing and predicting the content of images in videos to extract key information such as license plates, faces, and actions), traffic scene recognition (e.g., identifying illegally parked vehicles, pedestrians crossing the road in autonomous driving, lane positions in images, intersections, etc.), and image recognition in the medical field (e.g., identifying human tissue in images).

[0052] For example, in the field of natural language processing, based on the distributed contrastive learning architecture described above, the cloud server distributes global model parameters to each client device storing its private text corpus. Each client device updates its local model according to the global model parameters and uses the local model to extract the text representation of the enhanced text from its locally stored text corpus. The cloud server, based on the text representations of the enhanced text submitted by each client device, calculates the gradient of the contrastive loss for each client device, taking the text representations of enhanced text belonging to the same text corpus as positive samples and the text representations of enhanced text belonging to different client devices as negative samples, and sends this gradient to that client device. The client device receives the gradient of the contrastive loss sent by the cloud server, backpropagates according to this gradient to update its local model parameters, and sends the updated local model parameters to the cloud server. The cloud server aggregates the local model parameters from each client device to obtain new global model parameters, distributes the new global model parameters to each client device, and begins the next round of iterative training. This continues until the convergence condition is met, at which point the cloud server determines the final model parameters and distributes them to each client device. Each client device updates its local model based on the final model parameters to obtain a pre-trained text representation model.

[0053] Furthermore, when applied to downstream natural language processing tasks, the pre-trained text representation model can serve as an encoding module (backbone network) for extracting text features. Client devices can use the pre-trained text representation model as an encoding module to construct a natural language processing model. The model parameters are then fine-tuned (trained) using a local labeled text corpus to obtain the natural language processing model. When natural language processing is required on any target text, the target text to be recognized can be input into the natural language processing model. The encoding module extracts the text representation of the target text, and decoding is performed based on this representation to obtain the natural language processing result for the target text.

[0054] Natural language processing models can be used to implement various downstream tasks. Different downstream task models can be constructed using pre-trained text representation models. Specifically, they can be applied to tasks such as automatic summarization, opinion extraction, text classification, and text semantic comparison, but are not limited to these tasks.

[0055] For example, in text classification tasks applied to e-commerce scenarios, the user's input question can be fed into a natural language processing model. The encoding module generates a text representation of the question, and the classification decoder is used to classify and predict the text representation to determine the category of the question, such as product quality issues, express delivery issues, product description issues, etc.

[0056] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0057] Figure 2 This is a flowchart illustrating a data representation learning method provided in an exemplary embodiment of this application. In this embodiment, it is based on the distributed contrastive learning architecture described above in a federated scenario. Figure 2 As shown, the specific process of this method is as follows:

[0058] Step S201: The client device uses the local model to encode multiple augmented samples of the locally stored data samples to obtain data representations of multiple augmented samples.

[0059] In this embodiment, multiple client devices participating in data representation learning locally store private data sample sets. These data sample sets include data samples and may also include label information for the data samples. The feature domains of the data sample sets stored by each client device are roughly the same, but the label domains are different. The number of data samples is huge, but the number of labels is small and the labeling cost is high. The data distribution among the client devices is very unbalanced.

[0060] For example, in the field of graph representation learning, multiple client devices participating in graph representation learning locally store private graph datasets. These datasets include graph data and may also include the label information of the graph data. The feature domains of the graph datasets stored by each client device are roughly the same, but the label domains are different. The number of sub-graphs is huge, but the number of labels is small and the annotation cost is high. The data distribution among the client devices is very unbalanced.

[0061] For example, in the field of image recognition, multiple client devices participating in image representation learning locally store private image datasets. These datasets include sample images and may also include label information for those sample images. The feature domains of the image datasets stored by each client device are roughly the same, but the label domains differ. The number of sample images is enormous, but the number of labels is small, and the annotation workload is high. The data distribution among the client devices is highly unbalanced.

[0062] For example, in the field of natural language processing, multiple client devices participating in text representation learning locally store private text corpora. These corpora include text data and possibly label information. The feature domains of the text corpora stored on each client device are roughly the same, but the label domains differ. The amount of text data is enormous, but the number of labels is small, and the annotation workload is high. The data distribution among the client devices is very unbalanced.

[0063] During the data representation learning process, each client device augments its locally stored data samples using various data augmentation methods, resulting in multiple augmented samples for each data sample. The client device then encodes these augmented samples using its local model, obtaining the data representation for each augmented sample.

[0064] When performing data augmentation on data samples, different data augmentation methods can be used for different types of data samples to obtain augmented samples. For example, different augmentation methods can be used for graph data, images, and text.

[0065] For example, for graph data such as molecular graphs of proteins and compounds, personalized PageRank algorithms or hot kernel methods can be used to enhance the graph data; alternatively, existing graph data enhancement transformations can be used, including but not limited to: feature-based enhancement, structure-based enhancement, sampling-based enhancement, and adaptive enhancement.

[0066] For example, for image data samples (i.e., sample images), the client device can use various image enhancement methods to enhance them, resulting in multiple enhanced images for each sample image. Methods for enhancing sample images include, but are not limited to: flipping, rotating, scaling, cropping, and shifting. At least two of these enhancement methods can be used to enhance the sample images to obtain at least two enhanced images for each sample image.

[0067] For example, data augmentation methods for text corpora include, but are not limited to: synonym replacement, random insertion, random replacement, random deletion, text generation, adversarial generation, and conditional generation. At least two of these augmentation methods can be used to augment the text corpus to obtain at least two augmented texts for each text corpus.

[0068] In addition, in each iteration, the local model parameters of the client device's local model are determined based on the latest global model parameters issued by the cloud server.

[0069] Step S202: The client device sends the data representations of multiple augmented samples to the cloud server.

[0070] After obtaining the data representations of multiple augmented samples, the client device sends the data representations of multiple augmented samples to the cloud server, so that the cloud server will use the data representations of augmented samples belonging to the same data sample in the client device as positive samples and the data representations of other augmented samples in the same client device as negative samples, in order to calculate the gradient of the contrast loss of the client device.

[0071] For example, when sending the data representation of an enhanced sample to the cloud server, the corresponding data sample identifier can be carried simultaneously. This data sample identifier is only used to mark whether the enhanced samples belong to the same data sample and does not contain the content information of the data sample. Data samples on different client devices are different data samples with different data sample identifiers.

[0072] Step S203: The cloud server obtains the data representation of the augmented samples of the data samples extracted by multiple client devices using the local model. The data representation includes the data representation of multiple augmented samples of the same data sample.

[0073] The cloud server receives augmented sample data representations sent by various client devices. For any received augmented sample data representation from any client device, multiple data representations correspond to augmented samples obtained by augmenting the same data sample, thus establishing positive sample pairs.

[0074] Step S204: For each client device, the cloud server takes the data representation of augmented samples belonging to the same data sample as positive samples and the data representation of augmented samples belonging to different client devices as negative samples, calculates the gradient of the contrast loss of the client device, and sends it to the client device.

[0075] In this embodiment, the cloud server, based on the data representations of the augmented samples sent by each client device, constructs positive sample pairs for each client device by taking the data representations of augmented samples from the same data sample of that client device as its positive samples; and constructs negative sample pairs by taking the data representations of other client devices as its negative samples. Based on the constructed positive and negative sample pairs, the contrastive loss of the data representations is calculated, and the gradient of the contrastive loss is calculated to obtain the gradient corresponding to that client device, which is also the gradient of the contrastive loss corresponding to that client device. The cloud server then sends the calculated gradients for each client device to the corresponding client device.

[0076] Step S205: The client device optimizes the model parameters of the local model according to the gradient sent by the cloud server to obtain a pre-trained data representation model. The data representation model is used to encode the input target data to obtain the data representation of the target data. The data representation of the target data is used to perform data processing to obtain the data processing result.

[0077] The client device receives gradients sent by the cloud server and backpropagates based on these gradients to optimize the model parameters of the local model. These local model parameters are then aggregated to obtain the global model parameters.

[0078] In one alternative implementation, the client sends the optimized local model parameters to a single cloud server. The cloud server aggregates the local model parameters sent by each client device to obtain new global model parameters, which are then distributed to each client device. Upon receiving the new global model parameters from the cloud server, the client device updates its local model based on the new global model parameters and begins the next round of iterative learning.

[0079] In another optional implementation, the cloud server may include a first server and a second server, with the cloud server in steps S202-S205 being the first server. The client sends the optimized local model parameters to a second server, which is a different server from the first server used to calculate the contrastive loss and gradient. The second server aggregates the local model parameters sent by each client device to obtain new global model parameters, and then distributes these new global model parameters to each client device. The client device receives the new global model parameters from the other cloud server, updates its local model according to the new global model parameters, and begins the next round of iterative learning.

[0080] In this embodiment, the data representation learning process is a pre-training process of the data representation model. The obtained pre-trained data representation model can be applied to downstream data processing tasks. Specifically, the data representation model is used to extract the data representation of the target data to be processed, and the data representation is decoded to obtain the data processing result.

[0081] For example, by using client devices from multiple molecular research institutions and leveraging locally stored private molecular graph datasets, distributed data representation learning can be performed with a cloud server to obtain a pre-trained molecular graph representation model (i.e., a data representation model). This model is then used to encode the input molecular graph to obtain its graph representation. Downstream data processing tasks can include molecular property prediction tasks, specifically predicting whether a molecule possesses a specific property, or predicting which one or more properties from a given set of properties it possesses. Specifically, the molecular graph representation model is used to extract the graph representation of the target molecule to be predicted. This graph representation is then decoded by a classification decoder to achieve classification prediction, determining whether the predicted molecule possesses a specific property, or which one or more properties from a given set of properties it possesses.

[0082] Molecular property prediction is a key task in computer-aided drug development, playing a crucial role in many downstream applications such as drug screening and drug design. Its main purpose is to predict the physical and chemical properties of molecules using internal molecular information such as atomic coordinates and atomic numbers. This enables researchers to identify compounds with the expected properties from a large pool of candidate compounds, accelerating drug screening and design.

[0083] It should be noted that both the cloud server and each client device store a data representation model to be trained. This data representation model can be a graph convolutional network, a neural network model for image encoding, or a transformer model for natural language processing, etc. The data representation models stored on the cloud server and each client device have the same structure.

[0084] In this embodiment, during the data representation learning process, the cloud server integrates the data representations of augmented samples from data samples sent by various client devices to calculate the contrastive loss and gradient of the data representation for each client device. Specifically, by using the data representations of augmented samples belonging to the same data sample as positive samples and the data representations of augmented samples belonging to different client devices as negative samples for contrastive learning, this distributed contrastive learning framework can fully consider the imbalance in data distribution among client devices and the degree of data difference among global client devices, greatly improving the model's expressive power, generalization ability, and performance. At the same time, it makes the model's expressive power and the degree of difference in model evaluation metrics mutually reinforcing; the stronger the model's expressive power, the more accurate the degree of data difference represented by the output data representation, thus improving the accuracy of data processing results when used for downstream data processing tasks.

[0085] Figure 3 This document provides a detailed flowchart of a data representation learning method as an exemplary embodiment of this application. Based on the above data representation learning embodiments, this embodiment provides a relatively complete example. Figure 3 As shown, the specific process of this method is as follows:

[0086] Step S300: The cloud server sends global model parameters to each client device.

[0087] In this step, the cloud server first sends global model parameters to each client device so that each client device can use the global model parameters to update its local model.

[0088] Step S301: The client device updates the local model using the global model parameters sent by the cloud server.

[0089] The client device receives global model parameters from the cloud server and uses these global model parameters to update the local model, ensuring that the updated local model parameters are consistent with the global model parameters.

[0090] Step S302: The client device uses the local model to encode multiple augmented samples of the locally stored data samples to obtain data representations of multiple augmented samples.

[0091] After updating the local model, the client device uses the updated local model to encode multiple augmented samples of the locally stored data samples, resulting in a data representation of multiple augmented samples.

[0092] In this embodiment, the client device can pre-amplify locally stored data samples to obtain and store augmented samples. In each iteration of data representation learning, the client device directly uses the pre-stored augmented samples to reduce the time overhead of data augmentation and improve the efficiency of data representation learning.

[0093] For locally stored data samples, the client device can use a variety of different data augmentation methods to augment the data, resulting in multiple augmented samples for each data sample.

[0094] For example, for graph data such as molecular graphs of proteins and compounds, personalized PageRank algorithms or hot kernel methods can be used to enhance the molecular graph; alternatively, existing graph data enhancement transformations can be used, including but not limited to: feature-based enhancement, structure-based enhancement, sampling-based enhancement, and adaptive enhancement.

[0095] For example, for image data samples (i.e., sample images), the client device can use various image enhancement methods to enhance them, resulting in multiple enhanced images for each sample image. Methods for enhancing sample images include, but are not limited to: flipping, rotating, scaling, cropping, and shifting. At least two of these enhancement methods can be used to enhance the sample images to obtain at least two enhanced images for each sample image.

[0096] For example, data augmentation methods for text corpora include, but are not limited to: synonym replacement, random insertion, random replacement, random deletion, text generation, adversarial generation, and conditional generation. At least two of these augmentation methods can be used to augment the text corpus to obtain at least two augmented texts for each text corpus.

[0097] Step S303: The client device sends the graph representations of multiple enhanced graphs to the cloud server.

[0098] For example, when sending the data representation of an enhanced sample to the cloud server, the corresponding data sample identifier is also carried. This data sample identifier is only used to mark whether the enhanced samples belong to the same data sample and does not contain the content information of the data sample. Data samples on different client devices are different data samples with different data sample identifiers.

[0099] Step S304: The cloud server receives the augmented sample data representation of the data samples extracted by each client device using the local model.

[0100] The data representation here refers to the data representation of multiple augmented samples that contain the same data sample.

[0101] Step S305: For each client device, the cloud server takes the data representation of augmented samples belonging to the same data sample as positive samples and the data representation of augmented samples belonging to different client devices as negative samples, calculates the gradient of the contrast loss of the client device, and sends it to the client device.

[0102] In this embodiment, for any first client device, for the data representation sent by the first client device, any two enhanced samples of the same data sample are considered positive samples to construct a positive sample pair. Data representations sent by other client devices are used as negative samples of the data representation sent by the first client device, and any data representation from the first client device is paired with any data representation from other client devices to form a negative sample pair.

[0103] For example, any data representation sent by the first client device can be used as the target sample. Based on the data sample identifier corresponding to the target sample, other data representations corresponding to the same data sample are determined as positive samples of the target sample, and the target sample is paired with each positive sample to form a positive sample pair. Data representations sent by other clients are used as negative samples of the target sample, and the target sample is paired with each negative sample to form a negative sample pair.

[0104] Optionally, when calculating the gradient of the contrastive loss for each client device, the data representations of augmented samples belonging to the same client device but not to the same data sample can also be included as negative samples. Specifically, for any first client device, any data representation sent by the first client device is taken as the target sample. Based on the data sample identifier corresponding to the target sample, other data representations corresponding to different data samples are determined and used as negative samples of the target sample. The target sample and each negative sample are then paired to form a negative sample pair. This approach can obtain richer negative samples, improve the effect of data representation learning, and thus enhance the performance of the pre-trained data representation model.

[0105] After constructing positive and negative sample pairs, the cloud server calculates the distance (e.g., Euclidean distance) between the two data representations in each sample pair (including both positive and negative pairs). Based on the distance between the two data representations in each sample pair and a pre-set contrastive loss function, the contrastive loss of the data representation is calculated. This contrastive loss ensures that the data representations of augmented samples of the same data sample are as close as possible in the feature space, while maximizing the distance between data representations sent by different client devices in the feature space. Furthermore, the corresponding gradient can be calculated based on the contrastive loss.

[0106] The contrastive loss function is primarily used in feature extraction. It determines whether two samples that were originally similar remain similar in the feature space after feature extraction, or whether two samples that were originally dissimilar remain dissimilar. Similarly, this loss function can also effectively express the degree of matching between samples.

[0107] The specific implementation method for calculating the contrastive loss and gradient in this embodiment is consistent with the existing implementation method for calculating the contrastive loss and gradient in contrastive learning, and will not be described again here.

[0108] After calculating the gradients of each client device (i.e., the gradients of the contrastive loss), the cloud server sends the gradients of each client device to the corresponding client device.

[0109] Step S306: The client device optimizes the model parameters of the local model based on the gradient sent by the cloud server.

[0110] The client device receives gradients sent by the cloud server and backpropagates based on these gradients to optimize the model parameters of the local model. These local model parameters are then aggregated to obtain the global model parameters.

[0111] Optionally, before updating the local model based on the global model parameters, the client device can also use a locally stored validation set to verify the performance parameters of the model determined by the global model parameters, and determine whether the performance parameters meet the update conditions based on the validation results. The performance parameters can be time parameters representing the model's inference efficiency, or parameters representing the accuracy of the model's processing results, etc. Common metrics used in specific application domains / scenarios to verify model performance using validation sets can be used; no specific limitations are imposed here.

[0112] Specifically, the performance parameters of the model determined by the global model parameters are verified. If the performance parameters of the model determined by the global model parameters meet the update conditions, the client device updates the local model according to the global model parameters. If the performance parameters of the model determined by the global model parameters do not meet the update conditions, the client device does not update the local model in this iteration.

[0113] Optionally, before updating the local model based on the global model parameters, the client device can also verify whether the global model parameters meet the model parameter requirements based on the locally stored model parameter requirements. If the global model parameters meet the model parameter requirements, the client device updates the local model based on the global model parameters. If the global model parameters do not meet the model parameter requirements, the client device does not update the local model in this iteration. The model parameter requirements may include the range of model parameter values ​​and required constraints in the current application domain / scenario, and can be set and adjusted based on empirical values ​​for specific application domains / scenarios; no specific limitations are made here. Step S307: The client device sends the optimized local model parameters to the cloud server.

[0114] Step S308: The cloud server aggregates the optimized local model parameters from each client device to obtain the global model parameters.

[0115] In this embodiment, the cloud server aggregates the optimized local model parameters of each client device. This can be achieved by using any of the existing federated learning techniques to aggregate the model parameters of multiple participants. These methods will not be listed and detailed here.

[0116] For example, the optimized local model parameters of each client device can be averaged (or weighted averaged) to determine the aggregated global model parameters; or, the optimized local model parameters of each client device can be clustered based on the gradient offset to determine the aggregated global model parameters.

[0117] Step S309: The cloud server determines whether the iteration has ended.

[0118] The cloud server determines the end of the iteration when the pre-set convergence conditions are met. These convergence conditions can include reaching a pre-set number of iterations, or the comparison loss being less than a preset loss threshold and remaining stable. The convergence conditions can be set and adjusted according to the needs of the actual application scenario and empirical values, and will not be elaborated further here.

[0119] If the determination result indicates that the iteration is not yet complete, the cloud server executes step S300, distributing the optimized global model parameters from this iteration to each client device to begin the next iteration. If the determination result indicates that the iteration is complete, the cloud server executes step S310, determining the final model parameters and distributing the final model parameters to each client device.

[0120] In this embodiment, under the distributed learning architecture, all client devices participate in data representation learning by default. In practical applications, there may be situations where the data characteristics of a certain client device differ significantly from those of other client devices, making the private data of that client device unsuitable for participation in the current data representation learning.

[0121] In one alternative embodiment, the cloud server can screen client devices participating in data representation learning and exclude client devices that are not suitable for participation.

[0122] Optionally, if the gradient of a certain client device differs too much from the gradients of other client devices, it indicates that the data characteristics of that client device are significantly different from those of other client devices, and that client device can be removed from the set of client devices participating in data representation learning.

[0123] Specifically, the cloud server determines the gradient center based on the gradient of the client device; client devices whose gradient is greater than or equal to the gradient center are removed from the set of client devices participating in training, so as to prevent the client device from participating in subsequent iterative learning.

[0124] Optionally, during the representation learning process of molecular graphs of proteins, compounds, etc., if the structural features of the molecular graph of a certain client device are too different from the structural features of the molecular graphs of other client devices, it indicates that the data features of the client device are significantly different from the data features of other client devices, and the client device can be removed from the set of client devices participating in the molecular graph representation learning.

[0125] Specifically, the client device generates a molecular structure representation of the molecular graph used in this round of iterative learning, and sends this representation to the cloud server. The cloud server determines the molecular structure representation center corresponding to the client device based on the molecular structure representation sent by the client device; it then determines the center vector of the molecular structure representation center of the client device as the global center of the molecular structure representation; and removes client devices whose molecular structure representation center is greater than or equal to a second distance threshold from the set of client devices participating in training. If a client device uses multiple molecular graphs, it will send multiple molecular structure representations, and the cloud server will determine the molecular structure representation center corresponding to the client device based on these multiple representations. If a client device uses only one molecular graph, it will send only one molecular structure representation, and the cloud server will use this single representation as the molecular structure representation center corresponding to the client device.

[0126] In addition, in other application areas / scenarios, other filtering strategies can be added during the data representation learning process, taking into account the characteristics of the dataset in the specific area / scenario, to filter and control the client devices participating in data representation learning. No specific limitations are made here.

[0127] Furthermore, once the cloud server determines that a certain client device should be removed from the set of client devices participating in training, it will no longer send gradients to that client device, so that the client device will no longer participate in subsequent iterative learning.

[0128] Step S310: The cloud server sends the final model parameters to each client device.

[0129] The final model parameters can be determined by the cloud server based on the global model parameters aggregated from multiple iterations of learning, and the optimal global model parameters are selected through testing and evaluation. The method by which the cloud server determines the final model parameters is similar to the existing implementation method of selecting optimal model parameters in contrastive learning, and will not be elaborated here.

[0130] Step S311: The client device updates the local model using the final model parameters sent by the cloud server to obtain the pre-trained data representation model.

[0131] The client device receives the final model parameters sent by the cloud server and updates its local model using these parameters, thus obtaining the pre-trained data representation model. The pre-trained data representation model is used to encode the input target data, obtaining a data representation of the target data. This data representation is then used for data processing to obtain the processing results.

[0132] This embodiment presents a relatively complete process for data representation learning based on a distributed contrastive learning architecture in a federated scenario. It calculates the contrastive loss and gradient of each client device's data representation by integrating the augmented representations of data samples sent from various client devices via a cloud server. Specifically, it uses the augmented representations belonging to the same data sample as positive samples and the augmented representations belonging to different client devices as negative samples for contrastive learning. This distributed contrastive learning framework fully considers the imbalance in data distribution among client devices and the degree of data difference across all client devices, significantly improving the model's expressive power, generalization ability, and performance. Furthermore, it fosters a positive relationship between model expressive power and the degree of difference in model evaluation metrics; the stronger the model's expressive power, the more accurate the degree of data difference represented by the output data representation, thus improving the accuracy of data processing results when used in downstream data processing tasks.

[0133] In an optional embodiment, the above Figure 1 The cloud server in the distributed contrastive learning architecture shown may include a first server and a second server. The first server is responsible for aggregating local model parameters sent by multiple client devices, and the second server is responsible for calculating the contrastive loss and gradient. For example, Figure 4 Another distributed contrastive learning system architecture diagram provided for embodiments of this application, such as... Figure 4 As shown, the comparative learning system architecture includes a first server, a second server, and multiple client devices. No data is transmitted between the first server and the second server. The processing flow performed by each client device, as well as their interaction with the first and second servers, are similar. Figure 4 The steps shown illustrate the overall process of one iterative learning cycle, and the specific details of each step are as follows:

[0134] ① The second server broadcasts the global model parameters to each client device, so that each client device receives the global model parameters;

[0135] ② Each client device updates its local model using global model parameters and uses the updated local model to extract the augmented data representation of the data samples;

[0136] ③ Each client device sends its data representation to the first server;

[0137] ④ For each client device, the first server takes the data representation of augmented samples belonging to the same data sample as positive samples and the data representation of augmented samples belonging to different client devices as negative samples, and calculates the contrast loss and gradient for each client device.

[0138] ⑤ The first server sends the gradient of each client device to the corresponding client device;

[0139] ⑥ Each client device optimizes its local model parameters based on the received gradients through backpropagation;

[0140] ⑦ Each client device sends the optimized local model parameters to the second server;

[0141] ⑧ The second server aggregates the local model parameters from each client to obtain new global model parameters.

[0142] based on Figure 4 As shown in the system architecture, this application provides a data representation learning system, which includes a first server, a second server, and multiple client devices. Figure 5 As shown, this data represents the overall process of the learning system as follows:

[0143] Step S500: The second server sends global model parameters to each client device.

[0144] Step S501: The client device updates the local model using the global model parameters sent by the second server.

[0145] Step S502: The client device uses the local model to encode multiple augmented samples of the locally stored data samples to obtain data representations of multiple augmented samples.

[0146] Step S503: The client device sends the data representations of multiple enhanced samples to the first server.

[0147] The first server uses the data representations of augmented samples belonging to the same data sample as positive samples and the data representations of augmented samples belonging to different client devices as negative samples, and calculates the gradient of the contrast loss for the client devices.

[0148] Step S504: The first server obtains the data representation of the augmented samples of data samples extracted by multiple client devices using the local model.

[0149] Step S505: For each client device, the first server takes the data representation of augmented samples belonging to the same data sample as positive samples and the data representation of augmented samples belonging to different client devices as negative samples, calculates the gradient of the contrast loss of the client device, and sends it to the client device.

[0150] Step S506: The client device optimizes the model parameters of the local model based on the gradient sent by the first server.

[0151] Step S507: The client device sends the optimized local model parameters to the second server.

[0152] Step S508: The second server aggregates the optimized local model parameters from each client device to obtain the global model parameters.

[0153] Step S509: The second server determines whether the iteration has ended.

[0154] If the iteration is not completed, execute step S500 to start the next iteration.

[0155] If the iteration ends, proceed to step S510.

[0156] Step S510: The second server sends the final model parameters to each client device.

[0157] Step S511: The client device updates its local model using the final model parameters sent by the second server to obtain the pre-trained data representation model.

[0158] The data representation model is used to encode the input target data to obtain the data representation of the target data. The data representation of the target data is used for data processing to obtain the data processing result of the target data.

[0159] The specific implementation of each step in this embodiment differs from steps S300-S312 above in that the executing entities of some steps are different, but the specific implementation of the corresponding steps is similar, and will not be repeated here.

[0160] Optionally, when filtering the client devices participating in the learning process, the second server can filter them based on the gradient of each client device and stop sending gradients to the removed client devices, so that the removed client devices will no longer participate in subsequent iterative learning.

[0161] Furthermore, the client device is also used to apply the pre-trained data representation model to downstream data processing tasks. Specifically, the client device uses the pre-trained data representation model as an encoding module to construct a data processing model; it trains the data processing model using local training data to obtain a trained data processing model; it inputs the target data to be processed into the encoding module of the trained data processing model for encoding to obtain a data representation of the target data; and it uses the decoding module of the trained data processing model to perform data processing based on the data representation of the target data to obtain the data processing result.

[0162] The distributed contrastive learning architecture provided in this embodiment uses two different cloud servers. One cloud server is used to acquire the data representations of each client device and calculate the contrastive loss and gradient; the other cloud server is used to acquire and aggregate the local model parameters of each client device to obtain the global model parameters. The two cloud servers are completely decoupled, which ensures that the model parameters and data representations are not simultaneously on either cloud server. This guarantees that the cloud server cannot combine the model parameters and data representations to reconstruct the information of the data samples and augmented samples of the client devices, ensuring that the private data of each client device is not leaked. This greatly improves the privacy protection capability of distributed contrastive learning in federated scenarios and enhances data security.

[0163] Figure 6 This is a flowchart of a graph data processing method provided in an exemplary embodiment of this application. The method of this embodiment can be executed by any client device in any of the above embodiments. Different client devices can apply the data representation model of pre-trained graph data to the same or different downstream graph data processing tasks.

[0164] In this embodiment, the data representation learning method described above is applied to the field of graph data processing, such as molecular graph research. Based on the distributed contrastive learning architecture described above, graph data representation learning is performed to obtain a pre-trained graph data representation model. This graph data representation model is then used for downstream graph data processing tasks.

[0165] like Figure 6 As shown, the specific steps of the graph data processing method based on the graph data representation model are as follows:

[0166] Step S601: Obtain the pre-trained graph data representation model.

[0167] In this embodiment, the pre-trained graph data representation model is obtained by pre-training the graph data dataset through any of the above method embodiments.

[0168] Step S602: Use the pre-trained graph data representation model as an encoding module to construct a graph data processing model.

[0169] The pre-trained graph data representation model is used as the backbone network (i.e., the encoding module) of the graph data processing model to encode the input graph data and obtain the graph representation of the graph data.

[0170] Specifically, a pre-trained graph data representation model can be used as an encoding module, and a decoding module can be added to construct a graph data processing model. Graph data processing models can include classification models or regression models. For different types of graph data processing models, the added decoding module is different, but the same pre-trained graph data representation model can be used as the encoding module.

[0171] For example, for classification models, a pre-trained graph data representation model is used as an encoding module, and a classifier is added as a decoding module to construct a graph data processing model for classification tasks. For instance, a molecular property classification prediction model is used to predict whether the target molecule to which the input molecular graph belongs has a specified property (the category includes yes / no), or to predict which property(s) the molecule has from a given property category.

[0172] For example, for regression models, a pre-trained graph data representation model is used as an encoding module, and a regression prediction decoder is added as a decoding module to construct a graph data processing model for regression tasks. For instance, a regression-based molecular property prediction model is used to extract analytical property information from the input molecular graph.

[0173] Step S603: Train the graph data processing model using local training data to obtain the trained graph data processing model.

[0174] For the constructed downstream graph data processing model, the client device can use the training set in this scenario for training (fine-tuning), so that the trained graph data processing model can be applied to downstream sub-graph data processing tasks. For example, for a molecular property prediction model for classification prediction, the training set can include molecular graphs and classification labels. The molecular property prediction model can be fine-tuned based on the molecular graph with classification labels, so that the trained molecular property prediction model has the function of molecular property classification prediction.

[0175] Step S604: Input the target graph data to be processed into the encoding module of the trained graph data processing model for encoding to obtain the graph representation of the target graph data.

[0176] Step S605: Using the decoding module of the trained molecular property prediction model, perform graph data processing based on the graph representation of the target graph data to obtain the graph data processing result.

[0177] When performing graph data processing tasks using a trained graph data processing model, the target graph data to be processed is input into the model. The encoding module of the model encodes the target graph data to obtain its graph representation. Further, the decoding module of the model decodes the graph representation to obtain the graph data processing result.

[0178] This embodiment provides a specific method for using the pre-trained data representation model in the field of graph data processing. The pre-trained graph data representation model has strong feature extraction capabilities (i.e., expressive capabilities) and generalization capabilities. Using the pre-trained graph data representation model as the backbone network of the graph data processing model in downstream tasks can better extract the graph representation of the graph data. Based on this graph representation, graph data processing can improve the accuracy of the graph data processing results.

[0179] The above Figure 1 and Figure 4 The distributed contrastive learning architecture shown can also be applied to image representation learning in image recognition scenarios, that is, the pre-training of image representation models.

[0180] Figure 7 This is a flowchart of an image recognition method provided as an exemplary embodiment of this application. The method of this embodiment can be executed by any client device in any of the above embodiments. Different client devices can apply the pre-trained image representation model to the same or different image recognition tasks.

[0181] In this embodiment, the aforementioned data representation learning method is applied to the field of image recognition. Based on the distributed contrastive learning architecture described above, image representation learning is performed to obtain a pre-trained image representation model. During the image representation learning process, the cloud server integrates the image representations of the augmented images of sample images sent by various client devices to calculate the contrastive loss and gradient of the image representation for each client device. Specifically, the image representations of augmented images belonging to the same sample image are used as positive samples, and the image representations of augmented images belonging to different client devices are used as negative samples for contrastive learning. Under the distributed contrastive learning framework, the imbalance of data distribution among various client devices and the degree of data difference among global client devices can be fully considered, greatly improving the model's expressive power, generalization ability, and performance. At the same time, the model's expressive power and the degree of difference in model evaluation metrics are mutually reinforcing. The stronger the model's expressive power, the more accurate the degree of data difference represented by the output image representation, thus improving the accuracy of image recognition when used downstream.

[0182] This image representation model is used for downstream image recognition tasks. Specifically, the image representation model is used to extract an image representation of the target image to be recognized, and the image representation is decoded to determine the image recognition result of the target image.

[0183] For example, taking face recognition as an example, a pre-trained image representation model can be applied to downstream face recognition tasks. Specifically, the image representation model is used to encode the input face image, extract the image features (i.e., image representation) of the face image, decode the image features to achieve face recognition, and obtain the face recognition result.

[0184] In addition, the pre-trained image representation model can also be applied to object detection, intelligent video analysis (such as analyzing and predicting the image content in a video and extracting key information such as license plates, faces, and actions), traffic scene recognition (such as identifying illegally parked vehicles, pedestrians crossing the road in autonomous driving, lane positions in images, intersections, etc.), and image recognition in the medical field (such as identifying human tissue in images). This embodiment does not make specific limitations here.

[0185] like Figure 7 As shown, the specific steps of the image recognition method based on the image representation model are as follows:

[0186] Step S701: Obtain the pre-trained image representation model.

[0187] Step S702: Construct an image recognition model using the pre-trained image representation model as the encoding module.

[0188] A pre-trained image representation model is used as the backbone network (i.e., the encoding module) of the image recognition model to encode the input image and obtain its representation. Specifically, the pre-trained image representation model can be used as the encoding module, with a decoding module added to construct the image recognition model. The image recognition model can be a classification model or a regression model. Different types of image recognition models require different decoding modules, but the same image recognition model can be used as the encoding module.

[0189] Step S703: Train the image recognition model using local training data to obtain the trained image recognition model.

[0190] For the constructed downstream image recognition model, the client device can use the training set in this scenario for training (fine-tuning), so that the trained image recognition model can be applied to downstream image recognition tasks. For example, for a classification image recognition model, the training set can include sample images and classification labels. The image recognition model is fine-tuned based on sample images with classification labels, so that the trained image recognition model has the function of image classification and recognition.

[0191] Step S704: Input the target image to be processed into the encoding module of the trained image recognition model for encoding to obtain the image representation of the target image.

[0192] Step S705: Using the decoding module of the trained image recognition model, image recognition is performed based on the image representation of the target image to obtain the image recognition result.

[0193] When performing an image recognition task using a trained image recognition model, the target image to be recognized is input into the model. The encoding module of the image recognition model encodes the target image to obtain its image representation. Further, the decoding module of the image recognition model decodes the image representation to obtain the image recognition result.

[0194] This embodiment provides a specific method for using the pre-trained image representation model in the aforementioned method embodiment. The pre-trained image representation model has powerful feature extraction capabilities (i.e., expressive capabilities) and generalization capabilities. Using it as the backbone network of the image recognition model in downstream tasks can better extract the image representation of the image to be recognized. Based on this image representation, image recognition can be performed accurately, thus improving the accuracy of image recognition.

[0195] The above Figure 1 and Figure 4 The distributed contrastive learning architecture shown can also be applied to text representation learning in natural language processing scenarios, that is, the pre-training of text representation models.

[0196] Figure 8 This is a flowchart of a natural language processing method provided in an exemplary embodiment of this application. The method of this embodiment can be executed by any client device in any of the above embodiments. Different client devices can apply the pre-trained text representation model to the same or different natural language processing tasks.

[0197] In this embodiment, the aforementioned data representation learning method is applied to the field of natural language processing. Based on the distributed contrastive learning architecture described above, text representation learning is performed to obtain a pre-trained text representation model. This text representation model is used for downstream natural language processing tasks. During the text representation learning process, the cloud server integrates the text representations of the augmented texts from various client devices to calculate the contrastive loss and gradient of the text representation for each client device. Specifically, the text representations of augmented texts belonging to the same text corpus are used as positive samples, and the text representations of augmented texts belonging to different client devices are used as negative samples for contrastive learning. Under the distributed contrastive learning framework, the imbalance of data distribution among client devices and the degree of data difference among global client devices can be fully considered, greatly improving the model's expressive power, generalization ability, and performance. At the same time, the model's expressive power and the degree of difference in model evaluation metrics are mutually reinforcing. The stronger the model's expressive power, the more accurate the degree of data difference represented by the output text representation, thus improving the accuracy of natural language processing when used downstream.

[0198] In this embodiment, the pre-trained text representation model can be applied to downstream natural language processing tasks. Specifically, the text representation model is used to extract the text representation of the target text to be processed, and the text representation is decoded to obtain the natural language processing result of the target text.

[0199] For example, taking a text classification task applied in e-commerce scenarios, the user's input question can be fed into a natural language processing model. The encoding module generates a text representation of the question, and a classification decoder is used to classify and predict the question's category, such as product quality issues, delivery issues, or product description issues. Furthermore, the pre-trained text representation model can also be applied to automatic summarization, opinion extraction, text classification, and text semantic comparison, etc., but this embodiment does not specifically limit its application.

[0200] like Figure 8 As shown, the specific steps of the natural language processing method based on the text representation model are as follows:

[0201] Step S801: Obtain the pre-trained text representation model.

[0202] Step S802: Construct a natural language processing model based on the pre-trained text representation model as the encoding module.

[0203] A pre-trained text representation model is used as the backbone network (i.e., the encoding module) of a natural language processing (NLP) model to encode the input text and obtain its text representation. Specifically, a pre-trained text representation model can be used as the encoding module, with a decoding module added to construct the NLP model. The NLP model can be a classification model or a regression model. Different types of NLP models require different decoding modules, but the same NLP model can be used as the encoding module.

[0204] Step S803: Train the natural language processing model using local training data to obtain the trained natural language processing model.

[0205] For the constructed downstream natural language processing (NLP) model, the client device can use the training set specific to this scenario for training (fine-tuning), enabling the trained NLP model to be applicable to downstream NLP tasks. For example, for a classification-based NLP model, the training set can include text corpora and classification labels. The NLP model can be fine-tuned based on the text corpora with classification labels, giving the trained NLP model the ability to classify text.

[0206] Step S804: Input the target text to be processed into the encoding module of the trained natural language processing model for encoding to obtain the text representation of the target text.

[0207] Step S805: Using the decoding module of the trained natural language processing model, natural language processing is performed based on the text representation of the target text to obtain the natural language processing result.

[0208] When performing a natural language processing (NLP) task using a trained NLP model, the target text to be processed is input into the NLP model. The encoding module of the NLP model encodes the target text to obtain its text representation. Further, the decoding module of the NLP model decodes the text representation to obtain the NLP result of the target text.

[0209] This embodiment provides a specific method for using the pre-trained text representation model in the aforementioned method embodiment. The pre-trained text representation model has powerful feature extraction capabilities (i.e., expressive capabilities) and generalization capabilities. Using it as the backbone network of the natural language processing model in downstream tasks can better extract the text representation of the text to be processed. Based on this text representation, natural language processing can be performed accurately, thereby improving the accuracy of natural language processing.

[0210] Figure 9 This is a schematic diagram of the structure of a cloud server provided in an example embodiment of this application. Figure 9 As shown, the cloud server 90 includes a processor 901 and a memory 902 communicatively connected to the processor 901. The memory 902 stores computer-executable instructions. The processor executes the computer-executable instructions stored in the memory to implement the method flow executed by the cloud server or the first server in any of the above method embodiments. Specific functions and achievable technical effects are not elaborated here.

[0211] Figure 10 This is a schematic diagram of the structure of a client device provided in an example embodiment of this application. For example... Figure 10 As shown, the client device 100 includes a processor 1001 and a memory 1002 communicatively connected to the processor 1001. The memory 1002 stores computer-executable instructions. The processor executes the computer-executable instructions stored in the memory to implement the method flow executed by any client device in any of the above method embodiments. Specific functions and achievable technical effects are not elaborated here.

[0212] This application also provides a computer-readable storage medium storing computer-executable instructions. When executed by a processor, the computer-executable instructions are used to implement the method flow executed by any cloud server or client device in any of the above method embodiments. The specific functions and technical effects to be achieved are not described here.

[0213] This application also provides a computer program product, which includes a computer program stored in a readable storage medium. At least one processor of the electronic device can read the computer program from the readable storage medium. The at least one processor executes the computer program to cause the electronic device to perform the method flow executed by any cloud server or client device in any of the above method embodiments. The specific functions and technical effects that can be achieved are not described here.

[0214] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0215] Furthermore, in some of the processes described in the above embodiments and accompanying drawings, multiple operations appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or may be executed in parallel. The sequence numbers are merely used to distinguish different operations, and the sequence number itself does not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types. "Multiple" means two or more, unless otherwise explicitly specified.

[0216] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0217] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A data representation learning method, characterized in that, Applied to cloud servers, including: Obtain data representations of augmented samples from data samples extracted by multiple client devices using a local model, wherein the data representations contain data representations of multiple augmented samples of the same data sample; For each client device, the data representation of augmented samples belonging to the same data sample is taken as a positive sample, and the data representation of augmented samples belonging to different client devices is taken as a negative sample. The gradient of the contrastive loss of the data representation of the client device is calculated and sent to the client device. The gradient of the contrastive loss of the data representation of the client device is used to optimize the local model of the client device to obtain a pre-trained data representation model. The data representation model is used to encode the input target data to obtain a data representation of the target data. The data representation of the target data is used for data processing to obtain the data processing result of the target data.

2. The method according to claim 1, characterized in that, After calculating the gradient of the contrastive loss in the data representation of the client device and sending it to the client device, the method further includes: Receive local model parameters optimized according to the gradient from multiple client devices; By aggregating the local model parameters of multiple client devices, global model parameters are obtained; The global model parameters are sent to multiple client devices, so that each client device updates its local model according to the global model parameters and begins the next round of iterative learning.

3. The method according to claim 1, characterized in that, Also includes: When calculating the gradient of the contrast loss for the data representation of each client device, the data representations of augmented samples that belong to the same client device but not to the same data sample are also treated as negative samples.

4. The method according to any one of claims 1-3, characterized in that, Also includes: The gradient center is determined based on the gradient of the comparative loss represented by the data from the client device. Client devices whose distance from the gradient to the gradient center is greater than or equal to a first distance threshold are removed from the set of client devices participating in training.

5. The method according to any one of claims 1-3, characterized in that, The cloud server includes a first server and a second server; The first server is used to calculate the gradient of the contrast loss for the data representation of each of the client devices; The second server is used to aggregate the local model parameters of multiple client devices to obtain global model parameters.

6. A data representation learning method, characterized in that, Applied to client devices, including: The local model is used to encode multiple augmented samples of the locally stored data sample to obtain data representations of multiple augmented samples; The data representations of the multiple augmented samples are sent to the cloud server, so that the cloud server treats the data representations of augmented samples belonging to the same data sample as positive samples and the data representations of augmented samples belonging to different client devices as negative samples, and calculates the gradient of the contrast loss of the data representations of the client devices. Based on the gradient sent by the cloud server, optimize the model parameters of the local model to obtain a pre-trained data representation model; The data representation model is used to encode the input target data to obtain a data representation of the target data. The data representation of the target data is used for data processing to obtain the data processing result of the target data.

7. The method according to claim 6, characterized in that, After optimizing the model parameters of the local model based on the gradient sent by the cloud server, the process further includes: The optimized local model parameters are sent to the cloud server, and these local model parameters are used to aggregate and obtain global model parameters. Based on the global model parameters sent by the cloud server, the local model is updated, and the next round of iterative learning begins.

8. The method according to claim 6, characterized in that, The cloud server includes a first server and a second server. Sending the data representations of the multiple enhanced samples to the cloud server includes: The data representations of the multiple augmented samples are sent to the first server, so that the first server treats the data representations of augmented samples belonging to the same data sample as positive samples and the data representations of augmented samples belonging to different client devices as negative samples, and calculates the gradient of the contrast loss of the data representations of the client devices; After optimizing the model parameters of the local model based on the gradient sent by the first server, the process further includes: The optimized local model parameters are sent to the second server, and these local model parameters are used to aggregate and obtain global model parameters. Based on the global model parameters sent by the second server, the local model is updated, and the next round of iterative learning begins.

9. The method according to claim 7 or 8, characterized in that, Before updating the local model based on the global model parameters, the following is also included: Using a locally stored validation set, the performance parameters of the model determined by the global model parameters are validated, and the performance parameters are determined to meet the update conditions based on the validation results.

10. A data processing method, characterized in that, Applied to client devices, including: Obtain a pre-trained data representation model, wherein the data representation model is pre-trained by the method described in any one of claims 1-9; The pre-trained data representation model is used as an encoding module to construct the data processing model; The data processing model is trained using local training data to obtain a trained data processing model. The target data to be processed is input into the encoding module of the trained data processing model for encoding to obtain the data representation of the target data; The decoding module of the trained data processing model performs data processing based on the data representation of the target data to obtain the data processing result.

11. A data representation learning system, characterized in that, include: A first server, a second server, and multiple client devices. The client device is configured to encode multiple augmented samples of locally stored data samples using a local model to obtain a sample representation of multiple augmented samples for each data sample, and send the sample representation of the multiple augmented samples to a first server. The first server is configured to, for each client device, take the data representation of augmented samples belonging to the same data sample as a positive sample, and the data representation of augmented samples belonging to different client devices as a negative sample, calculate the gradient of the contrast loss of the data representation of the client device, and send it to the client device. The client device is further configured to optimize the model parameters of the local model based on the gradient of the contrast loss represented by the data sent by the first server and send it to the second server. The second server is used to aggregate the local model parameters sent by each of the client devices to obtain aggregated global model parameters, and then send the aggregated global model parameters to each of the client devices. The client device is also used to update the local model based on the aggregated global model parameters sent by the second server, and to start the next round of iterative learning to obtain a pre-trained data representation model.

12. The system according to claim 11, characterized in that, The client device is also used for: The pre-trained data representation model is used as an encoding module to construct the data processing model; The data processing model is trained using local training data to obtain a trained data processing model. The target data to be processed is input into the encoding module of the trained data processing model for encoding to obtain the data representation of the target data; The decoding module of the trained data processing model performs data processing based on the data representation of the target data to obtain the data processing result.

13. A cloud server, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-5.

14. A client device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 6-10.

Images