Model training method and device, data prediction method and device, equipment and storage medium

Abstract

The invention provides a model training method and device, a data prediction method and device, equipment and a storage medium. The method comprises the following steps: determining an input sample, and carrying out comparative learning processing on the input sample through a first encoder to obtain a comparison loss; encoding the input sample through a first encoder to obtain a first encoding feature, and encoding the input sample based on a second encoder to obtain a second encoding feature; performing supervised learning processing on the first coding feature and the second coding feature to obtain supervised loss; and training a to-be-trained model based on the comparison loss and the supervision loss to obtain a target model. According to the invention, the accuracy of data prediction can be improved.

Description

Technical Field

[0001] This application relates to artificial intelligence technology, and more particularly to a model training method, a data prediction method, an apparatus, a device, and a storage medium. Background Technology

[0002] In recommender systems, the prediction of recommendation metrics (such as click-through rate or conversion rate) has gradually received widespread attention, as they directly affect the efficiency and effectiveness of recommendations. However, the training process of these models used for data prediction often faces the problems of data sparsity and long-tail distribution. Data sparsity means that in recommender systems, although there are millions of data points, the explicit feedback from users on these data (such as clicks or conversions) is very limited, resulting in a far from sufficient amount of data available for training. The long-tail distribution problem is characterized by a small number of data points receiving a large amount of feedback, while most data points receive almost no feedback, making it difficult for the model to learn the features of data points with little feedback. Summary of the Invention

[0003] This application provides a model training method, a data prediction method, an apparatus, a device, and a storage medium that can improve the accuracy of data prediction.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] This application provides a model training method, the method comprising:

[0006] The input sample is determined, and the input sample is subjected to contrastive learning processing through the first encoder to obtain the contrastive loss;

[0007] The input sample is encoded by the first encoder to obtain a first encoded feature, and the input sample is encoded by the second encoder to obtain a second encoded feature;

[0008] Supervised learning processing is performed on the first encoded feature and the second encoded feature to obtain a supervised loss;

[0009] Based on the contrastive loss and the supervised loss, the model to be trained is trained to obtain the target model.

[0010] This application provides a data prediction method, the method comprising:

[0011] The input data is encoded based on the first encoder included in the target model to obtain the third encoded feature;

[0012] The input data is encoded based on the second encoder included in the target model to obtain the fourth encoded feature;

[0013] The third and fourth coding features are subjected to prediction processing to obtain a prediction result, wherein the target model is trained by the model training method provided in the embodiments of this application.

[0014] This application provides a model training apparatus, including:

[0015] The contrastive learning module is used to determine the input sample and perform contrastive learning processing on the input sample through the first encoder to obtain the contrastive loss;

[0016] The supervised learning module is used to encode the input sample using the first encoder to obtain a first encoded feature, and to encode the input sample using the second encoder to obtain a second encoded feature; and to perform supervised learning processing on the first encoded feature and the second encoded feature to obtain a supervised loss.

[0017] The model training module is used to train the model to be trained based on the contrastive loss and the supervised loss to obtain the target model.

[0018] This application provides a data prediction device, including:

[0019] A data prediction module is used to encode input data based on a first encoder included in the target model to obtain a third encoded feature; to encode the input data based on a second encoder included in the target model to obtain a fourth encoded feature; and to perform prediction processing on the third encoded feature and the fourth encoded feature to obtain a prediction result, wherein the target model is trained by the model training method according to any one of claims 1 to 12.

[0020] This application provides an electronic device, including:

[0021] Memory is used to store executable instructions for a computer;

[0022] The processor, when executing computer-executable instructions stored in the memory, implements the model training method or the data prediction method provided in the embodiments of this application.

[0023] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions, which, when executed by a processor, implements the model training method or the data prediction method provided in this application.

[0024] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the model training method or the data prediction method provided in this application.

[0025] The embodiments of this application have the following beneficial effects:

[0026] The input samples are subjected to contrastive learning processing using a first encoder to obtain a contrastive loss. The input samples are then encoded using the first encoder to obtain first encoded features, and then encoded using a second encoder to obtain second encoded features. Supervised learning processing is then performed on the first and second encoded features to obtain a supervised loss. Based on the contrastive and supervised losses, the model to be trained is trained to obtain the target model. The contrastive loss is used to learn feature representations, making similar samples closer together in the feature space and dissimilar samples further apart. The supervised loss is used to measure the model's predicted output against the true label. Since the contrastive loss helps the model learn more robust feature representations, and the supervised loss ensures the model can accurately predict labels, combining the contrastive and supervised losses to train the model improves the accuracy of data prediction while maintaining its generalization ability. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of the architecture of the model training and data prediction system provided in the embodiments of this application;

[0028] Figure 2A This is a first structural schematic diagram of the electronic device provided in an embodiment of this application;

[0029] Figure 2B This is a schematic diagram of the second structure of the electronic device provided in the embodiments of this application;

[0030] Figure 3A This is a schematic diagram of the first process of the model training method provided in the embodiments of this application;

[0031] Figure 3B This is a schematic diagram of the second process of the model training method provided in the embodiments of this application;

[0032] Figure 3C This is a schematic diagram of the third process of the model training method provided in the embodiments of this application;

[0033] Figure 3D This is a schematic diagram of the fourth process of the model training method provided in the embodiments of this application;

[0034] Figure 3E This is a schematic diagram of the fifth process of the model training method provided in the embodiments of this application;

[0035] Figure 3F This is a schematic diagram of the sixth process of the model training method provided in the embodiments of this application;

[0036] Figure 3G This is a schematic diagram of the seventh process of the model training method provided in the embodiments of this application;

[0037] Figure 4 This is a flowchart illustrating the data prediction method provided in an embodiment of this application;

[0038] Figure 5 This is a schematic diagram illustrating the principle of joint training of supervised learning and contrastive learning provided in an embodiment of this application;

[0039] Figure 6 This is a schematic diagram of the model inference stage provided in the embodiments of this application;

[0040] Figure 7 This is an illustration of an advertising recommendation provided in an embodiment of this application.

[0041] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0043] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0044] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0045] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0046] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant national laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0047] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0048] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0049] 1) Input samples are features obtained by mapping sample data. This application embodiment does not limit the sample data. Sample data includes at least one of the following: sample name, sample content, sample format, etc.

[0050] 2) Contrastive learning is an unsupervised learning method that learns the representation of input samples by comparing the similarity between different input samples, so that similar samples are closer in the representation space, while dissimilar samples are further apart in the representation space.

[0051] 3) Supervised learning is a machine learning paradigm that uses labeled input samples to learn how to predict the labels of new input data. The model's parameters are updated by minimizing the difference between the predicted labels obtained during the supervised learning process and the true labels of the input samples.

[0052] In related technologies, features encoded by a single encoder are used for supervised learning and contrastive learning respectively to obtain a model for data prediction, which reduces the accuracy of data prediction. To address the above problems, embodiments of this application provide a model training method, a data prediction method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product, in which features encoded by different encoders are used for supervised learning and contrastive learning respectively, thereby improving the generalization of the model and enhancing the accuracy of data prediction.

[0053] The model training method described in this application can be applied to various fields (such as image field, text field, video field, audio field, etc.). For example, it can be multiple fields such as product click-through rate prediction in the recommendation field, product conversion rate prediction in the recommendation field, image recognition field, anomaly detection field, etc. That is, the model training method in this application is not limited to a certain field.

[0054] The following describes exemplary applications of the electronic device provided in the embodiments of this application. The device provided in the embodiments of this application can be implemented as a terminal or as a server. The following will describe exemplary applications when the device is implemented as a server.

[0055] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of the model training and data prediction system 100 provided in the embodiments of this application. In order to support a model training application, the terminal (terminal 400 is shown as an example) connects to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0056] Terminal 400 is used to send input samples to server 200 via network 300. Server 200 is used to perform contrastive learning processing on the input samples using a first encoder to obtain contrastive loss, encode the input samples using the first encoder to obtain first encoded features, and encode the input samples using a second encoder to obtain second encoded features. Supervised learning processing is performed on the first and second encoded features to obtain supervised loss. Based on the contrastive loss and supervised loss, the model to be trained is trained to obtain the target model. Then, server 200 can receive input data sent by terminal 400, encode the input data using the first encoder included in the target model to obtain third encoded features, encode the input data using the second encoder included in the target model to obtain fourth encoded features, perform prediction processing on the third and fourth encoded features to obtain prediction results, and send the obtained prediction results to terminal 400. Terminal 400 displays the prediction results through graphical interface 410.

[0057] The following is an example of terminal 400 performing model training and data prediction.

[0058] In some embodiments, the terminal 400 can independently complete model training and data prediction tasks. For example, the terminal 400 determines the input sample, performs contrastive learning processing on the input sample through a first encoder to obtain a contrastive loss, encodes the input sample through the first encoder to obtain a first encoded feature, encodes the input sample based on a second encoder to obtain a second encoded feature, performs supervised learning processing on the first and second encoded features to obtain a supervised loss, trains the model to be trained based on the contrastive loss and the supervised loss to obtain a target model, then encodes the input data based on the first encoder included in the target model to obtain a third encoded feature, encodes the input data based on the second encoder included in the target model to obtain a fourth encoded feature, performs prediction processing on the third and fourth encoded features to obtain a prediction result, and displays the prediction result through a graphical interface 410.

[0059] In one implementation scenario, a server or terminal can train a click-through rate (CTR) prediction model to predict the CTR of advertisements. This involves identifying user data samples, performing contrastive learning on these samples using a first encoder to obtain a contrastive loss, encoding the user data samples using the first encoder to obtain a first encoded feature, and encoding the user data samples using a second encoder to obtain a second encoded feature. Supervised learning is then performed on the first and second encoded features to obtain a supervised loss. Based on the contrastive and supervised losses, the CTR prediction model is trained to obtain a trained CTR prediction model. Then, the user data is encoded using the first encoder included in the trained CTR prediction model to obtain a third encoded feature, and the user data is encoded using the second encoder included in the trained CTR prediction model to obtain a fourth encoded feature. CTR prediction is then performed on the third and fourth encoded features to obtain the predicted CTR. Advertisements are then ranked based on the predicted CTR to obtain a ranking result. A portion of the ranked advertisements are then pushed to users. Here, the CTR is the probability that a user clicks on an advertisement's display page after receiving the advertisement push message.

[0060] In one implementation scenario, a server or terminal can train a conversion rate prediction model to predict the conversion rate of advertisements. This involves identifying user data samples, performing contrastive learning on these samples using a first encoder to obtain a contrastive loss, encoding the user data samples using the first encoder to obtain a first encoded feature, and encoding the user data samples using a second encoder to obtain a second encoded feature. Supervised learning is then performed on the first and second encoded features to obtain a supervised loss. Based on the contrastive and supervised losses, the untrained conversion rate prediction model is trained to obtain a trained conversion rate prediction model. Then, the user data is encoded using the first encoder included in the trained conversion rate prediction model to obtain a third encoded feature, and the user data is encoded using the second encoder included in the trained conversion rate prediction model to obtain a fourth encoded feature. Conversion rate prediction is then performed on the third and fourth encoded features to obtain the predicted conversion rate. Advertisements are ranked based on the predicted conversion rate to obtain a ranking result, and a portion of the ranked advertisements are pushed to users. Here, the conversion rate represents the probability that a user purchases or uses a product based on the advertisement information displayed on the ad display page after clicking through to the ad's display page.

[0061] In some embodiments, server 200 may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0062] Terminal 400 can be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, smart voice interaction device, smart home appliance, vehicle terminal, aircraft, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0063] See Figure 2A , Figure 2A This is a first structural schematic diagram of the electronic device provided in an embodiment of this application. Figure 2A The electronic device 500 shown can be Figure 1The terminal 400 or server 200 in the figure, and the electronic device 500 include: at least one processor 510, memory 550, and at least one network interface 520. The various components in server 200 are coupled together via bus system 540. It is understood that bus system 540 is used to implement communication between these components. In addition to a data bus, bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus system 540 in Figure 2.

[0064] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0065] User interface 530 includes one or more output devices 531 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls;

[0066] In some embodiments, when the terminal 400 independently completes the model training task or data prediction task, the server 200 provided in this application embodiment does not include the user interface 530.

[0067] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 550 may optionally include one or more storage devices physically located away from the processor 510.

[0068] The memory 550 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory.

[0069] In some embodiments, memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0070] Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0071] The network communication module 552 is used to reach other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0072] Presentation module 553 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 531 (e.g., a display screen, a speaker, etc.) associated with user interface 530;

[0073] In some embodiments, when the terminal 400 independently completes the model training task or data prediction task, the server 200 provided in this application embodiment may not include the presentation module 553.

[0074] The input processing module 554 is used to detect and translate one or more user inputs or interactions from one or more input devices 532; in some embodiments, when the terminal 400 independently completes the model training task or data prediction task, the server 200 provided in this application embodiment may not include the presentation module 553.

[0075] In some embodiments, the model training apparatus provided in this application can be implemented in software. Figure 2A A model training apparatus 555 stored in memory 550 is shown. This apparatus can be software in the form of programs and plug-ins, and includes the following software modules: a supervised learning module 5551, a contrastive learning module 5552, a model training module 5553, and a data acquisition module 5554. These modules are logically connected and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.

[0076] In some embodiments, the data prediction apparatus provided in this application can also be implemented in software, see [link to relevant documentation]. Figure 2B , Figure 2B This is a schematic diagram of the second structure of the electronic device provided in the embodiments of this application. Figure 2B Apart from the image classification device 556 shown, the rest can all be related to Figure 2AThe same applies, and will not be repeated here. The image classification device 556 stored in memory 550 can be software in the form of programs and plug-ins, including the following software modules: data prediction module 5561. These modules are logically linked and can therefore be arbitrarily combined or further split according to the functions they implement. The functions of each module will be described below.

[0077] It should be noted that, in the model training examples below, those skilled in the art, based on their understanding of the following text, can apply the target model trained by the model training method provided in the embodiments of this application to data prediction.

[0078] See Figure 3A , Figure 3A This is a schematic diagram of the first process of the model training method provided in the embodiments of this application, which will be combined with Figure 3A The steps shown will be explained below. The model training method provided in this application embodiment can be implemented by the server or the terminal alone, or by the server and the terminal working together. The following will be an example of the server and the terminal working together.

[0079] In step 101, the input sample is determined, and the input sample is subjected to contrastive learning processing by the first encoder to obtain the contrastive loss.

[0080] Here, the input sample is the feature obtained by mapping the sample data. This application embodiment does not limit the sample data. The sample data includes at least one of the following: sample name, sample content, sample format, etc. The format of the sample data can be text, image, audio, video, etc. The first encoder is an algorithm or model used to convert the input sample into a specific format. It converts the input sample into a feature vector of a fixed size through a series of calculations. The feature vector is used to capture important information of the input sample. This application embodiment does not limit the first encoder. The first encoder can be a convolutional neural network, a recurrent neural network, etc. The feature vector encoded by the first encoder can be used for classification tasks, regression tasks, etc. Contrastive learning is an unsupervised learning method used to learn the representation of the input sample by comparing the similarity between different input samples, so that similar samples are closer in the representation space, while dissimilar samples are farther apart in the representation space.

[0081] In some embodiments, see Figure 3B , Figure 3B This is a schematic diagram of the second process of the model training method provided in the embodiments of this application, which is aimed at... Figure 3A The step 101 shown, which determines the input sample, can be done through... Figure 3B Steps 1011A to 1012A are implemented, and will be explained in detail below.

[0082] In step 1011A, sample data from multiple domains are acquired, and the sample data from each domain is mapped to obtain domain samples for each domain.

[0083] Here, this application embodiment does not limit the field or form of the sample data. The field can be mobile phones, computers, books, household goods, etc. Each field includes at least one sample data. Taking the mobile phone field as an example, the mobile phone's name and model, technical specifications, features, user reviews, and other detailed information constitute the text-based sample data. Among them, the mobile phone's name and model are key to identifying and distinguishing different mobile phones. Mobile phone illustrations or design drawings provide visual image-based sample data to show the mobile phone's appearance design. In the computer field, the sample data may include the computer's brand, configuration parameters, images of hardware components, screenshots of the operating system interface, and the text content of the user manual. In the book field, the sample data may cover the book's title, author, publication information, cover design, table of contents, book reviews, and abstracts. In the household goods field, the sample data may include product photos, size specifications, material composition, instructions for use, and customer feedback.

[0084] It should be noted that the mapping process is used to map the sample data of each domain to a new feature space to obtain the mapping features used to characterize the potential structure of the sample data. The embodiments of this application do not limit the mapping process, which can be linear mapping, nonlinear mapping, etc.

[0085] For example, the sample set contains sample data and the corresponding coding features of the sample data (e.g., the coding features of a mobile phone are [0.15, 0.03, 0.06], and the coding features of a computer are [0.15, 0.03, 0.06]).

[0086] In some embodiments, when the sample data is in the form of text, the above-mentioned "mapping the sample data of each domain to obtain the domain sample of each domain" can be implemented in the following way: performing word segmentation on the sample data of each domain to obtain domain word segments; performing stemming on each word segment to obtain restored word segments; filtering the restored word segments and vectorizing the filtered restored word segments to obtain the domain sample of each domain.

[0087] For example, the text sample data is segmented into words, breaking down continuous character sequences into meaningful words or phrases. For each segment, stemming or lemmatization is performed to convert the word into its basic form, reducing data dimensionality and eliminating the impact of lemmatization. For example, "running", "runs", and "ran" can all be converted back to "run". Stop words (such as "the") are removed. The filtered segments are then vectorized, converting the text into numerical vectors. Vectorization can be achieved by: extracting a vocabulary from all text data, containing all unique words; segmenting each text segment and matching the segmentation results with the vocabulary, counting the frequency of each word in the text; and converting each text into a vector, where each element corresponds to a word in the vocabulary, and the element value represents the probability of that word appearing in the text.

[0088] Continuing with the example above, given three texts: Text 1 is "The quick brown fox", Text 2 is "Jumped over the lazy dog", and Text 3 is "The quick dog outran the fox", and the given vocabulary is ["the", "quick", "brown", "fox", "jumped", "over", "lazy", "dog", "outran"]. The bag-of-words model vector representations of the three texts are as follows: Text 1: [0.166, 0.166, 0.166, 0, 0.166, 0, 0]; Text 2: [0, 0, 0, 0, 0.25, 0.25, 0.25, 0.25, 0]; Text 3: [0.2, 0.2, 0, 0.2, 0, 0, 0, 0.2, 0.2]. Each element represents the probability of the corresponding word appearing in the text, with 0 indicating that the word does not appear in the text.

[0089] In step 1012A, the domain samples corresponding to multiple domains are spliced ​​together to obtain the input sample.

[0090] Each domain corresponds to at least one domain sample.

[0091] For example, the domain samples corresponding to domain A (e.g., [0.14, 0.13, -0.06]) and the domain samples corresponding to domain B (e.g., [0.15, 0.03, 0.06]) are concatenated to obtain the input sample (e.g., [0.14, 0.13, -0.06, 0.15, 0.03, 0.06]).

[0092] Through the embodiments of this application, since data from different fields have different characteristics and distributions, the features of each field are extracted and integrated through mapping processing, which increases the information richness of the input samples and provides a more diverse data perspective.

[0093] In some embodiments, see Figure 3C , Figure 3C This is a schematic diagram of the third process of the model training method provided in the embodiments of this application, targeting... Figure 3A In step 101 shown, the input samples are subjected to contrastive learning processing through the first encoder to obtain the contrastive loss, which can be obtained through... Figure 3C Steps 1011B to 1013B are implemented, and will be explained in detail below.

[0094] In step 1011B, the input sample is masked to obtain multiple masked samples.

[0095] Here, masking is used to apply a mask template to a portion of the input sample to simulate an incomplete input sample, allowing the model to make predictions without complete information, thereby enhancing the robustness of the model.

[0096] It should be noted that the contrastive loss is obtained by performing contrastive learning on the input samples through at least one first encoder, and the number of first encoders is the same as the number of mask samples.

[0097] In some embodiments, step 1011B can be implemented by: obtaining multiple mask templates; performing the following processing on each mask template: performing mask processing on the input sample based on the mask template to obtain a mask sample.

[0098] Here, the mask template corresponds one-to-one with the mask sample. The mask template is used to hide or show specific parts of features (such as input samples). This application embodiment does not limit the mask template. The data in the mask template can be binary (i.e., only 0 and 1 values, where 0 represents transparent or hidden, that is, the replacement data is the data of the input sample at the corresponding position, and 1 represents opaque, that is, the replacement data is 1).

[0099] In some embodiments, the mask template includes replacement data, each replacement data corresponding to a position. The above-mentioned "masking the input sample based on the mask template to obtain a mask sample" can be achieved by replacing the data at the position of the input sample with the replacement data to obtain the mask sample.

[0100] It should be noted that the embodiments of this application do not limit the position. The position can be the index subscript corresponding to the input sample, and each index subscript corresponds to a piece of data in the input sample.

[0101] For example, given a mask template (e.g., [0, 0, 1, 0], where the replacement data at index 2 is 1, and when the index is 0, 1, or 3, the replacement data is the data of the input sample at the corresponding index), the data at position (e.g., index 2) of the input sample (e.g., [0.52, 0.13, -0.06, 0.15]) (e.g., "-0.06") is replaced with the replacement data (e.g., 1) to obtain the mask sample (e.g., [0.52, 0.13, 1, 0.15]).

[0102] Through the embodiments of this application, by using multiple mask templates, more accurate feature extraction and analysis can be achieved by focusing on specific regions or features of the input sample. At the same time, mask processing can spatially filter the input sample, which can not only enhance the useful information in the input sample, but also suppress noise and irrelevant features in the input sample, thereby improving the accuracy and efficiency of subsequent processing. In addition, mask processing can cope with the diversity and complexity of data, and determine different mask templates according to different scenarios, thereby improving the accuracy of data prediction in different scenarios.

[0103] In step 1012B, each mask sample is encoded by the first encoder to obtain the third encoded feature.

[0104] Here, the first encoder includes a first normalization layer, an attention layer, a second normalization layer, and a mapping layer.

[0105] In some embodiments, the first encoder includes a convolutional layer, a pooling layer, and a mapping layer. Step 1012B can be implemented by performing the following processing for each mask sample: convolving the mask sample through the convolutional layer to obtain convolutional features; pooling the convolutional features through the pooling layer to obtain pooled features; and mapping the pooled features through the mapping layer to obtain third encoded features.

[0106] For example, the mask sample (e.g., [0.52, *, -0.06, 0.15, -0.06, *, 0.52, 0.13, *, 0.15) is convolved through a convolutional layer to obtain convolutional features (e.g., [0.14, 0.13, -0.06, 0.15, 0.03, 0.06]). The convolutional features are then pooled through a pooling layer to obtain pooled features (e.g., [0.14, 0.13, -0.06]). Finally, the pooled features are mapped through a mapping layer to obtain the third encoded features (e.g., [0.52, 0.13, -0.06, 0.15]).

[0107] In step 1013B, a loss is constructed for multiple third-encoded features to obtain a contrastive loss.

[0108] In some embodiments, step 1013B can be implemented as follows: mapping each third coding feature to obtain a first mapping feature; constructing a sub-contrast loss based on any two first mapping features; and weighting and summing the sub-contrast losses to obtain the contrast loss.

[0109] Here, the embodiments of this application do not limit the mapping method, which can be linear mapping, nonlinear mapping, etc.

[0110] In some embodiments, the above-mentioned "mapping process for each third coding feature to obtain a first mapped feature" can be implemented by: determining the mapping weight and the mapping bias, and determining the first mapped feature by summing the product of each third coding feature and the mapping weight with the mapping bias.

[0111] For example, given a mapping weight (e.g., 0.5) and a mapping bias (e.g., 0.2), the product of each third coding feature (e.g., [0.15, 0.03, 0.06]) with the mapping weight (e.g., [0.075, 0.015, 0.03]) and the sum of the mapping bias (e.g., [0.275, 0.215, 0.23]) is determined as the first mapping feature.

[0112] Following the above embodiments, the above "constructing a sub-contrast loss based on any two first mapping features" can be achieved in the following way: determining the similarity between any two first mapping features; and determining the difference between the preset similarity (e.g., 1) and the similarity as the loss function.

[0113] It should be noted that the embodiments of this application do not limit the similarity. The similarity can be the cosine similarity, Euclidean distance, etc. between any two first mapping features.

[0114] For example, given a first mapping feature A (e.g., [0.14, 0.13, -0.06]) and a first mapping feature B (e.g., [0.14, 0.13, -0.06, 0.15, 0.03, 0.06]), the similarity between the first mapping feature A and the first mapping feature B (e.g., 0.85) is determined as the similarity between any two first mapping features.

[0115] For example, if sub-contrast loss A is 0.4 and sub-contrast loss B is 0.8, when sub-contrast loss A and sub-contrast loss B have the same weight, that is, when the weights of sub-contrast loss A and sub-contrast loss B are both 0.5, the resulting contrast loss is 0.6. When sub-contrast loss A and sub-contrast loss B have different weights, where the weight of sub-contrast loss A is 0.6 and the weight of sub-contrast loss B is 0.4, the resulting contrast loss is 0.56.

[0116] This application's embodiments enhance the regularization and generalization capabilities of deep learning models during feature learning. The contrastive learning framework strengthens feature discriminativeness. The construction of sub-contrastive loss ensures that the model not only learns to distinguish different categories during training but also learns to maintain close relationships between similar samples and distant relationships between different samples within the feature space. It also allows for flexible adjustments based on different contrastive relationships to adapt to the varying contributions of different features to data prediction, thereby improving the model's ability to capture key information. Furthermore, learning through the interrelationships between features, independent of the distribution of class labels, effectively alleviates the data imbalance problem.

[0117] See also Figure 3A In step 102, the input sample is encoded by the first encoder to obtain the first encoded feature, and the input sample is encoded by the second encoder to obtain the second encoded feature.

[0118] Here, the model structure and model parameters of the second encoder are different from those of the first encoder.

[0119] In some embodiments, the first encoder includes a first normalization layer, an attention layer, a second normalization layer, and a mapping layer. The normalization layer adjusts the data (such as input samples or attention features obtained through attention processing by the attention layer) to a uniform scale to avoid bias caused by different features having different units of measurement and numerical ranges, thereby accelerating the training process and improving model stability. This application embodiment does not limit the normalization method; normalization can be batch normalization, layer normalization, etc. The attention layer includes an attention model that performs attention processing on the data. This application embodiment does not limit the attention model; the attention model can be a multi-head self-attention network or a single-head self-attention network, etc. Attention processing is used to learn important information in the input samples from different perspectives simultaneously through the attention model, allowing any two data points in the input samples to influence each other. The mapping layer maps the fused features to a new feature space to obtain mapped features that characterize the latent structure of the input data (such as normalized features). This application embodiment does not limit the mapping method; the mapping method can be linear mapping or nonlinear mapping.

[0120] In some embodiments, see Figure 3D , Figure 3D This is a schematic diagram of the fourth process of the model training method provided in the embodiments of this application, targeting... Figure 3A The step 102 shown illustrates encoding the input sample using the first encoder to obtain the first encoded feature, which can be achieved through... Figure 3D Steps 1021A to 1024A are implemented, and will be explained in detail below.

[0121] In step 1021A, the input samples are normalized through the first normalization layer to obtain the first normalized features.

[0122] Here, normalization is used to adjust the data to a uniform scale to avoid deviations caused by different features having different units of measurement and numerical ranges. This application does not limit the normalization method; normalization can be batch normalization, layer normalization, etc.

[0123] In some embodiments, the first normalization layer includes a norm parameter, and the above-mentioned "normalizing the input sample to obtain the first normalized feature" can be achieved in the following way: based on the norm parameter, exponentiation is performed on each element of the input sample to obtain the exponent element; each exponent element is summed to obtain the summed element; the reciprocal of the norm parameter is used to exponentiation on the summed element to obtain the target element; the ratio of each element of the input sample to the target element is determined as the first normalized feature, wherein the norm parameter (i.e., the exponent) is used to characterize the number of times the elements are multiplied by themselves, and the norm parameter is a real number greater than 0.

[0124] For example, given an input sample (e.g., [2, 4]) and a norm parameter (e.g., 2), the square of each element in the input sample is calculated to obtain the power element (e.g., 2 squared is 4, 2 squared is 4). The sum of each power element is calculated to obtain the sum element (e.g., 20). The reciprocal of the norm parameter (e.g., 0.5, i.e., the square root) is used to exponentiate the sum element to obtain the target element (e.g., 4.5). The ratio of each element in the input sample to the target element (e.g., the ratio of 2 to 4.5 is 0.44, the ratio of 2 to 4.5 is 0.88) is determined as the first normalized feature (e.g., [0.44, 0.88]).

[0125] In step 1022A, the first normalized features are processed by the attention layer to obtain attention-encoded features.

[0126] In some embodiments, step 1022A can be implemented as follows: performing multi-head attention processing on the obtained first normalized features to obtain multi-head attention encoding features; fusing the first normalized features and the multi-head attention encoding features to obtain fused features; mapping the fused features to obtain mapped features; and fusing the mapped features and the fused features to obtain attention encoding features.

[0127] Here, multi-head attention processing is used to learn important information from the input data from different perspectives simultaneously through the model.

[0128] Following the above embodiments, the following explanation uses a multi-head self-attention model to perform attention processing on the first normalized feature as an example. The step of performing multi-head attention processing on the obtained first normalized feature to obtain multi-head attention encoded features can be implemented in the following ways: Encode the first normalized feature to obtain query features, key features, and value features; determine a similarity matrix based on the query features and key features; and determine the multi-head attention encoded feature by multiplying the similarity matrix with the value features. The step of fusing the first normalized feature and the multi-head attention encoded feature to obtain fused features can be implemented in the following ways: concatenate the first normalized feature and the multi-head attention encoded feature to obtain fused features, or perform a weighted summation of the first normalized feature and the multi-head attention encoded feature to obtain fused features.

[0129] Following the above embodiments, the step of fusing the mapping features and the fused features to obtain the attention encoding features is similar to the step of fusing the first normalized features and the multi-head attention encoding features to obtain the fused features, and will not be described in detail here.

[0130] For example, the first normalized feature (e.g., [0.2, 0.3, -0.1]) is weighted and interacted through an attention mechanism. The attention weight of the first normalized feature, i.e., the similarity matrix of the first normalized feature, is [0.2, 0.3, 0.5]. The first normalized feature and its similarity matrix are weighted and summed to obtain the multi-head attention encoding feature (e.g., [0.14, 0.13, -0.06]).

[0131] Continuing with the example above, the first normalized feature (e.g., [0.14, 0.13, -0.06]) and the multi-head attention encoding feature (e.g., [0.15, 0.03, 0.06]) are concatenated to obtain the fused feature (e.g., [0.14, 0.13, -0.06, 0.15, 0.03, 0.06]); or, the first normalized feature and the additional feature are weighted and summed to obtain the fused feature (taking the weights of the first normalized feature and the multi-head attention encoding feature as both being 0.5 as an example, the fused feature is obtained as [0.28, 0.16, 0]).

[0132] In step 1023A, the attention encoding features are normalized through the second normalization layer to obtain the second normalized features.

[0133] Here, step 1023A is similar to step 1021A, and will not be described in detail here.

[0134] In step 1024A, the second normalized feature is mapped through the mapping layer, and the mapped second normalized feature is determined as the first encoded feature.

[0135] Here, the step 1024A, "mapping the second normalized feature", is similar to the step described above, "mapping each third encoded feature to obtain the first mapped feature", and will not be repeated here.

[0136] For example, the second normalized feature (e.g., [0.14, 0.13, -0.06]) is mapped to obtain the first encoded feature (e.g., [0.52, 0.13, -0.06, 0.15]).

[0137] The embodiments of this application ensure data scale consistency and enable the model to focus on key information in the input features, thereby enhancing the representational power of the features. This allows the model to more accurately capture important patterns and trends in the data, thereby improving the generalization ability of the features and the accuracy of prediction using the model and retrograde data.

[0138] In some embodiments, the input samples include multiple domain samples, with different domain samples belonging to different domains, and the second encoder includes a feature extraction layer corresponding to each domain.

[0139] It should be noted that there is a one-to-one correspondence between the domain and the feature extraction layer. The feature extraction layer is used to extract features from the corresponding domain that are helpful for model learning and prediction.

[0140] In some embodiments, see Figure 3E , Figure 3E This is a schematic diagram of the fifth step of the model training method provided in the embodiments of this application, which is aimed at... Figure 3A The step 102 shown illustrates encoding the input sample based on the second encoder to obtain the second encoded features, which can be obtained through... Figure 3E Steps 1021B to 1023B are implemented, and will be explained in detail below.

[0141] In step 1021B, based on the feature extraction layer corresponding to the domain to which each domain sample belongs, feature extraction is performed on each domain sample to obtain domain features.

[0142] Here, this application embodiment does not limit the feature extraction layer corresponding to each domain. The feature extraction layer can be a convolutional layer, recurrent layer, fully connected layer, etc. of a neural network, used to optimize the data for the corresponding domain.

[0143] In some embodiments, step 1021B can be implemented by performing convolution processing on each domain sample to obtain convolutional features, performing pooling processing on the convolutional features to obtain pooled features, and performing mapping processing on the pooled features to obtain domain features.

[0144] For example, convolution is performed on each domain sample (e.g., "mobile phone") to obtain convolutional features (e.g., [0.14, 0.13, -0.06, 0.15, 0.03, 0.06]), pooling is performed on the convolutional features to obtain pooled features (e.g., [0.14, 0.13, -0.06]), and mapping is performed on the pooled features to obtain domain features (e.g., [0.52, 0.13, -0.06, 0.15]).

[0145] In step 1022B, any two neighborhood features are cross-processed to obtain cross features.

[0146] Here, cross-features combine any two domain features to construct a new feature. The new feature helps the model capture the interaction and correlation between different features, thereby improving the accuracy of prediction processing. This application does not limit the cross-processing method. Cross-processing can be feature hash cross, numerical cross, etc. Feature hash cross maps multiple domain features to a single value by hashing the domain features to reduce dimensionality. Numerical cross adds, multiplies, or concatenates two domain features.

[0147] In some embodiments, step 1022B can be implemented as follows: constructing a domain pair based on the domains to which any two domain features belong, and determining the first weight of the domain pair; performing a dot product on any two domain features to obtain a dot product feature; and determining the product of the first weight and the dot product feature as the cross feature.

[0148] Here, a domain pair contains two domains, which can be the same domain or different domains, and each domain pair corresponds one-to-one with the first weight.

[0149] It should be noted that the dot product is used to calculate the sum of the corresponding elements of two vectors.

[0150] In some embodiments, the above-mentioned "determining the first weight of a domain pair" can be achieved by querying a first mapping table based on the domain pair and determining the weight corresponding to the domain pair as the first weight, wherein the mapping table includes the correspondence between different candidate domain pairs and different weights.

[0151] Here, this application embodiment does not limit the correspondence between different candidate domain pairs and different weights. The correspondence can be multiple candidate domain pairs corresponding to one weight, or each candidate domain pair corresponding to its own weight.

[0152] It should be noted that the first mapping table is a data structure used to store the correspondence between candidate neighborhood pairs and weights. This application embodiment does not limit the mapping table. The first mapping table can be a data structure implemented as a hash table, a data structure implemented as an array, etc., used to quickly perform lookup, insertion, and deletion operations on the correspondence between candidate neighborhood pairs and weights. The hash table uses a hash function to calculate the index value and maps the input neighborhood pair to the position in the hash table to access the weight.

[0153] For example, determine the weight of domain pair A (e.g., [phone, computer]) (e.g., 0.2), determine the weight of domain pair B (e.g., [phone, phone]) (e.g., 0.6), and determine the weight (e.g., 0.2) corresponding to the domain pair (e.g., [phone, computer]) as the first weight. Given features of domain A (e.g., [2, 3, 5]) and features of domain B (e.g., [1, 4, 6]), perform dot product processing on the features of domain A and domain B to obtain the dot product feature (e.g., 44); determine the cross feature (e.g., 8.8) by multiplying the first weight (e.g., 0.2) and the dot product feature.

[0154] This application's embodiments promote inter-domain synergy, enhance the model's understanding and utilization of multi-source data, and by performing dot product operations on domain features, the model can capture the interactions and correlations between features from different domains. The first weight of the domain pairs allows the model to adjust the relative importance of different domain features, improving the model's sensitivity to key features and helping to suppress the influence of noise and irrelevant features, thereby improving the model's generalization ability and prediction accuracy, especially since the data comes from different sources and types. Through the generation of cross features, the model can more comprehensively capture the inherent structure and patterns of the data, achieving a deeper understanding of complex phenomena.

[0155] In step 1023B, the cross features are weighted and summed to obtain the second coding feature.

[0156] For example, if cross feature A is 0.4 and cross feature B is 0.8, and cross feature A and cross feature B have the same weight (i.e., both cross feature A and cross feature B have a weight of 0.5), the second coding feature is 0.6. If cross feature A and cross feature B have different weights (i.e., cross feature A has a weight of 0.6 and cross feature B has a weight of 0.4), the second coding feature is 0.56.

[0157] See also Figure 3A In step 103, supervised learning processing is performed on the first coding feature and the second coding feature to obtain the supervised loss.

[0158] Here, supervised learning is a machine learning paradigm that uses labeled input samples to learn how to predict the labels of new input data. By minimizing the difference between the predicted labels obtained during supervised learning and the true labels of the input samples, the parameters of the model are updated. The supervised loss is a metric used in supervised learning to measure the difference between the predicted labels obtained during supervised learning and the true labels of the input samples. This application does not limit the supervised loss, which can be mean squared error loss, cross-entropy loss, etc.

[0159] In some embodiments, see Figure 3F , Figure 3F This is a schematic diagram of the sixth process of the model training method provided in the embodiments of this application, targeting... Figure 3A Step 103 shown can be achieved through... Figure 3F Steps 1031 to 1032 are implemented, and will be explained in detail below.

[0160] In step 1031, the first coding feature and the second coding feature are subjected to prediction processing to obtain the prediction sample.

[0161] Here, predictive processing is the process of making predictions using historical data. In the recommendation field, the first and second encoded features can be encoded features obtained by encoding user attribute information or product attribute information. The prediction sample can be the product click-through rate or the product conversion rate. The product click-through rate is the probability that a user clicks to enter the product page after receiving the product page, and the product conversion rate is the probability that a user buys or uses the product after clicking to enter the product page.

[0162] In some embodiments, step 1031 can be implemented by fusing the first coding feature and the second coding feature to obtain a fused feature; performing mapping processing on the fused feature to obtain a second mapped feature; and decoding the second mapped feature to obtain a predicted sample.

[0163] It should be noted that the step of "fusing the first encoding feature and the second encoding feature to obtain the fused feature" is similar to the step of "fusing the first normalized feature and the multi-head attention encoding feature to obtain the fused feature", and will not be elaborated here. Similarly, the step of "mapping the fused feature to obtain the second mapped feature" is similar to the step of "mapping each third encoding feature to obtain the first mapped feature", and will not be elaborated here.

[0164] In some embodiments, the second mapping feature is decoded to obtain the prediction sample corresponding to the second mapping feature.

[0165] In some embodiments, the above-mentioned "decoding the second mapping feature to obtain the prediction sample corresponding to the second mapping feature" can be implemented in the following way: mapping the second mapping feature to obtain the mapping feature, upsampling the mapping feature to obtain the upsampled feature, and deconvolutioning the upsampled feature to obtain the prediction sample.

[0166] For example, the second mapping feature (e.g., [0.52, 0.13, -0.06, 0.15]) is mapped to obtain the mapping feature (e.g., [0.14, 0.13, -0.06]), the mapping feature is upsampled to obtain the upsampled feature (e.g., [0.14, 0.13, -0.06, 0.15, 0.03, 0.06]), and the upsampled feature is deconvolved to obtain the predicted sample (e.g., 0.85).

[0167] Through the embodiments of this application, the mapping process further optimizes the expressive power of features. By introducing new levels of abstraction through nonlinear transformation, the model can more accurately capture the complex structure of the data, improve the discriminative power of features, and enhance the model's ability to extract key information.

[0168] In step 1032, a supervised loss is constructed based on the predicted samples and the sample labels of the input samples.

[0169] In some embodiments, the above-mentioned "constructing a loss function based on the sample labels of the predicted sample and the input sample" can be achieved by: determining the similarity between the sample labels of the predicted sample and the input sample; and determining the difference between the preset similarity (e.g., 1) and the similarity as the loss function.

[0170] Following the above embodiments, the above "determining the similarity between the sample labels of the predicted sample and the input sample" can be achieved in the following way: determining the edit distance (e.g., 1) between the predicted sample (e.g., "coffee") and the sample label (e.g., "coffee cup") of the input sample; determining the maximum length (e.g., 3) between the length of the predicted sample (e.g., 2) and the length of the sample label (e.g., 3) of the input sample; and determining the ratio (e.g., 0.66) of the difference between the maximum length and the edit distance (e.g., 2) to the maximum length as the similarity between the sample labels of the predicted sample and the input sample.

[0171] It should be noted that edit distance is used to characterize the number of single-character edits (insertion, deletion, or replacement) required to transform a predicted sample into a sample label of the input sample. The number of single-character edits is the minimum number of edits required to transform a predicted sample into a sample label of the input sample. Taking the calculation of the edit distance between a predicted sample (e.g., "kitten") and the sample label of the input sample (e.g., "sitting") as an example, a total of 2 replacements and 1 insertion are performed in this process. The first replacement is to replace "k" in "kitten" with "s" to get "sitten"; the second replacement is to replace "e" in "sitten" with "i" to get "sittin"; the insertion is to insert a "g" at the end of "sittin" to get "sitting". Therefore, the edit distance between the sample labels of the predicted sample and the input sample is 3.

[0172] Following the above embodiments, the above-mentioned "determining the similarity between the sample labels of the predicted sample and the input sample" can also be achieved in the following way: encoding the predicted sample (e.g., "coffee") to obtain a first sample encoding feature (e.g., [0.14, 0.13, -0.06]); encoding the sample label of the input sample (e.g., "coffee cup") to obtain a second sample encoding feature (e.g., [0.14, 0.13, -0.06, 0.15, 0.03, 0.06]); and determining the similarity between the first sample encoding feature and the second sample encoding feature (e.g., 0.85) as the similarity between the sample labels of the predicted sample and the input sample.

[0173] It should be noted that the embodiments of this application do not limit the similarity. The similarity can be the cosine similarity, Euclidean distance, etc. between the coding features of the first sample and the coding features of the second sample.

[0174] For example, the sample data includes sample data 1 and sample data 2. Then, the encoding vector corresponding to the sample label of the input sample of sample data 1 is [0, 1], and the encoding vector corresponding to the sample label of the input sample of sample data 2 is [1, 0]. The first element in the encoding vector is used to indicate whether the sample data is sample data 1. When the first element is 0, it indicates that the sample data is not sample data 1. When the first element is 1, it indicates that the sample data is sample data 1. The second element in the encoding vector is used to indicate whether the sample data is sample data 2. When the second element is 0, it indicates that the sample data is not sample data 2. When the second element is 1, it indicates that the sample data is sample data 2.

[0175] See also Figure 3A In step 104, the model to be trained is trained based on contrastive loss and supervised loss to obtain the target model.

[0176] Here, the target model is the model obtained after the training process, which is able to make accurate predictions on new and unseen input data.

[0177] In some embodiments, see Figure 3G , Figure 3G This is a schematic diagram of the seventh process of the model training method provided in the embodiments of this application, which is aimed at... Figure 3A Step 104 shown can be achieved through... Figure 3G Steps 1041 to 1043 are implemented, and will be explained in detail below.

[0178] In step 1041, a second weight for the contrastive loss is determined based on multiple third coding features, and a third weight for the supervised loss is determined based on the second weight for the contrastive loss.

[0179] Among them, multiple third-encoded features are obtained through the comparative learning process of the input samples by the first encoder.

[0180] Here, multiple third-encoded features are used to construct the contrastive loss.

[0181] In some embodiments, determining the second weight of the contrast loss based on multiple third coding features in step 1041 can be achieved in at least one of the following ways: when the similarity between any two third coding features is greater than a similarity threshold, a preset weight is determined as the second weight of the contrast loss; the similarity between any two third coding features is weighted and summed to obtain the target similarity, and the weight corresponding to the target similarity is determined as the second weight of the contrast loss.

[0182] It should be noted that the embodiments of this application do not limit the similarity. The similarity can be the cosine similarity, Euclidean distance, etc. between the first coding feature and the second coding feature. The second weight of the comparison loss is inversely proportional to the target similarity and the similarity between any two third coding features. When the target similarity and the similarity between any two third coding features are greater, the second weight of the comparison loss is smaller.

[0183] For example, when the similarity between any two third coding features (e.g., 0.85) is greater than the similarity threshold (e.g., 0.8), a preset weight (e.g., 0.2) is determined as the second weight of the contrast loss.

[0184] For example, the similarity between third coding feature A (e.g., [0.14, 0.13, -0.06]) and third coding feature B (e.g., [0.14, 0.03, 0.06]) is 0.4, the similarity between third coding feature A and third coding feature C (e.g., [0.14, 0.15, 0.03]) is 0.8, and the similarity between third coding feature B and third coding feature C is 0.6. This is achieved when each similarity has the same weight, i.e., the similarity between third coding feature A and third coding feature B, and the similarity between third coding feature C and third coding feature C. The similarity between code feature A and third code feature C, and the similarity between third code feature B and third code feature C, are both weighted at 0.33, resulting in a target similarity of 0.6. When each similarity has a different weight, the similarity between third code feature A and third code feature B is weighted at 0.6, the similarity between third code feature A and third code feature C is weighted at 0.2, and the similarity between third code feature B and third code feature C is also weighted at 0.2, resulting in a target similarity of 0.52.

[0185] Following the above embodiments, the determination of the second weight of the contrast loss based on target similarity in step 1041 can be achieved in the following way: querying the second mapping table based on target similarity, and determining the weight corresponding to the target similarity as the second weight of the contrast loss. The second mapping table is a data structure that includes the correspondence between different similarities and different weights. This application embodiment does not limit the second mapping table. The data structure of the second mapping table is similar to that of the first mapping table, and will not be described in detail here.

[0186] In some embodiments, the above-mentioned "determining the third weight of the supervision loss based on the second weight of the contrast loss" can be achieved by determining the difference (e.g., 0.8) between the preset weight (e.g., 1) and the second weight of the contrast loss (e.g., 0.2) as the third weight of the supervision loss.

[0187] In this embodiment, when the similarity between any two third-encoded features exceeds a preset similarity threshold, it indicates that they are very close in the feature space and can be considered positive sample pairs. The preset weight is set as the second weight of the contrastive loss. During the optimization process, the similarity of such positive sample pairs will be given a higher weight, thereby encouraging the model to more closely cluster these similar samples in the feature space. By calculating the similarity of all third-encoded feature pairs that meet the similarity threshold and summing them by weight, the target similarity is obtained, which reflects the average similarity of all positive sample pairs in the feature space. This helps the model balance the similarity of different sample pairs globally, thereby achieving more balanced feature learning. At the same time, by adaptively adjusting the weights, the model not only focuses on those very similar sample pairs but also considers the overall sample distribution, thereby capturing the inherent structure of the data and improving the distinguishing ability of the features, enabling it to make accurate predictions when faced with new and unseen data.

[0188] In step 1042, the contrast loss and the supervision loss are weighted and summed based on the second weight and the third weight to obtain the target loss.

[0189] In some embodiments, the above-mentioned "weighted summation of contrast loss and supervision loss based on second weight and third weight to obtain target loss" can be implemented in the following way: multiplying the second weight with contrast loss to obtain first summation loss; multiplying the third weight with supervision loss to obtain second summation loss; and determining the sum of the first summation loss and the second summation loss as target loss.

[0190] For example, the contrast loss is 0.4, the supervision loss is 0.8, the second weight is 0.6, and the third weight is 0.4. The second weight is multiplied by the contrast loss to obtain the first summation loss (e.g., 0.24). The third weight is multiplied by the supervision loss to obtain the second summation loss (e.g., 0.32). The sum of the first and second summation losses is determined as the target loss (e.g., 0.32).

[0191] In step 1043, the parameters of the model to be trained are updated based on the target loss, and the updated parameters of the model to be trained are used as the parameters of the target model.

[0192] In some embodiments, the above-mentioned "updating the parameters of the model to be trained" can be achieved by: performing backpropagation in the model to be trained based on the loss function to obtain the gradient; and updating the parameters of the model to be trained based on the gradient.

[0193] It should be noted that backpropagation is implemented using the backpropagation algorithm, which calculates the gradient of the parameters of the model to be trained using the chain rule of derivatives.

[0194] In the field of image recognition, a server or terminal can train an image recognition model to identify the category of an image, determine the image sample, and perform contrastive learning processing on the image sample through a first encoder to obtain a contrastive loss. The image sample is then encoded using the first encoder to obtain a first encoded feature, and the image sample is encoded using a second encoder to obtain a second encoded feature. Supervised learning processing is then performed on the first and second encoded features to obtain a supervised loss. Based on the contrastive loss and the supervised loss, the image recognition model to be trained is trained to obtain a trained image recognition model. Then, the first encoder included in the trained image recognition model is used to encode the input image (such as a cat or dog image) to obtain a third encoded feature, and the second encoder included in the trained image recognition model is used to encode the input image to obtain a fourth encoded feature. The third and fourth encoded features are then used for prediction processing to obtain the category of the input image (such as a dog).

[0195] In the field of anomaly detection, a server or terminal can train an anomaly detection model to identify whether information is abnormal, determine information samples, and perform contrastive learning processing on the information samples through a first encoder to obtain a contrastive loss. The information samples are then encoded using the first encoder to obtain a first encoded feature, and a second encoder is used to encode the information samples to obtain a second encoded feature. Supervised learning processing is then performed on the first and second encoded features to obtain a supervised loss. Based on the contrastive loss and the supervised loss, the anomaly detection model to be trained is trained to obtain a trained anomaly detection model. Then, the first encoder included in the trained anomaly detection model encodes the input information (such as data packets in network transmission) to obtain a third encoded feature, and the second encoder included in the trained anomaly detection model encodes the input information to obtain a fourth encoded feature. Predictive processing is then performed on the third and fourth encoded features to obtain an anomaly prediction result (such as an anomaly). When the anomaly prediction result is anomaly, a risk assessment is performed on the input information to evaluate the potential risks and impacts of the input information, and security defenses are implemented for the input information.

[0196] See Figure 4 , Figure 4 This is a first flowchart illustrating the data prediction method provided in this application embodiment, which will be combined with... Figure 4The steps shown will be explained below. The data prediction method provided in this application embodiment can be implemented by the server or terminal alone, or by the server and terminal working together. The following will be an example of the server and terminal working together.

[0197] In step 201, the input data is encoded based on the first encoder included in the target model to obtain the third encoded feature.

[0198] The target model is trained using the model training method provided in the embodiments of this application.

[0199] Here, step 201 is similar to the step 102 of “encoding the input sample by the first encoder to obtain the first encoded feature”, and will not be described in detail here.

[0200] Before step 201, input data is acquired. Input data consists of features derived from raw data through mapping processing. Raw data includes both offline and online data. Offline data is independent of real-time transmission and can be stored and processed in batches after collection, such as historical transaction records and static database backups. Online data refers to data collected in real-time via the internet, relying on continuous data streams and instant data processing capabilities. Examples of online data include real-time transaction data, website access logs, and IoT device sensor data. This type of data is characterized by its dynamic nature, requiring rapid response and real-time analysis to capture its immediate value. Specifically, offline data may include customer feedback collected via email or face-to-face surveys. This data is digitized and stored in a database for subsequent analysis, such as user shopping preferences and satisfaction with different products. Online data involves real-time identification and tracking of user click behavior and purchasing patterns for personalized recommendations to improve user satisfaction and conversion rates. For example, based on the features obtained by mapping the user's browsing history on the website, data prediction is performed to obtain the probability of each product being purchased, thereby recommending relevant products in real time. This application embodiment does not limit the form of the original data, which can be numbers, text, images, sound, video, etc.

[0201] In step 202, the input data is encoded based on the second encoder included in the target model to obtain the fourth encoded feature.

[0202] Here, step 202 is similar to the step 102 of "encoding the input sample based on the second encoder to obtain the second encoded feature", and will not be described in detail here.

[0203] In step 203, the third and fourth coding features are subjected to prediction processing to obtain the prediction result. Here, step 203 is similar to step 1031, and will not be described in detail here.

[0204] The following will describe an exemplary application of the model training method and data prediction method provided in this application embodiment in a real-world application scenario.

[0205] In related technologies, a two-stage training method is used for model training. The training is carried out in a serial manner, which has a long training cycle and is not suitable for advertising recommendation scenarios based on real-time user data feedback. In the first stage, self-supervised contrastive learning is used to generate high-order feature codes through pre-training for the next stage task. In the second stage, the pre-trained high-order feature codes are applied to the supervised learning task.

[0206] To address the aforementioned issues, this application proposes a model training method and a data prediction method. By leveraging contrastive learning, it considers not only enhancing the expressive power of low-order feature representations but also enhancing the expressive power of high-order feature representations, thereby further improving the generalization of the model. Simultaneously, by conducting supervised learning and contrastive learning in parallel training, the training cycle is reduced and training efficiency is improved. Here, high-order feature representations are higher-order feature representations obtained after feature crossing.

[0207] Taking click-through rate in the recommendation field as an example, see Figure 5 , Figure 5 This is a schematic diagram illustrating the principle of joint training of supervised learning and contrastive learning provided in the embodiments of this application. Figure 5 middle, Figure 5 It includes multiple modules for joint modeling of supervised learning (i.e., supervised learning processing) and contrastive learning (i.e., contrastive learning processing) according to the embodiments of this application, such as feature representation module 301, supervised learning module 302 and contrastive learning module 303, etc. The multiple modules for joint training of supervised learning and contrastive learning provided in the embodiments of this application will be explained below.

[0208] The feature representation module 301 transforms the original features (such as user data, product data, i.e., sample data from multiple domains) into a vectorized representation (i.e., mapping process) from the original sparse category values ​​(taking product categories as an example, including category values ​​such as mobile phones, computers, books, and household goods), thus obtaining the feature vector.

[0209] Here, the feature vector can support any dimension (such as 64 / 128, etc.). The dimension of the feature vector can be a preset dimension or a dimension corresponding to the category of the original feature. All feature vectors are concatenated to obtain a large representation vector, which is denoted as the low-order feature representation (i.e., the input sample), and serves as the input to the supervised learning module 302 and the contrastive learning module 303.

[0210] Supervised learning module 302 is primarily based on deep learning neural networks, and incorporates explicit user feedback data (clicks / conversions, etc.) for model training. Figure 5 This application includes multiple sub-modules for supervised learning during model training, such as the feature cross layer 3021, encoder 3022, and task layer 3023. The following explains the multiple sub-modules for supervised learning provided in this application.

[0211] The feature cross layer 3021 (i.e., the second encoder) receives the output of the feature representation module 301 and is used to generate higher-order feature representations (i.e., the first encoded features).

[0212] Here, the feature cross layer 3021 is usually composed of a feature cross algorithm and multiple fully connected layers. The operation result is a high-order feature representation, which contains the abstract expression of the low-order feature representation.

[0213] The encoder 3022 (i.e., the first encoder) receives the output of the feature representation module 301 and uses it to generate an abstract expression from the low-order feature representation to obtain a high-order feature representation (i.e., the second encoded feature).

[0214] Here, the encoder 3022 in supervised learning and the encoders 3032A and 3032B in the contrastive learning module 303 adopt a parameter sharing method and use the same set of model parameters. That is, the encoder 3022 parameters can be fully learned by contrastive learning to assist the learning of the supervised learning module 302.

[0215] Meanwhile, the high-order feature representations generated after the feature cross layer 3021 are combined with the high-order feature representations generated by the encoder 3022 through splicing or addition, which plays a role in assisting supervised learning of high-order feature representations.

[0216] Task layer 3023 is used to provide model predictions.

[0217] Here, task layer 3023 employs various methods such as multi-layer fully connected networks and residual networks. Taking the click-through rate prediction model or conversion rate prediction model as an example, in the last layer of the model, an activation function is used to perform prediction processing on the fused features obtained by fusing the high-order feature representation generated after feature cross layer 3021 with the high-order feature representation generated by encoder 3022, to obtain the final model prediction value (i.e., the predicted sample). The model prediction value ranges from (0, 1), representing the probability of user clicks or the probability of user conversion.

[0218] Supervised learning loss functions are used to measure the degree of difference between the model's predicted values ​​and the true values ​​(i.e., sample labels).

[0219] Here, cross-entropy can be used as the loss function as shown in Equation (1).

[0220] L sup =-[ylog(pred)+(1-y)log(1-pred)] (1)

[0221] Among them, L sup This is the supervised learning loss function (i.e., supervised loss), where y∈{0,1} represents the sample label, i.e., the user's explicit feedback (click / conversion), and pred is the model's predicted value. Taking the conversion rate prediction model as an example, if a user clicks but no conversion occurs, then y is 0; if a user clicks and a conversion occurs, then y is 1. The range of y is (0,1).

[0222] Comparative learning module 303, Figure 5 It includes multiple sub-modules for comparative learning in the embodiments of this application, such as mask layer 3031, encoder 3022A, encoder 3022B, projection layer 3033A, and projection layer 3033B. The multiple sub-modules for comparative learning in the model training process provided in the embodiments of this application will be explained below.

[0223] The masking module 3031 uses a feature vector masking strategy to mask low-order feature representations.

[0224] Here, the feature vector masking strategy specifically includes: masking each neuron in each feature vector according to probability, where the probability of random masking can be a preset probability (e.g., the probability of random masking is 20%).

[0225] For low-order feature representations, a feature vector masking strategy is executed in two paths, with each path using the same masking probability, resulting in two mask vectors. i and mask j (i.e., mask sample).

[0226] Encoder 3032A receives the output from masking module 3031 and masks the result generated by masking module 3031. i The input is fed into encoder 3032A to generate high-order feature representation results h. i (i.e., the third coding feature).

[0227] Encoder 3032B receives the output from masking module 3031 and masks the result generated by masking module 3031. j The input is fed into encoder 3032B to generate high-order feature representation results h. j .

[0228] Here, encoders 3032A and 3032B are suitable for common feature encoder models, such as multilayer fully connected layers, residual networks, and Transformers. The self-attention mechanism in the encoder module of the Transformer model can be used to automatically learn the cross-information between features, or a multilayer fully connected layer, i.e., a multi-layer perceptron (MLP), can be used as the encoder.

[0229] Projection layer 3033A employs multiple fully connected layers and other methods to represent high-order features h. i Projected onto the contrastive learning application space, generating the projection result z. i .

[0230] Projection layer 3033B employs multiple fully connected layers and other methods to represent high-order features h. j Projected onto the contrastive learning application space, generating the projection result z. j .

[0231] Here, projection layers 3033A and 3033B are used to receive the high-order feature representation results output by the encoder, project the high-order feature representations onto the contrastive learning application space, and calculate the contrastive learning loss function. The projection result z... i and projection result z j As input to the contrastive learning loss function.

[0232] The output results of projection layer 3033A and projection layer 3033B are received, and the similarity evaluation function (such as cosine similarity) is used to evaluate the similarity between different feature representations, that is, to calculate the similarity between different projection results.

[0233] Here, the contrastive learning loss function is shown in Equation (2).

[0234]

[0235] Where N is the number of samples in the model training batch, τ is a temperature coefficient used to control the dispersion of sample features, and sim(z) i , z j ) is the projection result z i and projection result z j Cosine similarity between them.

[0236] The supervised learning loss function L sup With contrastive learning loss function L CLThe combination involves using supervised learning to learn feature representations and contrastive learning to assist in learning feature representations. The parameters of the model to be trained when the loss function converges are used as the parameters of the model (i.e., the target model), resulting in the loss function L as shown in formula (3).

[0237] L = L sup +αL CL (3)

[0238] Where L is the loss function and α is a hyperparameter used to control the weight of contrastive learning.

[0239] In the model inference phase, compared to the model training phase, no further action is required. Figure 5 The contrastive learning module 303 in the text, see Figure 6 , Figure 6 This is a schematic diagram of the model inference stage provided in an embodiment of this application. Figure 6 middle, Figure 6 It includes multiple modules for model inference in the embodiments of this application, such as feature representation module 401, supervised learning module 402, etc. The following explains the multiple modules for joint training of supervised learning and contrastive learning provided in the embodiments of this application.

[0240] The feature representation module 401 processes features in the same way as the feature representation module 301 during the training phase, and will not be described again here.

[0241] Supervised learning module 402, Figure 6 This application includes multiple sub-modules for supervised learning during model inference, such as the feature cross layer 4021, encoder 4022, and task layer 4023. The following explains the multiple sub-modules for supervised learning during model inference provided in this application.

[0242] The processing methods of the feature cross layer 4021, encoder 4022 and task layer 4023 are consistent with those of the feature cross layer 3021, encoder 3022 and task layer 3023 in the supervised learning module 301 during the training phase. Model inference is performed through the feature cross layer 4021, encoder 4022 and task layer 4023 to obtain the prediction result 4024, which will not be elaborated here.

[0243] The recommendation model trained by the model training method provided in this application is mainly applied to advertising recommendation scenarios. Online advertising is currently the main interaction method on the Internet. When users browse interactive web pages, the model predicts the probability of explicit feedback (clicks / conversions, etc.) after the advertisement is recommended to the user by combining the user's interests and preferences with the attribute information of the advertisement itself. The model inserts the advertisement that best matches the user's preferences into the page. After the user sees the advertisement that matches the user's preferences, they may exhibit explicit feedback (clicks / conversions, etc.). The model captures the user's explicit feedback information in real time and constructs label signals such as whether the user clicked or converted based on the obtained user feedback information. These label signals are provided to the recommendation model in real time. The recommendation model adjusts the model parameters based on the user's click / conversion label signals and updates the user's interests and preferences in real time. See [link to relevant documentation]. Figure 7 , Figure 7 This is an illustration of an advertising recommendation provided in an embodiment of this application, such as... Figure 7 As shown, based on the deep learning-based ad recommendation click-through rate / conversion rate prediction model provided in this application embodiment, the estimated click-through rate of each ad is obtained, and the ad 501 with the highest estimated click-through rate and the ad name 502 corresponding to ad 501 are sent to the display page 503.

[0244] In summary, in the recommendation system, by using the model training method provided in this application embodiment to verify the effect in the conversion rate prediction model, the cold start problem of new advertisements can be greatly alleviated, the prediction accuracy can be improved by 5.8%, and the user experience can be improved. At the same time, compared with the traditional supervised learning recommendation model trained based on user explicit feedback data (clicks / conversions, etc.), this application embodiment adopts a deep learning neural network structure and combines it with the cross-entropy loss function for training. It relies on a large amount of user explicit feedback data, uses the tag information constructed by user explicit feedback data to capture user interests and preferences, and can also use the contrastive learning method to assist in the learning of user interest representation based on the similarity and difference between samples, thereby improving the generalization of the model.

[0245] The following description continues to illustrate the exemplary structure of the model training device 555 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2A As shown, the software modules stored in the model training device 555 in the memory 550 may include:

[0246] The contrastive learning module 5551 is used to determine the input sample and perform contrastive learning processing on the input sample through the first encoder to obtain the contrastive loss.

[0247] The supervised learning module 5552 is used to encode the input sample through the first encoder to obtain the first encoded feature, and to encode the input sample based on the second encoder to obtain the second encoded feature; and to perform supervised learning processing on the first encoded feature and the second encoded feature to obtain the supervised loss.

[0248] The model training module 5553 is used to train the model to be trained based on contrastive loss and supervised loss to obtain the target model.

[0249] In some embodiments, the contrastive learning module 5551 is further configured to perform masking processing on the input samples to obtain multiple masked samples; encode each masked sample by a first encoder to obtain a third encoded feature; and construct a loss from the multiple third encoded features to obtain a contrastive loss.

[0250] In some embodiments, the contrast learning module 5551 is further configured to acquire multiple mask templates; and perform the following processing for each mask template: based on the mask template, perform mask processing on the input sample to obtain a mask sample.

[0251] In some embodiments, the contrastive learning module 5551 is further configured to perform mapping processing on each third coding feature to obtain a first mapping feature; construct a sub-contrast loss based on any two first mapping features; and perform weighted summation on the sub-contrast loss to obtain the contrastive loss.

[0252] In some embodiments, the supervised learning module 5552 is further configured to normalize the input samples through a first normalization layer to obtain a first normalized feature; to perform attention processing on the first normalized feature through an attention layer to obtain an attention-encoded feature; to normalize the attention-encoded feature through a second normalization layer to obtain a second normalized feature; and to map the second normalized feature through a mapping layer to determine the mapped second normalized feature as the first encoded feature, wherein the first encoder includes a first normalization layer, an attention layer, a second normalization layer, and a mapping layer.

[0253] In some embodiments, the supervised learning module 5552 is further configured to extract features from each domain sample based on the feature extraction layer corresponding to the domain to which each domain sample belongs, to obtain domain features; perform cross processing on any two domain features to obtain cross features; and perform weighted summation on the cross features to obtain second encoded features, wherein the input sample includes multiple domain samples, different domain samples belong to different domains, and the second encoder includes a feature extraction layer corresponding to each domain.

[0254] In some embodiments, the supervised learning module 5552 is further configured to construct a domain pair based on the domains to which any two domain features belong, and determine a first weight for the domain pair; perform a dot product on any two domain features to obtain a dot product feature; and determine the product of the first weight and the dot product feature as the cross feature.

[0255] In some embodiments, the supervised learning module 5552 is further configured to perform prediction processing on the first encoded features and the second encoded features to obtain predicted samples; and to construct a supervised loss based on the predicted samples and the sample labels of the input samples.

[0256] In some embodiments, the supervised learning module 5552 is further configured to fuse the first encoded feature and the second encoded feature to obtain a fused feature; perform mapping processing on the fused feature to obtain a second mapped feature; and decode the second mapped feature to obtain a predicted sample.

[0257] In some embodiments, the model training module 5553 is further configured to determine a second weight of the contrastive loss based on multiple third encoding features, and to determine a third weight of the supervised loss based on the second weight of the contrastive loss, wherein the multiple third encoding features are obtained during the contrastive learning process of the input samples by the first encoder; to obtain a target loss by weighted summation of the contrastive loss and the supervised loss based on the second weight and the third weight; and to update the parameters of the model to be trained based on the target loss, and to use the updated parameters of the model to be trained as the parameters of the target model.

[0258] In some embodiments, the model training module 5553 is further configured to determine the second weights of the contrastive loss by at least one of the following methods:

[0259] When the similarity between any two third coding features is greater than the similarity threshold, the preset weight is determined as the second weight of the contrast loss; the similarity between any two third coding features is weighted and summed to obtain the target similarity, and the weight corresponding to the target similarity is determined as the second weight of the contrast loss.

[0260] In some embodiments, the data acquisition module 5554 is used to acquire sample data from multiple fields, and to perform mapping processing on the sample data from each field to obtain a field sample for each field; and to concatenate the field samples corresponding to multiple fields to obtain an input sample.

[0261] The following description continues to illustrate the exemplary structure of the image classification device 556 provided in the embodiments of this application as a software module. In some embodiments, such as... Figure 2B As shown, the software module stored in the image classification device 556 in the memory 550 may include:

[0262] The data prediction module 5561 is used to encode the input data based on the first encoder included in the target model to obtain a third encoded feature; to encode the input data based on the second encoder included in the target model to obtain a fourth encoded feature; and to perform prediction processing on the third encoded feature and the fourth encoded feature to obtain a prediction result, wherein the target model is trained by the model training method provided in the embodiments of this application.

[0263] This application provides a computer program product, which includes computer-executable instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the model training method or data prediction method described in this application embodiment.

[0264] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the model training method or data prediction method provided in this application. For example, ... Figures 3A to 3G The model training method shown or Figure 4 The data prediction method is shown.

[0265] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0266] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0267] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0268] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0269] In summary, the input samples are determined, and a contrastive learning process is performed on them using a first encoder to obtain a contrastive loss. The first encoder encodes the input samples to obtain first encoded features, and a second encoder encodes them to obtain second encoded features. Supervised learning is then performed on both the first and second encoded features to obtain a supervised loss. Based on the contrastive and supervised losses, the model to be trained is trained to obtain the target model. Specifically, the contrastive loss is used to learn feature representations, making similar samples closer together in the feature space and dissimilar samples further apart. The supervised loss measures the model's predicted output against the true label. By combining the contrastive loss obtained from the contrastive learning process and the supervised loss obtained from the supervised learning process, the model to be trained learns more robust feature representations, while the supervised loss ensures that the model can accurately predict the label. The combination of these two methods allows the model to maintain its generalization ability while improving the accuracy of data prediction.

[0270] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A model training method, characterized in that, The method includes: The input sample is determined, and the input sample is subjected to contrastive learning processing through the first encoder to obtain the contrastive loss; The input sample is encoded by the first encoder to obtain a first encoded feature, and the input sample is encoded by the second encoder to obtain a second encoded feature; Supervised learning processing is performed on the first encoded feature and the second encoded feature to obtain a supervised loss; Based on the contrastive loss and the supervised loss, the model to be trained is trained to obtain the target model.

2. The method according to claim 1, characterized in that, The step of performing contrastive learning processing on the input samples through the first encoder to obtain the contrastive loss includes: The input sample is masked to obtain multiple mask samples; Each mask sample is encoded using a first encoder to obtain a third encoded feature; The contrastive loss is obtained by constructing a loss from multiple of the third encoded features.

3. The method according to claim 2, characterized in that, The input sample is masked to obtain multiple mask samples, including: Obtain multiple mask templates; For each of the aforementioned mask templates, the following processing is performed: Based on the mask template, the input sample is masked to obtain a mask sample.

4. The method according to claim 2, characterized in that, The step of constructing the contrastive loss by applying a loss to multiple of the third encoded features includes: Each of the third encoded features is mapped to obtain a first mapped feature; Construct a sub-contrast loss based on any two of the first mapping features; The contrast loss is obtained by weighted summation of the sub-contrast losses.

5. The method according to any one of claims 1-4, characterized in that, The first encoder includes a first normalization layer, an attention layer, a second normalization layer, and a mapping layer. Encoding the input sample using the first encoder to obtain the first encoded feature includes: The input sample is normalized through the first normalization layer to obtain the first normalized feature; The attention layer performs attention processing on the first normalized feature to obtain the attention-encoded feature; The attention encoding features are normalized through the second normalization layer to obtain the second normalized features; The mapping layer maps the second normalized feature, and the mapped second normalized feature is determined as the first encoded feature.

6. The method according to any one of claims 1-4, characterized in that, The input samples include multiple domain samples, with different domain samples belonging to different domains. The second encoder includes a feature extraction layer corresponding to each domain. Encoding the input samples based on the second encoder to obtain second encoded features includes: Based on the feature extraction layer corresponding to the domain to which each domain sample belongs, feature extraction is performed on each domain sample to obtain domain features; By performing cross processing on any two of the aforementioned domain features, cross features are obtained; The weighted summation of the cross features yields the second encoded feature.

7. The method according to claim 6, characterized in that, The step of performing cross-processing on any two of the domain features to obtain cross features includes: Based on the domains to which any two of the domain features belong, construct a domain pair and determine the first weight of the domain pair; Perform a dot product operation on any two of the aforementioned neighborhood features to obtain the dot product features; The product of the first weight and the dot product feature is determined as the cross feature.

8. The method according to any one of claims 1-4, characterized in that, The supervised learning process performed on the first encoded feature and the second encoded feature to obtain the supervised loss includes: The first encoded feature and the second encoded feature are subjected to prediction processing to obtain a prediction sample; The supervised loss is constructed based on the predicted samples and the sample labels of the input samples.

9. The method according to claim 8, characterized in that, The prediction processing of the first encoded feature and the second encoded feature to obtain the prediction sample includes: The first coding feature and the second coding feature are fused to obtain the fused feature; The fused features are mapped to obtain the second mapped features; The second mapping feature is decoded to obtain the predicted sample.

10. The method according to any one of claims 1-4, characterized in that, The process of training the model to be trained based on the contrastive loss and the supervised loss to obtain the target model includes: Based on multiple third coding features, a second weight of the contrastive loss is determined, and based on the second weight of the contrastive loss, a third weight of the supervised loss is determined, wherein the multiple third coding features are obtained during the contrastive learning process of the input samples by the first encoder; Based on the second weight and the third weight, the comparison loss and the supervision loss are weighted and summed to obtain the target loss; Based on the target loss, the parameters of the model to be trained are updated, and the updated parameters of the model to be trained are used as the parameters of the target model.

11. The method according to claim 10, characterized in that, The step of determining the second weight of the contrast loss based on multiple of the third encoded features includes: The second weight of the contrast loss is determined by at least one of the following methods: When the similarity between any two of the third coding features is greater than the similarity threshold, the preset weight is determined as the second weight of the comparison loss; The similarity between any two of the third coding features is weighted and summed to obtain the target similarity. The weight corresponding to the target similarity is determined as the second weight of the contrast loss.

12. The method according to claim 1, characterized in that, The determination of the input sample includes: Acquire sample data from multiple domains, and perform mapping processing on the sample data of each domain to obtain domain samples for each domain; The input sample is obtained by concatenating the domain samples corresponding to the multiple domains.

13. A data prediction method, characterized in that, The method includes: The input data is encoded based on the first encoder included in the target model to obtain the third encoded feature; The input data is encoded based on the second encoder included in the target model to obtain the fourth encoded feature; The third and fourth coding features are subjected to prediction processing to obtain a prediction result, wherein the target model is trained by the model training method according to any one of claims 1 to 12.

14. A model training device, characterized in that, The device includes: The contrastive learning module is used to determine the input sample and perform contrastive learning processing on the input sample through the first encoder to obtain the contrastive loss; The supervised learning module is used to encode the input sample using the first encoder to obtain a first encoded feature, and to encode the input sample using the second encoder to obtain a second encoded feature; and to perform supervised learning processing on the first encoded feature and the second encoded feature to obtain a supervised loss. The model training module is used to train the model to be trained based on the contrastive loss and the supervised loss to obtain the target model.

15. A data prediction device, characterized in that, The device includes: A data prediction module is used to encode input data based on a first encoder included in the target model to obtain a third encoded feature; to encode the input data based on a second encoder included in the target model to obtain a fourth encoded feature; and to perform prediction processing on the third encoded feature and the fourth encoded feature to obtain a prediction result, wherein the target model is trained by the model training method according to any one of claims 1 to 12.

16. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions for a computer; The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the model training method according to any one of claims 1 to 12, or implements the data prediction method according to claim 13.

17. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the model training method according to any one of claims 1 to 12, or the data prediction method according to claim 13.

18. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the model training method according to any one of claims 1 to 12, or the data prediction method according to claim 13.

Images