Data processing method and related apparatus

Abstract

A data processing method and a related apparatus, which can be applied to recommendation scenarios. The method comprises: acquiring a plurality of first items in a first scenario which are tapped by a user within a first time period and a plurality of second items in the first scenario which are not tapped by the user within the first time period; generating first viewing angle information of the user in the first scenario on the basis of the plurality of first items, the first viewing angle information comprising information of the user and information of the first scenario; extracting first media information of each first item and second media information of each second item; respectively fusing the first media information and the second media information with the first viewing angle information to obtain a first fused representation of each first item and a second fused representation of each second item; taking the first fused representations as positive samples and the second fused representations as negative samples for contrastive learning to obtain a pre-trained database; and extracting a media content representation part from the pre-trained database, the media content representation part comprising trained media information, and being used for executing a recommendation task.

Description

A data processing method and related apparatus

[0001] The present application claims priority from the Chinese patent application No. 202410637403.0 filed on May 21, 2024, and entitled "A data processing method and related apparatus", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence, and in particular to a data processing method and related apparatus. BACKGROUND

[0003] In recent years, intelligent recommendation technology has been widely applied in Internet products. For example, news information software recommends news to users, shopping software recommends goods to users, and music software recommends songs to users. The recommendation algorithm can be abstractly modeled as a click rate prediction algorithm, that is, predicting the probability of a user clicking a certain item in a specific environment. The specific process is to extract the features of the items clicked by the user within a period of time, and train a click rate prediction model based on the features of the clicked items. After the model is trained, it can be used to predict the probability of an item being clicked, and recommend items with high click rate to users.

[0004] When recommending to users, it is usually to recommend content in the same domain (scene), for example, recommending short video to users in the short video domain, and recommending news to users in the text information domain. Therefore, when training, the features of the clicked items in a domain are usually extracted to train a model for recommendation in the domain. However, when extracting the features of items in each domain, the domain information, user information and content information of the item are encoded together, which leads to poor cross-domain transferability of content representation of the item, and cannot be used for recommendation tasks in other scenes. SUMMARY

[0005] The present application provides a data processing method and related apparatus, which separates user information and scene information, so that the media content representation part focuses on the learning of content representation, thereby improving the cross-domain transfer accuracy of content representation.

[0006] The first aspect of the application provides a data processing method, which can be applied to a data processing device. The method comprises the following steps: obtaining a plurality of first items clicked by a user in a first time period; generating first perspective information of the user in a first scene based on the plurality of first items, the first scene being a scene to which the plurality of first items belong, and the first perspective information comprising information of the user and information of the first scene; obtaining a plurality of second items not clicked by the user in the first time period in the first scene; extracting first media information of each first item and second media information of each second item; fusing the first perspective information and the first media information to obtain first fusion representation of each first item; fusing the first perspective information and the second media information to obtain second fusion representation of each second item; performing comparative learning training on the plurality of first fusion representations as positive samples and the plurality of second fusion representations as negative samples to obtain a pre-training library; extracting a media content representation part in the pre-training library, the media content representation part comprising a plurality of third media information, and the plurality of third media information being obtained after training of the plurality of first media information; and performing a recommendation task based on the media content representation part.

[0007] The behavior data of the user in the first time period is obtained, including a plurality of first items clicked in the first time period and a plurality of second items not clicked. The behavior data can be obtained from a user behavior log, and the specific operation behaviors include clicking an item and behaviors such as browsing, commenting, downloading and purchasing after clicking the item. The first scene can be a short video scene, a picture-text information scene and a video scene, etc. The picture-text information scene includes content with pictures and / or texts, the short video scene includes short videos that can switch content by up and down sliding or left and right sliding, and the video scene includes video content with a title. Correspondingly, the first items and the second items in the first scene can be short videos, pictures, articles and commodities, etc. The first time period can be 3 days, 7 days or one month, and is not limited specifically.

[0008] The item features include user information, domain (i.e. scene) specific information and media information, and the user information and the scene information are separated from the media information. Specifically, the perspective information of the user in the first scene to which the first items belong is generated through the plurality of first items clicked, and the perspective information includes user information and scene information. Then, the media information of the first items and the second items is extracted, and the media information is the content information of the items, including videos, audios, images and texts, etc.

[0009] The first perspective information of the user in the first scene is fused with the media information of the project to obtain a fusion representation of the project. The first perspective information is multiplied by the first media information of the first project to obtain a first fusion representation of the first project, and the first perspective information is multiplied by the second media information of the second project to obtain a second fusion representation of the second project. The first fusion representation is taken as a positive sample, and the second fusion representation is taken as a negative sample for contrastive learning training, the distance between the first fusion representations is pulled in, and the distance between the second fusion representation and the first fusion representation is pulled away in the representation space. After training converges, the plurality of first fusion representations and the plurality of second fusion representations subjected to the contrastive learning training are saved in a pre-training database (pre-training library).

[0010] Then, a media content representation part is extracted from the pre-training library, and the media content representation part includes a plurality of third media information extracted from the plurality of first fusion representations after training, that is, the third media information is the first media information after training, in other words, the third media information is the first media information after the distance is pulled in. The media content representation part can also include a plurality of fourth media information, which is extracted from the plurality of second fusion representations after training, that is, the fourth media information is the second media information after training, in other words, the fourth media information is the second media information after the distance is pulled away.

[0011] The media content representation part is used to perform an online recommendation task, that is, based on the user's request access, the project features and the context information, the project with a close content representation distance is filtered out from the media content representation part, and is displayed in a recommendation list.

[0012] In the first aspect of the application, the decoupling of the project features is realized by respectively acquiring the perspective information and the media information, the user information and the scene-specific information are separated, so that when the fusion representation is subjected to contrastive learning training, the media content representation part can focus on the learning of the content itself, thereby the cross-domain migration accuracy and efficiency of the content representation (i.e. the media information) can be improved.

[0013] In a possible implementation manner of the first aspect, the first scene includes a second scene and a third scene, and the plurality of first projects includes a third project and a fourth project, wherein the third project belongs to the second scene, and the fourth project belongs to the third scene, and the step of generating the perspective information of the user in the first scene based on the plurality of first projects includes: generating second perspective information of the user in the second scene based on the third project; and generating third perspective information of the user in the third scene based on the fourth project.

[0014] In the possible implementation, the first scene includes multiple scenes, and the multiple first items are items clicked by the user in the multiple scenes, that is, the model can be trained using the co-occurrence information across scenes, so as to increase the amount of data for training and improve the training accuracy of the model. Moreover, compared with the items of other scenes, the items in each scene can better improve the training effect of the model in the scene, and therefore, using the items of multiple scenes can improve the training accuracy of the model in the corresponding multiple scenes and improve the generality of the model.

[0015] In the case where the first scene includes multiple scenes, the multiple first items belong to the multiple scenes, and generating the perspective information of the user in the first scene is to generate perspective information of the user in each scene based on the first item in the scene. For example, the first scene includes a second scene and a third scene, the multiple first items include a third item and a fourth item, the third item belongs to the second scene, and the fourth item belongs to the third scene. Generating the first perspective information of the user in the first scene includes generating second perspective information of the user in the second scene based on the third item and generating third perspective information of the user in the third scene based on the fourth item, and the first perspective information includes the second perspective information and the third perspective information.

[0016] In a possible implementation of the first aspect, the step of performing the recommendation task based on the media content representation part includes: obtaining a fifth item clicked by the user at a current time; extracting fifth media information of the fifth item; filtering target media information from the multiple third media information, the similarity of the target media information to the fifth media information being less than a preset value; and recommending an item corresponding to the target media information to the user.

[0017] After the user clicks the fifth item, the item recommendation is performed based on the media information of the fifth item and the media content representation part. Specifically, the fifth media information of the fifth item is extracted, and the similarity of the fifth media information to the multiple third media information included in the media content representation part is compared, and target media information with a similarity less than a preset value is filtered from the multiple third media information. The item corresponding to the target media information is the to-be-recommended item.

[0018] The fourth media information can also be included in the media content representation part, and the specific filtering process can be to filter media information with a similarity less than a preset value from the third media information and the fourth media information, and then to determine target media information close in the representation space from the filtering result.

[0019] In the possible implementation, the process of performing the recommendation task based on the media content guarantee part is limited, and the realizability of the scheme is improved.

[0020] In a possible implementation manner of the first aspect, the step of fusing the first view information and the first media information to obtain the first fusion representation of each first item comprises: converting the first view information into a first matrix; converting the first media information into a first vector; and multiplying the first matrix and the first vector to obtain the first fusion representation of each first item.

[0021] The first view information is mapped into a view matrix (i.e., the first matrix) by an encoder, and the first media information is converted into a vector (i.e., the first vector) by an encoder. Then the first view information and the first media information are fused, and the specific process is that the first matrix is multiplied by the first vector. After the matrix is multiplied by the vector, a vector is still obtained, which is the first fusion representation. In this possible implementation manner, the fusion manner of the view information and the media information is limited, and the realizability of the scheme is improved.

[0022] In a possible implementation manner of the first aspect, the step of generating the first view information of the user in the first scene based on the plurality of first items comprises: generating the view information of the user in the first scene based on the plurality of first items by a sparse mixture of experts (Sparse MOE) model.

[0023] In this possible implementation manner, the user behavior sequence is input into the sparse mixture of experts (sparse MOE) model, and the view information of the user in the first scene is generated by the sparse mixture of experts model. The user behavior sequence is the first items clicked by the user. The sparse mixture of experts model is a machine learning model used to solve multi-target prediction problems, aiming to improve the performance and generalization ability of neural networks by integrating multiple expert networks. Its main idea is to distribute input data to multiple expert networks for processing, and then weight and average the outputs of each expert network to obtain the final prediction result. The sparse mixture of experts model can reduce the number of parameters, thereby improving the training speed and generalization ability. Moreover, it can adapt to different data distributions because it can use different expert networks to process different data subsets, and can also process high-dimensional sparse data.

[0024] In a possible implementation manner of the first aspect, the step of extracting the first media information of each first item and the second media information of each second item comprises: extracting the first media information of each first item and the second media information of each second item by a fine-grained interactive image-text pre-training model (FILIP).

[0025] First, the multi-modal data of the first item and the second item are preliminarily extracted, such as text data, image data, video data and the like, and then the multi-modal data is input into a fine-grained interactive language-image pre-training (FILIP) model, and a representation vector of the media information is extracted by the FILIP model, that is, the first media information and the second media information are extracted by the FILIP. The FILIP model can solve the fine-grained matching problem in image-text matching, and realize more precise alignment through a cross-modal late interaction mechanism. The FILIP model successfully utilizes the fine-grained expression ability between image blocks and text words by modifying only the contrast loss, and at the same time obtains the ability to precompute image and text representations offline during inference, maintaining the efficiency of large-scale training and inference.

[0026] In a possible implementation of the first aspect, the first scene includes a short video scene, a picture-text information scene, and a video call scene.

[0027] In this possible implementation, the specific form of the first scene is limited, which improves the realizability of the scheme.

[0028] The second aspect of the present application provides a data processing apparatus, including an acquisition unit, a generation unit, an extraction unit, a fusion unit, a training unit and an execution unit. The acquisition unit is configured to acquire a plurality of first items clicked by a user in a first time period; the generation unit is configured to generate first perspective information of the user in a first scene based on the plurality of first items, the first scene being a scene to which the plurality of first items belong, and the first perspective information including information of the user and information of the first scene; the acquisition unit is further configured to acquire a plurality of second items not clicked by the user in the first time period in the first scene; the extraction unit is configured to extract first media information of each first item and second media information of each second item; the fusion unit is configured to fuse the first perspective information and the first media information to obtain first fusion representation of each first item; the fusion unit is further configured to fuse the first perspective information and the second media information to obtain second fusion representation of each second item; the training unit is configured to perform contrast learning training by taking the plurality of first fusion representations as positive samples and the plurality of second fusion representations as negative samples to obtain a pre-training library; the extraction unit is further configured to extract a media content representation part in the pre-training library, the media content representation part including a plurality of third media information, the plurality of third media information being obtained after training of the plurality of first media information; and the execution unit is configured to perform a recommendation task based on the media content representation part.

[0029] In a possible implementation manner of the second aspect, the first scene includes a second scene and a third scene, and the plurality of first items include a third item and a fourth item, where the third item belongs to the second scene, and the fourth item belongs to the third scene, and the generation unit is specifically configured to generate second view information of the user in the second scene based on the third item, and generate third view information of the user in the third scene based on the fourth item.

[0030] In a possible implementation manner of the second aspect, the execution unit is specifically configured to: acquire a fifth item clicked by the user at a current time; extract fifth media information of the fifth item; filter target media information from the plurality of third media information, where a similarity between the target media information and the fifth media information is less than a preset value; and recommend an item corresponding to the target media information to the user.

[0031] In a possible implementation manner of the second aspect, the fusion unit is specifically configured to: convert the first view information into a first matrix; convert the first media information into a first vector; and multiply the first matrix and the first vector to obtain first fusion representation of each first item.

[0032] In a possible implementation manner of the second aspect, the generation unit is specifically configured to: generate the first view information of the user in the first scene based on the plurality of first items by using a Sparse MOE (Mixed Expert) model.

[0033] In a possible implementation manner of the second aspect, the extraction unit is specifically configured to: extract the first media information of each first item and the second media information of each second item by using a fine-grained interactive image-text pre-training model FILIP (Fine-Grained Interactive Language-Image Pre-training).

[0034] In a possible implementation manner of the second aspect, the first scene includes a short video scene, an image-text information scene, and a video scene.

[0035] The data processing apparatus provided in the second aspect of the present application is used to execute the method in the first aspect or any possible implementation manner of the first aspect.

[0036] The third aspect of the present application provides a data processing apparatus, including a processor and a memory. The memory is used to store instructions, and the processor is used to acquire the instructions stored in the memory to execute the method in the first aspect or any possible implementation manner of the first aspect.

[0037] The fourth aspect of the present application provides a computer readable storage medium, including instructions, when the instructions run on a computer, causing the computer to execute the method in the first aspect or any possible implementation manner of the first aspect.

[0038] The fifth aspect of the present application provides a computer program product comprising instructions which, when the computer program product runs on a computer, enable the computer to perform the method in the first aspect or any possible implementation manner of the first aspect.

[0039] The sixth aspect of the present application provides a chip system, comprising at least one processor and a communication interface, the communication interface and the at least one processor are interconnected through a line, and the at least one processor is configured to run a computer program or instructions to perform the method in the first aspect or any possible implementation manner of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0040] FIG. 1 is a schematic diagram of an application scenario of the data processing method provided by the embodiments of the present application;

[0041] FIG. 2 is a schematic diagram of one embodiment of the data processing method provided by the embodiments of the present application;

[0042] FIG. 3a is a schematic diagram of generating view information in the embodiments of the present application;

[0043] FIG. 3b is another schematic diagram of generating view information in the embodiments of the present application;

[0044] FIG. 4a is a schematic diagram of extracting media information in the embodiments of the present application;

[0045] FIG. 4b is another schematic diagram of extracting media information in the embodiments of the present application;

[0046] FIG. 5 is a schematic diagram of obtaining fused representation in the embodiments of the present application;

[0047] FIG. 6 is a schematic diagram of contrastive learning in the embodiments of the present application;

[0048] FIG. 7 is a schematic diagram of another embodiment of the data processing method provided by the embodiments of the present application;

[0049] FIG. 8 is a schematic diagram of the recall effect of the data processing method provided by the embodiments of the present application;

[0050] FIG. 9 is a schematic diagram of a pruning experiment of the data processing method provided by the embodiments of the present application;

[0051] FIG. 10 is a schematic diagram of the structure of a data processing apparatus provided by the embodiments of the present application;

[0052] FIG. 11 is another schematic diagram of the structure of a data processing apparatus provided by the embodiments of the present application. DETAILED DESCRIPTION

[0053] The embodiments of the present application provide a data processing method, which can improve the cross-domain migration accuracy and efficiency of media content representation. The embodiments of the present application also provide corresponding devices, computer readable storage media, computer program products and the like. The following are described respectively.

[0054] The embodiments of the present application are described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Those skilled in the art can know that, with the development of technology and the appearance of new scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0055] The terms "domain" and "scenario", "model", "network", "media information" and "content representation" and the like in the specification and claims of the present application and the above-mentioned drawings can be used interchangeably. Unless otherwise specified, the ordinal numbers "first", "second" and the like are used to distinguish multiple objects, and are not used to limit the order, time sequence, priority or importance of the multiple objects. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to the process, method, product or device.

[0056] For ease of understanding, the related terms and concepts mainly involved in the embodiments of the present application are introduced below.

[0057] 1. Machine learning system

[0058] Based on the input features and labels, the parameters of the machine learning model are trained by gradient descent and other optimization methods, and finally the trained model is used to complete the label prediction of unknown data.

[0059] 2. Recommendation system

[0060] The recommendation system analyzes and learns by using machine learning algorithm according to the historical click behavior of the user, and then predicts the new request of the user and returns the item recommendation list.

[0061] 3. Click rate

[0062] It refers to the probability of a user clicking a certain display item in a specific environment.

[0063] 4. Content representation / multi-modal representation

[0064] A technique for representing multimedia information, such as input pictures, as vectors, through extraction by a neural network to obtain intermediate layer vectors.

[0065] 5. Transferability

[0066] The model or representation trained in one scenario can still be used in another scenario.

[0067] Please refer to FIG. 1, which is a schematic diagram of an application scenario of the data processing method provided by the embodiments of the present application.

[0068] As shown in FIG. 1, it is a schematic diagram of a recommendation scenario. The recommendation scenario refers to recommending content that a user may be interested in to the user, including short videos, graphic information, videos, and goods, etc. The recommendation technology can be simplified as prediction of item click rate. If the probability of clicking an item is high, the item is recommended to the user, otherwise, the item is not recommended. Specifically, the operations of the user in the front-end page generate behavior data, such as clicking an item and browsing, commenting, downloading, etc. after clicking, and the behavior data is written to the log. Then, the user behavior data in the log is used for offline model training, and a prediction model is generated after training convergence. Then, the prediction model is deployed in an online service environment, and a recommendation result is given based on user request access, item features and context information, etc. to recommend items with high click probability to the user, so as to increase the use time and activity level of the user. The recommended items are displayed in the form of a display list in the front-end page. After recommending the items to the user, feedback of the user on the recommended items is collected, i.e. whether the user clicks the recommended item, and the prediction model is adjusted according to the feedback data.

[0069] When training the prediction model, the behavior data of the user in one scenario is usually used to train the recommendation ability of the model in the scenario, and when recommending items, the items related to the current scenario are also recommended. For example, the data of the user browsing short videos is used to train the recommendation ability of the model in the short video scenario, and the data of the user browsing graphic information is used to train the recommendation ability of the model in the graphic information scenario. When recommending, if the user is browsing graphic news, the recommended items are also graphic information, and if the user is watching short videos, the user is also recommended short videos. It can be understood that a scenario can also be referred to as a domain, for example, the short video scenario can also be referred to as a short video domain. In the embodiments of the present application, the scenario is taken as an example for description, and the specific implementation is not limited.

[0070] The behavior data of the user in a scene is used to extract the features of the items clicked by the user in the scene. However, when extracting the features of each scene item, the scene information, the user information and the content information of the item are encoded together, thereby causing poor cross-domain migration of the content information of the item and failing to be used for the recommendation task in other scenes. For example, the user often clicks sports news in the information scene, and when extracting the features, the user information and the scene information are encoded into the sports news item together, thereby causing the sports news to be unable to be used for the recommendation in the small video scene.

[0071] Therefore, the embodiments of the present application provide a data processing method, which decouples the user information, the scene information and the media information in the features of the item, improves the learning effect of the model on the media information, and thereby improves the accuracy and effect of the cross-scene migration of the item. Further, the embodiments of the present application use the behavior data of the same time period in the cross-scene to train the model, thereby more fully utilizing the co-occurrence information, increasing the amount of data used for training, and thereby improving the generalization and accuracy of the model.

[0072] Please refer to FIG. 2, which is a schematic diagram of one embodiment of the data processing method provided by the embodiments of the present application. As shown in FIG. 2, the embodiment includes steps 201 to 209.

[0073] 201, obtaining a plurality of first items clicked by a user in a first time period.

[0074] The operation behavior of the user in each scene is recorded in the user behavior log, and the operation behavior of the user includes clicking the item and browsing, commenting, downloading or purchasing after clicking the item. The user behavior log is analyzed to extract the first item sequence (i.e. the plurality of first items) clicked by the user in the first time period. The first time period can be 3 days, 7 days or one month, and the length of the first time period is not limited. It can be understood that the module for storing the user behavior data can be called the user behavior log, or the user operation log or other names, and the specific name is not limited here.

[0075] The first items can be short videos, pictures, articles, and commodities, etc. In one possible solution, the first items cover multiple scenarios. For example, the information stream service provided by a mobile phone browser includes graphic information, short videos, and videos. The graphic information is content including pictures and / or text, the short videos are short videos that can switch content by up and down sliding or left and right sliding, and the videos are video content with titles. The graphic information is mainly picture modal and text modal, and the short videos and videos are mainly video modal and audio modal. The user can browse in multiple service scenarios in the first time period, and after clicking to browse the graphic information, the user can continue to browse the videos and short videos. Therefore, the multiple first items clicked in the first time period cover the graphic information scenario, the video scenario, and the short video scenario.

[0076] The first items clicked in a period of time can be considered as items that the user is more interested in, and the multiple first items clicked are also called co-occurring items. The multiple first items clicked are grouped into positive samples, which are used as labels in subsequent contrastive learning training.

[0077] 202. Generate first perspective information of the user in the first scenario based on the multiple first items, the first scenario being a scenario to which the multiple first items belong, and the first perspective information including information of the user and information of the first scenario.

[0078] The items clicked by the user in each scenario can reflect the user perspective of the user in the use scenario, and therefore the multiple first items obtained can describe the first perspective information of the user in the first scenario to which the first items belong. The first perspective information includes information of the user and information of the first scenario, which is actually the user information and scenario information (i.e., domain information) decoupled from the first item features.

[0079] In one possible solution, the multiple first items cover multiple use scenarios, in other words, the first scenario includes multiple scenarios. For example, the first scenario includes a second scenario and a third scenario, and the multiple first items include a third item and a fourth item, wherein the third item belongs to the second scenario and the fourth item belongs to the third scenario. Generating the first perspective information of the user in the first scenario based on the multiple first items specifically includes generating second perspective information of the user in the second scenario based on the third item, generating third perspective information of the user in the third scenario based on the fourth item, and the first perspective information including the second perspective information and the third perspective information. That is, when multiple scenarios are included, the perspective information of the scenario to which the clicked item belongs is generated based on the clicked item.

[0080] The process of generating perspective information based on multiple first items can be understood in combination with FIG. 3a and FIG. 3b. As shown in FIG. 3a, the multiple first items clicked by the user in the first scenario in the first time period include item 1, item 2, item 3, item 4 and item 5. The multiple first items are converted into corresponding embeddings by a shared item encoder. The embedding is a vector used to represent an item in a low-dimensional space, which can also be referred to as a vector mapping of the item. The vector mapping of item 1 is emb1, the vector mapping of item 2 is emb2, the vector mapping of item 3 is emb3, the vector mapping of item 4 is emb4, and the vector mapping of item 5 is emb5. Then, a projection matrix of the user in the first scenario is obtained by a domain encoder. The embedding is subjected to matrix transformation by the projection matrix, and finally the perspective information of the user in the first scenario is obtained. Since the perspective information is obtained by matrix transforming the embedding of the item, the perspective information can also be referred to as a perspective matrix.

[0081] FIG. 3b is a specific process of generating first perspective information when the first items cover multiple scenarios. Item 1 and item 2 clicked by the user in the first time period belong to a second scenario, and item 3, item 4 and item 5 clicked by the user in the first time period belong to a third scenario. The multiple first items are converted into corresponding embeddings by a shared item encoder, and then routed to corresponding domain encoders by a routing layer, i.e., embedding 1 and embedding 2 are routed to a second scenario encoder, and embedding 3, embedding 4 and embedding 5 are routed to a third scenario encoder, so as to obtain projection matrices of the second scenario and the third scenario. After the item embedding is subjected to matrix transformation processing by the projection matrix of the corresponding scenario, the perspective information of the corresponding scenario is obtained, i.e., second perspective information in the second scenario and third perspective information in the third scenario.

[0082] In one possible solution, the perspective information of the user in the first scene is generated by a sparse mixture of experts (sparse MOE), i.e., the shared item encoder, the scene encoder and the routing layer in FIGS. 3a and 3b are modules in the sparse MOE. Specifically, the user behavior sequence, i.e., the items clicked by the user, is input into the sparse MOE, so as to obtain the perspective matrix of the user in the corresponding scene. The sparse MOE is a machine learning model for solving multi-target prediction problems, aiming to improve the performance and generalization ability of neural networks by integrating multiple expert networks. The main idea is to distribute the input data to multiple expert networks for processing, and then perform weighted averaging on the outputs of the expert networks to obtain the final prediction result. The sparse MOE can reduce the number of parameters, thereby improving the training speed and generalization ability. Moreover, it can adapt to different data distributions, as it can use different expert networks to process different data subsets, and can also process high-dimensional sparse data.

[0083] In addition to the sparse MOE, other models can also be used to generate perspective information, such as a mixture of experts (MOE), a soft mixture of experts (soft MOE), etc., which are not limited here.

[0084] 203、obtain a plurality of second items in the first scene that the user does not click in the first time period.

[0085] The plurality of second items in the first scene that the user does not click in the first time period are obtained, and when the first scene only includes one scene, the second item is also only an item in the scene. For example, the first scene is a graphic information scene, and the second item is an article in the graphic information scene. When the first scene includes multiple scenes, the second item is an item in the corresponding multiple scenes. For example, the first scene includes a second scene and a third scene, the second scene is a graphic information scene, and the third scene is a small video scene, and the second item includes an article in the graphic information scene and a small video in the small video scene.

[0086] The items that are not clicked indicate that the user may be less interested in them, and the plurality of second items are used as negative samples for subsequent label training in contrastive learning.

[0087] 204、extract first media information of each first item and second media information of each second item.

[0088] The first media information of the clicked item and the second media information of the non-clicked item are extracted, and the media information is content information of the item, including text, image, audio and video, etc. The media information can reflect the content of the item in detail and intuitively, and the item can be more accurately recommended to the user in combination with the media information.

[0089] The first scene can be one scene, for example, the first scene is a picture-text information scene, and the modalities of the first item and the second item are image and / or text. The first scene can also include multiple scenes, for example, the first scene includes a second scene and a third scene, the second scene is a picture-text information scene, and the third scene is a short video scene, and the modalities of the media information of the first item and the second item include image, text, audio and video.

[0090] For multiple first items clicked by the user within a period of time, it can be considered that the representations of the multiple first items are similar, so that the representation alignment is performed after the media information is extracted, so that the user can be recommended related content of the multiple first items in subsequent recommendation. For example, the user clicks on basketball news and then clicks on football news, and it is considered that the two are similar, and the corresponding representations are aligned in space after the media information is extracted. After the model is trained using the aligned media information, the model can recommend related content to the user, for example, when the user browses the basketball news, the recommendation list will display other basketball news and also display football news. When the first scene only includes one scene, only the media information needs to be aligned within the domain after the media information is extracted. When the first scene includes multiple scenes, in addition to aligning the extracted media information within the domain, cross-domain alignment is also needed.

[0091] The process of extracting media information is shown in FIGS. 4a and 4b. It can be understood that the process of extracting the first media information and the second media information is similar, and FIGS. 4a and 4b take the first media information as an example for description.

[0092] As shown in FIG. 4a, the first scenario is a text and image information scenario, the first clicked item includes item 1, item 2 and item 3, and the modalities of the media information of the three items include images and texts. The multi-modal data is extracted from item 1 to obtain text 1 and image 1. The multi-modal data is extracted from item 2 to obtain text 2 and image 2. The multi-modal data is extracted from item 3 to obtain text 3 and image 3. The data is respectively input into the representation encoder of the corresponding modality, that is, the text is input into the text encoder, and the image is input into the image encoder. After the processing of the modality encoder, the data is input into the scene adapter to obtain the representation vector of only the media information. The scene adapter can be a scene mixed expert model adapter (domain MOE adapter). After the media information of each item is extracted, the information alignment in the scene is performed.

[0093] FIG. 4b is a case where the first scenario includes a short video scenario and a text and image information scenario, wherein the clicked item of the user includes item 1, item 2 and item 3. Item 1 belongs to the short video scenario, and the modality is audio and video. Item 2 and item 3 belong to the text and image information scenario, and the modality is text and image. The multi-modal data is extracted from item 1 to obtain audio 1 and video 1. The multi-modal data is extracted from item 2 to obtain text 1 and image 1. The multi-modal data is extracted from item 3 to obtain text 2 and image 2. The data is respectively input into the representation encoder of the corresponding modality, that is, the audio is input into the voice encoder, the video is input into the video encoder, the text is input into the text encoder, and the image is input into the image encoder. After the processing of the modality encoder, the data is input into the corresponding scene adapter, that is, audio 1 and video 1 are input into the short video scenario adapter, and text 1, image 1, text 2 and image 2 are input into the text and image information scenario adapter, to obtain the representation vector of only the media information. After the media information of each item is extracted, not only the media information is aligned in the domain, but also the cross-domain alignment is performed.

[0094] In one possible solution, the multimodal data (text data, video data, etc.) of the first item and the second item are preliminarily extracted, and then the multimodal data is input into a fine-grained interactive language-image pre-training (FILIP) model, and a representation vector containing only media information is extracted by the FILIP model, that is, the first media information and the second media information are extracted by the FILIP. The FILIP can solve the fine-grained matching problem in image-text matching, and realize more precise alignment through a cross-modal late interaction mechanism, which uses the maximum similarity between visual tokens and text tokens at the token level to guide the objective function of contrastive learning. The FILIP model successfully utilizes the fine-grained expression capability between image blocks and text words by modifying only the contrastive loss, while obtaining the ability to precompute image and text representations offline during inference, maintaining the efficiency of large-scale training and inference.

[0095] 205、fuse the first perspective information and the first media information to obtain a first fusion representation of each first item.

[0096] The clicked item can obtain the intra-domain user perspective of the domain to which the item belongs, that is, the perspective information, which specifically includes user information and scene information of the domain to which the item belongs, and the media information includes content information of the item, so that the perspective information and the media information are fused to obtain the complete representation of the item. The first perspective information and the first media information are fused to obtain the first fusion representation for representing the first item.

[0097] Optionally, the specific process of fusion is to map the first perspective information into a perspective matrix (i.e., a first matrix) through an encoder, and convert the first media information into a vector (i.e., a first vector) through an encoder. Then the first matrix is multiplied by the first vector, and the result of the matrix-vector multiplication is still a vector, which is the first fusion representation.

[0098] 206、fuse the first perspective information and the second media information to obtain a second fusion representation of each second item.

[0099] Similar to step 205, the first perspective information and the second media information are fused to obtain the second fusion representation for representing the second item.

[0100] Optionally, the specific process of fusion is to map the first perspective information into a first matrix through an encoder, and convert the second media information into a second vector through an encoder. Then the first matrix is multiplied by the second vector, and the result of the matrix-vector multiplication is still a vector, which is the second fusion representation.

[0101] The process of obtaining the first fusion representation and the second fusion representation can be understood in combination with FIG. 5, which takes obtaining the first fusion representation as an example. The lower dashed box in FIG. 5 is the process of extracting media information described in step 204. After the feature vector of the media information of each item is extracted, the feature vector is fused with the perspective information of the first scene to obtain the fusion representation of each item. When the first scene includes multiple scenes, the media information of each item is fused with the perspective information of the scene where the item is located. For example, item 1 belongs to the second scene, and the perspective information of the second scene is the second perspective information. Item 2 and item 3 belong to the third scene, and the perspective information of the third scene is the third perspective information. Then, the fusion process is to fuse the media information of item 1 with the second perspective information to obtain the fusion representation of item 1. The media information of item 2 and item 3 is fused with the third perspective information to obtain the fusion representation of item 2 and item 3. After obtaining the fusion representation, the feature vector also needs to be aligned in the representation space. When the first scene includes only one scene, only intra-domain alignment is needed. When the first scene includes multiple scenes, intra-domain alignment and cross-domain alignment are needed.

[0102] The process of obtaining the fusion representation can also be understood in combination with FIG. 3a and FIG. 3b. As shown in FIG. 3a and FIG. 3b, the dashed box connected with the perspective information is the media information of the item. After the matrix transformation obtains the perspective matrix, the media information is fused to obtain the fusion representation of the item.

[0103] The process of obtaining the fusion representation can be represented by the formula E=MxP, where E is the fusion representation, M is the media information, and P is the perspective information. That is, the feature vector of the media information of the item is multiplied by the perspective matrix of the corresponding scene to obtain the fusion representation vector of the item. After the vector is multiplied by the matrix, it is still a vector. The traditional multi-modal representation learning method directly learns E, but in the embodiment of the present application, E is decomposed into the product of M and P, so that the media information representation extraction network can focus on extracting and learning content representation. In other words, the learned content representation does not carry user information and domain information, so as to improve the cross-domain migration of media content and can be used for cross-domain recommendation tasks.

[0104] 207、The first fusion representation is taken as a positive sample, and the second fusion representation is taken as a negative sample for contrastive learning training to obtain a pre-training library.

[0105] After obtaining the first fusion representation of the first item and the second fusion representation of the second item, the positive and negative category labels constructed in the early stage are used for contrastive learning training. Specifically, in the process of contrastive learning, the plurality of first fusion representations corresponding to the plurality of first items are taken as positive samples, and the second fusion representation corresponding to the plurality of second items is taken as negative samples, the distance between the first fusion representations is shortened, and the distance between the second fusion representation and the first fusion representation is lengthened. After the training converges, the plurality of first fusion representations and the plurality of second fusion representations trained by contrastive learning are saved in a pre-training database (pre-training library). In the training process, the parameters of the decoupling module and the parameters of the media content representation module are learned at the same time. The decoupling module is a module for generating perspective information, and the media content representation module is a module for obtaining media information.

[0106] The process of contrastive learning can be understood in combination with FIG. 6. The fusion representation 1 and the fusion representation 2 are both first fusion representations, and the distance between the fusion representation 1 and the fusion representation 2 in space is shortened.

[0107] 208、Extract the media content representation part in the pre-training library, and the media content representation part includes a plurality of third media information, and the plurality of third media information is obtained after training of the plurality of first media information.

[0108] The media content representation part is extracted from the pre-training library, and the media content representation part includes a plurality of third media information extracted from the plurality of first fusion representations after training, that is, the third media information is the first media information after training. In other words, the third media information is the first media information after the distance is shortened. The media content representation part can also include a plurality of fourth media information, and the fourth media information is extracted from the second fusion representation after training, that is, the fourth media information is the second media information after training. In other words, the fourth media information is the second media information after the distance is lengthened.

[0109] 209、Based on the media content representation part, a recommendation task is performed.

[0110] The media content representation part is used to perform an online recommendation task, that is, based on the user's request access, item features and context information, items with close multi-modal representation distances are given, and the items are displayed in a recommendation list. Since the media content representation part focuses on content information, it can make recommendations across domains. For example, a user browses sports news in a text and image information scenario in a browser, and in addition to recommending sports news in a text and image information scenario, it will also recommend text and image news in a short video scenario or text and image news in a video scenario.

[0111] In a possible solution, the process of performing the recommendation task is to obtain a fifth item clicked by the user at the current time, and perform item recommendation based on the media information of the fifth item and the media content representation part. Specifically, the fifth media information of the fifth item is extracted, and similarity comparison is performed with the plurality of third media information included in the media content representation part, and target media information with a similarity less than a preset value is selected from the plurality of third media information, and the item corresponding to the target media information is the to-be-recommended item.

[0112] Optionally, the media content representation part can also include fourth media information, and the specific screening process can be to screen media information with a similarity less than a preset value from the third media information and the fourth media information, and then determine target media information close in the representation space from the screening result.

[0113] The recommendation task can be an item recall task and an item fine ranking task. The item recall task is to find a small part of items with close representation distances from a large number of candidate items, for example, to filter tens of thousands of items to hundreds of items. The item fine ranking task is to select a smaller number of items from the small number of items screened by the item recall task through fine ranking, for example, to select tens of items from the hundreds of items screened by the item recall task.

[0114] In addition, the recommendation task can also combine an item coarse ranking task and an item re-ranking task. The coarse ranking task is a step between the recall task and the fine ranking task, for example, to filter 500 items recalled to 100 items, and the fine ranking task further processes the 100 items. The re-ranking task is a step after the fine ranking task, which further processes the items screened by the fine ranking. For example, removing content that has been read by the user, removing duplicate content, and dispersing processing, etc., and inserting advertisements and operation content, etc.

[0115] In the embodiment, the decoupling of the item features is realized by respectively obtaining the perspective information and the media information, the user information and the scene-specific information are separated, the media content representation part focuses on the content itself, and therefore the cross-domain migration accuracy and efficiency of the content representation (i.e., the media information) can be improved, the model does not need to be fine-tuned when migrated to a new scene, and the training cost can be saved.

[0116] Optionally, in the embodiment, the co-occurrence information across scenes (i.e., the items clicked in different scenes in the same time period) is used for training, which increases the amount of data used for training, alleviates the problem of sparse collaborative signals, and can improve the accuracy and training effect of the model. Moreover, cross-domain training can avoid introducing too much domain information, thereby improving the transfer effect of media information. Although the decoupling of user information, scene information, and media information in the embodiment can improve the transfer effect of content representation, the items of each scene are still better for the training effect of the scene, and therefore, using the items of multiple scenes for training can better improve the accuracy of the model in the corresponding scene and improve the generality of the model. In addition, the improved cross-domain transferability of content representation in the embodiment can further improve the use efficiency of co-occurrence information, thereby improving the training efficiency.

[0117] In summary, the data processing method provided by the embodiment of the application is described below in combination with FIG. 7, which takes training using co-occurrence signals across scenes as an example. As shown in FIG. 7, the user clicks item 1 of the second scene and also clicks items 2 and 3 of the third scene in the first time period. The three items are input into the user domain decoupling module to separate the user information and the scene information, that is, to generate the perspective information corresponding to the scene. Specifically, the three items are first input into the all-domain item embedding layer (all domain id embedding) for vector mapping, and the output data is then processed by the user behavior encoder (user behavior encoder). After processing, each item goes to the scene encoder of the respective scene, that is, item 1 is input into the scene encoder of the second scene, and items 2 and 3 are input into the scene encoder of the third scene, thereby obtaining the prediction matrix of the second scene and the prediction matrix of the third scene. Finally, the second perspective information of the second scene and the third perspective information of the third scene are obtained through matrix transformation processing.

[0118] The media information of the three items is then extracted. Specifically, the multi-modal data of the three items is first extracted, the media information modalities of item 1 are audio and video, and audio 1 and video 1 are extracted from item 1. The media information modalities of items 2 and 3 are text and image, and text 1 and image 1 are extracted from item 2, and text 2 and image 2 are extracted from item 3. Each modality information is processed by the corresponding modality representation encoder, that is, the audio is input into the audio encoder, the video is input into the video encoder, the text is input into the text encoder, and the image is input into the image encoder. After processing by each encoder, the media information of item 1 is input into the second scene adapter, and the media information of items 2 and 3 is input into the third scene adapter, thereby obtaining the vector representation of the media information of each item.

[0119] Then the view information is fused with the media information to obtain the fusion representation of each item. Specifically, the media information of item 1 is multiplied by the second view information to obtain the fusion representation of item 1. The media information of item 2 is multiplied by the third view information to obtain the fusion representation of item 2. The media information of item 3 is multiplied by the third view information to obtain the fusion representation of item 3.

[0120] After obtaining the fusion representation of each item, cross-domain contrast learning is performed to narrow the distance of item 1, item 2 and item 3 in the representation space. After training convergence, the media content representation part is used for item recommendation tasks, such as performing recall tasks and precision tasks.

[0121] The data processing method provided in the embodiments of the present application has carried out recall experiments and pruning experiments on public data sets. First, the recall effect will be described in combination with FIG. 8. In FIG. 8, UniEmbedding is the data processing method provided in the embodiments of the present application, UniSRec and Item2Vec are existing methods. Testset is a test data set, including HR@10, HR@50, MRR@10, MRR@50, NDCG@10 and NDCG@50. Old seq represents an old sequence, and new seq represents a new sequence. The larger the number is, the better the recall effect is. As can be seen from FIG. 8, compared with UniSRec, UniEmbedding has better recall effect on the old and new sequences of each data set, and the improvement is large. Compared with Item2Vec, UniEmbedding also achieves better recall effect on each data set except HR@50.

[0122] FIG. 9 is a pruning experiment, that is, testing the recall effect after pruning part of the function of UniEmbedding. The UniEmbedding column represents no pruning, UniEmbedding(item only) represents the case of using only view information for contrast learning, and UniEmbedding(single domain) represents the case of using only co-occurrence information in a single scene for training. As can be seen from FIG. 9, compared with UniEmbedding, when only using view information, the recall effect on the old and new sequences of each data set has decreased a lot. When only using co-occurrence information in a single scene, the recall effect on the old and new sequences of each data set also has a significant decline.

[0123] The data processing method provided by the embodiments of the present application also achieves good results after practical application. In the browser video scene, the average visit view of the full population is increased by 1.55%, and the average use time of 4 days is increased by 32.53 seconds. In the browser text information scene, the average visit view of the low active population is increased by 1.89%, and the average use time is increased by 3.57 seconds. In the browser small video scene, the average visit view of the medium and high active population is increased by 1.1%, and the average use time of 7 days is increased by 13.99 seconds.

[0124] The embodiments of the present application are described above from the perspective of the method. The related devices in the embodiments of the present application are introduced from the perspective of the specific device implementation.

[0125] Please refer to FIG. 10, the present application embodiment provides a kind of schematic diagram of data processing device 1000.Therein, the data processing device 1000 includes acquisition unit 1001, generating unit 1002, extraction unit 1003, fusion unit 1004, training unit 1005 and execution unit 1006.

[0126] Acquisition unit 1001, for obtaining multiple first items clicked by user in first time period.

[0127] Generating unit 1002, for generating the first perspective information of user in first scene based on multiple first items, the first scene is the scene to which multiple first items belong, and the first perspective information includes the information of user and the information of first scene.

[0128] Acquisition unit 1001, for obtaining multiple second items not clicked by user in first time period in first scene.

[0129] Extraction unit 1003, for extracting the first media information of each first item and the second media information of each second item.

[0130] Fusion unit 1004, for fusing the first perspective information and the first media information, to obtain the first fusion representation of each first item.

[0131] Fusion unit 1004, for fusing the first perspective information and the second media information, to obtain the second fusion representation of each second item.

[0132] Training unit 1005, for comparing learning training with multiple first fusion representations as positive samples and multiple second fusion representations as negative samples, to obtain pre-training library.

[0133] Extraction unit 1003, for extracting media content representation part in pre-training library, and the media content representation part includes multiple third media information, and the multiple third media information is obtained after training multiple first media information.

[0134] The execution unit 1006 is configured to execute the recommendation task based on the media content representation part.

[0135] Optionally, the first scene includes a second scene and a third scene, and the plurality of first items include a third item and a fourth item, the third item belongs to the second scene, and the fourth item belongs to the third scene, and the generation unit 1002 is specifically configured to generate second perspective information of the user in the second scene based on the third item, and generate third perspective information of the user in the third scene based on the fourth item.

[0136] Optionally, the execution unit 1006 is specifically configured to obtain a fifth item clicked by the user at a current time, extract fifth media information of the fifth item, filter target media information from the plurality of third media information, the similarity between the target media information and the fifth media information being less than a preset value, and recommend an item corresponding to the target media information to the user.

[0137] Optionally, the fusion unit 1004 is specifically configured to convert the first perspective information into a first matrix, convert the first media information into a first vector, and multiply the first matrix and the first vector to obtain the first fusion representation of each first item.

[0138] Optionally, the generation unit 1002 is specifically configured to generate the first perspective information of the user in the first scene based on the plurality of first items by using a Sparse MOE model.

[0139] Optionally, the extraction unit 1003 is specifically configured to extract the first media information of each first item and the second media information of each second item by using a fine-grained interactive image-text pre-training model FILIP.

[0140] Optionally, the first scene includes a short video scene, a text information scene, and a video scene.

[0141] The modules in the data processing apparatus 1000 perform operations of the data processing apparatus as described in the foregoing embodiments of FIGS. 2-5 and 7, and details are not repeated here.

[0142] Please refer to Fig. 11, which is a possible structural schematic diagram of a data processing apparatus 1100 provided by an embodiment of the present application, including a processor 1101, a communication interface 1102, a memory 1103 and a bus 1104. The processor 1101, the communication interface 1102 and the memory 1103 are connected with each other through the bus 1104. In an embodiment of the present application, the processor 1101 is configured to control and manage the actions of the data processing apparatus, for example, the processor 1101 is configured to execute the steps performed by the data processing apparatus in the method embodiment shown in Fig. 2. The communication interface 1102 is configured to support the communication of the data processing apparatus. The memory 1103 is configured to store the program code and data of the data processing apparatus.

[0143] The processor 1101 can be a central processing unit, a general purpose processor, a digital signal processor, an application specific integrated circuit, a field programmable gate array, or other programmable logic device, transistor logic device, hardware component or any combination thereof. It can implement or execute the various exemplary logical blocks, modules and circuits described in connection with the disclosure of the present application. The processor can also be a combination of computing functions, such as a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, etc. The bus 1104 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, only one thick line is shown in Fig. 11, but it does not mean that there is only one bus or only one type of bus.

[0144] An embodiment of the present application also provides a computer readable storage medium, which includes instructions, when the instructions are run on a computer, causing the computer to execute the method in the above-mentioned embodiments shown in Figs. 2-5 and 7.

[0145] An embodiment of the present application also provides a computer program product including instructions, when the computer program product is run on a computer, causing the computer to execute the method in the above-mentioned embodiments shown in Figs. 2-5 and 7.

[0146] An embodiment of the present application also provides a chip system, which includes at least one processor and a communication interface, the communication interface and the at least one processor are connected through a circuit, and the at least one processor is configured to run a computer program or instructions to execute the method in the above-mentioned embodiments shown in Figs. 2-5 and 7.

[0147] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiment, and will not be repeated here.

[0148] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiment, and will not be repeated here.

[0149] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0150] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0151] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0152] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, read-only memory), a random access memory (RAM, random access memory), a magnetic disk or an optical disk, and various media that can store program codes.

Claims

1. A data processing method, characterized by, The method comprises: obtaining a plurality of first items clicked by a user in a first time period; generating first perspective information of the user in a first scene based on the plurality of first items, the first scene being a scene to which the plurality of first items belong, and the first perspective information comprising information of the user and information of the first scene; obtaining a plurality of second items not clicked by the user in the first time period in the first scene; extracting first media information of each first item and second media information of each second item; fusing the first perspective information and the first media information to obtain first fusion representation of each first item; fusing the first perspective information and the second media information to obtain second fusion representation of each second item; performing contrastive learning training on the plurality of first fusion representations as positive samples and the plurality of second fusion representations as negative samples to obtain a pre-training library; extracting a media content representation part in the pre-training library, the media content representation part comprising a plurality of third media information, the plurality of third media information being obtained after training of the plurality of first media information; performing a recommendation task based on the media content representation part.

2. The method of claim 1, wherein, The first scene comprises a second scene and a third scene, and the plurality of first items comprises a third item and a fourth item, wherein the third item belongs to the second scene, and the fourth item belongs to the third scene, and the generating of the first perspective information of the user in the first scene based on the plurality of first items comprises: generating second perspective information of the user in the second scene based on the third item; and generating third perspective information of the user in the third scene based on the fourth item.

3. The method according to claim 1 or 2, characterized in that, The performing of the recommendation task based on the media content representation part comprises: obtaining a fifth item clicked by the user at a current time; extracting fifth media information of the fifth item; filtering target media information from the plurality of third media information, the target media information having a similarity less than a preset value with the fifth media information; and recommending an item corresponding to the target media information to the user.

4. The method according to any one of claims 1 to 3, characterized in that, The fusing of the first perspective information and the first media information to obtain the first fusion representation of each first item comprises: converting the first perspective information into a first matrix; converting the first media information into a first vector; and multiplying the first matrix and the first vector to obtain the first fusion representation of each first item.

5. The method according to any one of claims 1 to 4, characterized in that, The generating of the first perspective information of the user in the first scene based on the plurality of first items comprises: generating the first perspective information of the user in the first scene based on the plurality of first items by a Sparse MOE model.

6. The method according to any one of claims 1 to 5, characterized in that, The extracting of the first media information of each first item and the second media information of each second item comprises: extracting the first media information of each first item and the second media information of each second item by a fine-grained interactive image-text pre-training model FILIP.

7. The method according to any one of claims 1 to 6, characterized in that, The first scene comprises a short video scene, an image-text information scene, and a video scene.

8. A data processing apparatus, characterized by, The method comprises: An acquisition unit is configured to acquire a plurality of first items clicked by a user in a first time period; A generation unit is configured to generate first perspective information of the user in a first scene based on the plurality of first items, the first scene being a scene to which the plurality of first items belong, and the first perspective information including information of the user and information of the first scene; The acquisition unit is further configured to acquire a plurality of second items not clicked by the user in the first time period in the first scene; An extraction unit is configured to extract first media information of each first item and second media information of each second item; A fusion unit is configured to fuse the first perspective information and the first media information to obtain first fusion representation of each first item; The fusion unit is further configured to fuse the first perspective information and the second media information to obtain second fusion representation of each second item; A training unit is configured to perform contrastive learning training on a plurality of first fusion representations as positive samples and a plurality of second fusion representations as negative samples to obtain a pre-training library; The extraction unit is further configured to extract a media content representation part in the pre-training library, the media content representation part including a plurality of third media information, the plurality of third media information being obtained after training of the plurality of first media information; An execution unit is configured to perform a recommendation task based on the media content representation part.

9. The apparatus of claim 8, wherein, The first scene includes a second scene and a third scene, and the plurality of first items includes a third item and a fourth item, wherein the third item belongs to the second scene, and the fourth item belongs to the third scene, and the generation unit is specifically configured to: generate second perspective information of the user in the second scene based on the third item; generate third perspective information of the user in the third scene based on the fourth item.

10. The apparatus of claim 8 or 9, wherein, The execution unit is specifically configured to: acquire a fifth item clicked by the user at a current time; extract fifth media information of the fifth item; filter target media information from the plurality of third media information, the target media information having a similarity less than a preset value with the fifth media information; recommend an item corresponding to the target media information to the user.

11. The apparatus of any one of claims 8 to 10, wherein, The fusion unit is specifically configured to: convert the first perspective information into a first matrix; convert the first media information into a first vector; multiply the first matrix and the first vector to obtain the first fusion representation of each first item.

12. The apparatus of any one of claims 8 to 11, wherein, The generation unit is specifically configured to: generate the first perspective information of the user in the first scene based on the plurality of first items by a Sparse MOE model.

13. The apparatus of any one of claims 8 to 12, wherein, The extraction unit is specifically configured to: extract the first media information of each first item and the second media information of each second item by a fine-grained interactive image-text pre-training model FILIP.

14. The apparatus of any one of claims 8 to 13, wherein, The first scene includes a short video scene, an image-text information scene, and a video scene.

15. A data processing apparatus, characterized by: It includes: a processor and a memory; the memory is configured to store instructions; the processor is configured to execute the instructions stored in the memory to implement the method in any one of claims 1 to 7.

16. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program, which when executed by one or more processors, implements the method of any one of claims 1 to 7.

17. A computer program product comprising instructions, characterized in that, The computer program product, when running on a computer, causes the computer to carry out the method of any one of claims 1 to 7.

Images