Information recommendation method and device, model training method and device, electronic equipment, computer readable storage medium and computer program product

Abstract

The invention provides an information recommendation method and device, a model training method and device, electronic equipment, a computer readable storage medium and a computer program product. The method comprises the steps of performing feature extraction on a first text to obtain a first text feature; performing feature decoupling on the first text feature to obtain a first semantic feature and a first language feature; based on the first language feature, performing attention adjustment on the first semantic feature to obtain a second semantic feature; according to the first text feature and a behavior feature of a corresponding user, determining a multi-modal feature of a to-be-recommended first object; and determining recommendation information based on the behavior characteristics, the second semantic characteristics and the multi-modal characteristics of the first object. According to the method and the device, the text content input by the user can be accurately understood, so that accurate recommendation information is pushed to the user.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to an information recommendation method, a model training method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the rapid development of information technology, personalized recommendations have become a crucial component in various fields, including e-commerce, online video, and social media. Providing users with accurate recommendations of products, works, and other information can enhance user experience and improve the platform's commercial profitability. However, due to language diversity, users may ask questions in different languages. Recommendation systems or models in related technologies are not well-suited to understanding questions entered in uncommon languages, which can easily lead to misunderstandings of user intent and inaccurate content being pushed. Summary of the Invention

[0003] The embodiments of the present application provide an information recommendation method, a model training method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product, which can accurately understand the text content input by the user, thereby pushing accurate recommendation information to the user.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] This embodiment of the present application provides an information recommendation method, the method comprising:

[0006] Perform feature extraction on a first text to obtain a first text feature; perform feature decoupling on the first text feature to obtain a first semantic feature and a first language feature; perform attention adjustment on the first semantic feature based on the first language feature to obtain a second semantic feature; determine a multimodal feature of a first object to be recommended based on the first text feature and a behavioral feature of a corresponding user; and determine recommendation information based on the behavioral feature, the second semantic feature, and the multimodal feature of the first object.

[0007] The present invention provides a model training method, which includes:

[0008] Feature extraction is performed on the first text sample to obtain a fourth text feature, and feature decoupling is performed on the fourth text feature to obtain a sixth semantic feature and a second language feature; based on the second language feature, attention adjustment is performed on the sixth semantic feature to obtain a seventh semantic feature; based on the fourth text feature and the behavioral characteristics of the corresponding user sample, the multimodal features of the third object to be recommended are determined; based on the behavioral characteristics of the user sample, the seventh semantic feature and the multimodal features of the third object, the first text sample is predicted to obtain predicted recommendation information; and based on the difference between the predicted recommendation information and the actual recommendation information of the first text sample, the model is trained.

[0009] The present invention provides an information recommendation device, including:

[0010] A first feature determination module is used to extract features from the first text to obtain first text features;

[0011] a first decoupling module, configured to perform feature decoupling on the first text feature to obtain a first semantic feature and a first language feature;

[0012] a first adjustment module, configured to perform attention adjustment on the first semantic feature based on the first language feature to obtain a second semantic feature;

[0013] a second feature determination module, configured to determine a multimodal feature of a first object to be recommended based on the first text feature and a corresponding user's behavioral feature;

[0014] An information determination module is used to determine recommended information based on the behavioral feature, the second semantic feature and the multimodal feature of the first object.

[0015] The present invention provides a model training device, comprising:

[0016] a second decoupling module, configured to perform feature extraction on the first text sample to obtain a fourth text feature, and perform feature decoupling on the fourth text feature to obtain a sixth semantic feature and a second language feature;

[0017] a second adjustment module, configured to perform attention adjustment on the sixth semantic feature based on the second language feature to obtain a seventh semantic feature;

[0018] a third feature determination module, configured to determine a multimodal feature of a third object to be recommended based on the fourth text feature and a behavioral feature of the corresponding user sample;

[0019] an information prediction module, configured to predict the first text sample based on the behavioral characteristics of the user sample, the seventh semantic feature, and the multimodal features of the third object, to obtain predicted recommendation information;

[0020] A training module is used to train a model based on the difference between the predicted recommendation information and the actual recommendation information of the first text sample.

[0021] An embodiment of the present application provides an electronic device, comprising:

[0022] a memory for storing computer-executable instructions or computer programs;

[0023] The processor is used to implement the information recommendation method or model training method provided in the embodiment of the present application when executing the computer-executable instructions or computer programs stored in the memory.

[0024] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions, which is used to implement the information recommendation method or model training method provided in the embodiment of the present application when executed by a processor.

[0025] An embodiment of the present application provides a computer program product, including a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, the information recommendation method or model training method provided in the embodiment of the present application is implemented.

[0026] The embodiments of the present application have the following beneficial effects:

[0027] Through the above embodiment, when recommending information to a user, feature extraction can be performed on the first text input by the user to determine the first text feature of the first text. In order to reduce the constraints of a specific language, the first semantic feature and the first language feature in the first text feature can be decoupled to form an independent feature representation, and the attention of the first semantic feature can be adjusted by the first language feature to obtain a second semantic feature with universal semantic information. Then, based on the first text feature and the behavioral characteristics of the corresponding user, the multimodal features of the first object to be recommended are determined, and the behavioral features, the second semantic features and the multimodal features of the first object are used to determine the recommended information for the user. This method can more effectively align semantic representations when processing tasks of different language types, without being restricted by a specific language type, accurately understand the semantic information in the text of a specific language type (that is, understand user needs), and use the multimodal features of the first object to more accurately understand the characteristics of the first object, thereby obtaining more accurate recommendation information to push to the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a structural diagram of the information recommendation system architecture provided by an embodiment of the present application;

[0029] Figures 2A-2B is a structural diagram of an electronic device provided in an embodiment of the present application;

[0030] Figure 3A Schematic diagram of the flow of the information recommendation method provided in the embodiment of the present application;

[0031] Figure 3B 1 is a flow chart of a first text feature determination method provided in an embodiment of the present application;

[0032] Figure 3C 1 is a flow chart of a method for determining a first language feature and a first semantic feature provided in an embodiment of the present application;

[0033] Figure 3D is a flowchart of a second semantic feature determination method provided in an embodiment of the present application;

[0034] Figure 3E Schematic diagram of the flow of the method for determining recommendation information provided in an embodiment of the present application;

[0035] Figure 4 is a schematic diagram of an information recommendation model provided in an embodiment of the present application;

[0036] Figure 5A This is a first flow chart of the model training method provided in an embodiment of the present application;

[0037] Figure 5B This is a second flow chart of the model training method provided in an embodiment of the present application.

[0038] It should be pointed out that the above-mentioned "first" and "second" are only used to distinguish different solutions, and do not represent the degree of distinction between the advantages and disadvantages of the solutions or the priority in the implementation process. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0040] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0041] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0042] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0043] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0044] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, and the informed consent or separate consent of the personal information subject should be obtained. Subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0045] Personalized recommendations have become a crucial component of various sectors, including e-commerce, online video, and social media. Providing users with precise recommendations on products, works, and other information improves user experience and the platform's commercial success. For example, in e-commerce platforms, users can enter questions, which then provide product recommendations based on the user's input. Another example is in intelligent telemarketing systems, where users can ask intelligent voice assistants questions to request recommendations. The intelligent voice assistants then respond to the user's voice input based on the user's input.

[0046] For example, in related art, there is an intelligent marketing customer service system and method based on an open intelligent model. The specific method includes: initializing user information and loading user business list information; collecting user input text in real time based on customer service conversations and saving chat history between the customer service system and the user; analyzing user intent through an open intelligent model, extracting key information using prompt templates and saving it to a business information list; matching knowledge base content based on user intent and key information, and extracting the first reply text from the matching content using an open intelligent model; traversing the vacant items in the business information list and generating the second reply text using an open intelligent model; splicing the first reply text and the second reply text, and using an open intelligent model to polish the language and logic of the text splicing result, and finally outputting feedback to the user. The solution in related art is low-cost and easy to migrate, achieving more efficient user interaction and saving enterprise labor costs.

[0047] However, with the advancement of globalization and the diversification of user needs, personalized recommendation systems and models face numerous challenges in multilingual processing and multimodal data integration. In particular, they struggle to understand questions in uncommon languages, which can easily lead to misunderstandings of user intent and inaccurate content. These language comprehension issues are often caused by a lack of training data for uncommon languages. This results in insufficient data support for training recommendation models, resulting in poor generalization and suboptimal performance on tasks in these uncommon languages.

[0048] While the aforementioned intelligent marketing customer service systems based on open intelligent models offer certain advantages in terms of low cost and ease of portability, their performance depends heavily on the scale and quality of pre-trained models. Insufficient model training data can result in poor response quality in certain areas or for specific user needs, making them incapable of meeting the needs of all business scenarios. While open intelligent models can analyze user intent and generate responses, they can err on the side of context in conversations conducted in uncommon languages, making it difficult to maintain coherence and accuracy.

[0049] The embodiments of the present application provide an information recommendation method, a model training method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product, which can accurately understand the text content input by a user, thereby pushing accurate recommendation information to the user.

[0050] See also Figure 1 , Figure 1This is a schematic diagram of the architecture of the information recommendation system 100 provided in an embodiment of the present application. In order to support an information recommendation application, the terminal (terminal 400-1 and terminal 400-2 are shown as examples) is connected to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0051] In some embodiments, a trained information recommendation model (hereinafter referred to as a model) is deployed in server 200, and terminals (e.g., terminal 400-1 and terminal 400-2) are configured with an application, mini-program, or webpage for recommending information to users. The terminals (e.g., terminal 400-1 and terminal 400-2) can display an interactive interface for information recommendation, and users can enter a first text requesting recommended information through the interactive interface displayed by the terminals (e.g., terminal 400-1 and terminal 400-2). In response to the information recommendation instruction sent by the user, the terminals (e.g., terminal 400-1 and terminal 400-2) obtain the first text entered by the user and send an information recommendation request carrying the first text to server 200. In response to the information recommendation request, server 200 extracts features from the first text to obtain first text features; performs feature decoupling on the first text features to obtain first semantic features and first language features; performs attention adjustment on the first semantic features based on the first language features to obtain second semantic features; determines multimodal features of the first object to be recommended based on the first text features and the corresponding user's behavioral features; and determines recommended information based on the behavioral features, the second semantic features, and the multimodal features of the first object. Subsequently, server 200 may transmit the determined recommended information to terminals (e.g., terminals 400-1 and 400-2), causing the terminals (e.g., terminals 400-1 and 400-2) to display the recommended information on an interactive interface.

[0052] In some embodiments, a trained information recommendation model may also be deployed in a terminal (e.g., terminal 400-1 and terminal 400-2), and an application, mini-program, or webpage configured to recommend information to a user may be configured. The terminal (e.g., terminal 400-1 and terminal 400-2) displays an interactive interface for information recommendation, and the user enters a first text requesting recommended information through the interactive interface of the terminal (e.g., terminal 400-1 and terminal 400-2). In response to the information recommendation instruction sent by the user, the terminal (e.g., terminal 400-1 and terminal 400-2) obtains the first text input by the user, performs feature extraction on the first text to obtain a first text feature, performs feature decoupling on the first text feature to obtain a first semantic feature and a first language feature, performs attention adjustment on the first semantic feature based on the first language feature to obtain a second semantic feature, determines the multimodal feature of the first object to be recommended based on the first text feature and the corresponding user's behavioral features, determines the recommended information based on the behavioral features, the second semantic feature, and the multimodal features of the first object, and displays the recommended information on the interactive interface.

[0053] In some embodiments, terminals (such as terminal 400-1 and terminal 400-2) can be implemented as various types of terminals such as laptops, tablet computers, desktop computers, set-top boxes, smart phones, smart speakers, smart watches, smart TVs, car terminals, etc., and can also be implemented as servers.

[0054] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal and the server may be connected directly or indirectly via wired or wireless communication, which is not limited in the embodiments of the present application.

[0055] See also Figures 2A-2B , Figures 2A-2B is a structural diagram of an electronic device 400 provided in an embodiment of the present application, Figures 2A-2B The electronic device 400 shown includes: at least one processor 410, a memory 450, at least one network interface 420 and a user interface 430. The various components in the electronic device 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to including a data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 440 is not shown in FIG. Figures 2A-2B Various buses are labeled as bus system 440 .

[0056] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0057] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0058] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0059] The memory 450 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0060] In some embodiments, the memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0061] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks;

[0062] A network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420. Exemplary network interfaces 420 include Bluetooth, Wi-Fi, and Universal Serial Bus (USB);

[0063] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., a display screen, a speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripheral devices and displaying content and information);

[0064] The input processing module 454 is configured to detect one or more user inputs or interactions from one of the one or more input devices 432 and to translate the detected inputs or interactions.

[0065] In some embodiments, the information recommendation device provided in the embodiments of the present application can be implemented in software. Figure 2A An information recommendation device 455A stored in memory 450 is shown. This device can be software in the form of a program or plug-in, and includes the following software modules: a first feature determination module 4551A, a first decoupling module 4552A, a first adjustment module 4553A, a second feature determination module 4554A, and an information determination module 4555A. These modules are logical and can be arbitrarily combined or further separated according to the functions they implement. The functions of each module will be described below.

[0066] In some embodiments, the model training device provided in the embodiments of the present application can be implemented in software. Figure 2B The model training device 455B stored in the memory 450 is shown. This device can be software in the form of a program or plug-in, and includes the following software modules: a second decoupling module 4551B, a second adjustment module 4552B, a third feature determination module 4553B, an information prediction module 4554B, and a training module 4555B. These modules are logical and can be arbitrarily combined or further separated according to the functions they implement. The functions of each module will be described below.

[0067] In other embodiments, the information recommendation device or model training device provided in the embodiments of the present application can be implemented in hardware. As an example, the information recommendation device or model training device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the information recommendation method or model training method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0068] In some embodiments, the terminal or server can implement the information recommendation method provided by the embodiment of the present application by running various computer executable instructions or computer programs. For example, computer executable instructions can be commands, machine instructions or software instructions at the microprogram level. The computer program can be a native program or software module in the operating system; it can be a local (Native) application (APPlication, APP), that is, a program that needs to be installed in the operating system to run, such as an information recommendation APP or an instant messaging APP; it can also be a small program that can be embedded in any APP, that is, a program that can be run only by downloading it to a browser environment. In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form.

[0069] Below, the information recommendation method provided by the embodiment of the present application will be described with reference to the accompanying drawings. As mentioned above, the electronic device that implements the information recommendation method of the embodiment of the present application can be a terminal (such as terminal 400-1 and terminal 400-2) or a server 200. In other words, the trained information recommendation model can be deployed in the terminal (such as terminal 400-1 and terminal 400-2) or in the server 200.

[0070] Below, the information recommendation method of the embodiment of the present application is described by first taking the execution subject as a terminal (such as terminal 400-1 and terminal 400-2) and deploying the trained information recommendation model on the terminal as an example. Figure 3A , Figure 3AThis is a flow chart of the information recommendation method provided by the embodiment of the present application, which will be combined with Figure 3A The steps shown are explained.

[0071] In step 101, feature extraction is performed on a first text to obtain first text features.

[0072] In actual implementation, the first text can be input by the user on an interactive interface. The user can enter the first text directly by typing text or by voice inputting the corresponding speech. After receiving the speech corresponding to the first text, the terminal performs speech recognition on the speech to obtain the first text. The first text represents the user's request for recommendation and can also be understood as the question text entered by the user based on the request. For example, the first text could be "I want to buy XX product. Please recommend me a good value XX product."

[0073] In actual implementation, since it is necessary to recommend information to users, referring to the user's own preference data can improve the accuracy of the recommended information. Therefore, after receiving the first text input by the user, the user's behavior data can be obtained and converted into behavioral features that can be processed by the information recommendation model.

[0074] For example, you can obtain user behavior data on a specified platform within a specified historical period, such as click data and browsing time data. Taking an e-commerce platform as an example, you can obtain product click data, browsing time data for each product, and product interaction records generated by users within a week before the current time as user behavior data.

[0075] In some embodiments, Figure 3B This is a flow chart of the first text feature determination method provided in the embodiment of the present application, see Figure 3B , “extracting features from the first text to obtain first text features” in step 101 can be implemented through steps 1011 to 1012 .

[0076] In step 1011, feature extraction is performed on the first text to obtain second text features.

[0077] In actual implementation, Figure 4 This is a schematic diagram of the information recommendation model provided in the embodiment of the present application, see Figure 4 After the first text is input into the information recommendation model, the first text can be feature extracted by the first processing layer of the third network (also called feature extraction network) of the information recommendation model to obtain the second text feature of the first text.

[0078] As an example, the first processing layer, which can also be called a feature extraction layer (i.e., input embedding layer), can contain an embedding matrix, where each row of the embedding matrix represents the embedding vector of a word or vocabulary. The dimension of the embedding matrix is ​​usually determined by the vocabulary size and the embedding dimension. The embedding matrix can be regarded as a lookup table. When the information recommendation model is running, it queries this table based on the index of the word or vocabulary contained in the input first text to obtain the embedding vector corresponding to the first text, that is, the second text feature.

[0079] In step 1012 , when the language type corresponding to the first text is the first language type, language analysis is performed on the second text feature based on the first language attribute of the first language type to obtain a first text feature with the first language attribute.

[0080] In actual implementation, continue to refer to Figure 4 The third network of the information recommendation model also includes at least two language understanding layers (e.g., a first language understanding layer and an nth language understanding layer), one for each language type, and each for performing language understanding on the second text features of the corresponding language type. Here, the second processing layer corresponding to the first language type (i.e., the language understanding layer corresponding to the first language type) can be used to perform language analysis (also referred to as language understanding) on ​​the second text features based on the first language attribute of the first language type, thereby obtaining first text features having the first language attribute.

[0081] Among them, the language parsing process can include performing semantic analysis on the second text feature based on the first language attribute, capturing the relationship between keywords, phrases, sentence structures, etc. in the vector sequence of the second text feature, and then understanding the relationship between words (such as keywords, phrases, sentences, etc.) and the deep meaning of the text, based on the context information of the words in the first text, linking the second text feature with the first language attribute, and performing contextual understanding of the first text based on the first language attribute to understand the intention or classification of the first text, thereby inputting the high-level feature representation of the second text feature to obtain the first text feature with the first language attribute.

[0082] As an example, if the third network includes two language understanding layers (the second processing layer and the third processing layer), the language type corresponding to the second processing layer is Chinese, and the language type corresponding to the third processing layer is English, when the first text is a text based on Chinese description, the second text feature of the first text can be language understood by the second processing layer to obtain the first text feature with Chinese language attributes.

[0083] Here, since the user can input the first text in various language types, and the semantic accuracy expressed by the second text features extracted by the input embedding layer (i.e., the first processing layer) for the first text through a general feature extraction method may be relatively weak, therefore, in order to improve the accuracy of understanding the first text, the corresponding language understanding layer can further perform language understanding based on the corresponding language attributes to obtain the first text features of the first text.

[0084] In actual implementation, the first language type may refer to an uncommon language type, that is, some low-resource language types with relatively less training data, such as a minority language that is not frequently used.

[0085] In actual implementation, after receiving the first text, the information recommendation model can identify the language type of the first text. When determining that the first text is a text described by the first language type, it determines the second processing layer corresponding to the first language type in the third network, and performs language analysis on the second text feature of the first text based on the first language attribute of the first language type through the second processing layer to obtain the first text feature.

[0086] Through the above method, since different language understanding layers are trained based on the training data of their corresponding language types, the language understanding layer can better understand and capture the grammatical structure, vocabulary usage and semantic details unique to the corresponding language, thereby more accurately understanding the contextual information and semantic changes in the first text, making the language understanding and semantic understanding of the first text more accurate.

[0087] Continue to see Figure 3A , continue with the above step 101 for explanation.

[0088] In step 102 , feature decoupling is performed on the first text feature to obtain a first semantic feature and a first language feature.

[0089] It's understandable that the first text features extracted by the second processing layer contain a mixture of language-related and semantic-related features of the first text. If the information recommendation model directly recommends information based on the first text features, it may be constrained by a specific language and thus inaccurately understand the user's needs. Therefore, by decoupling the first text features, the first semantic features and the first language features can be separated from the first text features to form independent feature representations. This allows the information recommendation model to understand universal semantic information and improve its semantic understanding capabilities.

[0090] In actual implementation, see Figure 4The first network in the information recommendation model (also called the semantic understanding network) can be set as a dual-stream encoder architecture, namely the semantic stream and the language stream. The semantic stream focuses on capturing general semantic features, and the language stream focuses on capturing the grammar, lexical and other language features of a specific language.

[0091] In some embodiments, Figure 3C This is a flow chart of the method for determining the first language feature and the first semantic feature provided in the embodiment of the present application, see Figure 3C , step 102 can be implemented through steps 1021 to 1023.

[0092] In step 1021 , semantic recognition is performed on the first text feature to obtain a first semantic feature of a semantic association in the first text feature.

[0093] It can be understood that the first text feature is a feature sequence, that is, a vector sequence, which includes feature vectors of words or single words in the first text.

[0094] In actual implementation, the first text feature can be semantically recognized by the encoder of the corresponding semantic stream. The process of semantic recognition may include using a self-attention mechanism to capture the internal dependencies within the sequence in the first text feature, so that the information recommendation model pays attention to different information at different positions in the sequence, thereby extracting the global contextual information of the first text feature and learning the semantic correlation between the contextual information.

[0095] Here, context information refers to the information carried by each element (usually a word or vocabulary) in the feature sequence, including the interactions and dependencies between it and the remaining elements in the global scope (i.e., the first text).

[0096] In practical implementation, the semantic stream encoder utilizes a multi-layered self-attention structure to extract deep contextual dependencies based on semantic associations between contextual information, thereby understanding the semantic information contained in the first text feature. Subsequently, multi-head attention is employed to capture semantic information from different perspectives of the first text feature based on the contextual dependencies, thereby extracting the first semantic feature. The first semantic feature is a general semantic feature represented in the first text.

[0097] In step 1022 , when the language type corresponding to the first text is the first language type, grammatical features and lexical features corresponding to the first language type are extracted from the first text features based on the language representation features of the first language type.

[0098] In actual implementation, the encoder of the corresponding language stream can pre-learn the language patterns, grammatical structures, vocabulary usage, etc. of each language type for different language types, so as to learn the language representation characteristics of each language type.

[0099] When performing language feature extraction through a language stream encoder, the language stream encoder may first determine the language type corresponding to the first text. When determining that the language type of the first text is the first language type, the encoder may extract grammatical features and lexical features corresponding to the first language type from the first text features based on the language representation features of the first language type. That is, the language stream encoder may extract grammatical features and lexical features from the first text features based on the language patterns, grammatical structures, lexical usage, and other content learned for the first language type.

[0100] In step 1023 , a first language feature is determined based on the grammatical feature and the lexical feature.

[0101] In actual implementation, the grammatical feature can represent the part-of-speech information of the corresponding first language type in the first text feature, such as nouns, verbs, adjectives, etc., for analyzing the part-of-speech distribution in the first text feature; it can also represent the syntactic information of the corresponding first language type in the first text feature, such as subject-verb-object structure, modifiers and other information; it can also represent the grammatical information of the corresponding first language type in the first text feature, such as tense, voice and other information; it can also represent the dependency relationship between each word in the first text feature to reveal the grammatical level of the sentence.

[0102] The lexical feature can represent the vocabulary frequency information in the first text feature, such as the unique vocabulary and expressions corresponding to the first language type; it can also represent the different roots and part-of-speech change information in the first text feature, such as prefix, suffix, and word ending changes; it can also represent the stop word information corresponding to the first language type in the first text feature.

[0103] In actual implementation, the first language features obtained by fusion of the extracted grammatical features and lexical features can reflect the unique information of the first language type.

[0104] Continue to see Figure 3A , continue with the above step 102 for description.

[0105] In step 103, attention adjustment is performed on the first semantic feature based on the first language feature to obtain a second semantic feature.

[0106] In actual implementation, a feature interaction layer can be configured after the dual-stream encoder of the first network. This layer coordinates the information interaction between the language stream and the semantic stream through a gating mechanism. During this interaction, a multi-head attention mechanism can be used to capture the interaction details between the first semantic feature and the first language feature. This information interaction can help the information recommendation model better understand the semantic content of the language.

[0107] In some embodiments, Figure 3DThis is a flow chart of the second semantic feature determination method provided in the embodiment of the present application, see Figure 3D , step 103 can be implemented through steps 1031 to 1033.

[0108] In step 1031, the first language feature is used as a query feature in the attention mechanism, and the first semantic feature is used as a key feature and a value feature in the attention mechanism.

[0109] The attention mechanism is a widely used technique in deep learning that allows models to selectively focus on important parts of information while ignoring less important information. This mechanism mimics the way humans focus their attention: when processing a task, they typically focus on the most critical information while ignoring other distracting information. In deep neural networks, the attention mechanism is often used in sequence-to-sequence (seq2seq) models, particularly in fields such as machine translation, speech recognition, and natural language processing. Its basic principle is to determine which parts are important by calculating the correlation between the query, key, and value.

[0110] Therefore, in actual implementation, the attention mechanism can be used to adjust the attention of the first semantic feature by adjusting the degree of attention of the first language feature to the first semantic feature. Through the mutual attention and mutual dependence between the first language feature and the first semantic feature, the second semantic feature obtained after adjustment can have the semantic information that the first language type pays attention to.

[0111] In actual implementation, the first language feature can be used as the query feature (Query feature) in the attention mechanism, and the first semantic feature can be used as the key feature (Key feature) and value feature (Value feature). At the same time, considering the degree of attention paid to different features, when the first language feature is determined as the query feature, a first weight can be set for the first language feature, and the product of the first language feature and the first weight can be used as the query feature; when the first semantic feature is determined as the key feature, a second weight can be set for the first semantic feature, and the product of the first semantic feature and the second weight can be used as the key feature; when the first semantic feature is determined as the value feature, a third weight can be set for the first semantic feature, and the product of the first semantic feature and the third weight can be used as the value feature.

[0112] As an example, assuming that the first language feature is Elang and the first semantic feature is Esemantic, then the query feature Q = Elang*W q , key feature K = Esmantic * W k , value feature V = Esmantic * W v , where W q Represents the first weight, Wk Represents the second weight, W v Represents the third weight.

[0113] In step 1032 , an attention weight between the first linguistic feature and the first semantic feature is determined based on a dot product operation of the query feature and the key feature.

[0114] In actual implementation, after performing a dot product operation on the query feature and the key feature, an attention feature that can reflect the degree of attention of the first language feature to the first semantic feature can be obtained. Here, since the first language feature and the first semantic feature are usually feature sequences, the dimension of the attention feature obtained after the dot product operation increases. Therefore, the attention feature can be scaled based on the feature dimension of the key feature to obtain the scaled attention feature, which can ensure the convergence of the information recommendation model.

[0115] Then, the scaled attention feature is normalized and the attention score in the attention feature is converted into a probability distribution. That is, the attention scores at all positions in the scaled attention feature are converted into a probability form so that the total probability is 1, thereby obtaining the attention weight between the first language feature and the first semantic feature.

[0116] In step 1033, a dot product operation is performed on the attention weight and the value feature to obtain a second semantic feature.

[0117] In actual implementation, when performing a dot product operation on the attention weight and the value feature, since the attention weight reflects the degree of attention paid by the first language feature to the first semantic feature, the probability distribution in the attention weight can influence the relative importance of each position in the value feature. The dot product operation of the attention weight and the value feature yields the second semantic feature. The feature vector corresponding to each output position in the second semantic feature is adjusted by the probability value of the corresponding position in the attention weight. This approach, based on the characteristics of the attention mechanism, can better capture the details of the interaction between language and semantics and learn the semantic information in text described in a specific language type.

[0118] In the above manner, the first text feature is separated through the first network, and the first semantic feature and the first language feature are dissociated from the first text feature to form an independent feature representation, so that the information recommendation model can understand the universal semantic information in the first text, thereby enabling the information recommendation model to more effectively align the representation of the encoder in the model when processing tasks of various language types, without being restricted or constrained by a specific language type, accurately understand and process texts of different language types, and improve the semantic understanding ability of the information recommendation model.

[0119] Continue to see Figure 3A , continue with the above step 103 for explanation.

[0120] In step 104 , the multimodal features of the first object to be recommended are determined based on the first text features and the behavioral features of the corresponding user.

[0121] In actual implementation, after determining the first text feature of the first text input by the user and obtaining the behavior feature of the user, multiple first objects that can be recommended to the user can be determined based on the first text feature and the behavior feature.

[0122] Here, the number of first objects is greater than or equal to two. The first object refers to a specific item or service to be pushed to the user in various information recommendation systems. The first object can be any type of goods or services, and its purpose is to filter out the needs or interests represented by the user's first text. In different application scenarios, the first object refers to different goods or services, as follows: in e-commerce systems, the first object usually refers to goods, such as books, clothing, electronic products, etc.; in content platforms, the first object may be articles, videos, music or blogs, etc.; in social networks, the first object may be friends, groups, activities or posts, etc.

[0123] In actual implementation, the process of determining the first object can be understood as a preliminary screening process, which is to preliminarily screen out some objects related to the demand information represented by the first text based on the first text features and the user's behavioral characteristics. Then, through step 105, the second object that more accurately meets the demand information represented by the first text is screened out from the first object and pushed to the user.

[0124] In actual implementation, corresponding multimodal features can be extracted for each object in advance. After the first object is determined, the multimodal features of each first object can be directly obtained. Alternatively, after the first object is determined, relevant introductory information of the first object can be obtained, and the multimodal features of the first object can be extracted based on the relevant introductory information. Here, the relevant introductory information of the first object can include textual introductory information, such as a textual introduction of the product; video introductory information, such as a product introduction video; audio introductory information, such as a product introduction audio; and image introductory information, such as a product poster.

[0125] Continue to see Figure 3A , continue with the above step 104 for description.

[0126] In step 105 , recommendation information is determined based on the behavioral feature, the second semantic feature, and the multimodal feature of the first object.

[0127] In actual implementation, the recommendation information can be related introductory information about the second object to be pushed to the user, as determined from the first object. The recommendation information can be presented in various forms, such as text, audio, video, and images. The specific form of presentation can be determined based on the method in which the user sends the first text, or based on the recommendation system or platform used by the user. For example, if the user sends the first text by inputting text into an e-commerce platform, the recommendation information can be displayed to the user on the e-commerce platform in the form of text, image, or video. If the user speaks the first text requesting recommendation information through an intelligent voice assistant, the recommendation information can be output to the user in the form of audio through the intelligent voice assistant.

[0128] In actual implementation, since the number of objects that can be recommended to users in each information recommendation system is relatively large, in order to improve the recommendation efficiency, a preliminary screening can be performed from all recommendable objects based on the user's behavioral characteristics to obtain a first object that meets the preset number. Then, based on the information recommendation method of the embodiment of the present application, a refined screening can be performed from the first object to obtain a more accurate second object, and the recommendation information of the second object is returned to the user.

[0129] In some embodiments, Figure 3E This is a flow chart of the method for determining recommendation information provided in the embodiment of the present application, see Figure 3E , step 105 can be implemented through steps 1051 to 1053.

[0130] In step 1051 , the behavioral feature is fused with the second semantic feature to obtain a first fused feature.

[0131] Continue to see Figure 4 The information recommendation model also includes a second network (also called an information recommendation network), which can be built using dense retrievers. Dense retrievers are a technology in the field of information retrieval, which is usually based on deep learning models and is used to find the most relevant content to the query in text retrieval tasks. Dense retrievers can use dense vector representations (usually high-dimensional embedding vectors of the first object and the first text) and evaluate relevance by calculating the similarity between these vectors.

[0132] In actual implementation, the second network can accurately understand the recommendation needs represented by the user's first text based on the second semantic features. After giving the user's behavioral characteristics as additional features to the second network, the second network can integrate behavioral information into the semantic representation by fusing the behavioral characteristics with the second semantic features, thereby better learning the user's real needs.

[0133] In actual implementation, the feature fusion method can be any of the following: both the behavioral feature and the second semantic feature can be converted into a unified feature space, and the behavioral feature and the first semantic feature can be fused in the converted feature space to obtain the first fused feature; or, the interaction information between the behavioral feature and the first semantic feature can be learned, for example, by using a multi-layer perceptron to learn the nonlinear relationship between the two features, thereby obtaining the first fused feature.

[0134] In step 1052 , the semantic similarity between the first fused feature and the multimodal feature of the first object is determined.

[0135] In some embodiments, the multimodal features of the first object include a third text feature, an image feature, and an audio feature; step 1052 can be implemented by: performing semantic feature extraction on the third text feature, the image feature, and the audio feature to obtain a third semantic feature of the third text feature, a fourth semantic feature of the image feature, and a fifth semantic feature of the audio feature; mapping the third semantic feature, the fourth semantic feature, the fifth semantic feature, and the first fusion feature to the same semantic space to obtain a first mapping feature of the third semantic feature, a second mapping feature of the fourth semantic feature, a third mapping feature of the fifth semantic feature, and a fourth mapping feature of the first fusion feature; performing feature fusion on the first mapping feature, the second mapping feature, and the third mapping feature to obtain a second fusion feature of the first object; determining the distance between the fourth mapping feature and the second fusion feature in the semantic space, and determining the distance as the semantic similarity between the first fusion feature and the multimodal features of the first object.

[0136] In actual implementation, the multimodal features of the first object may include a third text feature, an image feature, and an audio feature; wherein the third text feature may be obtained by extracting features from the text description of the first object based on a pre-trained text extraction network, the image feature may be obtained by extracting features from the appearance image of the first object based on a pre-trained audio extraction network, and the audio feature may be obtained by extracting features from the audio corresponding to the first object based on a pre-trained audio extraction network. Depending on the actual situation, the multimodal features of the first object may also include a video feature, and the video feature may be obtained by extracting features from the video corresponding to the first object based on a pre-trained video extraction network.

[0137] In actual implementation, due to the large differences in the distribution of data features of different modalities such as text, audio, and images, it is necessary to effectively fuse the information between the features of different modalities in a unified semantic space. When fusing the information between features of multiple modalities, it is usually necessary to pay attention to the semantic information therein. Therefore, semantic features can be extracted from the third text feature, image feature, and audio feature respectively to obtain the third semantic feature of the third text feature, the fourth semantic feature of the image feature, and the fifth semantic feature of the audio feature. Then, the semantic features of the various modal features can be used for subsequent feature processing.

[0138] In actual implementation, the method for extracting semantic features from the third text feature, image feature, and audio feature can be as follows: feature separation is performed on the third text feature, image feature, and audio feature, and the semantic information and corresponding modality-specific information in each feature are separated to obtain the third semantic feature and text modality feature of the third text feature, the fourth semantic feature and image modality feature of the image feature, and the fifth semantic feature and audio modality feature of the audio feature. In order to reduce the difference in feature distribution between different modalities, only the separated semantic features (i.e., the third semantic feature, the fourth semantic feature, and the fifth semantic feature) are used here and each semantic feature is mapped to a unified semantic space.

[0139] It is understandable that in a unified semantic space, feature points of information with the same or similar meanings but different modalities should be close to each other, and the distance between points in the semantic space can represent the semantic similarity between them. The closer the distance, the more similar the semantics, and the farther the distance, the greater the semantic difference. For example, the text representation, pronunciation, and image representation of the word "apple" should be close to each other in this space. Therefore, extracting semantic features from the features of each modality and mapping them to a unified semantic space can minimize dependence on specific modal features and achieve cross-modal generalization capabilities. It can enable the second network to capture modality-independent and higher-level semantic information in a unified semantic space, and in a unified semantic space, the semantic features of each modality are converted into a common framework, so that multimodal data fusion can be achieved more easily.

[0140] In actual implementation, since it is necessary to filter the first object based on the first fused feature generated for the user, in order to better compare the semantic information, the first fused feature and the semantic features of the multimodal feature can also be mapped to a unified semantic space. In other words, the third semantic feature, the fourth semantic feature, the fifth semantic feature, and the first fused feature can be mapped to the same semantic space, resulting in a first mapping feature of the third semantic feature, a second mapping feature of the fourth semantic feature, a third mapping feature of the fifth semantic feature, and a fourth mapping feature of the first fused feature.

[0141] Then, the first mapping feature, the second mapping feature, and the third mapping feature are fused to obtain a second fused feature of the information to be recommended. The second fused feature can reflect higher-level semantic information of the first object.

[0142] Since the distance between points in the semantic space can represent the semantic similarity between them, the distance between the first fused feature and the second fused feature of each first object in the semantic space can be calculated, and the distance between the first fused feature and each second fused feature can be used as the semantic similarity between the two.

[0143] In step 1053 , based on the semantic similarity, a second object that matches the first text is screened out from the first object, and recommendation information of the second object is output.

[0144] In actual implementation, a semantic relevance threshold can be preset based on actual screening needs, and the recommended objects in the first object whose semantic similarity is higher than the preset semantic relevance threshold can be used as the second objects that conform to the first text; the screening number can also be preset, for example 3, and the 3 recommended objects with the highest semantic similarity ranking in the first object can be used as the second objects that conform to the first text.

[0145] Through the above embodiment, when recommending information to a user, feature extraction can be performed on the first text input by the user to determine the first text feature of the first text. In order to reduce the constraints of a specific language, the first semantic feature and the first language feature in the first text feature can be decoupled to form an independent feature representation, and the attention of the first semantic feature can be adjusted by the first language feature to obtain a second semantic feature with universal semantic information. Then, based on the first text feature and the behavioral characteristics of the corresponding user, the multimodal features of the first object to be recommended are determined, and the behavioral features, the second semantic features and the multimodal features of the first object are used to determine the recommended information for the user. This method can more effectively align semantic representations when processing tasks of different language types, without being restricted by a specific language type, accurately understand the semantic information in the text of a specific language type (that is, understand user needs), and use the multimodal features of the first object to more accurately understand the characteristics of the first object, thereby obtaining more accurate recommendation information to push to the user.

[0146] Below, the model training method of the embodiment of the present application is first described by taking the execution subject as a terminal (such as terminal 400-1 and terminal 400-2) and training the information recommendation model at the terminal as an example. Figure 5A , Figure 5A This is a first flow chart of the model training method provided in the embodiment of the present application, which will be combined with Figure 5A The steps shown are explained.

[0147] In step 201 , feature extraction is performed on the first text sample to obtain a fourth text feature, and feature decoupling is performed on the fourth text feature to obtain a sixth semantic feature and a second language feature.

[0148] In actual implementation, a training sample may be constructed first. The training sample includes a user sample and a first text sample input by the user sample.

[0149] In actual implementation, corresponding sample data can be constructed for different user samples. For example, sample data corresponding to user A can be constructed for user A, and sample data corresponding to user B can be constructed for user B. The sample data can include a first text sample corresponding to the user sample. The first text sample can be a text described in a first language type. The first text sample is annotated with a first label. The first recommendation information represented by the first label is recommendation information determined based on the recommendation requirement represented by the first text sample, that is, recommendation information related to the first text sample. The first text sample is also annotated with a second label. The second recommendation information represented by the second label is information that is irrelevant to the recommendation requirement represented by the first text sample, that is, the second recommendation information is recommendation information that does not meet the recommendation requirement.

[0150] As an example, the first text sample may include multiple (i.e., at least two groups) text pairs described by the first language type, each text pair including a "question text" for expressing a recommendation requirement and an "answer text" annotated with the corresponding "question text"; each group of text pairs includes two different "answer texts" for the same "question text", wherein the "answer text" in one text pair is the first recommendation information that meets the recommendation requirement of the "question text", and the "answer text" in the other text pair is the second recommendation information that is unrelated to the "question text".

[0151] As an example, a first text sample for a user sample and the tagging process for the first text sample can be implemented as follows: historical conversation data for the user sample is obtained, and the historical conversation data is integrated into a "question-answer" format using a large language model. Here, the "question" is the "question text" in a text pair, and the "answer" is the initial recommendation information for the "question text." Conversational text data from a historical period is obtained, and the conversational text data is integrated into different conversation segments based on the conversation content. For each "question text" of the user sample, semantic screening is performed from several conversation segments in the conversational text data based on the "question text" and the initial recommendation information corresponding to the "question text." The probability of each conversation segment being an accurate recommendation information for the "question text" is obtained. The conversation segment with the highest probability is then integrated with the initial recommendation information for the "question text" to obtain first recommendation information for the "question text," and the first recommendation information is annotated to the "question text" with a first label. The remaining conversation segments, except for the conversation segment with the highest probability, are selected as second recommendation information for the "question text," and the second information is annotated to the "question text" with a second label. In this way, several text pairs are obtained for the user sample, thereby obtaining the first text sample of the user sample. It should be noted that each text in the first text sample is a text described based on the first language type.

[0152] In actual implementation, training samples may also include behavioral data of user samples within a specified historical period, such as click data, browsing data, and product interaction data of user samples on the e-commerce platform. Training samples may also include multimodal data of the first object. For example, if the first object is a product on the e-commerce platform, the multimodal data of the first object may include text description information of the first object, image data of the first object, and audio data of the first object.

[0153] In some embodiments, the information recommendation model includes a third network, the third network including a first processing layer, a second processing layer corresponding to a first language type, and a third processing layer corresponding to a second language type; the first text sample is a text described based on the first language type corresponding to the second processing layer. Before decoupling the fourth text feature, the fourth text feature of the first text sample can be determined by: performing feature extraction on the first text sample using the first processing layer to obtain a fifth text feature of the first text sample; determining a first parameter obtained by training the second processing layer based on the third processing layer, and performing language parsing on the fifth text feature based on a first language attribute of the first language type using a second processing layer having the first parameter to obtain a fourth text feature having the first language attribute.

[0154] In actual implementation, continue to refer to Figure 4The third network of the information recommendation model includes at least two language understanding layers. Different language understanding layers correspond to different language types. Each language understanding layer can be trained based on the training samples of the corresponding language type, so that the language understanding layer can accurately understand the text content of the corresponding language type. However, since some language types are not commonly used languages ​​of users, the volume of training data for this language type is small, resulting in poor training effect of the language understanding layer corresponding to the language type. Therefore, the language understanding layer of uncommon language types can be trained through cross-language transfer learning. It can be understood that the above-mentioned second processing layer and third processing layer are both language understanding layers, the second processing layer is the language understanding layer corresponding to the first language type, and the third processing layer is the language understanding layer corresponding to the second language type.

[0155] Cross-lingual transfer learning refers to the process of transferring knowledge or parameters learned in one or more languages ​​and applying them to other languages. The source language refers to the language used during initial training and typically has relatively rich training data; the target language is the language to which the model is intended to be applied and may lack sufficient training data. For the language understanding layer, the knowledge learned by the language understanding layer corresponding to the source language (such as language patterns, vocabulary representation, etc.) can be transferred to the language understanding layer corresponding to the target language, allowing the language understanding layer corresponding to the target language to perform well in the target language as well.

[0156] Here, the first language type is used to characterize an uncommon language type, that is, the target language, and the second language type is used to characterize a language type with rich data, that is, the source language. Therefore, for the second processing layer, it is necessary to transfer the knowledge learned by the language understanding layer (that is, the third processing layer) corresponding to the source language (that is, the second language type) to the second processing layer to realize the training of the second processing layer and obtain the first parameter of the second processing layer. The first parameter is the model parameter of the second processing layer after the training is completed. The second processing layer based on the first parameter can perform language analysis on the fifth text feature of the first text sample based on the first language attribute of the first language type to obtain the fourth text feature of the first text sample with the first language attribute. Among them, the process of language analysis can refer to the relevant description in the above embodiment, which will not be repeated here.

[0157] In actual implementation, the first processing layer can first perform feature extraction on the question text in the first text sample to obtain the fifth text feature of each question text. Then, based on the second processing layer using the first parameter, the fifth text feature of each question text is subjected to language analysis to obtain the fourth text feature of each question text. The feature extraction process of the first processing layer can be referred to the relevant description in the aforementioned embodiment and will not be repeated here.

[0158] Through the above-mentioned cross-language transfer learning method, data from high-resource languages ​​(i.e., source languages) can be used to improve the performance of the language understanding layer of low-resource languages ​​(i.e., target languages). Information recommendation models can learn more general language features, thereby showing good performance even in languages ​​with little or no annotated data. In addition, using data from high-resource languages ​​to assist low-resource language tasks can reduce reliance on large amounts of low-resource language data.

[0159] In some embodiments, the first parameter obtained by training the second processing layer based on the third processing layer can be determined in the following manner: training the third processing layer based on the second text sample to obtain the second parameter of the third processing layer; determining the third text sample in the second text sample that corresponds to the semantics of the first text sample based on the semantic correspondence between the first language type and the second language type; training the second processing layer based on the second parameter of the third processing layer, the third text sample, and the first text sample to obtain the first parameter of the second processing layer.

[0160] Here, the training samples also include second text samples input by the user sample in a second language type. Specifically, the second text samples are texts described in the second language type. The number of second text samples is greater than the number of first text samples. The third network also includes a third processing layer corresponding to the second language type. The second language type can be considered the source language, for which there is abundant training data. The construction method for the second text samples can be referenced to the construction method for the first text samples and will not be further elaborated here.

[0161] In actual implementation, each language understanding layer may include an embedding table and a language header. The embedding table is used to store or represent the fifth text feature of the input text. The language header is used to perform language understanding on each feature in the fifth text feature and generate a probability value for each feature. The probability value is used to represent the accuracy of semantic understanding. Each language understanding layer shares the underlying parameters, while the top-level parameters of the language understanding layer can be adjusted based on different languages ​​or tasks.

[0162] The bottom-level parameters refer to the parameters shared by each language understanding layer in the model. These parameters constitute the core structure of the language understanding layer. In the structure of the multilingual understanding layer, the bottom-level parameters are common to all language understanding layers and are responsible for learning the common linguistic patterns and features between different languages. The top-level parameters refer to the parameters designed specifically for each language understanding layer. These parameters are located at the top level of the language understanding layer and are responsible for processing data of a specific language. The top-level parameters are fine-tuned based on the characteristics of a specific language, allowing each language understanding layer to better process the text features of the corresponding language type. These top-level parameters can learn the unique characteristics of each language, such as grammatical structure and vocabulary usage habits. By adjusting the top-level parameters of each language understanding layer, each language understanding layer can be made more accurate when processing text of a specific language type.

[0163] In actual implementation, after the information recommendation model is built, the first processing layer and each language understanding layer can be pre-trained using large-scale training data of multiple second language types, so that each language understanding layer can understand and represent the common features of different languages. Then, the third processing layer of each second language type (i.e., the language understanding layer of each second language type) is trained separately using training samples of each second language type. Finally, for the second processing layer corresponding to the first language type (i.e., the language understanding layer corresponding to the first language type) where training resources are relatively scarce, knowledge and parameter transfer are performed through transfer learning to complete the training of the second processing layer.

[0164] Specifically, for the third processing layer, the second text sample can be used to train the third processing layer to obtain the second parameters of the third processing layer. The second parameters here may include shared bottom-level parameters and top-level parameters of the third processing layer. Then, the semantic correspondence between the first language type and the second language type is obtained. The semantic correspondence can represent the correspondence between each word of the first language type and the second language type, such as text-text. For each word in the question text in the first text sample, based on the semantic correspondence, a semantically corresponding third text sample is determined from the second text sample. Then, the second processing layer is trained based on the third text sample, the first text sample and the second parameters of the third processing layer. After the training is completed, the first parameters of the second processing layer are obtained.

[0165] In some embodiments, the above-mentioned "training the second processing layer based on the second parameter of the third processing layer, the third text sample, and the first text sample to obtain the first parameter of the second processing layer" can be achieved in the following way: using the text features of the third text sample as the input features of the first text sample in the second processing layer; using the second parameter of the third processing layer as the initial parameter of the second processing layer, performing language understanding on the first text sample through the second processing layer to obtain the sixth text feature for the first text sample; determining the fourth loss, wherein the fourth loss is used to characterize the degree of alignment between the sixth text feature and the input feature; training the second processing layer based on the fourth loss to obtain the first parameter of the second processing layer.

[0166] Here, the text features of the third text sample are used as input features of the first text sample in the second processing layer, that is, the text features of the third text sample are stored in the embedding table of the second processing layer.

[0167] In actual implementation, since the semantic correspondence between words of different language types may be a one-to-many relationship, for example, assuming that the first language type is English and the second language type is Chinese, the third text sample determined for a word "part" in the first text sample may include multiple words such as "some", "part", and "fragment". Therefore, the text features of each word in the third text sample can be obtained by weighted averaging to obtain the text features of the third text sample, and the text features of the third text sample are used as the input features of the first text sample in the second processing layer. Here, the input features refer to the initial embedding of the embedding table of the first text sample in the second processing layer. Here, when determining the text features of the third text sample, the weight corresponding to the text features of each word can be determined based on semantic similarity. For example, the semantic similarity between "part" and "part" is higher than the semantic similarity between "some" and "Part", so the weight corresponding to the text feature of "part" is higher than the weight of the text feature of "some".

[0168] In actual implementation, using the second parameters of the third processing layer as the initial parameters of the second processing layer may include: during training, the second processing layer is based on the shared bottom-level parameters, and the top-level parameters of the third processing layer are used as the initial parameters of the top-level parameters of the second processing layer, so that the second processing layer performs language understanding on the input features based on the training knowledge of the third processing layer, thereby obtaining the sixth text features predicted for the first text sample. In this case, because the second processing layer's understanding of the first language type has not yet reached an ideal state, a fourth loss may be calculated based on the difference between the sixth text features and the input features, and the top-level parameters of the second processing layer may be fine-tuned based on the fourth loss to obtain the first parameters of the second processing layer.

[0169] As an example, the fourth loss is used to characterize the degree of alignment between the sixth text feature and the input feature. The fourth loss can be used to minimize the feature distance between the first language type and the second language type. Specifically, the third loss can be calculated using the following formula:

[0170]

[0171] Wherein, Esource represents the feature representation of the source language, which here refers to the text feature (i.e., input feature) of the third text sample described in the second language type; Etarget represents the feature representation of the target language, which here refers to the sixth text feature of the first text sample described in the first language type; L 特征对齐 Indicates the fourth loss.

[0172] In some embodiments, "feature decoupling the fourth text feature to obtain the sixth semantic feature and the second language feature" in step 201 can be achieved by performing feature decoupling on the fourth text feature of the first text sample through the first network in the model to obtain the sixth semantic feature and the second language feature.

[0173] In actual implementation, see Figure 4 The information recommendation model also includes a first network (also called a semantic understanding network). The first network can be a dual-stream encoder architecture, comprising a language stream and a semantic stream. The language stream encoder can be used to extract the fourth text feature of the first text sample to obtain the second language feature, and the semantic stream encoder can be used to extract the fourth text feature to obtain the sixth semantic feature. The feature separation process of the first network can be referred to the relevant explanation of the first network in the above embodiment, and will not be repeated here.

[0174] In some embodiments, after "obtaining the sixth semantic feature and the second language feature", the first network can be trained in the following manner: determining a first loss and a second loss, wherein the first loss is used to characterize the degree of information interaction between the sixth semantic feature and the second language feature, and the second loss is used to characterize the consistency between the sixth semantic feature and the second language feature; and training the first network based on the first loss and the second loss.

[0175] In actual implementation, the first loss is used to constrain the learning process to maintain the separation of semantic features and language features. The first loss can also be called constraint loss. The first loss can be used to maximize the independence between language features and semantic features, that is, to constrain the independence between the sixth semantic feature and the second language feature.

[0176] As an example, the first loss may be calculated based on the degree of information interaction between the sixth semantic feature and the second language feature. Specifically, the first loss may be calculated using the following formula:

[0177] L对抗损失 =max(0,δ-I(Elanguage;Esemantic))

[0178] Among them, L 对抗损失 Represents the first loss; δ represents a hyperparameter used to control the independence strength; I (Elanguage; Esemantic) represents the degree of interaction between the sixth semantic feature and the second language feature.

[0179] In actual implementation, in order to ensure the consistency of feature representation in the semantic stream between different languages, a second loss, namely the semantic consistency loss, can be introduced. By minimizing the representation differences of the same semantic content in different languages, semantic consistency is achieved, thereby promoting the learning of language knowledge.

[0180] As an example, the second loss is used to characterize the consistency between the sixth semantic feature and the second language feature. The second loss can be determined by calculating the cross entropy loss. Specifically, the second loss can be calculated by the following formula:

[0181]

[0182] Among them, L 语义一致 Represents the second loss; Esource(i) represents the i-th feature representation in the second language feature; Esemantic(i) represents the i-th feature representation in the sixth semantic feature; N is the number of feature representations; p(yi|Esource(i), Esemantic(i)) represents the label probability predicted by the first network, and yi represents the true semantic label.

[0183] In some cases, feature alignment loss can also be introduced to minimize the correlation between the semantic stream and the language stream to ensure their independence. The maximum mean difference feature alignment loss can be used to constrain the decoupling of the semantic stream and the language stream.

[0184] After determining the first loss and the second loss, weight coefficients may be assigned to the two losses respectively, and the network loss of the first network may be obtained by weighted summation. The weight coefficients may be used to control the contribution of each loss to the network loss.

[0185] Continue to see Figure 5A , continue with the above step 201 for explanation.

[0186] In step 202 , attention adjustment is performed on the sixth semantic feature based on the second language feature to obtain a seventh semantic feature.

[0187] The process of adjusting attention to the sixth semantic feature based on the second language feature may refer to the process of adjusting attention to the first semantic feature based on the first language feature in the above embodiment, and will not be repeated here.

[0188] Through the above method, through the training method of disentanglement learning, the first network can be trained to better separate semantic features and linguistic features, improve the accuracy of understanding subtle differences in language, and the semantic feature representation after disentanglement is clearer, which helps the model maintain its generalization ability when faced with new language styles or content.

[0189] Continue to see Figure 5A , continue with the above step 202 for description.

[0190] In step 203 , based on the fourth text feature and the behavioral feature of the corresponding user sample, the multimodal feature of the third object to be recommended is determined.

[0191] Here, the behavioral features of the user sample can be obtained by extracting features from the behavioral data of the user sample, and the multimodal features of the third object can be obtained by extracting features from the multimodal data of the third object (e.g., textual description information, audio, video, etc.). The process of determining the third object can be understood as a preliminary screening process, that is, based on the fourth text features and the behavioral features of the user sample, a preliminary screening of third objects related to the demand information represented by the first text sample is performed. The number of third objects is multiple, that is, greater than two.

[0192] Continue to see Figure 5A , continue with the above step 203 for explanation.

[0193] In step 204 , the first text sample is predicted based on the behavioral features of the user sample, the seventh semantic feature, and the multimodal features of the third object to obtain predicted recommendation information.

[0194] In some embodiments, Figure 4 The information recommendation model shown also includes a second network (also referred to as an information recommendation network). Step 204 can be implemented in the following manner: using the second network in the model, the first text sample is predicted based on the behavioral characteristics of the user sample, the seventh semantic feature, and the multimodal features of the third object.

[0195] In actual implementation, the second network can fuse the behavioral feature with the seventh semantic feature to obtain a third fused feature, and then map the third fused feature and the multimodal feature to a unified semantic space. The distance between the third fused feature and each multimodal feature in the semantic space is used to make a prediction to obtain the predicted recommendation information. Here, the processing process of the multimodal features and the process of mapping them to the semantic space can be referred to the relevant explanations in the above embodiment and will not be repeated here.

[0196] The second network is built by a dense searcher. The most important task of the dense searcher is to understand the recommendation requirements represented by the seventh semantic feature and accurately predict the predicted recommendation information for the first text sample based on the seventh semantic feature. Therefore, based on the first label and the second label annotated for the first text sample, the second network can be trained through comparative learning, so that the second network can align semantically related texts. Here, the first label is used to represent the first recommendation information related to the first text sample, and the second label is used to represent the second recommendation information unrelated to the first text sample. The first recommendation information is also the actual recommendation information.

[0197] In some embodiments, before "predicting the first text sample" in step 204, the second network can be trained in the following manner: determining a first distance between the seventh semantic feature and the semantic feature of the first recommendation information in the semantic space; determining a second distance between the seventh semantic feature and the semantic feature of the second recommendation information in the semantic space; determining a third loss based on the first distance and the second distance, and training the second network based on the third loss.

[0198] The second recommendation information is irrelevant recommendation information annotated for the first text sample, that is, recommendation information that does not meet the recommendation requirements represented by the first text sample.

[0199] In actual implementation, the first text sample and the first recommendation information can be used as positive samples, and the first text sample and the second recommendation information can be used as negative samples. When the first text sample is input, the first recommendation information and the second recommendation information are respectively spliced ​​with the first text sample in a specified format to form an input sequence of the first text sample.

[0200] As an example, the specified format may be set to "[CLS] question [SEP] recommendation information [SEP]", where the [CLS] tag is used to indicate the start of each input sequence, and the [SEP] tag is used to separate the question and the recommendation information.

[0201] As an example, the first text sample may include a positive sample text pair and a negative sample text pair.

[0202] Here, two positive sample text pairs are exemplified as follows:

[0203] Text pair 1:

[0204] Question: “What is machine learning?”

[0205] Answer (i.e., the first recommended information): "Machine learning is a branch of artificial intelligence that enables computers to learn from data."

[0206] Text pair 2:

[0207] Question: “What are the applications of machine learning?”

[0208] Answer (i.e., the first recommended information): "Machine learning is widely used in image recognition, natural language processing, and recommendation systems."

[0209] The following two negative sample text pairs are exemplified:

[0210] Text pair 3:

[0211] Question: “What is machine learning?”

[0212] Answer (i.e., the second recommended information): "Banana is a yellow fruit."

[0213] Text pair 4:

[0214] Question: "What is quantum computing?"

[0215] Answer (second recommended information): "Quantum computing uses quantum bits to perform calculations and has the potential to surpass classical computers."

[0216] In actual implementation, by performing comparative learning on the second network using the positive samples and negative samples, the second network can distinguish between the first recommendation information related to the first text sample and the second recommendation information unrelated to the first text sample.

[0217] In actual implementation, since the closer the distance between two features in the semantic space, the closer their semantics are, the third loss can be designed based on this principle to train the second network to shorten the distance between the question text in the first text sample and the first recommendation information of the positive sample in the semantic space, while increasing the distance between the question text and the second recommendation information of the negative sample.

[0218] As an example, a first distance between the feature representation of the seventh semantic feature and the semantic feature representation of the first recommendation information in the semantic space can be calculated, and a second distance between the feature representation of the seventh semantic feature and the semantic feature representation of the second recommendation information in the semantic space can be calculated. Then, a third loss is determined based on the first and second distances. Specifically, the third loss can be calculated using the following formula:

[0219] L(q'p+'p-)=max(0'd(q'p+)-d(q'p-)+α)

[0220] Wherein, L(q'p+'p-) represents the third loss; q represents the seventh semantic feature; p+ represents the semantic feature of the first recommendation information; p- represents the semantic feature of the second recommendation information; α is the preset boundary value used to control the distance difference between positive and negative samples.

[0221] After determining the third loss, the second network can be trained based on the third loss. Then, the trained second network is used to predict the first text sample based on the behavioral characteristics of the user sample, the seventh semantic feature, and the multimodal features of the third object to obtain predicted recommendation information.

[0222] Continue to see Figure 5A , continue with the above step 204 for description.

[0223] In step 205 , the model is trained based on the difference between the predicted recommendation information and the actual recommendation information of the first text sample.

[0224] Here, the actual recommended information is the first recommended information represented by the first tag.

[0225] In some embodiments, the information recommendation model includes a first network, a second network, and a third network; the first network corresponds to a first loss and a second loss, the second network corresponds to a third loss, and the third network corresponds to a fourth loss. Step 205 can be implemented by determining a fifth loss based on the difference between the predicted recommendation information and the actual recommendation information of the first text sample; and training the model based on the fifth loss, the first and second losses of the first network, the third loss of the second network, and the fourth loss of the third network.

[0226] It is understandable that during the training process of the information recommendation model, the first network, the second network, and the third network can be trained independently, or the entire model can be jointly trained by integrating the loss conditions of each network.

[0227] In actual implementation, during training, the information recommendation model can also determine the fifth loss based on the difference between the predicted recommendation information and the first recommendation information in the positive sample (that is, the actual recommendation information), and use the fifth loss to constrain the prediction accuracy of the information recommendation model.

[0228] In actual implementation, the first loss and the second loss can be weighted and summed to obtain the semantic understanding loss for the first network. Then, weight coefficients are assigned to the semantic understanding loss, the third loss, the fourth loss, and the fifth loss respectively. The weight coefficients are used to control their contribution to the total loss. Specifically, the total loss of the information recommendation model can be calculated using the following formula:

[0229] Ltotal=αL a +βL b +γL c +εL d

[0230] Among them, Ltotal represents the total loss; L a represents semantic understanding loss, α represents L aThe weight coefficient of L b represents the third loss, β represents L b The weight coefficient of L c represents the fourth loss, γ represents L c The weight coefficient of L d represents the fifth loss, ε represents L d The weight coefficient of .

[0231] Through the above embodiment, a joint training strategy is adopted to jointly train the cross-language learning task in the third network, the disentanglement learning task of the language stream and the semantic stream in the first network, and the contrastive learning task in the second network. This enables different networks and tasks in the model to share information and integrate their respective learning results, so that each network promotes each other during the training process. Moreover, by flexibly adjusting the weights and balances of different tasks, the relevance of different tasks can be fully utilized, and the performance and robustness of the model can be improved, so that the information recommendation model can more accurately understand the text content input by the user in actual application and push accurate recommendation information to the user.

[0232] In a specific embodiment, see Figure 5B , Figure 5B This is a second flow chart of the model training method provided in an embodiment of the present application. The model training process for information recommendation may include the following:

[0233] S301, data preparation.

[0234] Collect behavioral data of user samples and conversation data of user samples, and collect multimodal data of the first object (including image, text, audio and other data).

[0235] S302, data preprocessing.

[0236] Data cleaning: filtering out noise data in collected data, such as meaningless characters in text, background noise in audio, etc.; marking typos, incorrect translations, and irrelevant content in the data.

[0237] Data augmentation uses a pre-trained large language model to generate more conversation data with user samples, including synonym replacement, sentence translation, and context expansion.

[0238] S303, feature coding.

[0239] Feature encoding is performed on the collected multimodal data and behavioral data to obtain behavioral features of the user sample and multimodal features of the first object. A first text sample of the user sample and an annotation of the first text sample can also be constructed based on the conversation data of the user sample.

[0240] Exemplarily, a pre-trained input embedding layer is used to generate a vector representation of a text sample in language A. For example, the vector representation of the English text “How to upgrade my phone plan?” is [0.23, -0.11, 0.76, ..., 0.45]. The English vector is used to initialize the Chinese embedding. For example, the embedding vector of the Chinese text “How to upgrade my phone plan?” is initialized to [0.21, -0.09, 0.74, ..., 0.47].

[0241] For example, behavioral data is encoded to generate behavioral features [1, 0, 1, 0, 0, 1, ..., 0]; text in multimodal features is encoded to obtain text features; audio data is encoded to obtain audio features, etc.

[0242] S304, model selection and modeling.

[0243] Build as Figure 4 The information recommendation model shown.

[0244] S305, loss function design.

[0245] The fourth loss is designed for the third network in the information recommendation model, the first loss and the second loss are designed for the first network, the third loss is designed for the second network, and the fifth loss is designed for the information recommendation model.

[0246] S306, model training.

[0247] Based on the above Figure 5A The model training method shown is used to train the information recommendation model.

[0248] The following continues to describe the exemplary structure of the information recommendation device 455A provided in the embodiment of the present application implemented as a software module. In some embodiments, such as Figure 2A As shown, the software modules stored in the information recommendation device 455A of the memory 450 may include:

[0249] The first feature determination module 4551A is used to extract features from the first text to obtain first text features.

[0250] The first decoupling module 4552A is used to perform feature decoupling on the first text feature to obtain a first semantic feature and a first language feature.

[0251] The first adjustment module 4553A is used to adjust the attention of the first semantic feature based on the first language feature to obtain a second semantic feature.

[0252] The second feature determination module 4554A is used to determine the multimodal features of the first object to be recommended based on the first text features and the behavioral features of the corresponding user.

[0253] The information determination module 4555A is used to determine the recommended information based on the behavioral feature, the second semantic feature and the multimodal feature of the first object.

[0254] In some embodiments, the first feature determination module 4551A is also used to extract features from the first text to obtain second text features of the first text; when the language type corresponding to the first text is the first language type, the second text features are language parsed based on the first language attribute of the first language type to obtain first text features with the first language attribute.

[0255] In some embodiments, the first decoupling module 4552A is also used to perform semantic recognition on the first text feature to obtain a first semantic feature with semantic association in the first text feature; when the language type corresponding to the first text is the first language type, based on the language representation features of the first language type, the grammatical features and lexical features corresponding to the first language type are extracted from the first text feature; and the first language feature is determined based on the grammatical features and lexical features.

[0256] In some embodiments, the first adjustment module 4553A is further used to use the first language feature as the query feature in the attention mechanism, and the first semantic feature as the key feature and value feature in the attention mechanism; determine the attention weight between the first language feature and the first semantic feature based on the dot product operation of the query feature and the key feature; and perform a dot product operation on the attention weight and the value feature to obtain the second semantic feature.

[0257] In some embodiments, the information determination module 4555A is also used to fuse the behavioral feature with the second semantic feature to obtain a first fused feature; determine the semantic similarity between the first fused feature and the multimodal feature of the first object; based on the semantic similarity, filter out the second object that conforms to the first text from the first object, and output recommended information for the second object.

[0258] In some embodiments, the multimodal features of the first object include a third text feature, an image feature, and an audio feature; the information determination module 4555A is also used to extract semantic features from the third text feature, the image feature, and the audio feature to obtain a third semantic feature of the third text feature, a fourth semantic feature of the image feature, and a fifth semantic feature of the audio feature; map the third semantic feature, the fourth semantic feature, the fifth semantic feature, and the first fusion feature into the same semantic space to obtain a first mapping feature of the third semantic feature, a second mapping feature of the fourth semantic feature, a third mapping feature of the fifth semantic feature, and a fourth mapping feature of the first fusion feature; perform feature fusion on the first mapping feature, the second mapping feature, and the third mapping feature to obtain a second fusion feature of the first object; determine the distance between the fourth mapping feature and the second fusion feature in the semantic space, and determine the distance as the semantic similarity between the first fusion feature and the multimodal features of the first object.

[0259] The following continues to describe the exemplary structure of the model training device 455B provided in the embodiment of the present application as a software module. In some embodiments, such as Figure 2B As shown, the software modules stored in the model training device 455B of the memory 440 may include:

[0260] The second decoupling module 4551B is configured to perform feature extraction on the first text sample to obtain a fourth text feature, and perform feature decoupling on the fourth text feature to obtain a sixth semantic feature and a second language feature.

[0261] The second adjustment module 4552B is configured to perform attention adjustment on the sixth semantic feature based on the second language feature to obtain a seventh semantic feature.

[0262] The third feature determination module 4553B is used to determine the multimodal features of the third object to be recommended based on the fourth text feature and the behavioral features of the corresponding user sample.

[0263] The information prediction module 4554B is used to predict the first text sample based on the behavioral characteristics of the user sample, the seventh semantic feature and the multimodal features of the third object to obtain predicted recommendation information.

[0264] The training module 4555B is used to train the model based on the difference between the predicted recommendation information and the actual recommendation information of the first text sample.

[0265] In some embodiments, the model includes a first network, a second decoupling module 4551B, and is further used to perform feature decoupling on the fourth text feature of the first text sample through the first network in the model to obtain a sixth semantic feature and a second language feature; the model training device 455B also includes a first network training module, which is used to determine a first loss and a second loss after obtaining the sixth semantic feature and the second language feature, wherein the first loss is used to characterize the degree of information interaction between the sixth semantic feature and the second language feature, and the second loss is used to characterize the consistency between the sixth semantic feature and the second language feature; the first network is trained based on the first loss and the second loss.

[0266] In some embodiments, the model includes a second network, and the first text sample is annotated with a first label and a second label, the first label is used to represent first recommendation information related to the first text sample, and the second label is used to represent second recommendation information unrelated to the first text sample; the information prediction module 4554B is also used to predict the first text sample based on the behavioral characteristics of the user sample, the seventh semantic feature and the multimodal characteristics of the third object through the second network in the model; the model training device 455B also includes a second network training module, which is used to determine the first distance between the seventh semantic feature and the semantic feature of the first recommendation information in the semantic space before predicting the first text sample; determine the second distance between the seventh semantic feature and the semantic feature of the second recommendation information in the semantic space; determine the third loss based on the first distance and the second distance, and train the second network based on the third loss.

[0267] In some embodiments, the model includes a third network, which includes a first processing layer, a second processing layer, and a third processing layer; the first text sample is a text described based on the first language type corresponding to the second processing layer; the model training device 455B also includes a text feature determination module, which is used to extract features from the first text sample through the first processing layer to obtain a fifth text feature of the first text sample; determine the first parameter obtained by training the second processing layer based on the third processing layer, and perform language analysis on the fifth text feature based on the first language attribute of the first language type through the second processing layer having the first parameter to obtain a fourth text feature with the first language attribute.

[0268] In some embodiments, the training sample also includes a second text sample input by the user sample, and the second text sample is a text described in the second language type based on the third processing layer; the number of second text samples is greater than the number of first text samples; the third network also includes a third processing layer corresponding to the second language type; the text feature determination module is also used to train the third processing layer based on the second text sample to obtain a second parameter of the third processing layer; based on the semantic correspondence between the first language type and the second language type, determine a third text sample in the second text sample that corresponds to the semantics of the first text sample; train the second processing layer based on the second parameter of the third processing layer, the third text sample and the first text sample to obtain the first parameter of the second processing layer.

[0269] In some embodiments, the text feature determination module is further used to use the text features of the third text sample as the input features of the first text sample in the second processing layer; use the second parameters of the third processing layer as the initial parameters of the second processing layer, perform language understanding on the first text sample through the second processing layer, and obtain a sixth text feature for the first text sample; determine a fourth loss, wherein the fourth loss is used to characterize the degree of alignment between the sixth text feature and the input feature; train the second processing layer based on the fourth loss to obtain the first parameter of the second processing layer.

[0270] In some embodiments, the information recommendation model includes a first network, a second network, and a third network; the training module 4555B is further used to determine the fifth loss based on the difference between the predicted recommendation information and the actual recommendation information of the first text sample; and train the model based on the fifth loss, the first loss and the second loss of the first network, the third loss of the second network, and the fourth loss of the third network.

[0271] The present invention provides a computer program product comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the information recommendation method and / or model training method described in the present invention.

[0272] The embodiment of the present application provides a computer-readable storage medium in which computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the information recommendation method and / or model training method provided by the embodiment of the present application, for example, Figure 3A The information recommendation method shown.

[0273] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0274] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0275] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0276] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0277] In summary, through the above embodiment, when recommending information to a user, feature extraction can be performed on the first text input by the user to determine the first text feature of the first text. In order to reduce the constraints of a specific language, the first semantic feature and the first language feature in the first text feature can be decoupled to form an independent feature representation, and the attention of the first semantic feature can be adjusted by the first language feature to obtain a second semantic feature with universal semantic information. Then, based on the first text feature and the behavioral characteristics of the corresponding user, the multimodal features of the first object to be recommended are determined, and the behavioral features, the second semantic features and the multimodal features of the first object are used to determine the recommended information for the user. In this way, semantic representations can be more effectively aligned when processing tasks of different language types, without being restricted by a specific language type, and the semantic information in the text of a specific language type can be accurately understood (that is, user needs can be understood). The multimodal features of the first object can be used to more accurately understand the characteristics of the first object, thereby obtaining more accurate recommendation information to push to the user.

[0278] When training the model, a joint training strategy is adopted to jointly train the cross-language learning task in the third network, the disentanglement learning task of the language stream and semantic stream in the first network, and the contrastive learning task in the second network. This enables different networks and tasks in the model to share information and integrate their respective learning results, so that each network can promote each other during the training process. By flexibly adjusting the weights and balance of different tasks, the relevance of different tasks can be fully utilized, improving the performance and robustness of the model, so that the information recommendation model can more accurately understand the text content input by the user in actual application and push accurate recommendation information to the user.

[0279] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. An information recommendation method, characterized in that: The method comprises: Performing feature extraction on the first text to obtain first text features; Performing feature decoupling on the first text feature to obtain a first semantic feature and a first language feature; Based on the first language feature, performing attention adjustment on the first semantic feature to obtain a second semantic feature; Determining a multimodal feature of a first object to be recommended based on the first text feature and a corresponding user behavior feature; Recommendation information is determined based on the behavioral feature, the second semantic feature, and the multimodal feature of the first object.

2. The method according to claim 1, characterized in that The extracting features of the first text to obtain first text features includes: Performing feature extraction on the first text to obtain second text features; When the language type corresponding to the first text is the first language type, language analysis is performed on the second text feature based on the first language attribute of the first language type to obtain a first text feature having the first language attribute.

3. The method according to claim 1, characterized in that The step of performing feature decoupling on the first text feature to obtain a first semantic feature and a first language feature includes: Performing semantic recognition on the first text feature to obtain a first semantic feature semantically associated with the first text feature; When the language type corresponding to the first text is the first language type, extracting grammatical features and lexical features corresponding to the first language type from the first text features based on the language representation features of the first language type; The first language feature is determined based on the grammatical feature and the lexical feature.

4. The method according to claim 1, wherein The step of performing attention adjustment on the first semantic feature based on the first language feature to obtain a second semantic feature includes: Using the first language feature as a query feature in an attention mechanism, and using the first semantic feature as a key feature and a value feature in the attention mechanism; determining an attention weight between the first language feature and the first semantic feature based on a dot product operation of the query feature and the key feature; Perform a dot product operation on the attention weight and the value feature to obtain a second semantic feature.

5. The method according to claim 1, wherein The determining of recommendation information based on the behavioral feature, the second semantic feature, and the multimodal feature of the first object includes: Fusing the behavioral feature with the second semantic feature to obtain a first fused feature; determining a semantic similarity between the first fused feature and the multimodal feature of the first object; Based on the semantic similarity, a second object that matches the first text is screened out from the first object, and recommendation information of the second object is output.

6. The method according to claim 5, characterized in that The multimodal features of the first object include a third text feature, an image feature, and an audio feature; The determining of the semantic similarity between the first fusion feature and the multimodal feature of the first object includes: Performing semantic feature extraction on the third text feature, the image feature, and the audio feature to obtain a third semantic feature of the third text feature, a fourth semantic feature of the image feature, and a fifth semantic feature of the audio feature; Mapping the third semantic feature, the fourth semantic feature, the fifth semantic feature, and the first fusion feature into the same semantic space to obtain a first mapping feature of the third semantic feature, a second mapping feature of the fourth semantic feature, a third mapping feature of the fifth semantic feature, and a fourth mapping feature of the first fusion feature; performing feature fusion on the first mapping feature, the second mapping feature, and the third mapping feature to obtain a second fused feature of the first object; A distance between the fourth mapping feature and the second fused feature in the semantic space is determined, and the distance is determined as a semantic similarity between the first fused feature and the multimodal feature of the first object.

7. A model training method, characterized in that: The method comprises: Performing feature extraction on the first text sample to obtain a fourth text feature, and performing feature decoupling on the fourth text feature to obtain a sixth semantic feature and a second language feature; performing attention adjustment on the sixth semantic feature based on the second language feature to obtain a seventh semantic feature; Determining a multimodal feature of a third object to be recommended based on the fourth text feature and the behavioral feature of the corresponding user sample; Based on the behavioral characteristics of the user sample, the seventh semantic feature, and the multimodal features of the third object, predicting the first text sample to obtain predicted recommendation information; The model is trained based on the difference between the predicted recommendation information and the actual recommendation information of the first text sample.

8. The method according to claim 7, characterized in that The model includes a first network, Decoupling the fourth text feature to obtain a sixth semantic feature and a second language feature includes: Decoupling the fourth text feature of the first text sample by the first network in the model to obtain a sixth semantic feature and a second language feature; After obtaining the sixth semantic feature and the second language feature, the method further includes: determining a first loss and a second loss, wherein the first loss is used to represent a degree of information interaction between the sixth semantic feature and the second language feature, and the second loss is used to represent a consistency between the sixth semantic feature and the second language feature; The first network is trained based on the first loss and the second loss.

9. The method according to claim 7, characterized in that The model includes a second network, the first text sample is annotated with a first label and a second label, the first label is used to represent first recommendation information related to the first text sample, and the second label is used to represent second recommendation information unrelated to the first text sample; The predicting of the first text sample based on the behavioral feature of the user sample, the seventh semantic feature, and the multimodal feature of the third object includes: Predicting the first text sample based on the behavioral characteristics of the user sample, the seventh semantic feature, and the multimodal features of the third object through the second network in the model; Before predicting the first text sample, the method further includes: determining a first distance between the seventh semantic feature and the semantic feature of the first recommendation information in a semantic space; determining a second distance between the seventh semantic feature and the semantic feature of the second recommendation information in the semantic space; A third loss is determined based on the first distance and the second distance, and the second network is trained based on the third loss.

10. The method according to claim 7, characterized in that The model includes a third network, the third network includes a first processing layer, a second processing layer, and a third processing layer; the first text sample is a text described based on a first language type corresponding to the second processing layer; Before performing feature decoupling on the fourth text feature, the method includes: performing feature extraction on the first text sample through the first processing layer to obtain a fifth text feature of the first text sample; Determine a first parameter obtained by training the second processing layer based on the third processing layer, and perform language parsing on the fifth text feature based on the first language attribute of the first language type through the second processing layer having the first parameter to obtain the fourth text feature having the first language attribute.

11. The method according to claim 10, wherein determining the first parameter obtained by training the second processing layer based on the third processing layer comprises: Training the third processing layer based on the second text sample to obtain a second parameter of the third processing layer; The second text samples are texts described in a second language type based on the third processing layer, and the number of the second text samples is greater than the number of the first text samples; Determining, based on the semantic correspondence between the first language type and the second language type, a third text sample in the second text sample that semantically corresponds to the first text sample; The second processing layer is trained based on the second parameter of the third processing layer, the third text sample, and the first text sample to obtain the first parameter of the second processing layer.

12. The method according to claim 11, characterized in that The training of the second processing layer based on the second parameter of the third processing layer, the third text sample, and the first text sample to obtain the second parameter of the second processing layer includes: Using the text features of the third text sample as input features of the first text sample in the second processing layer; Using the second parameter of the third processing layer as the initial parameter of the second processing layer, performing language understanding on the first text sample through the second processing layer to obtain a sixth text feature for the first text sample; determining a fourth loss, wherein the fourth loss is used to characterize the degree of alignment between the sixth text feature and the input feature; The second processing layer is trained based on the fourth loss to obtain first parameters of the second processing layer.

13. The method according to claim 7, characterized in that The model includes a first network, a second network, and a third network; The training of the model based on the difference between the predicted recommendation information and the actual recommendation information of the first text sample includes: determining a fifth loss based on a difference between the predicted recommendation information and actual recommendation information of the first text sample; The model is trained based on the fifth loss, the first loss and the second loss of the first network, the third loss of the second network, and the fourth loss of the third network.

14. An electronic device, characterized in that: The electronic device comprises: a memory for storing computer-executable instructions or computer programs; A processor, configured to implement the method according to any one of claims 1 to 13 when executing computer-executable instructions or computer programs stored in the memory.

15. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the method according to any one of claims 1 to 13 is implemented.

16. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the method according to any one of claims 1 to 13 is implemented.

Images