Information processing method and device, equipment, storage medium and program product

Abstract

The invention provides an information processing method and device, equipment, a storage medium and a program product. The method comprises the following steps: extracting text information and image information from first user information of a first user and object information of candidate objects; generating first prompt information for prompting the text information, and generating second prompt information for prompting the image information; performing first coding processing on the text information and the second prompt information to obtain text features, and performing second coding processing on the image information and the first prompt information to obtain image features; and determining an evaluation index for recommending the candidate object to the first user based on the text feature and the image feature. Through the information recommendation method and device, the accuracy of information recommendation for the user can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an information processing method, device, equipment, storage medium and program product. Background Art

[0002] With the rapid development of deep learning in the fields of computer vision and natural language, deep learning models can understand the content of text and images, thereby realizing the extraction of multimodal information, and then recommending object information to users in a targeted manner. Summary of the invention

[0003] The embodiments of the present application provide an information processing method, apparatus, device, storage medium, and program product, which can improve the accuracy of information recommendation to users.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present application provides an information processing method, the method comprising:

[0006] extracting text information and image information from first user information of the first user and object information of the candidate object;

[0007] generating first prompt information for prompting the text information, and generating second prompt information for prompting the image information;

[0008] Performing a first encoding process on the text information and the second prompt information to obtain text features, and performing a second encoding process on the image information and the first prompt information to obtain image features;

[0009] Based on the text features and the image features, an evaluation index for recommending the candidate object to the first user is determined.

[0010] In the above solution, the first user information includes a user description text, and the first encoding process is performed on the text information and the second prompt information to obtain text features, including:

[0011] Extracting label information from the text information, and performing word segmentation processing on the label information, the user description information and the second prompt information to obtain a first word;

[0012] Performing a first embedding process on the first word to obtain a first embedding feature;

[0013] Performing a first attention processing on the first embedded feature to obtain a first attention feature;

[0014] Perform multi-scale feature encoding on the first attention feature to obtain the text feature.

[0015] In the above solution, the image information includes spatial information and time information, and the second encoding process is performed on the image information and the first prompt information to obtain image features, including:

[0016] Performing word segmentation processing on the first prompt information to obtain a second word;

[0017] performing a second embedding process on the second word to obtain a second embedding feature;

[0018] Extracting a time feature from the time information, and extracting a space feature from the space information;

[0019] The spatial feature, the temporal feature and the second embedded feature are fused to obtain the image feature.

[0020] In the above scheme, the multi-scale feature encoding of the first attention feature to obtain the text feature includes:

[0021] Performing multi-scale convolution processing on the first attention feature to obtain convolution features of multiple scales;

[0022] Perform pooling on the convolutional features of each scale to obtain the pooled features of each scale;

[0023] The pooled features of the multiple scales are fused to obtain the text feature.

[0024] In the above solution, the process of extracting the time feature from the time information includes multiple stages, and the process of extracting the time feature from the time information includes:

[0025] For each of the stages, the time information is convolved through N cascaded convolutional layers to obtain output features of the N convolutional layers, wherein N is a positive integer greater than 1, the input of the first convolutional layer is the time information, the input of the nth convolutional layer is the output feature of the n-1th convolutional layer, and 1<n≤N;

[0026] For each of the stages, aggregating the output features of the N convolutional layers to obtain the output features of each stage;

[0027] The output features of each of the stages are aggregated to obtain the time features.

[0028] In the above solution, the fusing of the spatial feature, the temporal feature and the second embedded feature to obtain the image feature includes:

[0029] Performing a second attention process on the spatial feature, the temporal feature, and the second embedded feature to obtain a second attention feature;

[0030] Perform a linear transformation on the second attention feature to obtain the image feature.

[0031] The present application provides an information processing device, including:

[0032] A data acquisition module, used to extract text information and image information from the first user information of the first user and the object information of the candidate object;

[0033] A data processing module, used for generating first prompt information for prompting the text information, and generating second prompt information for prompting the image information;

[0034] The data processing module is further used to perform a first encoding process on the text information and the second prompt information to obtain text features, and to perform a second encoding process on the image information and the first prompt information to obtain image features;

[0035] The data processing module is further used to determine an evaluation index for recommending the candidate object to the first user based on the text feature and the image feature.

[0036] An embodiment of the present application provides an electronic device, the electronic device comprising:

[0037] A memory for storing computer executable instructions;

[0038] The processor is used to implement the information processing method provided in the embodiment of the present application when executing the computer executable instructions stored in the memory.

[0039] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the information processing method provided in the embodiment of the present application when executed by a processor.

[0040] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, the information processing method provided by the embodiment of the present application is implemented.

[0041] The embodiments of the present application have the following beneficial effects:

[0042] By extracting text information and image information from user information and object information, and generating first prompt information for prompting text information, and second prompt information for prompting image information, and encoding using the first prompt information and the second prompt information in the encoding stage, it is achieved to guide the generation of text features of the focused image information, and to guide the generation of image features of the focused text information. By utilizing the intersection of two prompt information from different modalities, that is, making the first prompt information describing the text information participate in the encoding of the image information, and making the second prompt information describing the image information participate in the encoding of the text information, it is achieved to fully utilize the text information and the image information, thereby improving the accuracy of the evaluation index for recommending candidate objects to the first user based on the fusion features. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a structural diagram of the information processing system architecture provided by an embodiment of the present application;

[0044] Figure 2 It is a schematic diagram of the structure of the server provided in the embodiment of the present application;

[0045] Figure 3A It is a first flow chart of the information processing method provided in the embodiment of the present application;

[0046] Figure 3B is a second flow chart of the information processing method provided in an embodiment of the present application;

[0047] Figure 3C is a third flow chart of the information processing method provided in the embodiment of the present application;

[0048] Figure 3D is a fourth flow chart of the information processing method provided in an embodiment of the present application;

[0049] Figure 3E is a fifth flow chart of the information processing method provided in the embodiment of the present application;

[0050] Figure 3F is a sixth flow chart of the information processing method provided in an embodiment of the present application;

[0051] Figure 3G is a seventh flow chart of the information processing method provided in the embodiment of the present application;

[0052] Figure 3H is an eighth flow chart of the information processing method provided in the embodiment of the present application;

[0053] Fig. 3I is a ninth flow chart of the information processing method provided in an embodiment of the present application;

[0054] Figure 3Jis a tenth flow chart of the information processing method provided in an embodiment of the present application;

[0055] Figure 4 It is a flowchart of an optional architecture of the information processing method provided in an embodiment of the present application.

[0056] It should be pointed out that the above-mentioned "first" and "second" are only used to distinguish different solutions, and do not represent the degree of superiority or inferiority of the solutions or the priority in the implementation process. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0058] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0059] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0060] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0061] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0062] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant national laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0063] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0064] 1) User information refers to personal information, activity records and interaction data about the first user on the Internet, in applications or physical environments, such as the user's personal identification (ID), browsing history, click behavior, purchase history, preference settings and user description text.

[0065] 2) User description text is a summary text used to describe the interests, preferences or personal characteristics of the first user, which can be used to create user profiles to help websites, applications or other services provide personalized experiences. For example, the user description text can be expressed as "like electronic products and technology news."

[0066] 3) Object information refers to a collection of information related to a candidate object (such as an advertisement to be recommended, a movie to be recommended, etc.). Taking the candidate object as an advertisement to be recommended as an example, the object information may include the text of the advertisement content (for example, "the latest smartphone is released, with powerful functions and affordable prices"), the time of advertisement delivery, and advertisement image data (such as the cover image and advertisement video of the advertisement, etc.).

[0067] 4) Text information refers to text data extracted from user information and text information. For example, text information may include user description text and advertising content text. Text information may also include derived text such as keywords and entity words extracted from text data.

[0068] 5) Image information refers to the image data contained in the object information. Taking the candidate object as the advertisement to be recommended as an example, the image information may include the cover image of the advertisement, the advertisement video (including multiple continuous video frames), etc.

[0069] 6) Spatial information refers to the two-dimensional spatial information of image data in image information, that is, the information of pixels in the width and height dimensions of the image. The information of each pixel contains color or grayscale value. These pixels are arranged in a certain spatial structure to form an image. Among them, spatial structure refers to the arrangement and organization of pixels in the image.

[0070] 7) Time information refers to a spatial information sequence composed of multiple continuous video frames in the image information, and there is a time dependency between the multiple continuous video frames.

[0071] 8) Transformer refers to a time series model based on the self-attention mechanism. The encoder part can effectively encode time series information. Its processing ability of time series information is much better than that of the long short-term memory network (LSTM), and its speed is fast. It is widely used in natural language processing, computer vision, machine translation, speech recognition and other fields.

[0072] In related technologies, information processing methods based on deep learning usually rely on a large amount of labeled data for training. However, there is a data scarcity problem in actual training and application, which limits the performance of the model and reduces the accuracy of information recommendations to users.

[0073] In order to solve the above problems, the embodiments of the present application provide an information processing method, apparatus, device, computer-readable storage medium and computer program product, which can improve the accuracy of information recommendation to users.

[0074] The following describes an exemplary application of the device provided in the embodiments of the present application. The electronic device provided in the embodiments of the present application can be implemented as various types of terminal devices such as laptop computers, tablet computers, desktop computers, set-top boxes, smart phones, smart speakers, smart watches, smart TVs, and vehicle-mounted terminals, and can also be implemented as a server.

[0075] See also Figure 1 , Figure 1 is a structural diagram of the information processing system architecture provided by an embodiment of the present application, Figure 1 The server 100, the terminal device 200 and the network 300 are involved. The terminal device 200 is connected to the server 100 via the network 300, wherein the network 300 can be a wide area network or a local area network, or a combination of the two.

[0076] In some embodiments, the embodiments of the present application can be implemented by a server and a terminal device in collaboration. For example, the terminal device 200 sends the first user information of the first user and the object information of the candidate object to the server 100, and the server 100 obtains the evaluation index for recommending the candidate object to the first user through the information processing method provided in the embodiments of the present application, and sends the evaluation index to the terminal device 200.

[0077] In other embodiments, the embodiments of the present application can be implemented by a terminal device alone. The terminal device 200 sends a request to the server 100, and the server 100 receives the request and sends the information processing model used to perform the information processing method provided in the embodiments of the present application to the terminal device 200. The terminal device 200 receives the information processing model sent by the server and downloads it locally, and obtains the evaluation index for recommending the candidate object to the first user through the information processing model.

[0078] In some embodiments, the terminal device or server can implement the information processing method provided by the embodiment of the present application by running various computer executable instructions or computer programs. For example, the computer executable instructions can be commands, machine instructions or software instructions at the microprogram level. The computer program can be a native program or software module in the operating system. In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form, and the terminal device includes but is not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, etc.

[0079] See also Figure 2 , Figure 2 is a schematic diagram of the structure of the server provided in the embodiment of the present application, Figure 2 The server 100 shown includes: at least one processor 110, a memory 130 and at least one network interface 120. The various components in the server 100 are coupled together via a bus system 140. It is understood that the bus system 140 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 140 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 140 is not described in detail. Figure 2 Various buses are labeled as bus system 140 .

[0080] The processor 110 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0081] The memory 130 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 130 may optionally include one or more storage devices that are physically remote from the processor 110.

[0082] The memory 130 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 130 described in the embodiments of the present application is intended to include any suitable type of memory.

[0083] In some embodiments, memory 130 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0084] Operating system 131, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0085] A network communication module 132, used to reach other electronic devices via one or more (wired or wireless) network interfaces 120, exemplary network interfaces 120 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.;

[0086] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 The information processing device 133 stored in the memory 130 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: a data acquisition module 1331 and a data processing module 1332. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0087] In other embodiments, the device provided in the embodiments of the present application can be implemented in hardware. As an example, the device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the information processing method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application specific integrated circuits (Application Specific Integrated Circuit, ASIC), digital signal processors (Digital Signal Processor, DSP), programmable logic devices (Programmable Logic Device, PLD), complex programmable logic devices (Complex Programmable Logic Device, CPLD), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other electronic components.

[0088] The following will combine the exemplary application and implementation of the server provided in the embodiment of the present application, taking the server as the execution subject, to illustrate the information processing method provided in the embodiment of the present application. Figure 3A , Figure 3A is a first flow chart of the information processing method provided in the embodiment of the present application, which will be combined with Figure 3A The steps shown are explained.

[0089] In step 101, text information and image information are extracted from first user information of a first user and object information of a candidate object.

[0090] In some embodiments, the candidate object refers to an object to be recommended to the first user, and the first user is the target audience of the candidate object. For example, the candidate object may be an advertisement, movie recommendation information, news recommendation information, etc.

[0091] Taking the advertising recommendation scenario as an example, the first user refers to the target audience in the advertising recommendation system, that is, the individual who receives the advertising content. The user information of the first user may include user ID, browsing history, purchase history, preference settings, and user description text, etc. The candidate object refers to the product, service or information that the advertiser wants to promote. The object information of the candidate object may include advertising content (such as advertising ID, advertising text, advertising images, videos and other multimedia elements) and advertising delivery time and other information.

[0092] In some embodiments, the first user information includes a user description text, see Figure 3B , Figure 3A The step 101 shown can be implemented by following the steps 1011 to 1014, which are described in detail below.

[0093] In step 1011, the words in the user description text are masked to obtain a first text.

[0094] In some embodiments, the user description text is a summary text that indicates the interests, preferences or personal characteristics of the first user. The user description text is randomly masked according to a preset ratio to obtain the first text. For example, assuming that the user description text is "like electronic products and technology news", the first text can be expressed as "like electronic products and ** news".

[0095] In step 1012, the masked words in the first text are predicted to obtain predicted words.

[0096] In some embodiments, the masked words in the first text are predicted by a pre-trained text rewriting model to obtain predicted words, for example, the word with the highest prediction probability is selected as the predicted word, or a predicted word is randomly selected.

[0097] For example, taking the text rewriting model as Bidirectional Encoder Repre sentation from Transformers (BERT) as an example, the training of the text rewriting model can be achieved in the following way: predict the masked words through the Masked Language Modeling (MLM) pre-training task to obtain the predicted words; obtain the mask loss value (such as cosine similarity, etc.) between the predicted words and the masked words; update the parameters of the text rewriting model based on the mask loss value to obtain the updated text rewriting model.

[0098] Here, Masked Language Modeling (MLM) is a pre-training method for natural language processing tasks, commonly used in models such as Transformer. In MLM, some words in the input text are randomly selected and replaced with special masked words. The goal of MLM is to predict the masked words based on the context.

[0099] In step 1013, the masked words in the first text are replaced with the predicted words to obtain a second text.

[0100] Continuing with the above example, assuming that the user description text is "like electronic products and technology news", the first text (or reconstructed text) is expressed as "like electronic products and ** news", that is, the masked word is "technology", and the predicted word can be "modern technology". Then, the masked word in the first text is replaced with the predicted word, and the obtained second text is expressed as "like electronic products and modern technology news".

[0101] In step 1014, based on the second text, text information and image information are determined from the first user information and the object information.

[0102] In some embodiments, see Figure 3C , Figure 3B The step 1014 shown can be implemented by following the steps 10141 to 10142, which are described in detail below.

[0103] In step 10141, when the similarity between the user description text and the second text is greater than a similarity threshold, the user description text in the first user information is replaced with the second text to obtain the second user information.

[0104] In some embodiments, the similarity between the user description text and the second text is obtained (for example, calculating cosine similarity, Euclidean distance, etc.), and when the similarity between the user description text and the second text is greater than a similarity threshold (for example, 85%, where 0% indicates no similarity and 100% indicates exactly the same), the user description text in the user information is replaced with the second text to obtain second user information.

[0105] In other embodiments, when the similarity between the user description text and the second text is less than or equal to a similarity threshold, the masked words in the first text are re-predicted to obtain new predicted words, and the masked words in the first text are replaced with the new predicted words to obtain a new second text, until the similarity between the user description text and the new second text is greater than the similarity threshold.

[0106] In step 10142, text information and image information are determined from the second user information and the object information of the candidate object.

[0107] For example, the second user information may include the user's personal identification (user ID), browsing history, click behavior, and new user description text (corresponding to the second text), etc. Taking the candidate object as an advertisement to be recommended as an example, the object information may include, for example, the advertisement ID, advertisement content text, advertisement delivery time, and advertisement image data, etc.

[0108] For example, the text information determined from the second user information and the object information of the candidate object may include a user ID, a user description text, an advertisement ID, and an advertisement content text; the image information determined from the second user information and the object information of the candidate object may include advertisement image data (such as an advertisement cover image and an advertisement video, etc.); the embodiments of the present application do not limit the specific content included in the text information and the image information.

[0109] Among them, the modality of text information is different from the modality of image information, that is, text information belongs to text modality, and image information belongs to image modality.

[0110] Continue to see Figure 3A In step 102, first prompt information for prompting text information is generated, and second prompt information for prompting image information is generated.

[0111] In some embodiments, first prompt information is generated to prompt at least part of the text information, and second prompt information is generated to prompt at least part of the image information.

[0112] In some embodiments, see Figure 3D , Figure 3A The first prompt information of generating the prompt text information in step 102 shown can be implemented by following steps 1021 to 1022, which are described in detail below.

[0113] In step 1021, first key information is extracted from the text information.

[0114] In some embodiments, the first key information includes at least one keyword. At least one keyword is extracted from the text information through a keyword extraction algorithm, and the extracted keyword is used as the first key information. These keywords are usually words that tend to express the theme or main content of the text information.

[0115] Taking obtaining at least one keyword through a keyword extraction algorithm as an example, it can be achieved in the following ways: first, the text information is cleaned and preprocessed, such as removing irrelevant characters, word segmentation, stop words, part-of-speech tagging, etc., and then part-of-speech filtering is performed, and certain parts of speech are filtered out as needed, and only parts of speech with practical meanings such as nouns, verbs, and adjectives are retained. For example, adverbs, prepositions, conjunctions and other parts of speech are filtered out. Next, the target keyword extraction algorithm is determined, such as the term frequency-inverse document frequency statistical method (Term Frequency-Inverse Document Frequency, TF-IDF), a keyword extraction model based on an attention mechanism, etc. Finally, the preprocessed text information is processed using the selected target keyword extraction algorithm to identify at least one keyword. The keyword can be a word with high frequency and strong relevance in the text information automatically identified by the algorithm.

[0116] Taking the training of a keyword extraction model based on deep learning as an example, text samples and manually annotated keywords can be used as training data sets to build a deep learning model architecture that includes an attention mechanism, for example, including an encoder (such as Transformer Encoder, etc.) and a decoder (such as Transformer Decoder, etc.), where the encoder is used to process the input text data, and the decoder is used to generate keywords. Finally, the prepared data set is used to train the keyword extraction model. During the training process, the keyword extraction model will learn how to automatically focus on the key parts in the text samples and generate keywords based on these key parts. After the training is completed, the model can be used to extract keywords from new texts.

[0117] In other embodiments, each word in the text information can be embedded (Embedding) through a large language model (LLM) to obtain an embedding vector for each word, and the weight of each word is calculated based on the embedding vector of each word and the attention mechanism, so as to determine the keyword according to the weight (such as taking the word with a weight greater than a preset threshold as the keyword). The embodiments of the present application do not limit the specific method of keyword extraction.

[0118] In other embodiments, the first key information includes summary information (or summary text), and the summary text can be obtained in the following manner: through a summary sentence generation model (such as BERT), the text information is input through a text input layer, and then processed through token embeddings, interval segment embeddings, position embeddings and summarization layers to generate summary text corresponding to the text information, and the summary text is used as the first key information.

[0119] In other embodiments, the first key information includes entity information, and the acquisition of the entity information can be achieved in the following manner: performing entity recognition (Named Entity Recognition, NER) on the text information to obtain an entity recognition result, wherein the entity recognition result includes entity information in the text information, such as people, places, organizations, etc.

[0120] In step 1022, first prompt information is generated based on the first key information and a preset first information template.

[0121] In some embodiments, the extracted first key information is filled into the corresponding position of the first information template to generate first prompt information, and the first prompt information prompts the content in the text information, thereby providing a prompt for the processing of the image data (image information).

[0122] Continuing with the above examples, taking the example that the first key information includes at least one keyword, the first prompt information can be expressed as "Please pay attention to <keyword 1> and <keyword 2> in the image information"; taking the example that the first key information includes summary information, the first prompt information can be expressed as "Please pay attention to <summary information> in the image information"; taking the example that the first key information includes entity information, the first prompt information can be expressed as "Please pay attention to <entity 1> and <entity 2> in the image information". The embodiments of the present application do not limit the text composition method or specific style of the first information template.

[0123] In some embodiments, generating the second prompt information of the prompt image information can be achieved by: performing image-text conversion processing on the image information to obtain image description text; extracting the second key information from the image description text; generating the second prompt information based on the second key information and a preset second information template.

[0124] For example, the image information can be converted into text using a large language model (or image-to-text model) to obtain image description text, that is, the image information is used as the input of the large language model to generate the image description text.

[0125] Taking the image-to-text model Pix2Text as an example, the image-to-text conversion of image information to obtain image description text can be achieved in the following ways: first, the image information is preprocessed, such as grayscale, normalization, etc.; next, the preprocessed image information is subjected to feature extraction processing, such as by using a convolutional neural network (CNN) or other image processing technology to identify texture, shape, color and other features in the image information, and encode this information into a high-dimensional vector to obtain image features; next, the processed image features are converted into a vector of fixed length, which contains rich information of the image information and is used for subsequent image description text generation; finally, text mapping processing is performed based on the vector to obtain the image description text, such as mapping the vector through a long short-term memory network to generate a text sequence. After obtaining the text sequence, it can also be sorted and corrected to improve the accuracy of the image description text.

[0126] For example, suppose an image in the image information shows a smart phone with a high-definition interface displayed on the phone screen, a brand logo at the top of the screen, a phone model and a slogan of "powerful functions and affordable prices" at the bottom, and a pure white background. The output image description text may be: "An advertising image of a smart phone with a striking brand logo and a high-definition interface. The phone model and the slogan of "powerful functions and affordable prices" are located at the bottom of the screen, with a pure white background and a simple design style."

[0127] In some embodiments, the specific implementation of extracting the second key information from the image description text can refer to the description of step 1021 above, which will not be repeated here.

[0128] In some embodiments, the extracted second key information is filled into the corresponding position of the second information template to obtain second prompt information, and the second prompt information prompts the content in the image information, thereby providing prompts for the processing of text data (text information).

[0129] For example, taking the example that the second key information includes at least one keyword, the second prompt information can be expressed as "Please pay attention to <keyword 1> and <keyword 2> in the text information"; taking the example that the second key information includes summary information, the second prompt information can be expressed as "Please pay attention to <summary information> in the text information"; taking the example that the second key information includes entity information, the second prompt information can be expressed as "Please pay attention to <entity 1> and <entity 2> in the text information". The embodiments of the present application do not limit the text composition method or specific style of the second information template.

[0130] Continue to see Figure 3A In step 103, a first encoding process is performed on the text information and the second prompt information to obtain text features, and a second encoding process is performed on the image information and the first prompt information to obtain image features.

[0131] In some embodiments, the first user information includes a user description text, see Figure 3E , Figure 3A The step 103 in which the text information and the second prompt information are subjected to a first encoding process to obtain text features can be implemented by following the steps 1031 to 1034, which are described in detail below.

[0132] In step 1031, the tag information is extracted from the text information, and the tag information, the user description information and the second prompt information are segmented to obtain a first word.

[0133] In some embodiments, the text information includes a plurality of third texts, see Figure 3F , Figure 3EExtracting tag information from text information in step 1031 shown can be achieved through the following steps 10311 to 10314, which are described in detail below.

[0134] In step 10311, based on the weight of each word in the text information, a first keyword is determined from the words in the text information.

[0135] In some embodiments, each word in the text information can be embedded (Embedding) through a large language model (LLM) to obtain an embedding vector for each word, and the weight of each word is calculated based on the embedding vector of each word and the attention mechanism, so as to determine the first keyword according to the weight (such as taking a word with a weight greater than a preset threshold as the first keyword). The embodiments of the present application do not limit the specific method of extracting the first keyword.

[0136] In step 10312, the first keyword is classified to obtain multiple categories.

[0137] In some embodiments, the extracted first keywords may be clustered (eg, using a K-means clustering method) to obtain a plurality of clusters, each cluster corresponding to a category.

[0138] In step 10313, the probability that each third text belongs to each category is determined, and based on the probability, a fourth text is determined from the plurality of third texts.

[0139] In some embodiments, the first keyword is classified and the obtained multiple categories are used as the first category (see the description in step 10312); the text category (or second category) of each third text is determined; the similarity between the text category and each first category is obtained, and the similarity is used as the probability that the third text belongs to each first category.

[0140] For example, each third text is classified to obtain the text category of each third text, and the similarity between the text category of each third text (such as the advertising content text, user description text, etc. mentioned above) and each first category is determined by a similarity calculation method (such as cosine similarity, Euclidean distance, etc.), and the similarity is used as the probability that the third text belongs to each first category, and the third text with a probability higher than a preset threshold is selected as the fourth text.

[0141] In step 10314, a second keyword of the fourth text is determined, and a plurality of categories and the second keyword are determined as tag information.

[0142] In some embodiments, the second keyword of the fourth text may be obtained through a large language model (LLM), and multiple categories and the second keyword may be combined into tag information (or referred to as a tag space).

[0143] Here, the second keyword can be obtained by combining one or more long-tail keywords. For example, the long-tail keywords extracted from the fourth text include "lightweight hiking shoes" and "suitable for hiking". The second keyword obtained by combining the long-tail keywords can be expressed as "lightweight hiking shoes-suitable for hiking".

[0144] In some other embodiments, each word in the text information may be clustered first to obtain a clustering result, and then the first keyword may be extracted based on the clustering result, and the first keyword and the second keyword may be combined as tag information.

[0145] Through steps 10311 to 10314, the label information obtained can be used as an important reference to guide the subsequent extraction of text features. By further extracting text features from the text in the label information, the encoding process is made more accurate. That is, extracting label information can help the information processing model better understand and classify different types of text, especially for the recognition of long-tail tags (second keywords), which significantly improves the performance of the model on rare categories.

[0146] In some embodiments, punctuation marks such as spaces, periods, and commas can be used as segmentation marks to segment the label information, user description information, and the second prompt information to obtain the first word, that is, to divide the text into input units (toke n). Third-party segmentation tools can also be used for word segmentation, such as Jieba, NLTK, SpaCy, etc. These tools can achieve accurate Chinese and English word segmentation through algorithms and language models.

[0147] Continue to see Figure 3E In step 1032, a first embedding process is performed on the first word to obtain a first embedding feature.

[0148] In some embodiments, a first embedding process (or called word embedding process) can be performed on the first word through a word embedding model (such as Word2Vec, GloVe, etc.), each first word is represented as a vector of a fixed length, and word embedding features (sequence) are composed in order to obtain a first embedding feature.

[0149] In step 1033, a first attention processing is performed on the first embedded feature to obtain a first attention feature.

[0150] In some embodiments, based on the first embedded feature, an attention weight is determined; the first embedded feature is weighted by the attention weight to obtain the first attention feature.

[0151] For example, the first embedded feature is used as a query vector (Query, Q), a key vector (Key, K) and a value vector (Value, V) at the same time, and an attention score is obtained by calculating the dot product of Q and K. Next, the attention score is normalized, for example, by applying a normalization function (such as a softmax function) to convert the attention score into a probability distribution to serve as an attention weight.

[0152] For example, the first embedded feature is weighted using the attention weight to generate a new feature representation, which can be linearly transformed through a linear layer to obtain a rich representation containing the correlations at different positions in the first embedded feature, namely, the first attention feature.

[0153] For example, the first attention processing can be implemented through attention encoding methods such as self-attention mechanism (Self-Attention), multi-head self-attention mechanism (Multi-Head Self-Attention), etc. The embodiments of the present application do not limit the specific attention encoding method.

[0154] In step 1034, multi-scale feature encoding is performed on the first attention feature to obtain a text feature.

[0155] In some embodiments, see Figure 3G , Figure 3E Step 1034 shown can be implemented by following steps 10341 to 10343, which are described in detail below.

[0156] In step 10341, multi-scale convolution processing is performed on the first attention feature to obtain convolution features of multiple scales.

[0157] In some embodiments, multi-scale convolution processing can be performed on the first attention feature in parallel through multiple convolution layers with convolution kernels of different sizes to obtain convolution features of multiple scales.

[0158] For example, convolution kernels of different sizes can be 1×1, 3×3, and 5×5. Through these three sizes of convolution kernels, the first attention features are convolved in parallel on the three branches to obtain the convolution features corresponding to the three branches.

[0159] In step 10342, the convolutional features of each scale are pooled to obtain pooled features of each scale.

[0160] In some embodiments, the convolutional features of each scale can be pooled separately through a pooling layer to obtain pooled features. For example, pooling methods such as maximum pooling (Max Pooling) and average pooling (Average Pooling) can be used for pooling.

[0161] In step 10343, pooled features of multiple scales are fused to obtain text features.

[0162] In some embodiments, pooled features at multiple scales are concatenated to obtain text features.

[0163] In other embodiments, the pooling features of multiple scales may be fused by means of attention mechanism weighting to obtain text features.

[0164] In other embodiments, the first attention feature can be encoded with multi-scale features by constructing a pyramid network structure, introducing skip connections, etc., to obtain text features.

[0165] In some embodiments, the image information includes spatial information and temporal information, see Figure 3H , Figure 3A The second encoding process of the image information and the first prompt information in step 103 to obtain the image features can be implemented through the following steps 1035 to 1038, which are described in detail below.

[0166] In step 1035, the first prompt information is segmented to obtain a second word.

[0167] In some embodiments, punctuation marks such as spaces, periods, and commas can be used as segmentation marks to segment the first prompt information to obtain second words, that is, to divide the first prompt information into input units (tokens). Third-party segmentation tools can also be used to segment the first prompt information, such as Jieba, NLTK, SpaCy, etc. These tools can achieve accurate Chinese and English segmentation through algorithms and language models.

[0168] In step 1036, a second embedding process is performed on the second word to obtain a second embedding feature.

[0169] In some embodiments, a second embedding process (or called word embedding encoding process) can be performed on the second word through a word embedding model (such as Word2Vec, GloVe, etc.), each second word is represented as a vector of a fixed length, and word embedding features (sequence) are composed in order to obtain the second embedding feature.

[0170] In step 1037, a temporal feature is extracted from the temporal information, and a spatial feature is extracted from the spatial information.

[0171] In some embodiments, the process of extracting temporal features from temporal information includes multiple stages. Fig. 3I , Figure 3HThe extraction of time features from time information in step 1037 shown can be achieved by following steps 10371 to 10373, which are described in detail below.

[0172] In step 10371, for each stage, the time information is convolved through N cascaded convolutional layers to obtain the output features of N convolutional layers, where N is a positive integer greater than 1, the input of the first convolutional layer is the time information, the input of the nth convolutional layer is the output feature of the n-1th convolutional layer, and 1<n≤N.

[0173] In some embodiments, the process of extracting time features from time information includes multiple stages, each stage may include multiple cascaded convolutional layers, and the time features in the time information are extracted through multiple cascaded convolutional layers. Convolution processing of the time information can be achieved through a temporal convolutional neural network (TCN).

[0174] In other embodiments, before extracting the time features from the time information, the time information may be preprocessed to obtain preprocessed time information, and the above-mentioned step 10371 may be executed based on the preprocessed time information.

[0175] For example, the time information can be represented as a spatial information sequence composed of multiple consecutive video frames. The spatial information refers to the two-dimensional spatial information of the video frame, that is, the information of the pixel points in the width and height dimensions of the video frame. The time information can be preprocessed by performing the following processing on each spatial information in the spatial information sequence: performing convolution processing on the spatial information to obtain convolution features; dividing the convolution features into multiple feature blocks, and arranging the multiple feature blocks into a feature block sequence; performing embedded coding processing on the feature block sequence to obtain preprocessed spatial information; combining each preprocessed spatial information to obtain preprocessed time information.

[0176] For example, a convolution neural network (CNN) can be used to convolve spatial information to obtain convolution features; the convolution features can be divided into multiple feature blocks (Patches) according to a preset feature block division size, and the multiple feature blocks can be arranged (such as in order from left to right and from top to bottom) as a feature block sequence, and the feature block sequence can be used as an input unit (Input Tokens) for encoding processing; the feature block sequence can be embedded and encoded through an embedding layer (Embedding Layer) to obtain preprocessed spatial information. For example, in the embedding layer, the feature block sequence is convoluted through a convolution operation, and the feature block sequence after the convolution is position encoded (Position Embedding) to obtain the preprocessed spatial information. The feature block sequence can also be attention encoded through an attention mechanism to obtain the preprocessed spatial information.

[0177] For example, assuming that the feature map size of the convolution feature is 224×224, it is divided into 16×16 patches, and the result is (224 / 16) 2 = 196 Patches, and then arrange the 196 Patches from the upper left to the lower right to obtain the feature block sequence.

[0178] For example, before performing embedded coding processing, the feature block corresponding to each Patch can be flattened into a one-dimensional vector. For example, for a 16×16×3 feature block, the length of the flattened vector is 768. The position coding processing can be generated by a fixed algorithm, for example, it can be implemented using a combination of sine and cosine functions. For each position (Position, pos) (for example, the position of each pixel point of each feature block) and the dimension of the feature (for example, there are i dimensions), the position coding of the even-numbered dimensions uses the sine function, while the position coding of the odd-numbered dimensions uses the cosine function. The embodiments of the present application do not limit the specific implementation method of the position coding.

[0179] In step 10372, for each stage, the output features of N convolutional layers are aggregated to obtain the output features of each stage.

[0180] In some embodiments, the output features of N convolutional layers are concatenated to obtain the output features of each stage.

[0181] In some other embodiments, the output features of N convolutional layers are convolved through local temporal convolution to obtain the output features of each stage.

[0182] In some other embodiments, different weights may be pre-assigned to the output features of each convolutional layer, and then weighted summation is performed to obtain the output features of each stage.

[0183] In some other embodiments, the output features of different convolutional layers may be added element-wise to obtain the output features of each stage.

[0184] In some other embodiments, an attention mechanism may be used to automatically learn the weights of the output features of different convolutional layers, and the output features of multiple convolutional layers are weighted summed according to the weights to obtain the output features of each stage.

[0185] In step 10373, the output features of each stage are aggregated to obtain the time features.

[0186] In some embodiments, the output features of each stage are concatenated to obtain the temporal features.

[0187] In other embodiments, the output features of each stage are convolved through global temporal convolution to obtain temporal features.

[0188] In some other embodiments, different weights may be pre-assigned to the output features of each stage, and then weighted summation is performed to obtain the time feature.

[0189] In some other embodiments, output features of different stages may be added element-wise to obtain time features.

[0190] In some other embodiments, an attention mechanism may be used to automatically learn the weights of output features of different stages, and perform weighted summation of output features of multiple stages according to the weights to obtain temporal features.

[0191] In some embodiments, extracting spatial features from spatial information includes multiple stages, each stage may include multiple cascaded convolutional layers, and the spatial features in the spatial information are extracted through the multiple cascaded convolutional layers.

[0192] For example, for each stage, the spatial information is convolved through N cascaded convolutional layers to obtain the output features of N convolutional layers, where N is a positive integer greater than 1, the input of the first convolutional layer is the spatial information, the input of the nth convolutional layer is the output feature of the n-1th convolutional layer, 1<n≤N, and the output of the last convolutional layer of the last stage is used as the spatial feature.

[0193] For example, before extracting the spatial features from the spatial information, the spatial information may be preprocessed to obtain preprocessed spatial information, and the spatial features may be extracted based on the preprocessed spatial information.

[0194] For example, the preprocessing of spatial information can be achieved in the following ways: convolution processing is performed on the spatial information to obtain convolution features; the convolution features are divided into multiple feature blocks, and the multiple feature blocks are arranged into a feature block sequence; embedding coding is performed on the feature block sequence to obtain preprocessed spatial information. The specific implementation method can be found in the description of preprocessing time information above, which will not be repeated here.

[0195] Continue to see Figure 3H In step 1038, the spatial feature, the temporal feature and the second embedded feature are fused to obtain the image feature.

[0196] In some embodiments, see Figure 3J , Figure 3H Step 1038 shown can be implemented by following steps 10381 to 10382, which are described in detail below.

[0197] In step 10381, second attention processing is performed on the spatial features, temporal features and second embedded features to obtain second attention features.

[0198] In some embodiments, the spatial feature, the temporal feature and the second embedded feature are concatenated to obtain a concatenated feature; and the concatenated feature is subjected to a second attention process to obtain a second attention feature.

[0199] For example, the concatenation is used as a query vector (Q), a key vector (K), and a value vector (V) at the same time, and an attention score is obtained by calculating the dot product of Q and K. Next, the attention score is normalized, for example, a normalization function (such as a softmax function) is applied to convert the attention score into a probability distribution as an attention weight, and the concatenated features are weighted using the attention weight to generate a new feature representation, namely, the second attention feature.

[0200] In step 10382, the second attention feature is linearly transformed to obtain the image feature.

[0201] In some embodiments, the second attention feature may be linearly transformed through a linear layer to obtain an image feature.

[0202] In some other embodiments, the spatial feature, the temporal feature and the second embedded feature are concatenated to obtain the image feature.

[0203] In some other embodiments, different weights are pre-assigned to the spatial feature, the temporal feature, and the second embedded feature, and then weighted summation is performed to obtain the image feature.

[0204] Continue to see Figure 3AIn step 104, based on the text features and the image features, an evaluation index for recommending the candidate object to the first user is determined.

[0205] In some embodiments, text features and image features are concatenated to obtain fused features.

[0206] In some other embodiments, an attention mechanism may be used to automatically learn the weights of text features and image features, and perform weighted summation of the text features and image features according to the weights to obtain fused features.

[0207] In other embodiments, after obtaining the fused features, the fused features may be further subjected to multi-scale feature encoding to obtain new fused features, wherein the specific implementation of the multi-scale feature encoding may refer to the description of step 1034 above, which will not be repeated here.

[0208] In some embodiments, based on the fused features, an evaluation index for recommending the candidate object to the first user is determined.

[0209] In some embodiments, the fused features are decoded and mapped to evaluation indicators for recommending candidate objects to the first user.

[0210] For example, the fusion feature can be multiplied by the weight matrix of the fully connected layer through the fully connected layer, and a bias term can be added. The result can be used as an evaluation index for recommending the candidate object to the first user. Taking the candidate object as the advertisement to be recommended as an example, the evaluation index can be the predicted probability value of the first user clicking on the advertisement, the user's satisfaction with the advertisement, the correlation score between the user and the advertisement, etc. The value of the evaluation index is proportional to the probability of recommending the candidate object to the first user.

[0211] Taking the candidate object as the advertisement to be recommended and the evaluation index represented by the relevance score as an example, the evaluation index for recommending the candidate object to the first user can be represented by a list (sorted by relevance score), for example, represented as "[{"Advertisement ID":"A1","Relevance score":0.95},{"Advertisement ID":"A3","Relevance score":0.87}]".

[0212] Through steps 101 to 104, it is possible to extract text information and image information from user information and object information, generate first prompt information describing the text information, and generate second prompt information describing the image information. By encoding using the first prompt information and the second prompt information in the encoding stage, it is possible to guide the generation of text features of the image information of interest, and guide the generation of image features of the text information of interest. By crossing information of different modalities, it is possible to fully utilize the text information and image information, thereby improving the accuracy of the evaluation index for recommending candidate objects to the first user based on the fused features.

[0213] In some embodiments, during the network forward reasoning stage, the following processing can be performed for each network layer: performing nonlinear transformation processing on the features output by the current network layer to obtain nonlinear features; obtaining the product of a preset adjustment coefficient and the nonlinear feature, adding the product and the features output by the current network layer as new output features, and using the new output features as input to the next network layer, wherein the nonlinear transformation processing can be implemented by an activation function (such as Gaussian Error Linear Units (GeLU)), etc.).

[0214] For example, the new output feature = the feature output by the current network layer + α × nonlinear feature, where α is a preset adjustment coefficient that controls the influence of the nonlinear feature.

[0215] By obtaining new output features, the beneficial effect of obtaining outputs that are more in line with expectations can be achieved without changing the original model weights.

[0216] In other embodiments, during the network forward reasoning stage, the following processing can also be performed for each network layer: feature normalization processing (such as Layer Normalization, etc.) is performed on the features output by the current network layer to obtain normalized features; the product of a preset adjustment coefficient and the normalized features is obtained, and the sum of the product and the features output by the current network layer is used as a new output feature, and the new output feature is used as the input of the next network layer.

[0217] In some embodiments, the information processing method provided by the embodiments of the present application is implemented through an information processing model. Before extracting text information and image information from the user information of the first user and the object information of the candidate object, the following processing can also be performed: extract text information samples and image information samples from the user information samples of the user sample (corresponding to the user information) and the object information samples of the candidate object sample (corresponding to the object information) (the specific implementation method can refer to the description of step 101 above); generate a first prompt information sample (corresponding to the first prompt information) of the prompt text information sample (corresponding to the text information) through the initial information processing model, and generate a second prompt information sample (corresponding to the second prompt information, the specific implementation method can refer to the description of step 102 above) of the prompt image information sample (corresponding to the image information); through Through the initial information processing model, the text information sample and the second prompt information sample are subjected to a first sample encoding process (corresponding to the first encoding process) to obtain a text feature sample, and the image information sample and the first prompt information sample are subjected to a second sample encoding process (corresponding to the second encoding process) to obtain an image feature sample (the specific implementation method can refer to the description of step 103 above); based on the text feature sample (corresponding to the text feature) and the image feature sample (corresponding to the image feature), a first prediction index for recommending a candidate object sample to the user sample is determined (corresponding to the evaluation index in step 104, the specific implementation method can refer to the description of step 104 above); a first loss value is determined through the first prediction index and the labeling index in the user information sample; the parameters of the initial information processing model are updated through the first loss value to obtain the information processing model.

[0218] In some embodiments, the first prediction indicator may include multiple sub-prediction indicators. The first loss value can be determined by the first prediction indicator and the labeling indicators in the user information sample in the following way: obtain the labeling indicators corresponding to each sub-prediction indicator, and obtain the loss value between each sub-prediction indicator and the corresponding labeling indicator through a preset loss function, and perform weighted summation processing on the multiple loss values ​​to obtain the first loss value.

[0219] For example, the first prediction indicator may include the predicted user click-through rate, the predicted user satisfaction, and the predicted relevance score between the user and the candidate object, etc. The labeling indicator can be obtained from the user information of the user sample, for example, the user's historical click records (such as whether the candidate object is clicked) and the user's score for the candidate object (for example, a score of 1-5) are obtained as labeling indicators. The relevance score between the user and the candidate object in the labeling indicator can be obtained by taking a weighted sum of the user's historical click records and the user's score for the candidate object.

[0220] For example, the loss value between the predicted user click rate and the user's historical click record (such as clicks are marked as 1 and no clicks are marked as 0) can be calculated using the cross entropy loss function; the loss value between the predicted user satisfaction and the user's rating of the candidate object can be calculated using the mean square error loss.

[0221] In some embodiments, a preset number of user information samples in the user sample do not carry the labeling indicator. After obtaining the information processing model, the following processing can also be performed: through the information processing model, a second prediction indicator corresponding to the user information sample that does not carry the labeling indicator is obtained; the prediction indicator that exceeds the preset prediction confidence threshold is used as the labeling indicator of the user information sample that does not carry the labeling indicator to form a new user information sample, wherein the new user information sample is used to re-execute the training of the information processing model.

[0222] For example, for user information samples that do not carry labeled indicators, the second prediction indicator corresponding to the user information samples that do not carry labeled indicators is obtained through the trained information processing model (see the first prediction indicator for the acquisition method), and the second prediction indicator that is higher than the preset confidence threshold is used as a pseudo-labeled indicator. After adding it to the training data set, the information processing model is retrained, thereby making full use of the information in the labeled data and unlabeled data, and improving the prediction accuracy of the evaluation indicators of the information model.

[0223] It should be noted that the embodiments of the present application can be applied to various recommendation scenarios, such as advertisement recommendation scenarios, video recommendation scenarios, etc. The following will describe an exemplary application of the embodiments of the present application in an advertisement recommendation scenario. Figure 4 , Figure 4 is a flowchart of an optional architecture of the information processing method provided in the embodiment of the present application, which will be combined with Figure 4 The steps shown are explained.

[0224] In step 201, data is input.

[0225] In some embodiments, data input includes input of user data (corresponding to user information) and input of advertisement data (corresponding to object information). Specifically, user data may include: user behavior data, such as browsing records, click records, purchase records, etc.; user description data, such as user description information, search records, etc. Advertisement data may include: advertisement content, advertisement display location, advertisement delivery time, advertisement image (such as static advertisement (advertisement cover) and dynamic advertisement (advertisement video)), etc.

[0226] For example, user data can be represented as:

[0227] User behavior data: viewed ad IDs: A1, A3, A5; clicked ad IDs: A2, A4; ad rating records: A2 score: 2, A4 score: 3.

[0228] User description data: User ID: 12345.

[0229] User description text: Likes electronics and technology news.

[0230] For example, the advertisement data can be represented as:

[0231] Ad ID: A1.

[0232] Ad content: The latest smartphone is released, with powerful features and affordable price.

[0233] Advertising images: static smartphone pictures, dynamic smartphone ads.

[0234] In step 202, data is pre-processed.

[0235] In some embodiments, text preprocessing is performed on the text data (corresponding to text information) in the user data and the advertisement data, and image preprocessing is performed on the advertisement image (corresponding to image information) in the advertisement data.

[0236] For example, text preprocessing of text data in user data and advertising data can be achieved in the following way: perform word segmentation on the user description text in the user data (see the description of the word segmentation process in step 1031 above) to obtain multiple words, perform embedding coding process on the multiple words (see the description in step 1032 above) to obtain an embedded feature vector of the text data, for example, perform word segmentation on [like electronic products and technology news] to obtain multiple words ["electronics", "products", "technology", "news"], perform embedding coding process (or word embedding process) on the multiple words to obtain corresponding embedded feature vectors [V1, V2, V3, V4].

[0237] For example, image preprocessing of the advertising image in the advertising data can be achieved in the following way: adjusting the size of the advertising image to 224×224 and normalizing the pixel values, performing image feature extraction processing on the advertising image after the normalized pixel values ​​to obtain the preprocessed advertising image, for example, using a pre-trained CNN model to extract image features, and representing the image features as an image embedding vector [I1, I2, I3].

[0238] In step 203, keywords are extracted and clustered.

[0239] In some embodiments, keywords are extracted from user description text and advertisement content through a large language model (LLM), that is, words or phrases that can represent the main content and theme are extracted, so as to better understand the text semantics.

[0240] For example, the attention mechanism can be used to obtain keywords with high word weights in user description texts and advertising content. Specifically, the preprocessed text is input into the LLM to obtain the embedded representation of the text, obtain the context embedding vector of each word, perform an importance score on each word (word weight), and identify the words that best represent the text theme. Here, the attention mechanism or other methods can be used to calculate the weight of each word. Finally, a number of keywords (such as a preset number of keywords with high weight rankings) are selected according to the importance score (weight) to form a keyword set for the text.

[0241] In some embodiments, clustering is performed based on the extracted keywords to construct an initial label space (corresponding to the classification of the first keyword above to obtain multiple categories).

[0242] For example, LLM is used to extract keywords from user description text and advertising content. For example, user description keywords are extracted from user description: ["electronic products", "technology news"], and advertising content keywords are extracted from advertising content: ["smartphone", "powerful functions", "affordable prices"]. Next, the extracted keywords are clustered and the similarity matrix of the keyword embedding representation is calculated (for example, using K-means clustering). The clustering results can be expressed as: Cluster 1: ["electronic products", "smartphone"]; Cluster 2: ["technology news", "powerful functions", "affordable prices"].

[0243] For example, after constructing the initial label space, the user description data and the keywords in the advertisement content document and the initial label space are converted into text embedding vectors, for example, a pre-trained model (large language model) is used to generate a high-dimensional vector representation to capture text semantic information. Next, the user description data and the advertisement content document embedding are similarly calculated with the initial label space embedding to obtain the similarity score of each tag in the initial label space, and the similarity score is used to determine the label of the document, and the label with the highest score is selected as the classification result of the document. All documents are classified by the above classification method, and the documents are sorted according to the similarity score. The document with the highest score (corresponding to the fourth text above) is selected, and the document with the highest score is analyzed in detail to re-extract keywords. More fine-grained keywords, especially long-tail tags, are extracted from the high-scoring documents (fourth text) using LLM, and these long-tail tags are added to the initial label space to enrich the label space. The above method of enriching the label space by long-tail tags can be repeated multiple times, for example 3-5 times), that is, the updated label space is used to reclassify the documents each time, and then the long-tail tags of the documents with the highest scores are re-obtained and added to the label space.

[0244] During the model training and reasoning process, the text information in the label space will serve as an important reference to guide the subsequent text feature encoding. By further extracting text features from the text information in the label space, the encoding process is made more accurate. The rich label space helps the model better understand and classify different types of text, especially the recognition of long-tail labels, which significantly improves the performance of the model on rare categories.

[0245] For example, other natural language processing techniques may also be used, such as term frequency-inverse document frequency statistical method (TF-IDF) combined with topic models (such as latent Dirichlet Allocation (LDA) model, etc.) to extract and cluster keywords. The order of keyword extraction and clustering can be interchanged, and preliminary clustering is performed first, and then keywords are extracted from the clustering results.

[0246] Through step 203, by extracting and clustering keywords and acquiring long-tail tags, long-tail users and long-tail advertisements can be effectively located, thereby achieving the beneficial effect of improving the accuracy of recommendations.

[0247] In step 204, model selection and modeling are performed.

[0248] In some embodiments, the model (corresponding to the information processing model) is structured as a dual-branch framework based on a transformer, including a text data processing branch and an image data processing branch.

[0249] For example, the text data processing branch includes an encoder and a decoder (such as an encoder-decoder structure based on a Transformer), the encoder is used to receive text data (corresponding to text information) input and generate a hidden representation (corresponding to text features), and the decoder is used to output a target text representation based on the hidden representation (that is, output a numerical representation (corresponding to a predicted evaluation indicator, such as predicting the click probability of an advertisement, predicting the user's rating of the advertisement, etc.)), thereby reducing the burden of language alignment; the image data processing branch is used to convert an advertisement image (corresponding to image information) into a short sequence of visual tags (corresponding to image features) to prevent the model from focusing on low-level visual features.

[0250] For example, the image data processing branch can construct an image tagger based on a combination of (CNN) and a transformer, wherein the CNN is used to extract spatial features of the advertising image and the transformer is used to convert the extracted features into a short sequence of visual tags.

[0251] In some embodiments, the model includes a multi-head attention module, which is used to capture global information and fine-grained details to improve the feature extraction capability of the model. Here, the multi-head attention module can be reused in the text data processing branch and the image data processing branch. The embodiment of the present application does not limit the number of times the multi-head attention module is superimposed in the network structure of the model.

[0252] For example, the multi-head self-attention mechanism is used to globally capture important information in text data and advertising images. For example, the extracted text features and image features are respectively summarized by the attention scores between the query vector, key vector and value vector, and the information is weighted by the attention scores.

[0253] In some embodiments, the model includes a fine-grained detail capture module, which captures fine-grained features through multi-scale feature extraction. Here, the fine-grained detail capture module can be reused in the text data processing branch and the image data processing branch. The embodiment of the present application does not limit the number of times the fine-grained detail capture module is superimposed in the network structure of the model.

[0254] For example, the input is processed at multiple levels to extract features of different scales. Multi-level features are fused to generate fine-grained representations. Specifically, first, multi-level features are obtained through feature extraction layers of different scales, such as using different convolution kernels and pooling operations to extract features of different scales. Next, features of different scales are fused to generate fine-grained representations, such as using splicing and other methods to fuse multi-scale features.

[0255] By adopting a dual-branch framework, combining the text data processing branch and the image data processing branch, and through a multi-head attention module and a fine-grained detail capture module, multimodal information can be fully extracted and integrated, thereby improving the recommendation effect.

[0256] In some embodiments, a cross-prompt strategy is applied between the text data processing branch and the image data processing branch, and a cross-prompt is generated between the data processing branch and the image data processing branch to enhance information exchange.

[0257] For example, a prompt (corresponding to the first prompt information) is generated from a text data processing branch and provided to an image data processing branch; a prompt (corresponding to the second prompt information) is generated from an image data processing branch and provided to a text data processing branch. Here, the prompt refers to information generated from one modality (such as text or image) that is instructive to another modality (such as a prompt in text form, or a prompt in feature vector form). It is a guide to visual features, contextual information or specific details, with the aim of enabling the two modalities to better complement and integrate.

[0258] By utilizing the cross-hint strategy in the encoding stage to guide the generation of text features that focus on image data, and guide the generation of image features that focus on text data, the accuracy of recommendations can be improved by cross-hinting information of different modalities to fully utilize text data and image data.

[0259] In step 205, features are encoded.

[0260] In some embodiments, feature encoding (corresponding to the first feature encoding and the second feature encoding described above) is performed on text data and image data according to the dual-branch structure (text data processing branch and image data processing branch) determined in model selection and modeling.

[0261] It should be noted that in the encoding process of the text data processing branch and the image data processing branch, the cross-hints generated by the cross-hint strategy (hints generated from the text data processing branch and hints generated from the image data processing branch) also participate in the encoding process. The cross-hint strategy can be repeatedly implemented in multiple stages of feature encoding. The embodiment of the present application does not limit the number of times the cross-hint strategy is implemented.

[0262] For example, the feature encoding of text data by the text data processing branch can be implemented in the following way: using the BERT pre-trained model to convert text (for example, the text information and user description text in the above label space) into a vector representation, and using the word embedding method to represent text features. Here, the specific implementation method can refer to the description of steps 1031 to 1032 above. Fine-grained text features can also be further obtained through a multi-head attention module and a fine-grained detail capture module. Here, the specific implementation method can refer to the description of steps 1033 and 1034 above.

[0263] For example, the feature encoding of the advertisement image by the image data processing branch can be implemented in the following manner: using a convolutional neural network (CNN) to extract image features from the advertisement image. The image marker converts a long sequence of pixels into a short sequence of visual tags to prevent the model from focusing on low-level visual features. In other words, the role of the image marker is to convert the features extracted by the convolutional neural network into a short sequence of visual tags to prevent the model from focusing on low-level visual features, thereby making more efficient use of image information.

[0264] For example, generating a short sequence of visual tags through an image tagger can be achieved in the following way: first, input a feature map, such as a feature map processed by a CNN, such as a size of H×W×D (height, width, depth), then generate visual tags, that is, divide the feature map output by the CNN into multiple small blocks (patches), each small block as a visual tag, specifically, each small block can be linearly transformed and mapped to a vector representation of a fixed length as a visual tag, then serialize the visual tags, that is, arrange the visual tags of all small blocks in a fixed order (the order of feature maps from left to right and from top to small), thereby generating a short sequence of visual tags. The short sequence of visual tags can also be processed by attention encoding as the final visual tag sequence (attention encoding result) to prevent attention to low-level visual features.

[0265] For example, after obtaining the visual marker sequence corresponding to the advertisement image, the image features can be extracted from the spatial and temporal dimensions based on the visual marker sequence. This can be achieved in the following ways:

[0266] First, determine the input data. For the spatial dimension, the input data is the visual marker sequence corresponding to the spatial data (corresponding to spatial information). For the temporal dimension, the input data is the visual marker sequence corresponding to the temporal data (corresponding to temporal information). Specifically, the visual marker sequence corresponding to the spatial data retains the spatial structure of the image through the relative positions between the visual markers. For example, after serialization, different parts of the image (such as the upper left corner and the lower right corner) retain their spatial relationships through position encoding; the visual marker sequence corresponding to the temporal data carries the order and dependency between frames. In subsequent feature extraction, this order information can be used to capture temporal dependency.

[0267] Next, convolution operation (CNN) is performed on the visual tag sequence corresponding to the spatial data to extract spatial features. Temporal convolution processing (TCN) is performed on the visual tag sequence corresponding to the temporal data, or the visual tag sequence is processed by methods such as recurrent neural network (RNN) and LSTM to extract temporal features. Here, the acquisition of spatial features and temporal features is achieved in parallel.

[0268] Finally, the spatial features and temporal features are fused through the splicing method to obtain the image features of the image data processing branch. For example, the splicing operation is performed in the early layers of the backbone network (such as the residual network (ResNet)) so that the subsequent layers can consider both spatial and temporal information. In other words, the fusion operation can be performed in the first few layers of the network to ensure that the spatial and temporal features are fully utilized in the early stages of the network.

[0269] For example, the network structure for performing splicing operations in the early layers of the backbone network (such as ResNet) may include: a spatial convolution layer (such as CNN) for extracting low-level spatial features of the input image; a temporal convolution layer (such as TCN) for extracting low-level temporal features of the input time series; a fusion layer for fusing (such as splicing) spatial and temporal features to generate fused features; and a backbone network for further processing and analyzing the fused features.

[0270] For example, the extraction of temporal features can be achieved by cross-stage feature aggregation, the purpose of which is to enhance temporal representation. By aggregating temporal features at different stages, the temporal dependencies and change patterns in the data can be captured. Specific implementation methods may include: first, performing stage division, such as dividing the backbone network into multiple stages, each stage corresponding to a specific feature extraction level. Next, performing temporal feature aggregation, that is, aggregating temporal features at each stage (concatenating the temporal features of each stage), enables the network to capture temporal information at different levels.

[0271] For example, temporal feature aggregation may include local temporal feature aggregation, such as aggregating temporal features using local temporal convolution within each stage to capture the temporal dependencies and change patterns of the current stage. For example, the temporal features of the current stage may be aggregated by adding a local temporal aggregation layer (temporal convolution layer or RNN layer) at each stage. Specifically, local temporal convolution is performed within each stage, and local temporal features are extracted by defining a convolution kernel and sliding it over a sequence of temporal features. The local features obtained are further aggregated through activation functions and pooling operations to form comprehensive temporal features within each stage. These features capture short-term temporal dynamics and provide a basis for subsequent global temporal feature aggregation.

[0272] For example, temporal feature aggregation may include global temporal feature aggregation, such as across multiple stages, adding a global temporal aggregation layer between multiple stages, and aggregating temporal features of multiple stages using global temporal convolution or RNN to capture temporal dependencies and change patterns across stages.

[0273] For example, the network structure for temporal feature aggregation can be expressed as:

[0274] Backbone network stage 1: Extract low-level spatial and temporal features.

[0275] Local temporal aggregation layer 1: Aggregate temporal features of stage 1.

[0276] Backbone network stage 2: Extract mid-level spatial and temporal features.

[0277] Local temporal aggregation layer 2: Aggregate temporal features of stage 2.

[0278] Backbone network stage 3: extracting high-level spatial and temporal features.

[0279] Local temporal aggregation layer 3: Aggregates temporal features of stage 3.

[0280] Global temporal aggregation layer: Aggregates temporal features across multiple stages (such as concatenating the features output from stages 1, 2, and 3 above, and then applying global temporal convolution) to generate aggregated temporal features.

[0281] For example, the input and output of the text data processing branch can be expressed as:

[0282] Input: Embedding vector of text data [V1, V2, V3, V4] (embedded feature vector of text data obtained in step 202).

[0283] Output: By processing the text embedding layer by layer, text features [T1, T2, T3, T4] are generated.

[0284] The input and output of the image data processing branch can be expressed as:

[0285] Input: Embedding vectors [I1, I2, I3] of ad images.

[0286] Image Processing: CNN processes image embeddings to generate visually labeled embedding representations [I1', I2', I3'].

[0287] Transformer layer processing: Generate image features based on the embedded representation of visual tags [G1, G2, G3].

[0288] Continuing with the above example, the following processing can also be performed on the image features [G1, G2, G3]:

[0289] Global information capture: Apply self-attention mechanism to the image features [G1, G2, G3] to generate global features [G1', G2', G3'].

[0290] Fine-grained detail capture: Multi-head attention mechanism is applied to the global features [G1', G2', G3'] to generate fine-grained features [F1, F2, F3].

[0291] Feature fusion: Combine global features [G1', G2', G3'] and fine-grained features [F1, F2, F3] to generate comprehensive feature representation [C1, C2, C3].

[0292] Multi-level feature fusion: Multi-scale feature fusion is applied to the comprehensive feature representation [C1, C2, C3], features at different levels are fused to generate a unified feature representation [M1, M2, M3], and the unified feature representation [M1, M2, M3] is used as the image feature output by the image data processing branch.

[0293] In some embodiments, the text features and image features obtained by the feature encoding of the text data processing branch and the image data processing branch are fused to obtain fused features. Here, the specific implementation method can refer to the description of step 104 above.

[0294] In step 206, a loss function is designed.

[0295] In some embodiments, the design of the loss function may include: using cross entropy loss for classification tasks (such as predicting advertisement click-through rate), using mean square error loss for regression tasks (such as predicting user satisfaction with advertisements), and combining multiple losses with a multi-task loss function (such as weighted summation of cross entropy loss and mean square error loss). The embodiments of the present application do not limit the specific loss function used.

[0296] Based on the above data preprocessing, keyword extraction and clustering, model selection and modeling, feature encoding and loss function design, semi-supervised training is performed to obtain the trained model. Specifically, labeled data is used for supervised training to learn clear labels, and unlabeled data is used for self-supervision or pseudo-label learning to mine potential information. The specific training process can be found in the above description.

[0297] In step 207, the model is inferred.

[0298] In some embodiments, the model inference result is obtained through differential private text rewriting and model forward pass adjustment, wherein differential private text rewriting is used to rewrite the user description text in the user data input in the model inference phase to protect user privacy, and model forward pass adjustment is used to obtain an output that is more in line with the expected without changing the original model weights, and the process is carried out in the model.

[0299] For example, loading the trained model, obtaining the user data and advertising data to be processed, and rewriting the user description text in the user data can be achieved in the following ways: rewriting the user description text in a semantically similar fuzzy way through a masked language model (MLM) to protect user privacy, and introducing appropriate noise so that the rewritten text is difficult to restore to the original text while maintaining the original semantics.

[0300] For example, rewriting the user description text in a semantically similar fuzzy way through a masked language model (MLM) can be achieved in the following way: first, mask some words in the input user description text, and the masking strategy is to randomly select a certain proportion of words for masking. Next, select the pre-trained BERT for rewriting generation. Specifically, input the masked text into the model to generate prediction results for the masked words. The MLM model predicts possible alternative words for the masked words based on the context. Next, based on the prediction results of the MLM model, replace the masked words to generate a rewritten text (corresponding to the second text), for example, select the word with the highest predicted probability for replacement, or randomly select a word from the top few high-probability words. By introducing randomness when selecting alternative words, the rewritten text is prevented from being completely consistent with the original text. Finally, through semantic similarity evaluation, ensure that the rewritten text is semantically similar to the original text, for example, using cosine similarity for evaluation, and then adjust the rewriting strategy according to the evaluation results to ensure semantic similarity. For example, when the similarity is lower than a threshold, rewrite the text until the similarity between the rewritten text and the original user description text is higher than the similarity threshold.

[0301] For example, the user description text "like electronic products and technology news" is masked by some words to obtain "like [MASK] and [MASK]", and MLM is used to generate prediction results. For example, the pre-trained BERT model is used to generate prediction results for the masked words: "technology products", "latest developments". According to the prediction results, the text is rewritten to obtain "like technology products and latest developments".

[0302] For example, privacy protection technology can be used to protect user data, such as protecting user data through differential privacy or federated learning, using semantic similarity algorithms (such as word embedding or sentence embedding) for text rewriting, etc. Here, privacy protection technology is not only applicable to the privacy protection of text data, but can also be applied to other forms of data, such as images, voice, etc., through similar methods to perform data fuzzification processing, privacy protection technology can be applied to different application scenarios, such as social media, search engines, and other scenarios that require user privacy protection.

[0303] For example, the forward transfer adjustment of the model can be implemented in the following way. The following processing can be performed for each network layer: a nonlinear transformation is performed on the features output by the current network layer to obtain a nonlinear feature; the product of a preset adjustment coefficient and the nonlinear feature is obtained, the sum of the product and the features output by the current network layer is used as a new output feature, and the new output feature is used as the input of the next network layer, wherein the nonlinear transformation processing can be implemented by an activation function (such as GeLU, etc.).

[0304] For example, the new output feature = the feature output by the current network layer + α × nonlinear feature, where α is a preset adjustment coefficient that controls the influence of the nonlinear feature.

[0305] For example, model reasoning can be expressed as follows: first, the user data and advertising data to be processed are preprocessed to obtain user features and advertising features, such as user ID: 12345; user description text embedding vector: [D1, D2, D3, D4]; user historical behavior (such as whether to click on the advertisement, the rating of the advertisement, etc.) embedding vector: [H1, H2, H3], and advertisement ID: A1; advertisement content embedding vector: [K1, K2, K3] and advertisement image embedding vector: [Y1, G2, G3]. Next, the user features and advertising features are processed based on the dual-branch framework of the model, such as the multi-head attention module and the multi-scale feature fusion module for feature processing and fusion, and the cross-cue is used to perform dual-branch information crossover, etc., to obtain fused features. The specific feature extraction process can be found in the above description, which will not be repeated here. Finally, a list of recommended ads and their relevance scores are generated based on the fusion features (see the description of step 104 above), for example, represented as: [{"ad ID":"A1","relevance score":0.95}, {"ad ID":"A3","relevance score":0.87}], where the relevance score can be viewed as a comprehensive score, for example, the higher the relevance score, the greater the click probability, and the higher the user's score for the ad.

[0306] Through steps 201 to 207, a complete process of performing advertisement recommendation in an advertisement recommendation scenario is implemented, including data preparation, data preprocessing, model selection and modeling, feature encoding, loss function design, and model reasoning, thereby achieving the beneficial effect of improving the efficiency, accuracy, and security of the advertisement recommendation system.

[0307] The following is a description of an exemplary structure of the information processing device 133 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules stored in the information processing device 133 of the memory 130 may include:

[0308] The data acquisition module 1331 is used to extract text information and image information from the first user information of the first user and the object information of the candidate object.

[0309] The data processing module 1332 is used to generate first prompt information for prompting the text information and generate second prompt information for prompting the image information.

[0310] In some embodiments, the data processing module 1332 is further used to perform a first encoding process on the text information and the second prompt information to obtain text features, and to perform a second encoding process on the image information and the first prompt information to obtain image features.

[0311] In some embodiments, the data processing module 1332 is further used to determine an evaluation index for recommending the candidate object to the first user based on the text feature and the image feature.

[0312] In some embodiments, the data processing module 1332 is further used to extract first key information from the text information; and generate the first prompt information based on the first key information and a preset first information template.

[0313] In some embodiments, the data processing module 1332 is further used to perform image-to-text conversion on the image information to obtain image description text; extract second key information from the image description text; and generate the second prompt information based on the second key information and a preset second information template.

[0314] In some embodiments, the first user information includes user description text, and the data acquisition module 1331 is further used to mask words in the user description text to obtain a first text; predict the masked words in the first text to obtain predicted words; replace the masked words in the first text with the predicted words to obtain a second text; and based on the second text, determine the text information and the image information from the first user information and the object information.

[0315] In some embodiments, the data acquisition module 1331 is also used to replace the user description text in the first user information with the second text to obtain the second user information when the similarity between the user description text and the second text is greater than a similarity threshold; and determine the text information and image information from the second user information and the object information of the candidate object.

[0316] In some embodiments, the first user information includes user description text, and the data processing module 1332 is further used to extract label information from the text information, and perform word segmentation processing on the label information, the user description information and the second prompt information to obtain a first word; perform a first embedding processing on the first word to obtain a first embedding feature; perform a first attention processing on the first embedding feature to obtain a first attention feature; perform multi-scale feature encoding on the first attention feature to obtain the text feature.

[0317] In some embodiments, the text information includes multiple third texts, and the data processing module 1332 is also used to determine a first keyword from the words in the text information based on the weight of each word in the text information; classify the first keyword to obtain multiple categories; determine the probability that each third text belongs to each category, and based on the probability, determine a fourth text from the multiple third texts; determine the second keyword of the fourth text, and determine the multiple categories and the second keyword as the label information.

[0318] In some embodiments, the data processing module 1332 is also used to perform multi-scale convolution processing on the first attention feature to obtain convolution features of multiple scales; perform pooling processing on the convolution features of each scale to obtain pooling features of each scale; and fuse the pooling features of multiple scales to obtain the text feature.

[0319] In some embodiments, the image information includes spatial information and temporal information, and the data processing module 1332 is further used to perform word segmentation processing on the first prompt information to obtain a second word; perform a second embedding processing on the second word to obtain a second embedded feature; extract a temporal feature from the temporal information, and extract a spatial feature from the spatial information; and fuse the spatial feature, the temporal feature and the second embedded feature to obtain the image feature.

[0320] In some embodiments, the process of extracting time features from the time information includes multiple stages, and the data processing module 1332 is also used to, for each of the stages, perform convolution processing on the time information through N cascaded convolution layers to obtain output features of the N convolution layers, wherein N is a positive integer greater than 1, the input of the first convolution layer is the time information, the input of the nth convolution layer is the output feature of the n-1th convolution layer, and 1<n≤N; for each of the stages, aggregate the output features of the N convolution layers to obtain the output features of each stage; aggregate the output features of each stage to obtain the time features.

[0321] In some embodiments, the data processing module 1332 is also used to perform convolution processing on the spatial information through N cascaded convolution layers for each of the stages to obtain output features of the N convolution layers, wherein N is a positive integer greater than 1, the input of the first convolution layer is the spatial information, the input of the nth convolution layer is the output feature of the n-1th convolution layer, 1<n≤N, and the output of the last convolution layer of the last stage is used as the spatial feature.

[0322] In some embodiments, the data processing module 1332 is also used to perform second attention processing on the spatial feature, the temporal feature and the second embedded feature to obtain a second attention feature; and perform linear transformation processing on the second attention feature to obtain the image feature.

[0323] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the electronic device executes the information processing method described in the embodiment of the present application.

[0324] The present application embodiment provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will be caused to execute the information processing method provided by the present application embodiment, for example, Figure 3A The information processing method shown.

[0325] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0326] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0327] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0328] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0329] To summarize, through the embodiments of the present application, text information and image information are extracted from user information and object information, and first prompt information for prompting text information and second prompt information for prompting image information are generated. By encoding using the first prompt information and the second prompt information in the encoding stage, it is achieved to guide the generation of text features of the focused image information and to guide the generation of image features of the focused text information. By utilizing the intersection of two prompt information from different modalities, that is, making the first prompt information describing the text information participate in the encoding of the image information, and making the second prompt information describing the image information participate in the encoding of the text information, it is achieved to fully utilize the text information and image information, thereby improving the accuracy of the evaluation index for recommending candidate objects to the first user based on the fused features.

[0330] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent replacements and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. An information processing method, characterized in that: The method comprises: extracting text information and image information from first user information of the first user and object information of the candidate object; generating first prompt information for prompting the text information, and generating second prompt information for prompting the image information; Performing a first encoding process on the text information and the second prompt information to obtain text features, and performing a second encoding process on the image information and the first prompt information to obtain image features; Based on the text features and the image features, an evaluation index for recommending the candidate object to the first user is determined.

2. The method according to claim 1, characterized in that The generating of the first prompt information for prompting the text information includes: Extracting first key information from the text information; The first prompt information is generated based on the first key information and a preset first information template.

3. The method according to claim 1, characterized in that The first user information includes a user description text, and extracting text information and image information from the first user information of the first user and the object information of the candidate object includes: Masking the words in the user description text to obtain a first text; Predicting the masked words in the first text to obtain predicted words; Replacing the masked words in the first text with the predicted words to obtain a second text; Based on the second text, the text information and the image information are determined from the first user information and the object information.

4. The method according to claim 3, characterized in that The determining, based on the second text, the text information and the image information from the first user information and the object information includes: When the similarity between the user description text and the second text is greater than a similarity threshold, the user description text in the first user information is replaced with the second text to obtain second user information; Text information and image information are determined from the second user information and the object information of the candidate object.

5. The method according to claim 1, characterized in that The first user information includes a user description text, and the first encoding process is performed on the text information and the second prompt information to obtain text features, including: Extracting label information from the text information, and performing word segmentation processing on the label information, the user description information and the second prompt information to obtain a first word; Performing a first embedding process on the first word to obtain a first embedding feature; Performing a first attention processing on the first embedded feature to obtain a first attention feature; Perform multi-scale feature encoding on the first attention feature to obtain the text feature.

6. The method according to claim 5, characterized in that The text information includes a plurality of third texts, and extracting tag information from the text information includes: determining a first keyword from the words in the text information based on the weight of each word in the text information; Classifying the first keyword to obtain multiple categories; determining a probability that each third text belongs to each category, and determining a fourth text from the plurality of third texts based on the probability; A second keyword of the fourth text is determined, and the plurality of categories and the second keyword are determined as the tag information.

7. The method according to claim 1, characterized in that The image information includes spatial information and time information, and the second encoding process is performed on the image information and the first prompt information to obtain image features, including: Performing word segmentation processing on the first prompt information to obtain a second word; performing a second embedding process on the second word to obtain a second embedding feature; Extracting a time feature from the time information, and extracting a space feature from the space information; The spatial feature, the temporal feature and the second embedded feature are fused to obtain the image feature.

8. An information processing device, characterized in that: The device comprises: A data acquisition module, used to extract text information and image information from the first user information of the first user and the object information of the candidate object; A data processing module, used for generating first prompt information for prompting the text information, and generating second prompt information for prompting the image information; The data processing module is further used to perform a first encoding process on the text information and the second prompt information to obtain text features, and to perform a second encoding process on the image information and the first prompt information to obtain image features; The data processing module is further used to determine an evaluation index for recommending the candidate object to the first user based on the text feature and the image feature.

9. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions; A processor, configured to implement the information processing method according to any one of claims 1 to 7 when executing the computer executable instructions or computer programs stored in the memory.

10. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the information processing method according to any one of claims 1 to 7 is implemented.

Images