Information retrieval method and device, computer program product and electronic equipment

Abstract

The embodiment of the invention provides an information retrieval method and device, a behavior prediction method and device, a computer program product and electronic equipment, and relates to the technical field of computers. The information retrieval method comprises the following steps: acquiring multi-modal data corresponding to a target user in an application platform; performing feature extraction on the multi-modal data based on a large language model to obtain a feature vector of the multi-modal data; performing dimension reduction processing on the feature vector to determine a feature vector after dimension reduction, and generating a retrieval vector based on the feature vector after dimension reduction; and performing information retrieval according to the retrieval vector, and determining a retrieval result corresponding to the multi-modal data. According to the technical scheme, the accuracy of the retrieval result can be improved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer technology, and more specifically, relate to an information retrieval method and device, a computer program product, and an electronic device. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the disclosure that are recited in the claims. No description herein is admitted to be prior art by inclusion in this section.

[0003] With the development of big data, recommendation systems can be used to recommend corresponding content or information to users.

[0004] In the related art, information recommendation is generally implemented by relying on the user's historical interaction sequence or based on user identification data, such as recommending music or other results to the user. Summary of the invention

[0005] In the related art, information recommendation relies on the user's historical interaction sequence or ID-based data. Since it can only rely on information of a single dimension, it has certain limitations and restricts the comprehensive understanding of user information, and the accuracy of the retrieval results obtained is low.

[0006] To this end, an improved information retrieval method is urgently needed to generate a retrieval vector based on the extracted multimodal data, and then perform information retrieval based on the retrieval vector to obtain retrieval results for target users in the application platform.

[0007] In this context, embodiments of the present disclosure are intended to provide an information retrieval method, an information retrieval device, a computer program product, and an electronic device.

[0008] According to one aspect of the present disclosure, there is provided an information retrieval method, including: acquiring multimodal data corresponding to a target user in an application platform; performing feature extraction on the multimodal data based on a large language model to obtain a feature vector of the multimodal data; performing dimensionality reduction processing on the feature vector to determine a feature vector after dimensionality reduction, and generating a retrieval vector based on the feature vector after dimensionality reduction; performing information retrieval according to the retrieval vector to determine a retrieval result corresponding to the multimodal data.

[0009] In an exemplary embodiment of the present disclosure, multimodal data includes text data and / or image data; feature extraction is performed on the multimodal data based on a large language model to obtain a feature vector of the multimodal data, including: feature extraction of text data in the multimodal data to obtain text features; feature extraction of image data in the multimodal data to obtain image features.

[0010] In an exemplary embodiment of the present disclosure, feature extraction is performed on text data in multimodal data to obtain text features, including: if the length of the text data is less than or equal to a length threshold, feature extraction is performed on the text data through a large language model to obtain text features; if the length of the text data is greater than the length threshold, the text data is compressed, and feature extraction is performed on the compressed text data through a large language model to obtain text features.

[0011] In an exemplary embodiment of the present disclosure, feature extraction is performed on text data through a large language model to obtain text features, including: segmenting the text data through a word segmenter to obtain segmented text data, and processing the segmented text data into a text sequence; inputting the text sequence into the large language model for feature extraction to obtain a hidden layer output result; and performing average pooling on the hidden layer output result to obtain text features.

[0012] In an exemplary embodiment of the present disclosure, a text sequence is input into a large language model for feature extraction to obtain a hidden layer output result, including: normalizing the text sequence to determine a normalized text sequence; extracting features from the normalized text sequence through a multi-head attention mechanism layer to obtain intermediate features; fusing the text sequence and the intermediate features to determine fused features; normalizing the fused features, and performing weighted summation and nonlinear transformation on the normalized fused features based on a feedforward neural network to obtain a hidden layer output result.

[0013] In an exemplary embodiment of the present disclosure, compressing text data includes: determining prompt words; inputting the prompt words into a large language model to extract key texts from the text data to compress the text data.

[0014] In an exemplary embodiment of the present disclosure, feature extraction is performed on image data in the multimodal data to obtain image features, including: dividing the image data into multiple image blocks; adding the embedding vector and the image mark vector of each image block to the corresponding position vector respectively to determine the image sequence; performing feature extraction on the image sequence to obtain sequence features; and performing linear processing on the sequence features to generate image features.

[0015] In an exemplary embodiment of the present disclosure, a feature vector includes text features and image features; the feature vector is subjected to dimensionality reduction processing to determine the reduced feature vector, including: mapping the text features through an encoder in a trained autoencoder to generate a text latent vector of a target dimension; mapping the image features through an encoder in a trained autoencoder to generate an image latent vector of a target dimension.

[0016] In an exemplary embodiment of the present disclosure, generating a retrieval vector based on a feature vector after dimensionality reduction includes: concatenating a text latent vector of a target dimension and an image latent vector of a target dimension to determine the retrieval vector.

[0017] In an exemplary embodiment of the present disclosure, the method further includes: when there is incremental multimodal data, performing dimensionality reduction processing on the feature vector corresponding to the incremental multimodal data by an encoder in a trained autoencoder.

[0018] In an exemplary embodiment of the present disclosure, the method also includes: mapping the sample feature vector corresponding to the sample data through an encoder in the autoencoder to generate a latent vector of the target dimension; the sample data includes sample text data or sample image data; reconstructing the latent vector of the target dimension into a reconstructed vector with the same dimension as the sample feature vector through a decoder in the autoencoder; determining the loss between the sample feature vector and the reconstructed vector, and updating the parameters of the encoder and the decoder in the autoencoder based on the loss to obtain a trained autoencoder.

[0019] In an exemplary embodiment of the present disclosure, after generating the retrieval vector, the method further includes: evaluating the quality of the retrieval vector by the similarity between the retrieval vector and a reference vector.

[0020] In an exemplary embodiment of the present disclosure, the quality of the retrieval vector is evaluated by the similarity between the retrieval vector and a reference vector, including: building an index for each retrieval vector, the index including multiple reference vectors; determining the similarity between the retrieval vector and each reference vector in the index; determining at least one candidate vector similar to the retrieval vector from the multiple reference vectors based on the similarity; and evaluating the quality of the retrieval vector based on whether the similarity of the at least one candidate vector meets a similarity condition.

[0021] In an exemplary embodiment of the present disclosure, information retrieval is performed based on a retrieval vector to determine retrieval results corresponding to multimodal data, including: determining retrieval results that match the retrieval vector in an application platform, and displaying a cover image and description text corresponding to the retrieval results on a display interface of the application platform.

[0022] According to one aspect of the present disclosure, there is provided an information retrieval device, comprising: a data acquisition module, used to acquire multimodal data corresponding to a target user in an application platform; a feature extraction module, used to perform feature extraction on the multimodal data based on a large language model to obtain a feature vector of the multimodal data; a dimensionality reduction processing module, used to perform dimensionality reduction processing on the feature vector to determine the feature vector after dimensionality reduction, and generate a retrieval vector based on the feature vector after dimensionality reduction; and a retrieval result determination module, used to perform information retrieval according to the retrieval vector to determine the retrieval result corresponding to the multimodal data.

[0023] According to one aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the information retrieval method as described above is implemented.

[0024] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions; wherein the processor is configured to execute any one of the above-mentioned information retrieval methods by executing the executable instructions.

[0025] According to the information retrieval method and device, computer program product, and electronic device of the disclosed embodiment, on the one hand, since feature vectors can be extracted through the multimodal data of the target user on the application platform, and the multimodal data can be feature processed using a large language model, the problem of not being able to fully understand visual information that may result from using only single-dimensional data is avoided, and multiple types of data can be comprehensively used to extract feature vectors, avoiding the limitations of a single dimension, increasing the comprehensiveness of feature vectors, and improving the reliability and accuracy of retrieval results. On the other hand, by reducing the dimension of the feature vector to obtain the reduced-dimensional feature vector, the compatibility of the vector dimension generated by the large language model with the recommendation system used to determine the retrieval results can be improved, thereby improving the accuracy of the retrieval results. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, in which:

[0027] Figure 1 A schematic diagram of a system architecture of an application scenario of an embodiment of the present disclosure is schematically shown.

[0028] Figure 2 The flowchart of the information retrieval method in the embodiment of the present disclosure is schematically shown.

[0029] Figure 3 The flowchart of text feature extraction for short text data in the embodiment of the present disclosure is schematically shown.

[0030] Figure 4 The flowchart of text feature extraction for long text data in the embodiment of the present disclosure is schematically shown.

[0031] Figure 5 The flowchart of obtaining image features in an embodiment of the present disclosure is schematically shown.

[0032] Figure 6 The flowchart of feature extraction of image data in the embodiment of the present disclosure is schematically shown.

[0033] Figure 7 The structural block diagram of the automatic encoder according to the embodiment of the present disclosure is schematically shown.

[0034] Figure 8 The schematic diagram of determining a similarity vector according to an embodiment of the present disclosure is schematically shown.

[0035] Fig. 9 The schematic diagram of determining the search image and the search text according to the embodiment of the present disclosure is schematically shown.

[0036] Fig.10 The flowchart of the recommended search result of the embodiment of the present disclosure is schematically shown.

[0037] Fig.11 A schematic block diagram of an information retrieval device according to an embodiment of the present disclosure is schematically shown.

[0038] Fig.12 A block diagram of an electronic device according to an embodiment of the present disclosure is schematically shown.

[0039] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0040] The principles and spirit of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0041] Those skilled in the art will appreciate that the embodiments of the present disclosure may be implemented as a system, device, apparatus, method or computer program product. Therefore, the present disclosure may be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0042] According to an embodiment of the present disclosure, an information retrieval method, an information retrieval device, a computer program product, and an electronic device are proposed.

[0043] In addition, any number of elements in the drawings is for illustration and not limitation, and any naming is only for distinction and does not have any limiting meaning.

[0044] The principle and spirit of the present disclosure are explained in detail below with reference to several representative embodiments of the present disclosure.

[0045] The inventors have found that information recommendation based on historical interaction sequences of users or data based on user identification IDs has certain limitations and restricts the comprehensive understanding of user information, resulting in low accuracy of retrieval results because it can only rely on information of a single dimension.

[0046] Based on the above content, the basic idea of ​​the present disclosure is to extract multimodal data of target users in the application platform, perform feature extraction based on the multimodal data, and then perform information retrieval in the application platform according to the extracted retrieval vector to obtain retrieval results in the application platform, so as to use the retrieval results as recommendation results for the target users in the application platform. The recommendation results can be recommended music or recommended playlists in the application platform.

[0047] More specifically, in the present disclosure, multimodal data of target users in the application platform is extracted, and then feature extraction is performed on the multimodal data based on a large language model, and the extracted feature vector is reduced in dimension to generate a retrieval vector, and then the retrieval result in the application platform is obtained based on the retrieval vector. Therefore, according to the embodiments of the present disclosure, the accuracy of the retrieval vector can be improved, thereby improving the accuracy and comprehensiveness of the retrieval results, and the reliability of the retrieval results can be improved.

[0048] After introducing the basic principles of the present disclosure, various non-limiting embodiments of the present disclosure are described in detail below.

[0049] It should be noted that the following application scenarios are only shown to facilitate understanding of the spirit and principle of the present disclosure, and the embodiments of the present disclosure are not limited in this respect. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario.

[0050] First reference Figure 1 , Figure 1 The system architecture diagram of an exemplary application scenario of an embodiment of the present disclosure is shown. The information retrieval method can be used for recommendation functions in various application platforms. For example, music or playlists can be recommended to users, or other types of information such as items, news, novels, videos, etc. can be recommended to users.

[0051] like Figure 1As shown, the system architecture 100 includes a terminal 101, a network 102 and a server 103. The server 103 analyzes the multimodal data extracted from the terminal 101, so that the server generates a feature vector according to the multimodal data, processes the feature vector into a search vector, and then determines a search result for each target user as a recommendation result according to the search vector. Those skilled in the art should understand that Figure 1 The schematic framework shown is only an example in which the embodiments of the present disclosure can be implemented. The scope of application of the embodiments of the present disclosure is not limited in any aspect by the framework.

[0052] It should be noted that the terminal 101 can be an intelligent device with data processing functions, such as monitoring equipment, smart phones, computers, tablet computers, vehicle-mounted devices, wearable devices and other intelligent devices. The terminal device can also be called a mobile terminal, terminal, mobile device, etc. The present disclosure does not limit the type of terminal device.

[0053] The server 103 may be a local server or a remote server. In addition, the server 103 may also be other products that can provide storage functions or processing functions, such as a cloud server. The embodiments of the present disclosure are not specifically limited here. The server may also be a terminal device with fast computing capabilities or a vehicle-mounted device, etc., which is not limited here.

[0054] It should be understood that the technical solution provided in the embodiment of the present disclosure can be executed by a terminal or a server. Correspondingly, in this mode of execution by a server, the server can start executing the steps in the technical solution of the embodiment of the present disclosure in response to a trigger instruction, wherein the trigger instruction can be sent by a terminal used by a user, or can be triggered locally by the server in response to some automation events. The server can be a background system that provides related services in the embodiment of the present disclosure, and can include a portable computer, a desktop computer, a smart phone, and other electronic devices with computing functions, or a cluster formed by multiple electronic devices.

[0055] In addition, the technical solution of the embodiment of the present disclosure can also be executed by the terminal and the server in collaboration. It should be noted that in this mode of collaborative execution by the terminal and the server, the steps executed by the terminal and the server respectively can be dynamically adjusted according to actual conditions, and the embodiment of the present disclosure does not impose any special restrictions on this.

[0056] Combine the following Figure 1 For application scenarios, refer to Figure 2 To describe the information retrieval method according to the exemplary embodiment of the present disclosure. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principle of the present disclosure, and the embodiments of the present disclosure are not limited in this respect. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario.

[0057] It should be noted that in the technical solution of the present disclosure, the collection / gathering, updating, analysis, use, transmission, storage and other aspects of user personal information involved are all authorized, comply with the provisions of relevant laws and regulations, and are used for legal and reasonable purposes, and are not shared, disclosed or sold outside of these legal uses.

[0058] Figure 2 FIG. 1 is a flowchart of an information retrieval method according to an embodiment of the present disclosure. Figure 2 As shown, the information retrieval method may include the following steps:

[0059] In step S210, multimodal data corresponding to the target user in the application platform is obtained;

[0060] In step S220, feature extraction is performed on the multimodal data based on the large language model to obtain a feature vector of the multimodal data;

[0061] In step S230, the feature vector is subjected to dimensionality reduction processing to determine a feature vector after dimensionality reduction, and a search vector is generated based on the feature vector after dimensionality reduction;

[0062] In step S240, information retrieval is performed according to the retrieval vector to determine the retrieval result corresponding to the multimodal data.

[0063] The technical solution in the disclosed embodiments can obtain feature vectors through multimodal data, improve the accuracy and comprehensiveness of feature vectors from multiple dimensions, so that the feature vectors cover multiple dimensions, avoid the limitations of a single dimension, and improve the comprehensiveness of the feature vectors. On this basis, it can improve the accuracy and comprehensiveness of the retrieval results, and can improve the reliability of the retrieval results.

[0064] Next, the information retrieval method in the embodiment of the present disclosure is explained in detail with reference to the accompanying drawings.

[0065] In step S210, multimodal data corresponding to the target user in the application platform is obtained.

[0066] In the disclosed embodiments, the application platform may be any type of application platform, for example, a music application platform, a shopping application platform, a reading application platform, a video application platform, etc., and the application platform is taken as an example for explanation herein. The target user may be every user who logs into the application platform. For the target user, its user behavior information on the application platform may be obtained, and the user behavior information may be, for example, the target user's playback record, search record, collection record, comment record, etc. on the music application platform. The playback record may include, for example, a list of music recently played by the target user, the name of the music, the artist, the album, etc.; the collection record may include, for example, a list of favorite music, albums, or playlists, etc.; the search record may include, for example, a list of searched music, artists, albums, or playlist keywords, etc. The comment record may include, for example, commented music, etc.

[0067] Multimodal data refers to various types of data corresponding to the reference information of the target user in the application platform. The reference information can be historical behavior information in the application platform, and the reference information can be determined based on the user behavior information. When the application platform is a music application platform, the reference information can be the historical music and historical playlists played by the target user, which can be specifically determined based on the playback records in the user behavior information of the target user. On this basis, the multimodal data can be text data and / or image data corresponding to the historical music and historical playlists. Among them, the text data can be the title, description or lyrics of the historical music and historical playlists, etc. The image data can be the cover image of the historical music and historical playlists.

[0068] In step S220, feature extraction is performed on the multimodal data based on the large language model to obtain a feature vector of the multimodal data.

[0069] In the disclosed embodiments, for the received multimodal data, feature extraction can be performed on the text data and image data in the multimodal data based on the large language model, respectively, to obtain text features corresponding to the text data and image features corresponding to the image data, so as to extract feature vectors of the multimodal data. For example, the text data is converted into text features through the Baichuan model for text data, and the image data is converted into image features through the ViT (Visual Transformer) model for image data. By extracting features from multimodal data through a large language model, it is possible to capture deep semantic information using a variety of parameters of the large language model, thereby improving the accuracy and richness of the feature vector.

[0070] In some embodiments, first, feature extraction is performed on text data in multimodal data to obtain the process of text features. Exemplarily, feature extraction can be performed directly on text data according to the length of the text data, or the text data can be compressed and feature extraction can be performed on the compressed text data. Specifically, when the length of the text data is less than or equal to the length threshold, feature extraction is performed directly on the text data through a large language model to obtain text features; when the length of the text data is greater than the length threshold, the text data is compressed, and feature extraction is performed on the compressed text data through a large language model to obtain text features. Among them, the length threshold can be determined according to actual needs, for example, it can be 100 or other values, which are not specifically limited here. Text data with a length less than or equal to the length threshold can be short text data, and text data with a length greater than the length threshold can be long text data. For example, short text data can be a music title or a music description; long text data can be lyrics.

[0071] The large language model learns rich language knowledge and semantic information by pre-training on a large amount of text data. Therefore, the large language model can capture the complex relationship between words, phrases and sentences in the text data, and then generate text vectors with high semantic relevance.

[0072] In some embodiments, for short text data, the large language model can be used to directly extract features from text data whose length is less than or equal to the length threshold. For example, the text data can be split by a word segmenter to obtain split text data, and the split text data can be processed into a text sequence; the text sequence can be input into the large language model for feature extraction to obtain a hidden layer output result; the hidden layer output result can be average pooled based on the average pooling layer to obtain the text features of the text data.

[0073] The word segmenter can be a word segmenter trained in Chinese. The large language model can be a large language model based on the Transformer architecture, represented by GPT. The large language model adopts a decoder architecture, refer to Figure 3 As shown in , the large language model may include a normalization layer, a multi-head attention network, a feedforward neural network layer, and a linear layer.

[0074] Figure 3 A flowchart for extracting text features from short text data is schematically shown in FIG. Figure 3As shown in , for each text data, the text data is first segmented using a word segmenter trained in Chinese to obtain the segmented text data, which can be represented as a token. Furthermore, each segmented text data token can be mapped to a unique integer or vector. In the face of text data of different lengths, in order to make the input sequences of all large language models have the same length, the word segmenter will add a special padding representation, namely a padding token, at the end of the shorter sequence, thereby generating a text sequence token sequence that can be processed by the large language model.

[0075] After obtaining the text sequence, the text sequence can be input into a large language model, and feature extraction can be performed through the large language model. Exemplarily, the text sequence can be normalized to determine a normalized text sequence. The normalized text sequence is further feature extracted through a multi-head attention network to obtain intermediate features. In the intermediate features, the output of each position contains the contextual information of the position. Next, the text sequence and the intermediate features are added to achieve the fusion of the text sequence and the intermediate features to obtain fused features. On this basis, the fused features are normalized through a normalization layer to obtain normalized fused features, and the normalized fused features are input into a feedforward neural network, and the normalized fused features are weighted summed and nonlinearly transformed based on the feedforward neural network to obtain the hidden layer output result.

[0076] Exemplarily, the forward propagation process can be performed in a feedforward neural network, as follows: the input data first enters the input layer, and is passed to the hidden layer through weights and biases. The nodes in the hidden layer perform weighted summation on the input and perform nonlinear transformation through the activation function. Finally, the output layer receives the signal processed by the hidden layer and generates the final output. In the disclosed embodiment, the input of the feedforward neural network can be a normalized fusion feature, and the activation function is used to introduce nonlinearity in the feedforward neural network, so that the feedforward neural network can learn and simulate complex function mappings. The weight is the connection strength between the input layer and the hidden layer, and the hidden layer and the output layer. The bias is a constant added to the input to adjust the output of the activation function.

[0077] For each text sequence input into the large language model, each text sequence, i.e., token sequence, can be expressed as Where l is the sequence length, and d is the embedding vector length corresponding to the token sequence, i.e., the embedding length. After the text sequence is processed by the large language model, the hidden layer output result can be expressed as follows: The vector of each position in the text sequence can be regarded as the potential representation of each text sequence token. Furthermore, in order to obtain the text feature E of the entire input text data c, the hidden layer output of the large language model can be averaged and pooled through the average pooling layer to obtain text features. The text features obtained by feature extraction from the large language model can be expressed by formula (1):

[0078]

[0079] In other embodiments, for text data whose length is greater than the length threshold, it may be long text data. Long text data contains more information, and directly using the average pooling method in short text data may cause more information loss. Therefore, when processing text data whose length is greater than the length threshold, the summarization ability of the large model is first used to configure a reasonable prompt word prompt, and the prompt word is input into the large language model. The input long text data is extracted by the large language model, so as to achieve text compression for the text data whose length is greater than the length threshold through key text extraction, and obtain compressed text data.

[0080] Among them, when determining the prompt word for long text data such as lyrics content, the goal is to simplify and compress the text data through a large language model, which helps to improve the compactness of the text data and the concentration of information. In view of the direct correlation between the length of the text generated by the large language model and the processing time, it is necessary to limit the length of the output text to ensure that the running time of the large language model remains within an acceptable range. By limiting the length of the text data, it is possible to balance the quality of text generation and the computing resource consumption of the model, thereby maintaining the overall performance and user experience of the system while ensuring the efficiency of information transmission. For example, the prompt word prompt can be determined as "summarize the main idea of ​​the lyrics below, less than 20 words: lyrics: #lrc_lyric_content, song name: #song_name". Of course, the prompt word prompt can also be in other forms, as long as the key text extraction can be achieved.

[0081] Figure 4 The flowchart of extracting text features from long text data is schematically shown in FIG. Figure 4 As shown in , the input text data can first be segmented by a word segmenter, and the segmented text data can be input into a large language model. The large language model can extract key texts through prompt words to obtain a text summary, so as to compress the text data and obtain compressed text data. Furthermore, the compressed text data can be segmented by a word segmenter to obtain segmented text data, and the segmented text data can be processed into a text sequence; the text sequence can be input into a large language model for feature extraction to obtain a hidden layer output result; the hidden layer output result can be average pooled by an average pooling layer to obtain text features.

[0082] In the disclosed embodiments, by combining the relationship between the length of text data and a length threshold and combining with a large language model to extract features from the text data to obtain text features, different methods can be used to extract text features for text data of different lengths, thereby improving the accuracy of text features and the efficiency of extracting text features.

[0083] For image data in multimodal data, the ViT model can be used to extract features from the image data to obtain image features. The image data may be a cover image. For example, in a music application platform, the image data may be a music cover image.

[0084] As a visual expression of a musical work, music cover images can be used to reflect the style and emotion of the music, and can also reflect the visual image of the characters. Compared with pure text data, music cover images can reduce the time cost of users when screening music. It should be noted that the music cover images of the same type of music works can be similar, for example, they can share similar visual design languages, such as color matching, composition layout and graphic elements, thereby providing users with an intuitive classification basis.

[0085] In some embodiments, the input image data may be subjected to feature extraction by a ViT model. Exemplarily, the ViT model may include a Transformer encoder and a linear layer. Figure 5 As shown in , obtaining image features mainly includes the following steps:

[0086] In step S510, the image data is divided into a plurality of image blocks.

[0087] In this step, the image data can be divided into patches of fixed size. Each patch is regarded as an embedding vector, and the attention mechanism in the ViT model can capture the local information in the image data based on the embedding vector.

[0088] In step S520, the embedding vector and the image mark vector of each image block are added to the corresponding position vector to determine the image sequence.

[0089] In this step, the image token vector CLS token can be a vector representation used to represent the entire image data, and classification prediction can be performed through the image token vector. Based on this, the embedding vector of each image block can be added to the corresponding position vector. In addition, the image token vector can also be added to the corresponding position vector, and an image sequence can be obtained according to the vector addition result to map the image data into a high-dimensional space to form a new image representation.

[0090] In step S530, feature extraction is performed on the image sequence to obtain sequence features.

[0091] In the disclosed embodiment, an image sequence is input into a Transformer encoder, and the image sequence is processed using a self-attention mechanism layer in the Transformer encoder to obtain global context information between image blocks to determine sequence features.

[0092] In step S540, linear processing is performed on the sequence features to generate image features.

[0093] In this step, the linear layer is used to perform linear processing on the sequence features, and the vector corresponding to the image block and the overall image vector are output, and then the image features can be obtained according to the overall image vector. Among them, the overall image vector is the vector related to CLStoken, which is used to represent the features of the entire image data.

[0094] refer to Figure 6 As shown in , the image data can be used as input, the image data can be divided into multiple image blocks, the embedding vector of the image block and the image mark vector are added to the corresponding position vector respectively to obtain the image sequence. The image sequence is input to the Transformer encoder, and the sequence features output by the Transformer encoder are input to the linear layer to obtain the vector of each image block and the overall image vector. The overall image vector can be further used as the output of the ViT model to obtain the image feature E of the image data image .

[0095] In the disclosed embodiment, by extracting features from multi-modal data represented by text data and image data through a large language model, feature vectors can be obtained from multiple dimensions, thereby improving the comprehensiveness of feature vectors. In addition, different methods are used to extract features for different types of data, thereby improving the accuracy and efficiency of feature extraction and increasing the richness of feature vectors.

[0096] Next, continue to refer to Figure 2 As shown in , in step S230 , the feature vector is subjected to dimensionality reduction processing to determine a feature vector after dimensionality reduction, and a retrieval vector is generated based on the feature vector after dimensionality reduction.

[0097] In the disclosed embodiment, after obtaining the feature vector, since the dimension of the feature vector generated by the large language model is larger than the dimension used by the recommendation system, such as the output dimension of the Baichuan model is 4096 dimensions, which is much larger than the 100 dimensions used by the recommendation system, a dimensionality reduction method can be used to perform dimensionality reduction processing on the feature vector output by the large language model, so as to process the feature vector output by the large language model into a dimension that can be used by the recommendation system, thereby improving the adaptability of the two.

[0098] In some embodiments, data dimensionality reduction can be achieved through PCA (Principal Component Analysis). Specifically, high-dimensional data can be mapped to a low-dimensional space through linear transformation, while retaining as much information as possible in the original data. However, there is a problem with using PCA directly: for incremental data, in order to ensure that the new data and the previously generated data are in the same feature space, the two need to be spliced ​​together and calculated simultaneously during dimensionality reduction. This process reduces the efficiency of vector output, and the total number of vectors will increase with the accumulation of incremental data, and there are also high requirements for memory.

[0099] In order to solve the problem of data dimensionality reduction in principal component analysis, in the embodiment of the present disclosure, the dimensionality reduction of feature vectors can be achieved through a trained autoencoder (AE). Autoencoder is an unsupervised learning algorithm, which is mainly used for data dimensionality reduction and feature learning. Figure 6 As shown in Figure 1, the autoencoder consists of an encoder and a decoder. The encoder compresses the input feature vector into a low-dimensional representation, and then the decoder reconstructs the low-dimensional representation back to the original feature vector. The goal of the autoencoder is to minimize the reconstruction error, which refers to the difference between the original feature vector and the reconstructed vector reconstructed by the decoder.

[0100] First, the autoencoder can be trained to obtain a trained autoencoder. Figure 7 As shown in , the autoencoder includes an encoder and a decoder, and the encoder and the decoder can be trained respectively to obtain a trained autoencoder. In some embodiments, the encoder is used to map the original vector to obtain a latent vector of the target dimension. The dimension of the latent vector is lower than the vector dimension of the input original vector. The decoder is used to reconstruct the latent vector of the target dimension to obtain a reconstructed vector, wherein the dimension of the reconstructed vector is the same as the vector dimension of the original vector.

[0101] Based on this, the original vector can be a sample feature vector corresponding to the sample data. The sample feature vector corresponding to the sample data can be mapped by the encoder in the autoencoder to generate a latent vector of the target dimension; the sample data includes sample text data or sample image data. Based on this, the encoder can map the sample feature vector corresponding to the sample data to generate a latent vector of the target dimension. For example, the original vector is a sample feature vector output by a large language model, and the dimension of the sample feature vector can be, for example, 4096 dimensions. The encoder can map the sample feature vector to a latent vector of the target dimension. The encoder's mapping process of the sample feature vector can be expressed as formula (2):

[0102]

[0103] in, is the input sample feature vector, is the latent vector of the target dimension output by the encoder.

[0104] Next, the latent vector of the target dimension is reconstructed into a reconstructed vector with the same dimension as the sample feature vector through the decoder in the autoencoder.

[0105] After the encoder in the autoencoder reduces the dimension of the sample feature vector, the decoder also constructed by MLP is required to reduce the latent vector of the target dimension output by the encoder. Reconstruct the latent vector of the target dimension into a reconstructed vector with the same dimension as the original sample feature vector The decoder's reconstruction process of the latent vector of the target dimension can be expressed by formula (3):

[0106]

[0107] Furthermore, after the reconstruction vector is determined, the loss between the sample feature vector and the reconstruction vector can be determined, and the parameters of the encoder and the decoder in the autoencoder are updated based on the loss to train the encoder and the decoder to obtain a trained autoencoder. The loss is used to represent the reconstruction error, which can be determined based on the square value of the difference between the sample feature vector and the reconstruction vector, and can be specifically calculated according to formula (4):

[0108]

[0109] Exemplarily, the loss between the sample feature vector and the reconstructed vector can be minimized as a training objective, and the parameters of the encoder and the decoder can be updated to obtain a trained autoencoder. After obtaining the trained autoencoder, the structural parameters of the autoencoder can be frozen to perform dimensionality reduction processing through the trained autoencoder.

[0110] It should be noted that the sample data may include sample text data and sample image data. For the sample text data and sample image data, the corresponding automatic encoders may be trained independently.

[0111] After obtaining the trained autoencoder, the feature vector can be mapped by the encoder in the trained autoencoder to generate a latent vector of the target dimension. Exemplarily, the text features in the feature vector can be mapped by the encoder in the trained autoencoder to generate a text latent vector of the target dimension. The image features in the feature vector can be mapped by the encoder in the trained autoencoder to generate an image latent vector of the target dimension. The dimensions of the text latent vector of the target dimension and the image latent vector of the target dimension are both lower than the dimension of the feature vector. Moreover, the dimensions of the text latent vector of the target dimension and the image latent vector of the target dimension can be determined according to the dimensions that the recommendation system can handle.

[0112] If there is incremental multimodal data, the encoder in the trained autoencoder can be used to directly perform dimensionality reduction processing on the feature vectors corresponding to the incremental multimodal data, without the need to concatenate the incremental multimodal data with previously existing historical data to achieve dimensionality reduction processing. Incremental multimodal data refers to newly added multimodal data, such as text data and image data that may include newly added music or newly added playlists. For incremental multimodal data, the encoder in the trained autoencoder can be used to directly perform dimensionality reduction processing on the text features of the incremental multimodal data to obtain the text latent vector of the target dimension; and the image features of the incremental multimodal data can be directly reduced in dimension to obtain the image latent vector of the target dimension.

[0113] In the disclosed embodiment, since the feature vector can be directly mapped through the encoder in the trained autoencoder to obtain the text latent vector of the target dimension and / or the image latent vector of the target dimension, there is no need to use old historical data to jointly implement dimensionality reduction processing, thereby achieving an efficient and accurate dimensionality reduction process.

[0114] After obtaining the reduced-dimensional feature vector, a retrieval vector may be generated based on the reduced-dimensional feature vector. For example, the target-dimensional text latent vector and the target-dimensional image latent vector in the reduced-dimensional feature vector may be concatenated to determine the retrieval vector. For example, a 64-dimensional text latent vector and a 64-dimensional image latent vector may be concatenated to generate a 128-dimensional retrieval vector.

[0115] It should be added that after obtaining the retrieval vector, the quality of the retrieval vector generated by the large language model can be evaluated by the similarity between the retrieval vector and the reference vector. The process of evaluating the quality of the retrieval vector can be implemented through Faiss (Facebook AI Similarity Search). Faiss represents a library for efficient similarity retrieval and clustering. Exemplarily, an index can be constructed for each retrieval vector, the index including multiple reference vectors; the similarity between the retrieval vector and each reference vector in the index is determined; based on the similarity, at least one candidate vector similar to the retrieval vector is determined from the multiple reference vectors; and the quality of the retrieval vector is evaluated based on whether the similarity of at least one candidate vector meets the similarity condition.

[0116] Among them, for music application platforms, the search vector can be a feature vector corresponding to a certain piece of music. The index refers to a list of music that is compared with the music. The index may include multiple pieces of music, and each piece of music may correspond to a reference vector. Based on this, the similarity between the search vector and the reference vector corresponding to each piece of music in the index can be calculated. The similarity here can be cosine similarity or other forms. After determining the similarity between the search vector and each reference vector in the index, the similarities can be sorted in descending order, and the reference vectors corresponding to the similarities ranked in the first K positions are used as at least one candidate vector similar to the search vector. Reference Figure 8 As shown in , for a retrieval vector, at least one candidate vector can be searched from the index as a similar vector.

[0117] Furthermore, the quality of the retrieval vector generated by the large language model may be evaluated based on whether the similarity of the at least one selected candidate vector satisfies a similarity condition. Specifically, when the similarity satisfies the similarity condition, it may be considered that the quality of the retrieval vector meets the requirement. The similarity condition may be, for example, greater than a similarity threshold, and the similarity threshold may be determined according to actual conditions.

[0118] Furthermore, the accuracy of the search vector can also be evaluated based on the candidate vector obtained based on the search vector. Fig. 9As shown in , when the search vector corresponds to the search image, if the image corresponding to the retrieved candidate vector is candidate image 901 instead of candidate image 902, the search vector generated by the large language model can be considered to be more accurate. For the search vector corresponding to the search text, the search results not only need to capture the emotions and feelings in the text, but also need to go deep into the specific emotional content understanding. For example, if the search text is "Warm-hearted folk songs | Save happiness to cope with hard days", the similarity of the candidate text "Turn off the trivial troubles, may you have a good sleep" is higher than the candidate text "Folk song selection Your name is as beautiful as poetry". When the text corresponding to the retrieved candidate vector is "Turn off the trivial troubles, may you have a good sleep", the search vector generated by the large language model can be considered to be more accurate.

[0119] In the disclosed embodiment, by using a library for efficient similarity retrieval and clustering represented by Faiss, similarity retrieval verification is performed on the retrieval vector, which can accurately verify the accuracy of the vector generated by the large language model and the vector after dimensionality reduction.

[0120] In step S240, information retrieval is performed according to the retrieval vector to determine the retrieval result corresponding to the multimodal data.

[0121] In the disclosed embodiment, the target user has multimodal data for representing historical music and historical playlists in the application platform, and the text features and image features of the multimodal data can be extracted through a large language model, and then a retrieval vector is generated based on the text features and image features. In the application platform, multiple retrieval results matching the retrieval vector are searched according to the retrieval vector. Exemplarily, multiple similarity vectors similar to the retrieval vector can be obtained, and the results corresponding to the similarity vectors are further used as retrieval results. In addition, the description text of the retrieval result and the cover image of the retrieval result can also be displayed at a preset position of the retrieval result. The preset position can be the lower side of the retrieval result or other suitable position, and the description text and cover image are displayed independently.

[0122] Exemplarily, the application platform is a music application platform. The target user has historical music and historical playlists in the application platform, and both the historical music and the historical playlists contain multimodal data composed of text data and image data. Based on this, the feature vector of the multimodal data can be extracted through a large language model, and the feature vector is reduced in dimension, and a retrieval vector is generated according to the feature vector after the dimension reduction. According to the text features and image features in the retrieval vector, music retrieval is performed in the application platform through multiple dimensions, and the function of automatically recommending retrieval results to the target user based on historical music and historical playlists is realized. Among them, the retrieval results can be used to represent the recommendation results automatically recommended based on historical music and historical playlists, which can specifically be music and playlists similar to historical music and historical playlists.

[0123] In the disclosed embodiments, the target user does not need to input anything, and the search results similar to the historical music and historical playlists can be automatically recommended based on the historical music and historical playlists of the target user in the application platform. Furthermore, the search results can be displayed on the display interface of the application platform, and the description text of the search results can be displayed at the bottom of the search results. In addition, the cover image of the search results can also be displayed, and the cover image can be an image that meets the preferences of the target user, for example, it can be similar to the image data of the historical music and historical playlists. Since personalized recommendations can be made based on the text data and image data of the historical playlists and historical music, the accuracy and comprehensiveness of the recommended search results can be improved.

[0124] Fig.10 The flowchart of recommending search results is schematically shown in FIG. Fig.10 As shown in , it mainly includes the following steps:

[0125] In step S1002, a data set is constructed, wherein the data set includes multimodal data of the target user in the application platform, and the multimodal data includes text data and image data.

[0126] In step S1004, feature extraction is performed on the multimodal data based on the large language model to obtain a feature vector. Exemplarily, feature extraction is performed on the text data based on the large language model to obtain text features. Feature extraction is performed on the image data based on the large language model to obtain image features. Among them, feature extraction can be directly performed on text data whose length is less than or equal to the length threshold, and text data whose length is greater than the length threshold is compressed and feature extraction is performed on the compressed text data based on the large language model. For image data, the image data can be divided into multiple image blocks, and feature extraction is performed based on the ViT model to obtain image features.

[0127] In step S1006, the feature vector is subjected to dimensionality reduction processing to obtain a reduced-dimensional feature vector. For example, the feature vector is mapped by an encoder in a trained autoencoder to obtain a text latent vector of a target dimension and an image latent vector of a target dimension.

[0128] In step S1008, a retrieval vector is generated based on the feature vector after dimensionality reduction. For example, the text latent vector of the target dimension and the image latent vector of the target dimension may be concatenated to obtain the retrieval vector.

[0129] In step S1010, the quality of the retrieval vector is verified and evaluated.

[0130] In step S1012, information retrieval is performed based on the retrieval vector to obtain the retrieval result of the target user. For example, the retrieval result may be music or a playlist similar to the retrieval vector.

[0131] In the disclosed embodiment, since feature vectors can be extracted through multimodal data such as text data and image data, the problem of not being able to fully understand visual information due to using only single-dimensional data is avoided, feature vectors can be accurately extracted, limitations can be avoided, and the comprehensiveness of feature vectors can be improved. By directly mapping feature vectors through trained autoencoders, there is no need to reduce dimensions at the same time as old data to achieve data dimensionality reduction, which can improve the efficiency and accuracy of data dimensionality reduction. In addition, for long text data, short text data, and image data, different methods are used to implement feature extraction based on a large language model, which can improve the extraction efficiency of feature vectors of multimodal data, so that the generation speed of feature vectors can meet the requirements of low time consumption of the recommendation system. In addition, it can capture more complex language patterns and deeper semantic information, which can improve the accuracy and richness of feature extraction. The accuracy, comprehensiveness and reliability of retrieval results can be further improved.

[0132] Exemplary Devices

[0133] Next, refer to Fig.11 The information retrieval device according to the exemplary embodiment of the present disclosure is described. Fig.11 As shown, the information retrieval device 1100 may include:

[0134] The data acquisition module 1101 is used to acquire multimodal data corresponding to the target user in the application platform;

[0135] A feature extraction module 1102 is used to extract features from the multimodal data based on a large language model to obtain a feature vector of the multimodal data;

[0136] A dimension reduction processing module 1103 is used to perform dimension reduction processing on the feature vector to determine a feature vector after dimension reduction, and generate a search vector based on the feature vector after dimension reduction;

[0137] The search result determination module 1104 is used to perform information search according to the search vector and determine the search result corresponding to the multimodal data.

[0138] In an exemplary embodiment of the present disclosure, the multimodal data includes text data and / or image data; the feature extraction module includes: a text feature extraction module, used to extract features from the text data in the multimodal data to obtain text features; an image feature extraction module, used to extract features from the image data in the multimodal data to obtain image features.

[0139] In an exemplary embodiment of the present disclosure, the text feature extraction module includes: a short text feature extraction module, which is used to extract features of the text data through a large language model to obtain text features if the length of the text data is less than or equal to a length threshold; and a long text feature extraction module, which is used to compress the text data if the length of the text data is greater than the length threshold, and extract features of the compressed text data through a large language model to obtain text features.

[0140] In an exemplary embodiment of the present disclosure, the short text feature extraction module includes: a word segmentation module, which is used to segment text data through a word segmenter to obtain segmented text data, and process the segmented text data into a text sequence; a hidden layer processing module, which is used to input the text sequence into a large language model for feature extraction to obtain a hidden layer output result; and a pooling module, which is used to average pool the hidden layer output result to obtain text features.

[0141] In an exemplary embodiment of the present disclosure, the hidden layer processing module includes: a normalization module, which is used to normalize the text sequence and determine the normalized text sequence; an intermediate feature extraction module, which is used to extract features from the normalized text sequence through a multi-head attention mechanism layer to obtain intermediate features; a fusion module, which is used to fuse the text sequence and the intermediate features to determine the fusion features; a feedforward neural network processing module, which is used to normalize the fusion features, and perform weighted summation and nonlinear transformation on the normalized fusion features based on the feedforward neural network to obtain the hidden layer output result.

[0142] In an exemplary embodiment of the present disclosure, the long text feature extraction module includes: a prompt word determination module for determining prompt words; and a text compression module for inputting prompt words into a large language model to extract key texts from text data to compress the text data.

[0143] In an exemplary embodiment of the present disclosure, the image feature extraction module is configured to: divide the image data into multiple image blocks; add the embedding vector and the image mark vector of each image block to the corresponding position vector to determine the image sequence; perform feature extraction on the image sequence to obtain sequence features; perform linear processing on the sequence features to generate image features.

[0144] In an exemplary embodiment of the present disclosure, the feature vector includes text features and image features; the dimensionality reduction processing module includes: a text dimensionality reduction module, which is used to map the text features through the encoder in the trained autoencoder to generate a text latent vector of the target dimension; an image dimensionality reduction module, which is used to map the image features through the encoder in the trained autoencoder to generate an image latent vector of the target dimension.

[0145] In an exemplary embodiment of the present disclosure, the dimensionality reduction processing module includes: a splicing module, which is used to splice the text latent vector of the target dimension and the image latent vector of the target dimension to determine the retrieval vector.

[0146] In an exemplary embodiment of the present disclosure, the device also includes: an incremental dimensionality reduction module, which is used to perform dimensionality reduction processing on the feature vector corresponding to the incremental multimodal data through an encoder in a trained autoencoder when there is incremental multimodal data.

[0147] In an exemplary embodiment of the present disclosure, the device also includes: a dimensionality reduction module, which is used to map the sample feature vector corresponding to the sample data through an encoder in an autoencoder to generate a latent vector of a target dimension; the sample data includes sample text data or sample image data; a reconstruction module, which is used to reconstruct the latent vector of the target dimension into a reconstructed vector with the same dimension as the sample feature vector through a decoder in the autoencoder; and an autoencoder update module, which is used to determine the loss between the sample feature vector and the reconstructed vector, and update the parameters of the encoder and the decoder in the autoencoder based on the loss to obtain a trained autoencoder.

[0148] In an exemplary embodiment of the present disclosure, after generating the retrieval vector, the apparatus further includes: a retrieval vector evaluation module, configured to evaluate the quality of the retrieval vector according to the similarity between the retrieval vector and a reference vector.

[0149] In an exemplary embodiment of the present disclosure, a retrieval vector evaluation module includes: an index construction module, which is used to construct an index for each retrieval vector, and the index includes multiple reference vectors; a similarity determination module, which is used to determine the similarity between the retrieval vector and each reference vector in the index; a candidate vector determination module, which is used to determine at least one candidate vector similar to the retrieval vector from multiple reference vectors based on the similarity; and an evaluation module, which is used to evaluate the quality of the retrieval vector based on whether the similarity of at least one candidate vector meets a similarity condition.

[0150] In an exemplary embodiment of the present disclosure, the search result determination module is configured to: determine the search results matching the search vector in the application platform, and display the cover image and description text corresponding to the search results on the display interface of the application platform.

[0151] It should be noted that the specific details of each module of the information retrieval device have been described in detail in the steps of the corresponding information retrieval method, and the specific details of each module of the information retrieval device have been described in detail in the steps of the corresponding information retrieval method, so they will not be repeated here.

[0152] Refer to the following Fig.121200 according to this embodiment of the present disclosure is described. Fig.12 The electronic device 1200 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0153] like Fig.12 As shown, the electronic device 1200 is in the form of a general computing device. The components of the electronic device 1200 may include, but are not limited to: the at least one processing unit 1210, the at least one storage unit 1220, a bus 1230 connecting different system components (including the storage unit 1220 and the processing unit 1210), and a display unit 1240. The bus 1230 may include a data bus, an address bus, and a control bus.

[0154] The storage unit stores program codes, which can be executed by the processing unit 1210, so that the processing unit 1210 performs the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 1210 can perform the following steps: Figure 2 Steps shown.

[0155] The storage unit 1220 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 12201 and / or a cache storage unit 12202 , and may further include a read-only storage unit (ROM) 12203 .

[0156] The storage unit 1220 may also include a program / utility 12204 having a set (at least one) of program modules 12205, such program modules 12205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0157] The bus 1230 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, a graphics acceleration interface, a processing unit, or a local bus using any of a variety of bus architectures.

[0158] The electronic device 1200 may also communicate with one or more external devices 1300 (e.g., keyboards, pointing devices, Bluetooth devices, etc.). Such communication may be performed via an input / output (I / O) interface 1250. Furthermore, the electronic device 1200 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 1260. As shown, the network adapter 1260 communicates with other modules of the electronic device 1200 via a bus 1230. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0159] It should be noted that, in some embodiments of the present disclosure, a computer program product is also provided. The computer program product includes a computer program. When the computer program is executed by a processor, the above method is implemented.

[0160] In one embodiment, the computer program product may be a tangible product containing a computer program, such as a computer-readable storage medium storing a computer program. The readable storage medium may be a storage medium based on electrical, magnetic, optical, electromagnetic, infrared, or other signals, including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory (Flash), mechanical hard disk (HDD), solid-state drive (SSD), and the like. Exemplarily, the computer program product may be implemented as a non-volatile storage medium storing a computer program, such as a read-only memory, a NAND flash memory, and the like.

[0161] In one embodiment, the computer program product may be an intangible product including a computer program. Exemplarily, the computer program product may be implemented as a virtual digital product, such as a digital file storing an executable file, an installation package, etc. of the computer program.

[0162] The code of the computer program can be written in one or more programming languages. Programming languages ​​such as C language, Java, C++, etc. The program code can be executed entirely on the user computing device, or partially on the user computing device, or as a separate software package, or partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, such as a local area network (LAN), a wide area network (WAN), etc., or can be connected to an external computing device (e.g., an Internet connection provided by an operator).

[0163] The computer program may be carried or transmitted via electrical, magnetic, optical, electromagnetic, infrared, or other signals. The electronic device may convert the signal carrying the computer program into a digital signal, thereby running the computer program. When the computer program is run on an electronic device, its code is used to enable the electronic device to execute (more specifically, the processor of the electronic device may execute) the method steps of various exemplary embodiments of the present disclosure.

[0164] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0165] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0166] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0167] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing what is disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and embodiments are to be considered merely as exemplary, and the true scope and spirit of the present disclosure are indicated by the claims.

[0168] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An information retrieval method, characterized in that: include: Obtain multimodal data corresponding to target users in the application platform; Extracting features from the multimodal data based on a large language model to obtain feature vectors of the multimodal data; Performing dimensionality reduction processing on the feature vector to determine a feature vector after dimensionality reduction, and generating a search vector based on the feature vector after dimensionality reduction; Information retrieval is performed according to the retrieval vector to determine a retrieval result corresponding to the multimodal data.

2. The information retrieval method according to claim 1, characterized in that: The multimodal data includes text data and / or image data; The extracting features of the multimodal data based on the large language model to obtain a feature vector of the multimodal data includes: Performing feature extraction on text data in the multimodal data to obtain text features; Feature extraction is performed on the image data in the multimodal data to obtain image features.

3. The information retrieval method according to claim 2, characterized in that: The extracting features of the text data in the multimodal data to obtain text features includes: If the length of the text data is less than or equal to the length threshold, extracting features from the text data using a large language model to obtain text features; If the length of the text data is greater than the length threshold, the text data is compressed, and feature extraction is performed on the compressed text data through a large language model to obtain text features.

4. The information retrieval method according to claim 3, characterized in that: The extracting features of the text data by using a large language model to obtain text features includes: Segmenting the text data by a word segmenter to obtain segmented text data, and processing the segmented text data into a text sequence; Inputting the text sequence into a large language model for feature extraction to obtain a hidden layer output result; Average pooling is performed on the hidden layer output result to obtain the text feature.

5. The information retrieval method according to claim 4, characterized in that: The step of inputting the text sequence into a large language model for feature extraction to obtain a hidden layer output result includes: Normalizing the text sequence to determine a normalized text sequence; Extracting features from the normalized text sequence through a multi-head attention mechanism layer to obtain intermediate features; Fusing the text sequence and the intermediate features to determine a fused feature; The fusion features are normalized, and weighted summation and nonlinear transformation are performed on the normalized fusion features based on the feedforward neural network to obtain the hidden layer output result.

6. The information retrieval method according to claim 3, characterized in that: The compressing the text data comprises: Determine the prompt word; The prompt words are input into a large language model to extract key texts from the text data, so as to compress the text data.

7. The information retrieval method according to claim 2, characterized in that: The extracting features of the image data in the multimodal data to obtain image features includes: dividing the image data into a plurality of image blocks; Adding the embedding vector and the image mark vector of each image block to the corresponding position vector respectively to determine an image sequence; Extracting features from the image sequence to obtain sequence features; The sequence features are linearly processed to generate the image features.

8. The information retrieval method according to claim 1, characterized in that: The feature vector includes text features and image features; The step of performing dimensionality reduction processing on the feature vector to determine a feature vector after dimensionality reduction includes: Mapping the text features through the encoder in the trained autoencoder to generate a text latent vector of the target dimension; The image features are mapped through the encoder in the trained autoencoder to generate an image latent vector of the target dimension.

9. The information retrieval method according to claim 8, characterized in that: The step of generating a retrieval vector based on the feature vector after dimension reduction includes: The text latent vector of the target dimension and the image latent vector of the target dimension are concatenated to determine the retrieval vector.

10. The information retrieval method according to claim 8, characterized in that: The method further comprises: When there is incremental multimodal data, the encoder in the trained autoencoder is used to perform dimension reduction processing on the feature vector corresponding to the incremental multimodal data.

Images