Large model and multi-mode recommendation method and device, electronic equipment and medium

Abstract

The invention provides a large-model and multi-mode recommendation method and device, electronic equipment and a medium. The recommendation method comprises the steps that online feature extraction is carried out, and extracted features are combined to obtain fusion features; creating a cue word project of the sorting task according to the fusion features; the cue word project of the sorting task is input into a large language model to obtain a sorting result, and the sorting result comprises the interpretability of the sorting result. According to the technical scheme, the working efficiency is greatly improved, the sorting task can be completed in a zero-sample and / or few-sample mode, the interpretability of the sorting result is provided by the sorting method provided by the invention, so that the credibility of the user on the sorting result is enhanced, and the user experience is improved. And research and development personnel can conveniently explain the internal mechanism of the model in the business direction when putting into use in the future.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence, and specifically relates to a large-model and multimodal recommendation method, device, electronic device and medium. Background Art

[0002] The existing recommendation algorithms or ranking methods have a heavy workload. This is because the existing recommendation algorithms or ranking methods are constructed in the following two ways.

[0003] First, models must be trained to complete their tasks, but these trained models lack interpretability. Because these trained models lack interpretability (interpretability of large models refers to the ability to understand and explain the model's decision-making process and prediction results. As models become more complex, especially deep learning models, their "black box" nature makes understanding their internal mechanisms difficult, making interpretability a key issue), they cannot provide feedback to developers, leaving them unsure which parameters to modify to achieve the desired model results.

[0004] Second, in addition to the large language model (large language model: a large language model is a deep learning model trained based on large-scale text data that can understand and generate natural language text. Through large-scale pre-training, Transformer architecture, multi-task learning, and contextual understanding, the large language model performs well in tasks such as text generation, machine translation, question-answering systems, text summarization, sentiment analysis, and dialogue systems.), a database is built using the recall ranking method. However, building a database often relies on a lot of manual work, such as underlying data collection, data cleaning, feature engineering, online services, model training, and model launch. After switching business scenarios, the above processes need to be rebuilt, and the overall iteration cost and engineering resources required are very high, often requiring a team of hundreds of people to support. Summary of the Invention

[0005] In order to solve the problem in the prior art that model training consumes a lot of manpower, the purpose of the present invention is to provide a large-model and multi-modal recommendation method, device, electronic device and medium.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions: Acquire user-side features online, acquire resource-side features online, extract title features from the resource-side features using a large language model based on the prompt word project, and extract image features from the resource-side features using a multimodal large model based on the prompt word project, combine the extracted title features and image features to obtain fused features; and create a prompt word project for the sorting task based on the fused features. The prompt word engineering of the sorting task is input into a large language model to obtain a sorting result, wherein the sorting result includes the explainability of the sorting result.

[0007] Furthermore, extracting title features from the resource-side features using a large language model based on the prompt word project; and extracting image features from the resource-side features using a multimodal large model based on the prompt word project, and combining the extracted title features and image features to obtain fused features includes: Predetermining a title feature extraction standard, using the predetermined title feature extraction standard as part of a prompt word project, and extracting title features through a large language model based on the prompt word project; Predetermining an extraction standard for image features, using the predetermined extraction standard as part of a prompt word project, extracting image features using a multimodal large model based on the prompt word project, and extracting second text information from the image features based on the image features; Furthermore, vectorizing the extracted title features and the image features to obtain vector features, and combining the vector features to obtain fusion features includes: Processing the first text information in the title feature into a numerical feature; Processing the second text information into numerical features; The numerical features of the first text information and the numerical features of the second text information are input into a semantic model for vectorization processing to obtain vector features, and the vector features are combined to obtain fusion features.

[0008] Furthermore, in the step of creating a prompt word project for the sorting task based on the fusion features: The cue word engineering uses the COT framework.

[0009] Furthermore, in the step of creating a prompt word project for the sorting task based on the fusion features: The prompt word project uses the ICIO framework.

[0010] In a second aspect, the present invention provides a recommendation device based on a large model fusing multimodal features, comprising: A feature acquisition device for online feature extraction, wherein the feature acquisition device includes a user feature acquisition device and a resource feature acquisition device, wherein the user feature acquisition device is used to online acquire user-side features, and the resource feature acquisition device is used to online acquire resource-side features, including: extracting title features from resource-side features, wherein the title features extracted from the resource-side features are obtained through a large language model based on a prompt word project; and extracting image features from the resource-side features, wherein the image features extracted from the resource-side features are obtained through a multimodal large model based on a prompt word project, and combining the extracted title features and image features to obtain a fused feature; A prompt word device, used for creating a prompt word project for the sorting task based on the fusion features; A large language model sorting device is used to input the prompt word engineering of the sorting task into the large language model to obtain a sorting result, wherein the sorting result includes the interpretability of the sorting result. Furthermore, the resource feature acquisition device includes: A title feature extraction device, which predetermines a title feature extraction standard, uses the predetermined title feature extraction standard as part of a prompt word project, and extracts title features based on the prompt word project using a large language model; An image feature extraction device predefines an extraction standard for image features, uses the predetermined image feature extraction standard as part of a prompt word project, extracts image features based on the prompt word project using a multimodal large model, and extracts second text information from the image features based on the image features; The fusion device is used to perform vectorization processing on the extracted title features and the image features to obtain vector features, and combine the vector features to obtain fusion features.

[0011] Furthermore, the fusion device comprises: a vector processing device for processing the first text information in the title feature into a numerical feature, processing the second text information into a numerical feature, and inputting the numerical features of the first text information and the numerical features of the second text information into a semantic model for vectorization processing to obtain vector features; The vector fusion device is used to combine the vector features to obtain fusion features.

[0012] According to a third aspect of the present invention, an electronic device is provided, comprising a processor and a memory, wherein the processor is configured to execute a computer program stored in the memory to implement a large model and multimodal recommendation method.

[0013] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores at least one instruction, and when the at least one instruction is executed by a processor, a large model and multimodal recommendation method is implemented.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. The large model and multimodal recommendation method of the present invention extracts features online, combines the extracted features to obtain fused features, creates a prompt word project for the sorting task based on the fused features, and inputs the prompt word project for the sorting task into the large language model to obtain a sorting result, wherein the sorting result includes the sorting result, which can greatly shorten the iteration cycle.

[0015] 2. The large model and multimodal recommendation method of the present invention extracts title features through a large language model according to predetermined information extraction standards, and processes the keywords into numerical features; according to predetermined information extraction standards, extracts pictures through a multimodal large model, extracts second text information from the pictures, and processes the second text information into numerical features; there is no need to train the model, and the sorting task can be completed with zero samples and / or few samples.

[0016] 3. Due to the large model and multimodal recommendation method provided by the present invention, through online feature extraction, the extracted features are combined to obtain fused features; a prompt word project for the sorting task is created based on the fused features; the prompt word project for the sorting task is input into the large language model to obtain a sorting result, wherein the sorting result includes the interpretability of the sorting result, which enhances the user's trust in the sorting result, and can assist R&D personnel in understanding the model, selecting the model, and providing an important basis for optimizing model parameters, explaining the internal mechanism of the model to the business direction when it is put into use in the future. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are intended to explain the present invention and do not constitute an undue limitation of the present invention. In the accompanying drawings: Figure 1 A flowchart of a large model and multimodal recommendation method according to an embodiment of the present invention; Figure 2 A flowchart of online feature acquisition according to an embodiment of the present invention; Figure 3 This is a structural block diagram of a recommendation device based on a large model fusing multimodal features according to an embodiment of the present invention; Figure 4 This is a structural block diagram of an electronic device for a large model and multimodal recommendation method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The present application will be described in detail below with reference to the attached drawings and embodiments. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0019] The following detailed description is exemplary in nature and is intended to provide further description of the present application. Unless otherwise defined, all technical terms used herein are to be interpreted according to their ordinary meaning in the technical field of the present application. The terminology used in the present application is for the purpose of describing particular embodiments only and is not intended to limit the example embodiments according to the present application.

[0020] Example 1 In order to solve the problems of unexplainability, large amount of manual work and long iteration cycle of the pre-trained model in the prior art, the present application provides a large model and multi-modal recommendation method. The present application extracts the features of the user side and the resource side, fuses the extracted features, adopts COT framework (COT, Chain of Thought, the "thought chain" of the large model refers to the ability of the model to solve problems step by step by generating a series of intermediate reasoning steps when dealing with complex tasks. This ability enables the model to perform multi-step reasoning like humans, thereby better understanding and solving tasks that require high-level cognition.) or ICIO framework (Instruction, Context, Input Data, Output indicator, instruction, background information, input data, output data) to create prompt engineering, input the large language model from the prompt engineering (Prompt Engineering, the prompt engineering of the large model refers to the technology of carefully designing and optimizing input prompts to guide the model to generate expected output. Prompt engineering is crucial in the application of large models, as it directly affects the performance and output quality of the model.), so as to complete the collection of resource side data in a very short time, and use the reasoning ability of the large language model to build a recommendation method (or ranking method) in a zero-shot (Zero-shot Learning, a machine learning method that aims to let the model complete the inference and prediction of new tasks or new categories without any task-related training data. The core idea of this method is to use the general knowledge learned by the model in the pre-training stage to infer how to solve new tasks through task description or semantic association.) or few-shot (Few-shot Learning, a machine learning method that aims to let the model quickly learn and complete new tasks with only a small amount of task-related training data, usually a few to tens of samples. Unlike zero-shot learning, few-shot learning provides a small amount of examples as context to help the model understand the task pattern and generate output that meets the requirements.) way. Through the method of the present application, the iteration cycle for recommendation can be greatly shortened, and the explainability of the model for ranking can be obtained, which facilitates the improvement of the ranking result by the technical personnel, thereby greatly improving the work efficiency compared with the prior art.

[0021] The large model and multi-modal recommendation method of the application is suitable for recommendation in various scenes. For example, it can be used for business travel scene recommendation in the scenes of air tickets, train tickets and hotels, and can also be used for e-commerce shopping, video recommendation and news recommendation. Taking a business travel recommendation as an example, the user inputs various conditions of the hotel expected to stay in, and the recommendation method of the application fuses the features by acquiring the user-side features and the resource-side features (features related to multiple hotels), constructs a prompt word engineering, inputs the prompt word engineering into a large language model, and the large language model gives a recommendation result. Meanwhile, the large language model also gives the explainability of the recommendation result given by the technical personnel. Therefore, the recommendation method of the application is extremely lightweight. If it is necessary to iterate the recommendation algorithm, the technical personnel can optimize the recommendation result according to the specific content given by the explainability, solving the defect that the traditional pre-training model does not have explainability.

[0022] One embodiment of the application provides a large model and multi-modal recommendation method, comprising: online acquisition of user-side features, online acquisition of resource-side features, extraction of title features from the resource-side features by a large language model according to a prompt word engineering, and extraction of picture features from the resource-side features by a multi-modal large model according to the prompt word engineering, combination of the extracted title features and picture features to obtain fused features; creation of a prompt word engineering of a sorting task according to the fused features; input of the prompt word engineering of the sorting task into a large language model to obtain a sorting result; wherein the sorting result includes the explainability of the sorting result.

[0023] The above technical solution extracts title features from the resource-side features by a large language model according to a prompt word engineering, and extracts picture features from the resource-side features by a multi-modal large model according to the prompt word engineering, solving the problem of low feature extraction efficiency in the prior art and realizing the demand for rapid construction of small and medium-sized sorting scenes. Meanwhile, the technical features of the application, that is, the creation of a prompt word engineering of a sorting task according to fused features, the input of the prompt word engineering of the sorting task into a large language model to obtain a sorting result, solve the technical problem of the prior art that a large amount of manual work is required to build a small and medium-sized sorting scene. The application can solve the sorting problem by inputting the prompt word engineering of the sorting task into the large language model, which greatly shortens the online cycle of the small and medium-sized sorting scene. Meanwhile, the sorting result of the application can provide the explainability of the sorting task, which provides a technical basis for researchers to make improvements. When the recommendation method of the application is put into use in the future, it is convenient for researchers to explain the internal mechanism of the model to business personnel. In order to further explain and describe the application, another specific embodiment based on the same concept as the above-mentioned embodiment is provided. Please refer to Figure 1 . Figure 1 is a flowchart of a large model and a multi-modal recommendation method of an embodiment of the present application.

[0024] Step 100, online feature extraction, the extracted features are combined to obtain fused features.

[0025] The features of the present application include but are not limited to: user features, resource features (material features).

[0026] The online feature extraction of the present application includes two sub-steps, steps 110 and 120.

[0027] Step 110, online acquisition of user-side features. The user-side features of the present application are, for example, the user's age, gender, preferences, etc. Step 120, online acquisition of resource-side features. Taking business travel as an example, resource-side features are, for example, the geographic location, price, star rating, etc. of a hotel. In step 120, online acquisition of resource-side features specifically includes: extracting title features from resource-side features, wherein the title features are extracted from resource-side features according to prompt word engineering and obtained by a large language model; and extracting picture features from resource-side features, wherein the picture features are extracted from resource-side features according to prompt word engineering and obtained by a multi-modal large model. That is, the present application inputs prompt word engineering into a large language model and inputs prompt words into a multi-modal large model when extracting features. By inputting prompt word engineering into a large language model and inputting prompt word engineering into a multi-modal large model for feature extraction, the efficiency of feature extraction is greatly improved. In the present application, online acquisition of resource-side features specifically includes three sub-steps: steps 121, 122 and 123.

[0028] Figure 2 is a flowchart of online feature acquisition of an embodiment of the present application.

[0029] The technical solution of the present application adds two groups of features compared to the prior art, one is the title feature, and the other is the picture feature. The prior art only includes: user features, hotel features and interaction features. Among them, the user features include: age, gender, region, education level, income level, etc.; the hotel features include: brand, location, business district, etc.; the interaction features specifically include: user's price preference, location preference, brand preference, etc. of the hotel. Specifically, the title features, user features, hotel features, interaction features and picture features can be obtained through steps 121 and 122.

[0030] Compared with the prior art, the title feature of the present application includes the basic title features extracted by the large language model (such as brand words, location, star rating, etc.) and vector features, that is, the title feature of the present application includes not only the text attributes of the hotel rating, popular time point, surrounding scenic spots, traffic conditions and the like of the prior art, but also vector attributes. This makes the title feature of the present application have better expressiveness.

[0031] Compared with the prior art, the picture feature of the present application can further extract the theme, hotel rating, cleanliness, supporting equipment conditions (such as air conditioning, washroom) and other features that are not easy to be mined through conventional technical means. That is, the picture feature of the present application considers the information (such as title and picture) that can be intuitively perceived by the user, and after further extraction and processing by the large language model, the potential interest preferences of the user can be deeply expressed, which makes the picture feature of the present application more capable of conveying emotions and atmosphere than the prior art.

[0032] Referring to Figure 2 , step 121, the extraction standard of the title feature is prepared in advance, and the first text information in the title feature is processed into a numerical feature according to the extraction standard of the title feature prepared in advance as part of the prompt word engineering, and the title feature is extracted by the large language model according to the prompt word engineering.

[0033] Specifically, the design (or extraction, extraction) standard of the title feature is prepared in advance, that is, the title feature extraction standard of the present application can be prepared in advance according to the business type. This step is used to process text information (the title itself belongs to a kind of text, and the title feature is extracted from a plurality of text information). In other words, the title feature required by the embodiment of the present application is extracted from a plurality of text information. The title feature required by the embodiment of the present application is extracted by the prompt word engineering and the large language model of the embodiment of the present application according to the pre-prepared title feature extraction standard. The pre-prepared title feature extraction standard is, for example, the extraction standard of key identification, the extraction standard of spatial positioning, the extraction standard of operating attribute and the extraction standard of auxiliary information.

[0034] The title feature extraction standard of the present application can be: (1) the extraction standard of key identification, including: must be extracted: legal registered name (excluding promotional / descriptive suffix); must be identified: chain brand / group (including implied brand such as "Beijing Guomao Hotel (Shangri-La)"), error example: "Sanya special luxury sea view room" is an illegal name.

[0035] (2) The extraction criteria for spatial positioning include: city and administrative district where the city is located (100% required), landmark, business district and / or scenic spot (where scenic spot extraction takes precedence over administrative district). Incorrect example: "Shanghai Hotel" is an incorrect example because it lacks a specific location.

[0036] (3) The criteria for drawing up business attribute grants include: Type: business, vacation, homestay and / or apartment, etc., Grade: five-star, luxury and / or economy, etc. Incorrect example: "boutique hotel" is an incorrect example because it does not reflect the type or grade.

[0037] (4) The extraction criteria of auxiliary information include: branch logo (such as “Pudong Store”), architectural features (such as “tower”, “lakeside”), and promotional words.

[0038] Step 121 of the present invention uses a large language model to extract title features. This method significantly improves extraction efficiency by extracting title features using a large language model. For example, using a business travel scenario as an example, based on the aforementioned criteria for title feature extraction, the present invention uses a large language model to extract title features including, but not limited to, key identifiers, spatial location, business attributes, and auxiliary information.

[0039] Specifically, the title feature extraction standard can be used to make a prompt word engineering, the prompt word engineering is input into a large language model, and the large language model extracts the title feature according to the prompt word engineering. For example, one prompt word engineering of an embodiment of the present application can be "You are a professional hotel information structured expert, please extract the core information from the title according to the following priority and scoring standard: the key identification includes: must extract: the legal registered name (exclude promotional / descriptive suffix); must identify: chain brand / group (including implied brands such as "Beijing Guomao Hotel (Shangri-La)"), the error example is "Sanya special luxury sea view room", which is an illegal name. Spatial positioning includes: city and administrative district where the city is located (100% required), landmark, business district and / or scenic spot (among them, scenic spots are preferred to administrative districts), the error example is "Shanghai hotel", which is an error example because it lacks a specific location. Operating attribute includes: type: business, vacation, homestay and / or apartment, etc., grade: five-star, luxury and / or economy, etc., the error example is "boutique hotel", which is an error example because it does not reflect the type or grade. Auxiliary information includes: branch identification (such as "Pudong store"), architectural features (such as "tower", "lakefront"), promotional words. That is, through the prompt word engineering, the large language model is told which information to extract (the title feature of the embodiment of the present application), and the standard for extracting the information (the extraction standard of the title feature of the embodiment of the present application) is also told to the large language model, so that accurate title features can be extracted. Specifically, the title features extracted by the large language model after the prompt word engineering is input into the large language model can be output in JSON (JavaScript Object Notation, a data exchange format) format to the prompt word engineering of the following step 200.

[0040] The first text information of the title feature is processed into a numerical feature. The first text information is extracted by the large language model: key word identification, spatial positioning, operating attribute and auxiliary information. The first text information is processed into a numerical feature. For example, a general algorithm can be used to process the first text information into a numerical feature. Among them, the one-hot method can be used to process the first text information into a numerical feature.

[0041] Step 122, the extraction standard of the picture feature is prepared in advance, the picture feature is extracted by a multi-modal large model according to the prompt word engineering according to the extraction standard of the picture feature as part of the prompt word engineering, the second text information is extracted from the picture feature, and the second text information is processed into a numerical feature.

[0042] Specifically, an extraction standard of picture features is prepared in advance, and the extraction standard of picture features can be prepared in advance according to a business type. This step is used to process image type information. That is, the extraction of picture features from the image type file is completed by the prompt word engineering and the multi-modal large model according to the extraction standard of picture features prepared in advance.

[0043] The step 122 of the present application extracts picture features by a multi-modal large model. Taking a business travel scene as an example, the extraction standard of picture features prepared in advance of the present application specifically includes the following standards prepared in advance: a visual effect extraction standard, an extraction standard of appearance pictures or lobby pictures highlighting hotel features, an extraction standard of hotel pictures conveying emotions and atmosphere, an extraction standard of factors to be avoided in picture selection, and an extraction standard of user perspectives.

[0044] Examples of the above extraction standards of picture features: (1) the visual effect extraction standard, including: (1.1) clarity and resolution: high definition without blur, resolution not less than 1920x1080 (16:9) or 1280x960 (4:3); avoid pictures with excessive compression or obvious noise. (1.2) color and light: rich color, which can truly restore the appearance characteristics of the hotel. Daytime pictures need sufficient light, avoiding overcast or too dark shooting effects; nighttime pictures need bright lights to create a warm atmosphere. (1.3) composition and angle: use a front panoramic or slightly tilted angle to show the overall style of the hotel. Avoid local close-ups or too narrow perspectives. (2) the extraction standard of appearance pictures or lobby pictures highlighting hotel features, including: (2.1) iconic architecture or design (hotel appearance picture as the first picture): if the hotel has unique architectural style, sculpture or decoration, ensure that these elements occupy a prominent position in the first picture. Modern hotels can show glass curtain walls or light designs; resort hotels can highlight swimming pools, gardens or natural landscapes. (2.2) iconic architecture or design (lobby as the first picture): the central area of the lobby (such as the front desk, decorations) should be located at the center of the picture or the golden section point. Front desk decorations, lights and staff images can help convey service quality. Show the comfort of the lobby, such as sofas, rest areas or art decorations. If the lobby has unique design elements (such as sculptures, murals), ensure that these details are clearly visible in the picture. (2.3) environment and surroundings: if the hotel is located by the sea, in the city center or in a natural scenic area, try to integrate the surrounding environment (such as sea views, city skylines, greenery) into the picture. But avoid cluttered backgrounds to ensure that the hotel theme is prominent. (2.4) for hotels with seasonal characteristics (such as winter snow scenes, spring flower fields), prefer pictures that reflect the seasonal atmosphere.

[0045] Similarly, the extraction criteria of the hotel pictures conveying emotions and atmosphere, the extraction criteria of the factors to be avoided in the selection of the pictures, and the extraction criteria of the user's perspective can be defined according to the extraction criteria of the picture features in the specific scene, and details are not repeated here. Thus, the picture features can be extracted by the multi-modal large model according to the extraction criteria of the picture features. According to the second text information extracted from the picture, that is, the picture features extracted according to the extraction criteria of the picture features in the above embodiment of the application, the brand and location information of the hotel and the like are extracted from the picture by the multi-modal large model of the application. That is, the brand and location information and the like are the second text information, which is extracted from the picture by the multi-modal large model of the application. More specifically, the extraction criteria of the picture features are prepared as part of the prompt word engineering of the application, and the prompt word engineering is input into the multi-modal large model of the application, so that the multi-modal large model of the application extracts the picture features from the picture, including the brand and location information of the hotel and the like. These brand and location information of the hotel and the like are the second text information of the application.

[0046] The second text information is processed into a numerical feature. For example, a general algorithm can be used to process the second text information into a numerical feature. Specifically, the second text information can be processed into a numerical feature using one-hot (one-hot encoding) method.

[0047] In step 123, the extracted title features and picture features are vectorized to obtain vector features, and the vector features are combined to obtain fusion features.

[0048] In step 123, the numerical features are input into the semantic model for vectorization to obtain vector features.

[0049] In step 100, the extracted features are combined to obtain fusion features.

[0050] Specifically, first, step 100 includes combining the numerical features to obtain fusion features. That is, the fusion features obtained by combining the numerical features can obtain more comprehensive data representation capability.

[0051] Secondly, step 100 further includes combining the vector features to obtain fusion features, and the combination of the vector features can obtain the association relationship between images.

[0052] Thirdly, step 100 further includes combining the numerical features and the vector features to obtain fusion features. The association relationship between text and image can be obtained.

[0053] The fusion features of the embodiments of the present application include but are not limited to interaction features and context features. The fusion features can represent interaction features and context features at the application layer. The interaction features are used to express which products the user has accessed. The context features are used to express the time of user access, the type of mobile phone used by the user, etc.

[0054] The online feature extraction method of step 100 is used to combine the extracted features to obtain fusion features. According to the given extraction standards of title features and picture features, the method of the present application can extract user features and resource features by using a large language model and a multi-modal large model. By fusing these features, comprehensive data representation capabilities can be obtained, and the association between images and texts can also be obtained. These title features and picture features are processed into numerical features, and the numerical features are processed into vector features. By combining the vector features, interaction features and context features can also be obtained. Therefore, by using the method of the present application, the extraction efficiency of user features and resource features is greatly improved in a very lightweight manner, and the efficiency of obtaining fusion features from user features and resource features through steps 121, 122 and 123 is also improved.

[0055] Step 200, creating a prompt word engineering of a sorting task according to the fusion features; The prompt word engineering of the present application can use a COT framework or an ICIO framework. Taking the COT framework as an example, the COT framework includes four parts, namely: task, background, input data and output data. The following is an example of a prompt word engineering of a COT framework: "Please become a recommendation system expert and provide sorting algorithm services for me. Use the pointwise method to sort and provide personalized sorting algorithm capabilities for me.

[0056] Specifically, I will provide you with fixed format data, and you will help me return the hotel sorting result.

[0057] Wherein user_group includes user-related information, and hotel_group includes hotel-related features.

[0058] In more detail, user_id represents the user id, which does not affect the model as a feature.

[0059] user_sex represents the gender information of the user, 0 represents female, and 1 represents male, which can be one-hot encoded; user_price represents the user's price preference, such as 100 representing that the user likes hotels with a price of about 100 yuan, and the default value is 0.0, indicating that the data is missing, so the effect of this feature can not be considered.

[0060] user_level represents the level of the user, the larger the number, the higher the level, 0 represents ordinary employees, 1 represents middle managers, and 2 represents high-level managers.

[0061] hotel_id represents the hotel id, which does not affect the model as a feature.

[0062] hotel_sex represents the hotel gender, 0 represents that female users are high-frequency, and 1 represents that male users are high-frequency.

[0063] hotel_price represents the hotel price, which is a fixed value and not a range value, and can be processed by bucketing.

[0064] Please refer to the data and its physical meaning, and use the ranking ability of the large model to directly return the ranked results. The returned data format can be [(hotel_id1, scorel), (hotel_id2, score2),...], and the value range of score is 0~1, representing the predicted probability value.

[0065] Input data: { "user_group": {"user_id": 1, "user_sex": @, "user_level":1, "user_price": 300},{"hotel_id": 1003, "hotel_sex": @, "hotel_price": 150},{"hotel_id": 1004, "hotel_sex": 1, "hotel_price": 240} Result: [(1002, 0.5), (1003, 0.6), (1001, 0.2), (1004, @.8)] The result only needs to return the final ranking result, and does not need other reasoning and any dialogue related information.

[0066] Result: "。

[0067] The above prompt words of the application are input into a large language model, and the ranking result can be obtained through the following steps 300.

[0068] Step 300, input the prompt words of the ranking task into the large language model to obtain the ranking result, wherein the ranking result includes the explainability of the ranking result.

[0069] The present invention can obtain a ranking result through step 300. In a business travel scenario, the ranking result can be a sequentially arranged hotel recommendation. For example: Hotel 1004 scores 0.8; Hotel 1001 scores 0.6; Hotel 1003 scores 0.5; Hotel 1002 scores 0.2.

[0070] In addition to providing a ranking of hotels based on user searches, the present invention can also provide interpretability of the ranking results.

[0071] An example of the interpretability of the ranking results is as follows: “Sorting Effect Explained (SHAP-Style Feature Attribution Analysis) 1. Core feature weight distribution Price matching weight: 0.65 (dominated by user price sensitivity); Gender matching weight: 0.25 (user gender preference is second); User-level adjustment factor: 0.1 (middle-level managers adjust for price and gender balance); 2. Attribution breakdown of hotel scores (1) Hotel 1004 (score 0.8) Price matching contribution: +0.72; The hotel price of 240 → the difference from the user's target price of 300 (△ = 60) is transformed using the Gaussian kernel function:

[0072] Gender mismatch penalty: -0.10; Hotel gender 1 ≠ user gender 0, triggering the reverse correction factor.

[0073] User level gain: +0.18; The user is a middle-level manager (Level=1), whose price sensitivity is weakened, thus amplifying the price matching benefits.

[0074] (2) Hotel 1001 (score 0.6) Gender matching bonus: +0.25; Hotel gender 0 = user gender 0, basic matching benefit.

[0075] Price overrun penalty: -0.30; The hotel price 323 > the user's target price 300. The penalty term is calculated using a piecewise function:

[0076] User level compensation: +0.15; Middle managers compensate for their tolerance of minor overspending.

[0077] (3) Hotel 1003 (score 0.5) Gender match gain: +0.25; Base gender match gain.

[0078] Price deviation penalty: -0.25; Hotel price 150 -> absolute difference delta = 150, penalized by linear decay:

[0079] User level suppression: -0.10; Mid-level manager negative perception of too low price (possibly linked to quality expectation); User level suppression: -0.10; Mid-level manager negative perception of too low price (possibly linked to quality expectation); (4) Hotel 1002 (score 0.2); Price match contribution: +0.24; Price 240 -> same as hotel 1004, but gender mismatch does not trigger gain.

[0080] Gender mismatch penalty: -0.30; Base mismatch penalty stacked with user gender preference reinforcement factor.

[0081] User level neutral adjustment: +0.06; No significant adjustment effect.

[0082] Ranking logic summary 1. Price sensitivity priority: user price target (300) quantified by Gaussian kernel function, low difference hotels (like 240) score significantly higher.

[0083] 2. Gender dynamic trade-off: gender match provides base gain, but is overridden by large price deviation (like hotel 1003).

[0084] 3. Level adjustment effect: mid-level manager balances price and quality expectation, more tolerant to moderate deviation (like hotel 1001), but sensitive to extreme low price (like hotel 1003).

[0085] In terms of model explainability, the application can also provide the explainability of the model ranking results under different loss functions. For example, the model of the embodiment of the application can provide the explainability of pointwise ranking results, the explainability of pairwise ranking results, and the explainability of listwise ranking results. This solves the defect that the pre-trained model cannot provide explainability. Since the ranking method provided by the application provides the explainability of the ranking results, this enhances the user's trust in the ranking results, and can assist the research and development personnel in understanding the model, selecting the model, and optimizing the model parameters, and provides an important basis for explaining the internal mechanism of the model to the business direction in the future.

[0086] Embodiment 2 As Figure 3 shown, based on the same inventive concept as the above embodiment, the application also provides a large model fusion multi-modal feature recommendation device, comprising: A feature acquisition device 200 is used for online feature extraction, wherein the feature acquisition device 200 comprises a user feature acquisition device and a resource feature acquisition device, wherein the user feature acquisition device 240 is used for online acquisition of user-side features, and the resource feature acquisition device 230 is used for online acquisition of resource-side features, comprising: extracting title features from resource-side features, wherein the title features are extracted from the resource-side features by a large language model according to a prompt word engineering; and extracting picture features from the resource-side features, wherein the picture features are extracted from the resource-side features by a multi-modal large model according to a prompt word engineering, and the extracted title features and picture features are combined to obtain fusion features; A prompt word device 300 is used for creating a prompt word engineering of a ranking task according to the fusion features; A large language model ranking device 400 is used for inputting the prompt word engineering of the ranking task into a large language model to obtain ranking results, wherein the ranking results comprise the explainability of the ranking results.

[0087] The feature acquisition device 200 of the embodiment of the application comprises a user feature acquisition device 240 and a resource feature acquisition device 230. The resource feature acquisition device 230 comprises a title feature extraction device 210 and a picture feature extraction device 220.

[0088] The feature acquisition device 200 is used to perform step 100 to realize feature extraction. Step 100 and its sub-steps have been described in detail in Embodiment 1, please refer to Embodiment 1, which will not be repeated here.

[0089] The title feature extraction device 210 comprises a first prompt word engineering module 211 and a large language model extraction module 212.

[0090] The picture feature extraction device 220 comprises a second prompt word engineering module 221 and a multi-modal large model extraction module 222.

[0091] The prompt word device 300 is used to obtain the fusion features described above. The prompt word device 300 is used to perform step 200 of embodiment 1.

[0092] The large language model ranking device 400 is used to create a prompt word engineering of a ranking task by the third prompt word engineering module 410 according to the fusion features described above, and input the prompt word engineering of the ranking task into the large language model ranking module 420, and give a ranking result by the large language model ranking module 420. Specifically, the large language model ranking device 400 is used to perform step 300 of embodiment 1.

[0093] The embodiment of the present application extracts the title features by using the first prompt word and the large language model in the feature acquisition device 200, extracts the picture features by using the second prompt word and the multi-modal large model, fuses the extracted features, makes the third prompt word according to the fusion result, inputs the third prompt word into the large language model for ranking, and gives the ranking result. Therefore, the technical scheme of the present application is completed by the model in the feature extraction and ranking stages, which greatly improves the efficiency.

[0094] Embodiment 3 As shown in Figure 4 The present application also provides an electronic device 100 for implementing the recommendation method of the large model and the multi-modal; The electronic device 100 comprises a memory 101, at least one processor 102, a computer program 103 stored in the memory 101 and executable on the at least one processor 102, and at least one communication bus 104.

[0095] The memory 101 can be used to store the computer program 103, and the processor 102 realizes the recommendation method of the large model and the multi-modal of embodiment 1 by running or executing the computer program stored in the memory 101, and calling the data stored in the memory 101.

[0096] The memory 101 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program (such as a sound playing function, an image playing function, etc.) required by a function, etc.; and the data storage area can store data (such as audio data) created according to the use of the electronic device 100, etc. In addition, the memory 101 can include a non-volatile memory, for example, a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device.

[0097] The at least one processor 102 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 102 can be a microprocessor or the processor 102 can also be any conventional processor, etc. The processor 102 is the control center of the electronic device 100, and connects all parts of the electronic device 100 through various interfaces and lines.

[0098] The memory 101 in the electronic device 100 stores a plurality of instructions to implement a large model and a multi-modal recommendation method, and the processor 102 can execute the plurality of instructions to implement: online acquisition of user side features, online acquisition of resource side features, extraction of title features from the resource side features through a large language model according to a prompt word engineering, and extraction of picture features from the resource side features through a multi-modal large model according to a prompt word engineering, combination of the extracted title features and picture features to obtain fusion features; creation of a prompt word engineering of a sorting task according to the fusion features; input of the prompt word engineering of the sorting task into a large language model to obtain a sorting result, wherein the sorting result includes explainability of the sorting result.

[0099] Embodiment 4 If the module / unit integrated in the electronic device 100 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. Computer-readable media may include: any entity or device that can carry computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory and read-only memory (ROM, Read-Only Memory).

[0100] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0101] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0102] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0103] These computer program instructions can also be loaded into a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 one or more flowcharts and / or blocks

[0104] In the description of the present specification, the description of the terms "one embodiment", "an example", "a specific example" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application but not to limit it, and although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that the specific embodiments of the present application can be modified or replaced by the equivalent, without departing from the spirit and scope of the present application, and any modification or equivalent replacement should be covered in the protection scope of the claims of the present application.

Claims

1. A large-model and multimodal recommendation method, characterized in that: include: Obtain user-side features online, and obtain resource-side features online; Extracting title features from the resource-side features using a large language model according to the prompt word project; Furthermore, according to the prompt word project, the image features are extracted from the resource side features through the multimodal large model, and the extracted title features and the image features are combined to obtain fusion features; Creating a prompt word project for a sorting task based on the fused features; The prompt word engineering of the sorting task is input into a large language model to obtain a sorting result; wherein the sorting result includes the explainability of the sorting result.

2. The large model and multimodal recommendation method according to claim 1, characterized in that: Extracting title features from the resource-side features using a large language model based on the prompt word project; and extracting image features from the resource-side features using a multimodal large model based on the prompt word project, and combining the extracted title features and image features to obtain fused features includes: Predetermining a title feature extraction standard, using the predetermined title feature extraction standard as part of a prompt word project, and extracting title features through a large language model based on the prompt word project; Predetermining an extraction standard for image features, using the predetermined extraction standard as part of a prompt word project, and extracting image features using a multimodal large model based on the prompt word project; The extracted title features and the image features are vectorized to obtain vector features, and the vector features are combined to obtain fusion features.

3. The large model and multimodal recommendation method according to claim 2, characterized in that: Performing vectorization processing on the extracted title features and the image features to obtain vector features, and combining the vector features to obtain fusion features includes: Processing the first text information in the title feature into a numerical feature; extracting second text information from the image based on the image features, and processing the second text information into numerical features; The numerical features of the first text information and the numerical features of the second text information are input into a semantic model for vectorization processing to obtain vector features, and the vector features are combined to obtain fusion features.

4. The large model and multimodal recommendation method according to claim 1, characterized in that In the step of creating a prompt word project for the sorting task based on the fusion features, The cue word engineering uses the COT framework.

5. The large model and multimodal recommendation method according to claim 1, characterized in that: In the step of creating a prompt word project for the sorting task based on the fusion features, The prompt word project uses the ICIO framework.

6. A large-scale and multimodal recommendation device, characterized in that: include: A feature acquisition device for online feature extraction, wherein the feature acquisition device includes a user feature acquisition device and a resource feature acquisition device, wherein the user feature acquisition device is used to online acquire user-side features, and the resource feature acquisition device is used to online acquire resource-side features, and online acquiring resource-side features includes: extracting title features from resource-side features, wherein the title features are extracted from the resource-side features using a large language model based on a prompt word project; and extracting image features from the resource-side features, wherein the image features are extracted from the resource-side features using a multimodal large model based on the prompt word project, and combining the extracted title features and image features to obtain a fused feature; A prompt word device, used for creating a prompt word project for the sorting task based on the fusion features; A large language model sorting device is used to input the prompt word engineering of the sorting task into the large language model to obtain a sorting result, wherein the sorting result includes the interpretability of the sorting result.

7. The large model and multimodal recommendation device according to claim 6, characterized in that The resource feature acquisition device includes: A title feature extraction device, which predetermines a title feature extraction standard, uses the predetermined title feature extraction standard as part of a prompt word project, and extracts title features based on the prompt word project using a large language model; An image feature extraction device predefines an extraction standard for image features, uses the predetermined image feature extraction standard as part of a prompt word project, extracts image features based on the prompt word project using a multimodal large model, and extracts second text information from the image features based on the image features; The fusion device is used to perform vectorization processing on the extracted title features and the image features to obtain vector features, and combine the vector features to obtain fusion features.

8. The large model and multimodal recommendation device according to claim 7, characterized in that The fusion device comprises: a vector processing device for processing the first text information in the title feature into a numerical feature, processing the second text information into a numerical feature, and inputting the numerical features of the first text information and the numerical features of the second text information into a semantic model for vectorization processing to obtain vector features; The vector fusion device is used to combine the vector features to obtain fusion features.

9. An electronic device, characterized in that: The invention comprises a processor and a memory, wherein the processor is configured to execute a computer program stored in the memory to implement the large model and multimodal recommendation method according to any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one instruction, and when the at least one instruction is executed by a processor, the large model and multimodal recommendation method according to any one of claims 1 to 5 is implemented.

Images