A recommendation method and device based on a large language model

Abstract

Embodiments of the present specification provide a recommendation method and device based on a large language model, the method comprising: obtaining at least one subject of a user query, the at least one subject being identified from a target image; obtaining first information of the user, the first information being used to indicate an interest tendency of the user; determining a target subject in the at least one subject based on the first information; and generating at least one recommended content for the user based on the target subject through the large language model.

Description

TECHNICAL FIELD

[0001] Embodiments of the present specification relate to the technical field of artificial intelligence, in particular, to a recommendation method and device based on a large language model. BACKGROUND

[0002] With the continuous evolution of mobile Internet, people's life is increasingly inseparable from the query and recommendation functions of the network. The current query and recommendation functions rely on the user to search by typing input. For the vulnerable groups with low technical sensitivity, since the current "giant App" or "super App" usually deeply integrates multiple service functions on the same platform, resulting in complex function entry and excessive levels, the threshold for these vulnerable groups to use query and recommendation functions is significantly increased. For example, they may need to spend a lot of time to find the input box and input text, and the recognition accuracy is not stable when using voice input, and the error rate is high when typing. Therefore, the current recommendation system is difficult to provide reasonable service or content recommendation for the vulnerable groups according to their real query requirements. In addition, even for groups with high technical sensitivity, the information provided by typing input is limited, and it is difficult to meet the rich and diverse query requirements.

[0003] Therefore, how to ensure the fairness of network service use and provide a fast and accessible rich experience of information and services has become a real challenge that needs to be solved. SUMMARY

[0004] Embodiments of the present specification provide a recommendation method based on a large language model, which effectively improves the query experience of users by combining target images and user interest tendencies to identify user intent.

[0005] In a first aspect, embodiments of the present specification provide a recommendation method based on a large language model, comprising: obtaining at least one subject of a user query, the at least one subject being identified from a target image; obtaining first information of the user, the first information being used to indicate the interest tendency of the user; determining a target subject in the at least one subject based on the first information; generating at least one recommended content for the user based on the target subject through the large language model.

[0006] In some embodiments, the obtaining the first information of the user comprises: obtaining associated information of the user; generating the first information of the user based on the associated information through the large language model.

[0007] In some embodiments, the associated information of the user comprises at least one of the following information: attribute information of the user, historical behavior data of the user, location information of the user, and current time information of the user.

[0008] In some embodiments, the determining, based on the first information, a target subject in the at least one subject, comprises: determining, based on the first information, the target subject in the at least one subject and a vocabulary having an association relationship with the target subject; and the generating, by the large language model, at least one recommended content for the user based on the target subject, comprises: generating, by the large language model, at least one recommended content for the user based on the target subject and the vocabulary having an association relationship with the target subject.

[0009] In some embodiments, the determining, based on the first information, a target subject in the at least one subject and a vocabulary having an association relationship with the target subject, comprises: retrieving, based on a preset corpus, association information and / or context information of the at least one subject, the context information including an association vocabulary obtained based on the subject and / or the association information; generating, by a first multi-modal large language model, the vocabulary having an association relationship with the subject based on the association information, the at least one subject, and the context information; and determining, based on the first information, the target subject and the vocabulary having an association relationship with the target subject in the at least one subject and the vocabulary having an association relationship with the subject.

[0010] In some embodiments, the determining, based on the first information, a target subject in the at least one subject and a vocabulary having an association relationship with the target subject, comprises: performing semantic encoding on the first information to obtain a first semantic vector; performing semantic encoding on each of the subject and the vocabulary having an association relationship with the subject to obtain a second semantic vector; and filtering the second semantic vector based on a similarity between the first semantic vector and the second semantic vector to determine the target subject in the at least one subject and the vocabulary having an association relationship with the target subject.

[0011] In some embodiments, the vocabulary includes a hypernym of the target subject

[0012] In some embodiments, the large language model is a spatiotemporal large language model, and the spatiotemporal large language model is fine-tuned in the following manner: obtaining a spatiotemporal dataset and a label of a sample user, the spatiotemporal dataset including association information of the sample user, and the label including a service used by the sample user; predicting, by the spatiotemporal large language model, the service used by the sample user based on the spatiotemporal dataset; and adjusting network parameters of the spatiotemporal large language model based on a prediction result and the label.

[0013] In some embodiments, the acquiring the at least one subject of the user query comprises: acquiring the target image input by the user, analyzing a structure in the target image based on the target image by a second multi-modal large model, and generating a scene graph, the scene graph comprising the at least one subject and a relationship between the subjects.

[0014] In some embodiments, the target image is an image collected from a scene in which the user is currently located.

[0015] In some embodiments, the generating, by the large language model, at least one recommended content for the user based on the target subject comprises: generating, by the large language model, at least one recommended content for the user based on the target subject and associated information.

[0016] In a second aspect, the embodiments of the present specification provide a recommendation method based on a large language model, comprising: acquiring a target image of a user; displaying, to the user, recommended content corresponding to the target image, the recommended content being generated by the large language model for the user based on a target subject, the target subject being determined from at least one subject based on first information, the first information being used to indicate an interest tendency of the user, and the at least one subject being identified from the target image.

[0017] In some embodiments, a content generation direction is acquired, the content generation direction being used to indicate an information type of the recommended content, and the recommended content being generated by the large language model for the user based on the target subject and the content generation direction.

[0018] In some embodiments, after the displaying, to the user, of the recommended content corresponding to the target image, the method further comprises: in response to a first operation of the user on the recommended content in an interaction interface, storing the recommended content in a preference record of the user, the first operation being used to represent an interest of the user in the recommended content, and the preference record being used to generate the first information.

[0019] In some embodiments, after the displaying, to the user, of the recommended content corresponding to the target image, the method further comprises: storing the recommended content in a historical browsing record of the user, the historical browsing record being used to generate the first information.

[0020] In a third aspect, an embodiment of the present specification provides a recommendation device based on a large language model, comprising: a subject query module configured to obtain at least one subject of a user query, the at least one subject being identified from a target image; an information acquisition module configured to obtain first information of the user, the first information being used to indicate the interest tendency of the user; a target determination module configured to determine a target subject in the at least one subject based on the first information; and a content generation module configured to generate at least one recommended content for the user based on the target subject through the large language model.

[0021] In a fourth aspect, an embodiment of the present specification provides a computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement the method described in any implementation manner of the first aspect or the second aspect.

[0022] In the scheme provided by the above embodiments of the present specification, by identifying at least one subject in the target image, combining the first information indicating the user's interest tendency, using the large language model to predict the search intent, and then generating the recommended content that the user may be interested in, the use threshold is low, ensuring the fairness of network service use, and the generated recommended content is more in line with the user's interest and actual scene, improving the user experience. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the multiple embodiments disclosed in the present specification, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only a part of the embodiments disclosed in the present specification, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0024] Figure 1 is a schematic diagram of an application scenario to which the embodiments of the present specification can be applied;

[0025] Figure 2 is a schematic diagram of a mobile phone interface in an embodiment of the present specification;

[0026] Figure 3 is a flowchart of a recommendation method based on a large language model in an embodiment of the present specification;

[0027] Figure 4 is a flowchart of the acquisition process of a spatiotemporal large language model in an embodiment of the present specification;

[0028] Figure 5 is a flowchart of a recommendation process in an embodiment of the present specification;

[0029] Figure 6 is another flowchart of a recommendation method based on a large language model in an embodiment of the present specification;

[0030] Figure 7 is a flowchart of still another recommendation method based on a large language model in an embodiment of the present specification;

[0031] Figure 8 is a schematic diagram of an interactive interface in an embodiment of the present specification;

[0032] Figure 9 is a schematic diagram of still another interactive interface in an embodiment of the present specification;

[0033] Figure 10 is a structural schematic diagram of a recommendation device based on a large language model in an embodiment of the present specification;

[0034] Figure 11 is a structural schematic diagram of still another recommendation device based on a large language model in an embodiment of the present specification. DETAILED DESCRIPTION

[0035] In order to enable persons skilled in the art to better understand the technical solutions in the present specification, the technical solutions in the embodiments of the present specification will be described clearly and completely in conjunction with the drawings in the embodiments of the present specification. Obviously, the described embodiments are only some of the embodiments of the present specification, not all the embodiments. Based on the embodiments in the present specification, all other embodiments obtained by persons skilled in the art without creative labor should fall within the scope of protection of the present specification.

[0036] As mentioned earlier, compared with querying by typing, users are more likely to express their needs and provide more intent information through low-threshold methods such as taking pictures. For this purpose, the present specification proposes a recommendation scheme based on a large language model (LLM) to identify user intent based on target images combined with user interest preferences and generate recommended content for users. This scheme is based on the idea of multi-modal human-computer interaction: let users use more natural interaction methods such as taking pictures to reflect their query needs in different scenarios in a real, direct and convenient way. Users only need to provide images as queries, and the system can explore and generate corresponding recommended content according to the visual content combined with user interest preferences to meet the user's query needs.

[0037] Figure 1 A schematic diagram showing an application scenario in which the embodiments of the present specification can be applied is shown. As shown in Figure 1 Figure 1 ​In the application scenario shown, the user is near a self-service terminal in a hospital, and the user's possible needs are registration, report printing, and information inquiry, etc. The target image of the user can be obtained by the client installed on the terminal device of the user, which can be obtained by the user taking a photo of the self-service terminal in the hospital environment through the terminal device (such as a mobile phone), and then the recommended content corresponding to the target image is displayed to the user, such as Figure 2 As shown, the recommended content displayed on the mobile phone interface includes: "How to perform self-service medical insurance registration?" and "How does the self-service terminal change the service experience?" The recommended content here is generated by the large language model based on the self-service terminal as the target subject for the user. The self-service terminal determines at least one subject from the target image based on the first information. In addition to the subject of the self-service terminal, the target image can also contain subjects such as the hospital's green plants and water dispensers. The first information is used to indicate the user's interest inclination, which in this example can be an inclination related to low blood pressure. The at least one subject is identified from the target image taken by the user.

[0038] In the above process, the client can send the obtained target image to the remote server, and the server with stronger computing power can process the target image to finally generate at least one recommended content for the user, and send the recommended content to the client. It can also be that the target image is identified on the client side to obtain at least one subject, and then the at least one subject is sent to the server, and the server processes the at least one subject to finally generate at least one recommended content for the user, and sends the recommended content to the client. It can be understood that when the computing power of the terminal device where the client is located is sufficient, the target image can also be processed on the client side to finally generate at least one recommended content for the user.

[0039] Referring back to Figure 3 , Figure 3 A flowchart of a recommendation method based on a large language model according to an embodiment is shown. The method can be performed by any device, platform or device cluster with computing and processing capabilities, and can be applied to the client side or the server side as mentioned above, including the following steps S301-S304.

[0040] As Figure 3 shown, in step S301, at least one subject of the user query is obtained.

[0041] At least one subject is identified from the target image. A subject refers to an independent object or entity existing in the image or video. These objects or entities can be people, animals, objects, or any other elements that can be identified and distinguished in visual content. The target image is an image that the user has a query for. In practice, the target image can be an image that the user takes, screenshots, selects, or otherwise determines based on their query and uploads to the client. The target image usually contains at least one subject that the user is querying. For example, when the target image is an image taken by the user of a desktop, the teacup, paper, pen, laptop, tea leaves in the teacup, and the user's exposed pants on the desktop can all be considered subjects.

[0042] This embodiment does not limit the method of obtaining at least one subject from the user query. For example, the specific implementation method can be to extract the subject from the target image input by the user through image analysis, entity recognition and other technologies.

[0043] In step S302, the user's first information is obtained.

[0044] The first piece of information is used to indicate the user's interests. In practice, the first piece of information can be topics or services that the user is interested in. Specifically, it can be topics or online services related to people's livelihood and government affairs, product information, cultural and technological fields, historical and geographical knowledge, and current affairs. For example, the first piece of information can be football matches that the user cares about, traditional culture that the user is interested in, and housing provident fund withdrawal services.

[0045] This embodiment does not limit the method of obtaining the user's primary information. For example, it can be obtained by analyzing the user's historical behavior data, or it can be obtained by analyzing the attribute information uploaded by the user. Taking the user's billing records as an example of historical behavior data, if items such as running shoes, knee braces, and sports drinks appear multiple times in the billing records, it can be analyzed to obtain the user's primary information as topic words related to sports and fitness, indicating the user's interest in sports and fitness.

[0046] As one approach, to accurately obtain the user's primary information, this step could involve acquiring the user's related information and generating the user's primary information based on that related information using a large language model.

[0047] The user's associated information refers to personal information related to the user, such as the user's occupation, user demographics, and historical behavior. For example, the user's associated information may include at least one of the following: user attribute information, user historical behavior data, user location information, and user current time information.

[0048] Specifically, the attribute information of the user is used to describe the basic situation, interest preference, and behavior habit of the user, such as education level, social role, and exercise habit, so as to better model the user portrait; the historical behavior data of the user is used to describe the online behavior of the user, which can include the billing record of the user, the record of using the applet, the search history record, and the historical record of accessing the specific point of the client at a specific time, so as to obtain the continuous behavior of the user in different space-time scenes and better analyze the user behavior pattern; the location information of the user can be the geographic location information of the user when shooting or uploading the target image, or the shooting location information carried by the target image itself, so as to better predict the behavior pattern through the location information; the current time information of the user can be the time when the user shoots the target image, or the time when the user uploads the target image to the client, so as to better predict the behavior pattern through the time information.

[0049] The large language model used herein is an artificial intelligence model based on deep learning, which is specially used for understanding and generating natural language. It can be obtained by fine-tuning an open-source large language model, or it can be trained to enable the large language model to learn the deep association between the associated information and the first information of the user through supervised training. The acquisition process of the large language model used in the embodiments of the present specification will be described in detail later, and will not be described in detail here.

[0050] In practice, the prompt information can be constructed based on the associated information, and the prompt information is input into the large language model to obtain the first information of the user output by the large language model. For example, the constructed prompt information can be: according to the portrait information {p} and the recent purchase history {h} of the user, combined with the current time {t} and the location information {l}, predict what service {f} the user needs, so as to obtain the service prediction {f} output by the model.

[0051] It can be understood that the present embodiment does not limit the execution order of steps S301 and S302.

[0052] In one example, in order to reduce the calculation amount when the present scheme is applied online, the above process of generating the first information of the user can be completed offline in advance and stored in an offline database, and the first information can be obtained from the offline database when the first information is obtained.

[0053] Then, in step S303, based on the first information, a target subject in the at least one subject is determined.

[0054] Specifically, in the case where there are multiple identified subjects, the target subject can be determined from the user interest tendency indicated by the first information, that is, the target subject is the subject that the user is interested in, and the present specification does not limit the number of target subjects.

[0055] In this embodiment, the target subject associated with the first information can be determined from the plurality of subjects based on the association between the semantics of the first information and the semantics of the subjects. Still taking the desktop object as an example, in the case where there are multiple subjects on the desktop, such as a cup, paper, a pen, a notebook computer, tea leaves in the cup, and the user's exposed trousers, when the first information of the user indicates that the user is interested in Chinese traditional culture, because the cup and the tea leaves are not only daily necessities, but also have semantic association with Chinese traditional culture, and carry rich cultural connotations and symbolic meanings, the determined target subject can be the cup and the tea leaves, and when the first information of the user indicates that the user is interested in technology products, because the notebook computer has semantic association with technology products, the determined target subject can be the notebook computer.

[0056] In addition, in the case where there is one identified subject, the subject can be directly determined as the target subject.

[0057] Finally, in step S304, at least one recommended content for the user is generated based on the target subject by a large language model.

[0058] In practice, the large language model can be constructed based on the target subject and the user, and the large language model can output corresponding recommended content.

[0059] The prompt word contains information of the target subject, and can also contain information related to the user, so that the large language model generates recommended content that the user can be interested in, and the recommended content is semantically related to the target subject. The recommended content can be personalized text information, goods, services, or other resources recommended to the user.

[0060] For example, when the target subject is a computer host of brand A, the constructed prompt word can be "ask questions that the user can be interested in based on the target subject", and the recommended content generated by the large language model based on the prompt word can be "how does the computer host of brand A improve work efficiency?" and "how is the heat dissipation system of the computer host of brand A designed?" and other text information. When the user clicks on the text information, the text information can be input into the large language model or other large language models to obtain more related text or picture content generated by the large language model to answer related questions and satisfy the user's curiosity.

[0061] In some embodiments, this step can generate at least one recommended content for the user based on the target subject and the associated information by a large language model.

[0062] For example, when the recommended content is text information, the constructed prompt word can be "integrate the known background information of the user and the target subject in the target image, and propose an interactive question for the user. The question content must be related to the content in the target image provided by the user, combined with the user's interest, background and scene characteristics". For example, when the target subject is tea and a tea cup, and the user is interested in Chinese traditional culture, the generated recommended content can be "What kind of tea is suitable for this tea cup?", "What are the types and materials of Chinese traditional tea cups?", and "What are the types of Chinese tea?", thereby arousing the interest of the user.

[0063] Further, the control where the recommended content is located can be associated with the online service interface provided by the client. When the user clicks on the recommended content, the user can enter the page of the corresponding online service, thereby quickly mapping the user's spatio-temporal demand in the real scene to the function entry of the related application program, and obtaining timely feedback and guidance. For example, when the recommended content is "how to make an online medical insurance registration?", the user can jump to the mini-program interface providing the registration function after clicking on the recommended content; when the recommended content is "how to make an online public accumulation fund withdrawal?", the user can jump to the mini-program interface providing the public accumulation fund withdrawal.

[0064] Such multi-modal interaction not only expands the application scenarios of large language models from pure text to visual fields, but also reflects the potential of large models in image-text understanding and multi-modal search. Through multi-modal search as an entry, the personalized intention of the user can be connected with the diversified functions in the network service. Even for similar picture queries, due to the differences in user preferences and needs, personalized responses can be obtained through the scheme of the embodiment, which is important for satisfying user needs and improving user experience.

[0065] The current mainstream recommendation system mainly relies on ID (Identity Document, identity number) recommendation and historical behavior analysis, is good at mining user preferences from structured data, and can provide relatively accurate targeted content recommendation for existing and patterned user behavior. However, such systems often appear to be inadequate when faced with real-time multi-modal requests, especially in image-dominated query scenarios. For example, when the user takes and uploads a certain scene in daily life for query, such unstructured visual information exceeds the modeling ability of traditional recommendation systems or retrieval systems, making it difficult to quickly generate reasonable service or information recommendation.

[0066] Considering the above issues and addressing the dynamic needs of users, the large language model used in the embodiments of this specification can be a Spatial Temporal-LLM (ST-LLM). This model is obtained by training or fine-tuning the large language model using large-scale spatiotemporal data. It has a certain spatiotemporal prediction capability and can capture the specific needs of users in different scenarios. The generated recommended content is more in line with the user's current needs.

[0067] The following is combined with Figure 4 The following example illustrates the process of obtaining the spatiotemporal large language model used in the embodiments of this specification. Figure 4 The example shown may include steps S401-S403, specifically:

[0068] In step S401, the spatiotemporal dataset and labels of the sample users are obtained.

[0069] First, a spatiotemporal dataset of a certain scale needs to be constructed, containing the association information of sample users. Spatiotemporal prediction tasks typically require fine-grained, multi-dimensional user behavior data, which existing publicly available datasets often cannot meet. Therefore, it is necessary to collect user behavior information from online platforms and construct a corpus as the spatiotemporal dataset.

[0070] When constructing a spatiotemporal dataset, relevant information about the sample users can be obtained from various data sources with their authorization and permission. For example, data sources could be social networks, forums and communities, data provided by the sample users by creating accounts or filling out forms, and the sample users' browsing behavior and interaction history on online platforms.

[0071] For example, the billing records and historical interaction information of sample users can be selected as data sources. Historical interaction information mainly includes records of using mini-programs, search history, and records of accessing specific locations on the client at specific times. The reason for choosing these two types of data is that users' continuous consumption behavior in different spatiotemporal scenarios can explicitly or implicitly reflect their goals, intentions, interests, and knowledge depth. Based on this long-term accumulated information, not only can user behavior patterns be accurately characterized, but it can also provide crucial knowledge support for generative recommendation systems.

[0072] For example, for any unsupervised document d from a document set D, a corpus building system F can be used to extract four types of key information: the attribute information P of the sample users, time T, location L, and interaction history H. The corpus building process can be formalized as follows:

[0073] (1)

[0074] wherein the document set D refers to a set of unsupervised documents, i.e., text data without labeled tags, for example, billing records, historical interaction information, etc. of sample users; the unsupervised document d is any one document in the document set D, for example, the consumption record of a user on a certain day; the attribute information P of the sample user can be the sample user ID, occupation, etc.; the time T can be the consumption time or interaction time; the location L can be the consumption location or interaction location; the interaction history H can be the behavior record of the sample user such as clicking, purchasing, commenting, etc.

[0075] The four types of information are associated with each other and collectively provide rich background context for the spatiotemporal prediction and generative recommendation model. Based on this large-scale and high-quality corpus, the system can not only more accurately capture the spatiotemporal behavior patterns of users, but also provide more accurate and in-depth support for the generative recommendation model.

[0076] Then, for each sample user in the spatiotemporal dataset, a label can be obtained by labeling, which contains the services used by the sample user. For example, it can be manually labeled, or the services used by the sample user after the time period covered by the spatiotemporal dataset can be used as the label.

[0077] In step S402, the service used by the sample user is predicted based on the spatiotemporal dataset by the spatiotemporal large language model.

[0078] In practice, as shown in the spatial temporal guidance module (Spatial Temporal Guidance) in Figure 5 The spatial temporal guidance module depicts the fine-tuning process of the spatiotemporal large language model before actual application. For each sample user, a prompt word can be constructed according to the corresponding data in the spatiotemporal dataset, and the prompt word is input into the spatiotemporal large language model. The spatiotemporal large language model will output the predicted services that the sample user may use, such as which products to purchase.

[0079] The predetermined format of the prompt word can be set by staff as needed. For example, the embodiment designs a set of comprehensive input construction scheme, and the prompt word follows the following instruction template: given the attribute information of the sample user and the historical interaction data corresponding in the spatiotemporal dataset, combined with the current time and location information, predict the function services that the sample user may use. For example, when the sample user is near a restaurant, it is predicted that the sample user may use online ordering or delivery services, and when the sample user is at a gym, it is predicted that the sample user may need to purchase healthy food online. By providing large-scale labeled examples in the supervised learning stage, the model can learn the deep relationship between user features and spatiotemporal factors, so as to more accurately understand the user context in the reasoning process, and thus provide highly targeted and practical service suggestions.

[0080] In step S403, the network parameters of the spatiotemporal large language model are adjusted based on the prediction results and labels.

[0081] In practice, a loss function can be used to measure the gap between the model's predictions and the labels. Prediction accuracy can be improved by adjusting network parameters to minimize the loss function. This process can be fine-tuned using supervised learning examples, enabling the spatiotemporal large language model to predict service behavior based on user information and historical behavior data in specific spatiotemporal scenarios. This embodiment does not restrict the specific method of adjusting network parameters; for example, some or all network parameters can be fine-tuned based on a pre-trained model.

[0082] By providing large-scale labeled examples during the supervised learning phase, the model can learn the deep correlation between user characteristics and spatiotemporal factors, thereby more accurately understanding the user context during inference and providing highly targeted and practical service suggestions.

[0083] Below, in conjunction with Figure 5 The flowchart illustrating the recommendation process is provided below, which further details the recommendation process based on a large language model provided in the embodiments of this specification. Among them, Figure 5 The main components include: an input module (Input), used to display the input data required during the implementation of this solution; a spatiotemporal guidance module, used to fine-tune the spatiotemporal large language model; a preference discovery module, consisting of steps 1, 2, 3, and 4, where the step numbers are only used to identify different steps and do not restrict the execution order of the steps. Specifically, in step 1: the Recognizer generates a scene graph based on the target image; in step 2: the Generator generates a knowledge graph based on the user's attribute information and the scene graph; in step 3: the pre-trained spatiotemporal large language model generates first information based on the user's attribute information and historical behavior data; in step 4: the Retriever retrieves and sorts information in the knowledge graph based on the first information to determine the target subject and words related to the target subject; and a personalized recommendation module, used to generate recommended content using the spatiotemporal large language model.

[0084] Figure 6 This is a flowchart of another recommendation method based on a large language model, as described in the embodiments of this specification. This method can be executed by any device, platform, or cluster of devices with computing and processing capabilities, and includes steps S601-S604 as shown below.

[0085] like Figure 6As shown, in step S601, the target image input by the user is obtained, and the structure in the target image is analyzed based on the target image through the second multimodal large model to generate a scene graph.

[0086] The scene graph includes at least one subject and the relationships between subjects, and the scene graph can be represented using ternary sets.

[0087] In some embodiments, a Multilingual Large Language Model (MLLM) can be used to directly identify the subject in the target image, obtaining information about at least one subject. However, when used for visual content understanding, MLLM typically returns recognition results in text form, containing rich descriptive text information. This descriptive text information is often nested within complex grammatical and sentence structures, making it difficult to efficiently retrieve structured knowledge directly from visual queries. It requires processing complex grammar and sentence structures to extract concise and effective information. To address this issue, this embodiment proposes using Knowledge Graph (KG) triples to represent visual entities (i.e., subjects) and their relationships, thereby explicitly modeling the complex relationships between visual objects. This method, by effectively constructing a scene graph, provides greater flexibility for subsequent knowledge relationship filtering and expansion for subsequent reasoning and recommendation.

[0088] Specifically, such as Figure 5 The recognizer shown in step 1 can incorporate a second multimodal large model (e.g., an open-source large model) to perform visual queries on the input target image and extract entity information, resulting in a scene graph generated by MLLM, as shown below:

[0089] (2)

[0090] in, Composed of a series of triples, for example Each triple represents a visual entity. With another entity Relationship The set of triples can be represented as follows:

[0091] (3)

[0092] Different from natural language description, knowledge graph converts information into structured representation in the form of triples, which not only can express complex knowledge concisely and clearly, but also has strong flexibility and scalability, facilitating subsequent information retrieval, reasoning, relationship discovery, knowledge integration and other tasks. By storing information in the form of standardized triples, knowledge graph can more efficiently support a variety of subsequent tasks, including information retrieval, reasoning, relationship discovery, etc. In addition, the form of triples makes it more convenient to add new knowledge and integrate existing knowledge, greatly enhancing the scalability of knowledge graph and further improving its application value in complex tasks.

[0093] Exemplarily, Figure 5 Step 1 of the embodiment further shows a visual scene graph, which contains subjects of a tea cup, a paper, a pen, a notebook computer, tea leaves in the tea cup and trousers, and arrows represent the relationship between two subjects, such as tea leaves in the tea cup and the pen on the paper.

[0094] It can be understood that the target image of the user query in the embodiment can be an image obtained by collecting the scene in which the user is currently located. For example, as shown in the input module of Figure 5 The user uses the application program of the mobile phone to capture the target image of the desktop in front and upload it, so as to make recommendations based on the real-time needs of the user in the real-time space-time scene.

[0095] In other embodiments, in order to reduce the online calculation amount, the process of obtaining the scene graph can be image recognition on the target image to obtain the information of at least one subject in the target image, vectorize the information of the at least one subject to obtain a subject vector, and then match in a scene graph library. The scene graph library pre-stores a large number of scene graphs corresponding to different types of images. By matching to the scene graph closest to the subject vector, the scene graph corresponding to the target image can be determined, avoiding the use of large models online to generate the scene graph and speeding up the online response speed.

[0096] Next, in step S602, the associated information of the user is obtained, and the first information of the user is generated based on the associated information by the space-time large language model.

[0097] Exemplarily, as shown in step 3 of the preference discovery module of Figure 5 The associated information input into the space-time large language model can include attribute information of the user and historical behavior data of the user. After inputting the associated information into the space-time large language model, the space-time large language model can model the user's interest and predict the first information indicating the user's interest tendency.

[0098] The detailed explanation of the association information and the first information can refer to the related description in the foregoing, and will not be repeated here.

[0099] In step S603, based on the first information, a target subject in the at least one subject and a vocabulary having an association relationship with the target subject are determined.

[0100] The vocabulary having an association relationship with the target subject can be a vocabulary having a semantic association with the target subject, and the vocabulary is used to provide diversified information to expand the coverage of the recommended content. For example, when the target subject is tea, the vocabulary having a semantic association with the tea can be “green tea”, “tea art”, “tea culture”, “health preservation”, etc. The vocabulary having an association relationship with the target subject can also be a vocabulary related to the user's interest, so that the recommended content is more targeted. For example, when the target subject is a notebook computer and the user is a bank employee, the vocabulary related to the user can be “productivity tool”, “data analysis”, “efficient office”, etc. It can be understood that the above association relationship can be a direct association relationship or an indirect association relationship. In some embodiments, the vocabulary can include a hypernym of the target subject. For example, when the target subject is tea, the hypernym can be “plant”, “beverage”, “tea”, “leaf of plant”, etc. to introduce a larger range of knowledge background information.

[0101] In some embodiments, the vocabulary can be obtained by expanding the association for each subject, and then the target subject and the corresponding vocabulary are determined based on the first information from the plurality of subjects and the corresponding vocabulary. In yet some embodiments, the target subject can be determined based on the first information, and then the vocabulary having an association relationship with the target subject is obtained by expanding the target subject.

[0102] As an implementation manner, the present step can specifically include the following steps a) - c):

[0103] Step a), based on a preset corpus, the association information and / or the context information of the at least one subject are retrieved.

[0104] The context information includes an association word obtained by expanding the subject and / or the association information. For example, the association word can be a hypernym or a hyponym. For example, the hypernym of a bank employee can be bank, finance, securities, and the hyponym can be a clerk, a customer manager, etc.

[0105] The preset corpus can be various open knowledge bases or private corpora. For example, the corpus can be a public corpus such as Wikipedia, a private corpus such as a user's own knowledge base, or a combination of the public corpus and the private corpus. Figure 5The generator shown in step 2 of the method, in the scene graph containing the entity, can use an Open Knowledge Retriever to retrieve the most relevant contextual information from multiple open corpora to provide semantically rich background knowledge, ensuring that the information source relied on by the scene expansion is extensive and diverse, thereby improving the coverage and accuracy of the knowledge.

[0106] Step b) generating, by the first multi-modal large model, a vocabulary related to the subject based on the association information, the at least one subject, and the context information.

[0107] In practice, a prompt word can be constructed based on the association information, the at least one subject, and the context information, which is used to prompt the large model to generate a vocabulary related to both the user and the subject. The prompt word is input into the first multi-modal large model to obtain the vocabulary output by the large model. Exemplarily, the first multi-modal large model can be an existing open-source large model.

[0108] It can be understood that the vocabulary is essentially knowledge enhancement based on the scene graph to better utilize large language models as user agents to simulate user preferences and interests.

[0109] Continuing to refer to step 2 of the method, Figure 5 The process of generating the vocabulary in this step can be regarded as contextualized expansion of the original scene graph, thereby generating a knowledge graph, which can also be represented using a set of triples, each triple representing an association between a subject and a vocabulary, and multiple triples forming a multi-hop chain relationship.

[0110] Taking the knowledge graph in Figure 5 as an example, the vocabulary in the blue module is expanded from the subject, and the vocabulary in the yellow module is expanded from the association information. As can be seen, there is a one-hop relationship between tea and tea cup, and the relationship between the two subjects is represented by one triple. There is a two-hop relationship between tea and tableware, and the relationship between the two subjects is represented by two triples, i.e., the triple composed of tea and tea cup and the triple composed of tea cup and tableware. There is a three-hop relationship between tea and idea expansion, and the relationship between the two subjects is represented by a combination of multiple triples. This process not only preserves the original structure of the scene, but also combines personalized adjustments of user attribute information (such as long-term interests, recent concerns, etc.) to ensure that the generated knowledge graph is not only rich but also controllable and targeted.

[0111] The predetermined format of the prompt word can be set by staff as needed, and is generally set to a format with strong readability. For example, the constructed prompt word A can be as follows:

[0112] You are an advanced multi-modal AI assistant capable of analyzing image content, structured data, and textual descriptions to provide precise and targeted outputs. You will think as a user, extracting relevant knowledge from images based on the user's interests and background, and outputting in a specific format. Your responsibility is to ensure the accuracy, logic, and high relevance of the output.

[0113] **Task Description**:

[0114] Given a picture taken by a user, and multiple triples describing the relationships between the subjects in the picture, in the format (Head Subject, Relation, Tail Subject), and the encyclopedic information of these subjects, the user's basic information (such as occupation, demographic profile, region, and interests).

[0115] 1. Multiple triples describing the relationships between the subjects in the picture: {content}

[0116] 2. Encyclopedic information of each subject: {entities}

[0117] 3. User information:

[0118] - Occupation: {work}

[0119] - Demographic profile: {crowd}

[0120] - Region: {region_name}-{prov_name}-{city_name}-{city_degree}

[0121] - Interests: {interest}

[0122] Your task is:

[0123] 1. Extract relevant content from the encyclopedic information of the subjects that align with the user's interests.

[0124] 2. The extracted content should revolve around the relationships between the subjects in the picture and be highly relevant to the user's interests and background.

[0125] 3. The final output format is a triple: (Subject, Relation, Other Object or Information). The "Subject" in the output must be one of the head subjects or tail subjects from the given triples.

[0126] **Hints and Rules**:

[0127] 1. Filter information based on user interest: Based on the user's basic information (interests, profession, etc.), extract the subject encyclopedia content that the user may be interested in. You need to highlight the content highly related to the user's interest and ignore irrelevant or unnecessary details.

[0128] 2. Ensure the format and logic of the output: Finally output in triple format: (subject, relationship, other object or information). Where:

[0129] - The head subject or tail subject must come from the given triple.

[0130] - "Relationship" needs to be a concise and accurate description of the associated content.

[0131] - "Other objects or information" can include encyclopedia-related knowledge about the subject or other information that matches the user's interests.

[0132] 3. Rigor and coherence: Ensure that the output triplets are logically clear, accurate, and highly consistent with the user's interest points.

[0133] **Output requirements**:

[0134] - Only output triplets with reasonable facts.

[0135] - Directly give triple results in the example output format, and prohibit the addition of explanatory words.

[0136] **Example format**:

[0137] (XXX, Relationship A, YYY)

[0138] (XXX, Relationship B, ZZZ)

[0139] Then, after inputting the prompt word A into the first multi-modal large model, a set of triples consisting of multiple subjects and words associated with them is obtained, that is, a knowledge graph, so that the words associated with the subject are obtained. It can be understood that the number of words associated with each subject in this embodiment is not limited.

[0140] In other embodiments, to reduce online computing, steps a) and b) can be replaced by an offline execution scheme. Specifically, a query vector can be constructed according to the association information and at least one subject, and then matched in a knowledge graph library. The knowledge graph library pre-stores a large number of different types of images and corresponding knowledge graphs of users. By matching to the knowledge graph closest to the query vector, the knowledge graph can be determined, avoiding the use of large models for knowledge graph generation online, and speeding up the online response speed.

[0141] Step c), based on the first information, determining the target subject and the words having the association relationship with the target subject from the at least one subject and the words having the association relationship with the subject.

[0142] In practice, the semantic similarity of the first information and each subject-word pair (i.e., the subject and the words having the association relationship with the subject) can be calculated, and then one or more subject-word pairs with the highest similarity are determined as the target subject and the words having the association relationship with the target subject to find the target subject and the words closest to the user's interest tendency. The calculation manner of the similarity in this embodiment is not limited, and for example, a neural network can be used for calculation.

[0143] As an implementation manner, this step can specifically include the following steps 1) to 3):

[0144] Step 1), performing semantic encoding on the first information to obtain a first semantic vector.

[0145] Step 2), performing semantic encoding on each subject and the words having the association relationship with the subject to obtain a second semantic vector.

[0146] Still taking Figure 5 as an example, referring to the retriever shown in step 4 of Figure 5 , in order to further construct a personalized knowledge graph, a hop-by-hop semantic index can be performed on the knowledge graph to effectively retrieve related information and efficiently capture the first-order neighbors having clear semantic association, so as to use the first-order neighbors as a new basis for knowledge retrieval, thereby enhancing the knowledge matching capability of the user's interest.

[0147] For example, a pre-trained language model (such as a certain generative encoder) can be introduced to perform semantic encoding on the first information and the triples in the knowledge graph respectively to obtain the first semantic vector and the second semantic vector, thereby capturing the potential semantic information of the first information and the triples.

[0148] Step 3), based on the similarity of the first semantic vector and the second semantic vector, screening the second semantic vector to determine the target subject from the at least one subject and the words having the association relationship with the target subject.

[0149] Then, the cosine similarity between the first semantic vector and the second semantic vector can be calculated, and in other examples, the similarity can also be measured by using the Euclidean distance and the like. In the knowledge graph, since the relationships between the multiple subjects and the words are connected through the chain relationship composed of multiple triples, the semantic similarity between the first semantic vector corresponding to the first information and the second semantic vector corresponding to the triples can be calculated hop by hop.

[0150] For example, as Figure 5For the chain relationship of tea- pen- recording tool- expression medium, the first jump is the association relationship between tea and pen, and the corresponding second semantic vector can be obtained by performing semantic coding on the triple consisting of tea and pen. The second jump is the association relationship between tea and recording tool, and the corresponding second semantic vector can be obtained by performing semantic coding on the splicing of the triple consisting of tea and pen and the triple consisting of pen and recording tool. The third jump is the association relationship between tea and expression medium, and the corresponding second semantic vector can be obtained by performing semantic coding on the splicing of the triple consisting of tea and pen, the triple consisting of pen and recording tool, and the triple consisting of recording tool and expression medium. For the chain relationship of tea- beverage- cultural experience, the first jump is the association relationship between tea and beverage, and the corresponding second semantic vector can be obtained by performing semantic coding on the triple consisting of tea and beverage. The second jump is the association relationship between tea and cultural experience, and the corresponding second semantic vector can be obtained by performing semantic coding on the splicing of the triple consisting of tea and beverage and the triple consisting of beverage and cultural experience.

[0151] In the hop-by-hop calculation, the similarity between the first semantic vector and the second semantic vector corresponding to the first jump can be calculated first. If the similarity is higher than a set threshold, the triple is considered to be highly relevant and is retained. Otherwise, the propagation is continued along the graph structure, and the similarity of the next-hop adjacent triple is recursively calculated to ensure the expansibility of the retrieval range. For example, the similarity between the first semantic vector and the second semantic vector corresponding to the second jump is calculated, and it is determined whether it is higher than the set threshold. The cycle is repeated until the entire knowledge graph is traversed.

[0152] Finally, after completing the hop-by-hop indexing, the retained association relationships are screened, and the screened association relationships are sorted according to the semantic similarity. For example, the top 5 association relationships are selected, the subject and the vocabulary associated with the association relationship are taken as the target subject, and the vocabulary having an association relationship with the target subject, so as to be used for subsequent personalized recommendation or knowledge enhancement tasks.

[0153] As Figure 1As shown in the reordering module, by comparing the similarity between the first semantic vector and the second semantic vector corresponding to the association relationship between tea and beverage, the similarity score of the association relationship between tea and beverage is 0.9, by comparing the similarity between the first semantic vector and the second semantic vector corresponding to the association relationship between tea and cultural experience, the similarity score of the association relationship between tea and cultural experience is 0.7, the higher the similarity score, the closer the semantic relationship between the two, it can be seen that the user has a high interest in beverage, and based on this, question generation can meet the user's needs, and if only the target image is analyzed without considering the user's interest, it is difficult to guide the concept of beverage. Through the above method, structured knowledge can be effectively extracted from complex visual content, and personalized knowledge graph can be generated according to the user's preference, thereby improving the accuracy and relevance of information retrieval and recommendation.

[0154] It can be understood that in the embodiment, when the subject in the target image is one, the subject is the target subject, and there are multiple words associated with the subject. In this case, the above processing also needs to be performed to determine a limited number of words according to the requirements.

[0155] In addition, in step S604, at least one recommended content for the user is generated based on the target subject and the words associated with the target subject through the large language model.

[0156] In practice, the prompt word input large language model can be constructed based on the target subject and the words associated with the target subject, and the large language model can output the corresponding recommended content. The prompt word can contain the attribute information of the user and the historical behavior data of the user to provide rich background information to the large model, and can also contain the location information of the user and the current time information of the user to assist the large model to capture the complex and real dynamic needs of the user according to the actual time and geographical environment.

[0157] For example, the constructed prompt word B is as follows:

[0158] You are a multi-modal artificial intelligence assistant, your main task is to generate an attractive question for the user based on the user uploaded picture, known background information and user behavior data, so as to guide the user to click interaction. The generated question needs to be highly related to the picture content, and combined with the user's interest, background and current scene characteristics, it has attraction and personalization. The following is your task requirement:

[0159] **Task Description**:

[0160] 1. The user uploads a photo, we know the target subject of the picture and the words associated with the target subject, and provide these information in the form of triplets: {triplets}.

[0161] 2. User's background information includes:

[0162] - Occupation: {work}

[0163] - Crowd profile: {crowd}

[0164] - Region: {region_name} - {prov_name} - {city_name} - {city_degree}

[0165] - Topics of interest: {interest}

[0166] 3. User's recent purchase history: {history}.

[0167] 4. User's current photo-taking time and location.

[0168] - Time: {time}

[0169] - Location: {location}

[0170] 5. Based on the above multi-dimensional information, you need to analyze the user's background and current scenario (time, location) and propose an interactive question for the user.

[0171] **Hints and rules**:

[0172] 1. The question content must be related to the picture content provided by the user, combining user interest, background, and scene characteristics.

[0173] 2. Prefer to highlight directions that may interest the user, such as combining their occupation, region, purchase record, and interest topics.

[0174] 3. If the target subject of the picture is a certain type of commodity or scenery, try to guide the user to think from the perspective of practicality or emotional association.

[0175] 4. If the picture involves multiple target subjects, prefer to choose the angle with higher relevance to the user's interest or the shooting scene.

[0176] 5. The generated question must be concise and clear, not more than 18 characters.

[0177] 6. The question must be attractive and encourage users to click or further interact.

[0178] **Output format**:

[0179] Directly output a final generated interactive question, no explanation or additional instructions are required. The question must not exceed 18 characters.

[0180] Thus, when the prompt word B is input into the space-time large language model, the output recommendation content is an interactive question , and the generation process can be shown in the following formula:

[0181] (4)

[0182] wherein, represents the user, represents the target image, represents the associated information of the user, represents the determined five target subjects and the words associated with the target subjects, such as tea and drinks, tea and cultural experience, notebook computer and productivity tools, tea cup and tableware, and tea cup and idea expansion.

[0183] The interactive question can timely and individually respond to the multi-modal request of the user, and return to Figure 7 the example, since the user has historical behavior data related to low blood pressure, the generated recommendation content contains “What are the daily care for low blood pressure?”, which can timely meet the user's demand.

[0184] Figure 8 A flowchart of a recommendation method based on a large language model is shown, which can be executed by any computing device. The following describes an example of execution on a client, including the following steps S701-S702.

[0185] In step S701, the target image of the user is obtained.

[0186] In this embodiment, the client can obtain the target image of the user in various ways, for example, from the Internet, from a local database, or by calling an image acquisition device (such as a camera). For example, the user can call the camera through the client to collect the target image of the surrounding environment.

[0187] In step S702, the recommendation content corresponding to the target image is displayed to the user.

[0188] The recommendation content is generated by the large language model based on the target subject for the user, the target subject is determined from at least one subject based on the first information, the first information is used to indicate the interest tendency of the user, and the at least one subject is identified from the target image. For the generation process of the recommendation content, refer to the description in the previous embodiment, which will not be repeated here.

[0189] In this step, the recommendation content corresponding to the target image can be displayed on the interface of the client, and each recommendation content can be displayed at different positions on the interface.

[0190] In some embodiments, in order to make recommendations more accurately according to user needs, a content generation direction can also be obtained before generating the recommended content. The content generation direction is used to indicate the information type of the recommended content, and the recommended content is generated by the large language model based on the target subject and the content generation direction.

[0191] For example, the content generation direction can be "knowledge", indicating that the information type of the recommended content is knowledge type, and the prompt word input into the large language model is adjusted to generate the recommended content from the perspective of popular science and rigor; the content generation direction can also be "inspiration", indicating that the information type of the recommended content is knowledge type, and the prompt word input into the large language model is adjusted to generate the recommended content from the perspective of interesting and humorous.

[0192] In one example, the preset content generation direction can be obtained, or the specified content generation direction can be obtained, such as displaying different content generation direction controls on the interface, and the user can select or switch the content generation direction by clicking the control, Figure 9 An interface schematic diagram when the content generation direction is knowledge is shown.

[0193] In other embodiments, after step S702, the recommended content can also be stored in the user's preference record in response to the first operation of the user on the interactive interface.

[0194] The first operation is used to indicate the user's interest in the recommended content, for example, the first operation can be the user clicking the favorite or like control on the interface, as shown in Figure 7 The control can be located at the top right of the interface, and the preference record can be used as historical behavior data of the user to improve the accuracy of the first information indicating the user's interest tendency in subsequent generation.

[0195] In other embodiments, after step S702, the recommended content can also be stored in the user's historical browsing record, and the historical browsing record is used to generate the first information.

[0196] In practice, the recommended content generated each time can be recorded in the user's historical browsing record, and the historical browsing record can be used as historical behavior data of the user to improve the accuracy of the first information indicating the user's interest tendency in subsequent generation.

[0197] In other embodiments, the user can also be displayed new recommended content corresponding to the target image in response to a second operation of the user on the interactive interface. For example, the second operation can be that the user clicks a control on the interface for regenerating recommended content, or that the user swipes right on the interface. After receiving an instruction triggered by the second operation, the client can request the remote server to regenerate recommended content, and then receive and display the regenerated recommended content.

[0198] Figure 10 The scheme provided by the corresponding embodiments ensures the fairness of network service use by using a low threshold to show the user recommended content that the user may be interested in based on the target image of the user, and improves the user experience by making the generated recommended content more in line with the user's interests and actual scenarios.

[0199] Figure 11 FIG. 1 is a structural schematic diagram of a recommendation device based on a large language model in an embodiment of the present specification. The device can be applied to any device, platform or device cluster with computing and processing capabilities. The device comprises:

[0200] The subject query module 101 is configured to obtain at least one subject of a user query, and the at least one subject is identified from a target image;

[0201] The information acquisition module 102 is configured to obtain first information of the user, and the first information is used to indicate the interest tendency of the user;

[0202] The target determination module 103 is configured to determine a target subject in the at least one subject based on the first information;

[0203] The content generation module 104 is configured to generate at least one recommended content for the user based on the target subject through a large language model.

[0204] In some embodiments, the subject query module 101 is specifically configured to: obtain associated information of the user; and generate the first information of the user based on the associated information through a large language model.

[0205] In some embodiments, the associated information of the user comprises at least one of the following information: attribute information of the user, historical behavior data of the user, location information of the user, and current time information of the user.

[0206] In some embodiments, the target determination module 103 is specifically configured to: determine the target subject in the at least one subject and a word having an association relationship with the target subject based on the first information; and the content generation module 104 is specifically configured to: generate at least one recommended content for the user based on the target subject and the word having the association relationship with the target subject through a large language model.

[0207] In some embodiments, the target determination module 103, when configured to determine the target subject and the words having the association relationship with the target subject from the at least one subject based on the first information, is specifically configured to: retrieve the association information and / or the context information of the at least one subject based on a preset corpus, the context information including the association words obtained based on the subject and / or the association information; generate the words having the association relationship with the subject based on the association information, the at least one subject, and the context information through the first multi-modal large model; and determine the target subject and the words having the association relationship with the target subject from the at least one subject and the words having the association relationship with the subject based on the first information.

[0208] In some embodiments, the target determination module 103, when configured to determine the target subject and the words having the association relationship with the target subject from the at least one subject based on the first information, is specifically configured to: perform semantic encoding on the first information to obtain a first semantic vector; perform semantic encoding on each of the subject and the words having the association relationship with the subject to obtain a second semantic vector; and filter the second semantic vector based on the similarity between the first semantic vector and the second semantic vector to determine the target subject and the words having the association relationship with the target subject from the at least one subject.

[0209] In some embodiments, the words include hypernyms of the target subject.

[0210] In some embodiments, the large language model is a spatiotemporal large language model, and the spatiotemporal large language model is fine-tuned in the following manner: obtaining a spatiotemporal data set and a label of a sample user, the spatiotemporal data set including association information of the sample user, and the label including a service used by the sample user; predicting the service used by the sample user based on the spatiotemporal data set through the spatiotemporal large language model; and adjusting network parameters of the spatiotemporal large language model based on a prediction result and the label.

[0211] In some embodiments, the subject query module 101 is specifically configured to: obtain a target image input by a user, and generate a scene graph including the at least one subject and the relationship between the subjects by analyzing the structure in the target image based on the target image through the second multi-modal large model.

[0212] In some embodiments, the target image is an image obtained by capturing a scene in which the user is currently located.

[0213] In some embodiments, the content generation module 104 is specifically configured to: generate at least one recommended content for the user based on the target subject and the association information through the large language model.

[0214] Figure 3is a structural schematic diagram of still another recommendation device based on a large language model in the embodiments of the present specification. The device can be applied to any device, platform or device cluster with computing and processing capabilities. The device comprises:

[0215] The image acquisition module 111 is configured to acquire a target image of a user.

[0216] The interface display module 112 is configured to display, to the user, recommendation content corresponding to the target image, the recommendation content being generated by the large language model based on a target subject and a content generation direction, the target subject being determined from at least one subject based on first information, the first information being used to indicate the interest tendency of the user, the at least one subject being identified from the target image.

[0217] In some embodiments, the device further comprises a direction determination module (not shown in the figure) configured to acquire a content generation direction, the content generation direction being used to indicate the information type of the recommendation content, the recommendation content being generated by the large language model based on the target subject and the content generation direction.

[0218] In some embodiments, the device further comprises a preference recording module (not shown in the figure) configured to, after displaying, to the user, the recommendation content corresponding to the target image, store the recommendation content into a preference record of the user in response to a first operation of the user on the recommendation content in the interactive interface, the first operation being used to indicate the interest of the user in the recommendation content, the preference record being used to generate the first information.

[0219] In some embodiments, the device further comprises a history recording module (not shown in the figure) configured to, after displaying, to the user, the recommendation content corresponding to the target image, store the recommendation content into a history browsing record of the user, the history browsing record being used to generate the first information.

[0220] The embodiments of the present specification also provide a computer readable storage medium having a computer program stored thereon, wherein when the computer program is executed in a computer, the computer executes the method as described above. Figure 7 or Figure 3 described above.

[0221] The embodiments of the present specification also provide a computing device comprising a memory and a processor, wherein the memory has executable code stored therein, and the processor executes the executable code to implement the method as described above. Figure 7 or Figure 3 described above.

[0222] The embodiments of the present specification also provide a computer program product comprising computer programs / instructions, which, when executed by a processor, implement the steps of the method as described above. Figure 7 or ​ described above.

[0223] It can be understood that, before or while using the technical solutions of the various embodiments in the specification, the type, use range, use scenario, etc. of the personal information involved will be informed to the user in a proper manner according to relevant laws and regulations, and the authorization of the user will be obtained.

[0224] For example, in response to receiving an active request of a user, prompt information is sent to the user to explicitly prompt the user that the operation requested to be performed will require obtaining and using personal information of the user. Thus, the user can voluntarily choose whether to provide personal information to the software or hardware such as an electronic device, an application program, a server or a storage medium, etc. that performs the operation of the technical solutions of the specification according to the prompt information.

[0225] As an optional but non-limiting implementation manner, in response to receiving an active request of a user, the manner of sending prompt information to the user may, for example, be a pop-up window manner, in which the prompt information can be presented in a textual manner. In addition, the pop-up window can also carry selection controls for the user to select “agree” or “disagree” to provide personal information to the electronic device.

[0226] It can be understood that the above notification and obtaining of user authorization process is only illustrative, and does not limit the implementation manners of the specification, and other manners meeting relevant laws and regulations can also be applied to the implementation manners of the specification.

[0227] Those skilled in the art should appreciate that, in one or more examples described above, the functions described in the various embodiments disclosed in the specification can be implemented in hardware, software, firmware or any combination thereof. When implemented in software, these functions can be stored in a computer readable medium or transmitted as one or more instructions or codes on a computer readable medium.

[0228] In some cases, the acts or steps recited in the claims can be performed in a different order from those in the embodiments and still achieve the desired result. In addition, the processes depicted in the figures do not necessarily require the particular order shown or sequential order to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

[0229] The above detailed description sets forth the purposes, technical solutions and beneficial effects of the various embodiments disclosed in the specification. It should be understood that the above detailed description is only a specific implementation of the various embodiments disclosed in the specification, and is not intended to limit the protection scope of the various embodiments disclosed in the specification. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solutions of the various embodiments disclosed in the specification shall be included in the protection scope of the various embodiments disclosed in the specification.

Claims

1. A recommendation method based on a large language model, the method comprising: obtaining at least one subject of a user query, the at least one subject being identified from a target image; obtaining first information of the user, the first information being generated by a pre-trained large language model based on attribute information and historical behavior data of the user, the first information being used to indicate an interest tendency of the user; determining context information corresponding to the at least one subject based on the at least one subject, the context information containing associated words obtained by expanding based on the subject; determining an association relationship between the subject and a vocabulary based on the associated information of the user and the context information corresponding to the at least one subject, a plurality of the association relationships forming a chain relationship of multiple hops, the vocabulary including the context information and vocabulary obtained by expanding based on the associated information; performing semantic encoding on the first information to obtain a first semantic vector, and performing semantic encoding on each of the subject and the vocabulary associated with the subject to obtain a second semantic vector, the subject and the vocabulary associated with the subject having a chain relationship of at least one hop; calculating semantic similarity between the first semantic vector and the second semantic vector hop by hop, and in the calculation process, if the semantic similarity corresponding to a certain hop is higher than a set threshold, the semantic similarity is retained, otherwise, the semantic similarity corresponding to the next hop is continued to be calculated; based on the ranking of the semantic similarity, determining the subject and the vocabulary associated with the target subject associated with the association relationship corresponding to the second semantic vector that meets the preset ranking requirement as a target subject and a vocabulary associated with the target subject; generating at least one recommended content for the user based on the target subject and the vocabulary associated with the target subject by the large language model, the recommended content being used to connect to an online service corresponding to the recommended content.

2. The method of claim 1, wherein, The obtaining of the first information of the user comprises: obtaining associated information of the user; generating the first information of the user based on the associated information by the large language model.

3. The method of claim 2, wherein, The associated information of the user includes at least one of the following information: attribute information of the user, historical behavior data of the user, location information of the user, and current time information of the user.

4. The method of claim 2, wherein, The determination of the context information corresponding to the at least one subject based on the at least one subject comprises: retrieving the context information corresponding to the at least one subject based on a preset corpus, the context information containing associated words obtained by expanding based on the subject; The determination of the association relationship between the subject and the vocabulary based on the associated information of the user and the context information corresponding to the at least one subject comprises: generating the vocabulary associated with the at least one subject by a first multi-modal large model based on the associated information, the at least one subject, and the context information.

5. The method of claim 1, wherein, The large language model is a spatiotemporal large language model, and the spatiotemporal large language model is obtained by fine-tuning in the following manner: Obtaining a spatio-temporal data set and a label of a sample user, the spatio-temporal data set comprising: associated information of the sample user, and the label comprising a service used by the sample user; Predicting, by the spatio-temporal large language model, the service used by the sample user based on the spatio-temporal data set; Adjusting network parameters of the spatio-temporal large language model based on the prediction result and the label.

6. The method of claim 1, wherein, The at least one subject of the user query is obtained, including: Obtaining the target image input by the user, and generating a scene graph by a second multi-modal large model based on the structure in the target image, the scene graph comprising the at least one subject and the relationship between the subjects.

7. The method of claim 2, wherein, The at least one recommended content for the user is generated by the large language model based on the target subject and the words having an association relationship with the target subject, including: The at least one recommended content for the user is generated by the large language model based on the target subject, the words having an association relationship with the target subject, and the associated information.

8. The method of claim 1, wherein, The target image is an image collected from a scene currently occupied by the user.

9. A recommendation method based on a large language model, the method comprising: Obtaining a target image of a user; Displaying recommended content corresponding to the target image to the user, the recommended content being generated by the large language model for the user based on a target subject and words having an association relationship with the target subject, the recommended content being used to connect to an online service corresponding to the recommended content; Wherein, the target subject and the words having an association relationship with the target subject are determined by the following way: Based on at least one subject, determining context information corresponding to the at least one subject, the context information comprising associated words obtained based on the expansion of the subject; Based on the associated information of the user and the context information corresponding to the at least one subject, determining the association relationship between the subject and the words, a multi-hop chain relationship being formed between multiple association relationships, the words comprising the context information and words obtained based on the expansion of the associated information; Semantically encoding the first information to obtain a first semantic vector, and semantically encoding each of the subject and the words having an association relationship with the subject to obtain a second semantic vector, the subject and the words having an association relationship with the subject having at least one-hop chain relationship therebetween; Calculating the semantic similarity between the first semantic vector and the second semantic vector hop by hop, in the calculation process, if the semantic similarity corresponding to a certain hop is higher than a set threshold, the semantic similarity is retained, otherwise, the semantic similarity corresponding to the next hop is continued to be calculated; Based on the ranking of the semantic similarity, the subject and the words associated by the association relationship corresponding to the second semantic vector satisfying the preset ranking requirement are determined as the target subject and the words having an association relationship with the target subject. The at least one subject is obtained by recognizing the target image. The first information is generated by the large language model based on the user's attribute information and historical behavior data. The first information is used to indicate the user's interest tendencies.

10. The method of claim 9, wherein, The method further includes: The content generation direction is obtained, which indicates the information type of the recommended content. The recommended content is generated by the large language model for the user based on the target subject and the content generation direction.

11. The method of claim 9, wherein, After displaying recommended content corresponding to the target image to the user, the method further includes: In response to the user's first operation on the recommended content on the interactive interface, the recommended content is stored in the user's preference record, the first operation being used to indicate the user's interest in the recommended content, and the preference record being used to generate the first information.

12. The method of claim 9, wherein, After displaying recommended content corresponding to the target image to the user, the method further includes: The recommended content is stored in the user's browsing history, which is used to generate the first information.

13. A recommendation device based on a large language model, the device comprising: The subject query module is configured to obtain at least one subject queried by the user, the at least one subject being identified from the target image; The information acquisition module is configured to acquire the user’s first information, which is generated by a pre-trained large language model based on the user’s attribute information and historical behavior data. The first information is used to indicate the user’s interest tendencies. The target determination module is configured to determine context information corresponding to the at least one subject based on the at least one subject, wherein the context information includes related words obtained by expanding based on the subject; Based on the user's association information and the context information corresponding to the at least one subject, the association relationship between the subject and the words is determined. Multiple association relationships form a multi-hop chain relationship. The words include the context information and words expanded based on the association information. The first information is semantically encoded to obtain a first semantic vector, and each subject and the words associated with it are semantically encoded to obtain a second semantic vector. There is at least one hop chain relationship between the subject and the words associated with it. The semantic similarity between the first semantic vector and the second semantic vector is calculated hop-by-hop. During the calculation, if the semantic similarity corresponding to a certain hop is higher than a set threshold, the semantic similarity is retained; otherwise, the semantic similarity corresponding to the next hop is calculated. Based on the ranking of the semantic similarity, the subjects and words associated with the association relationships corresponding to the second semantic vectors that meet the preset ranking requirements are determined as the target subject and the words associated with the target subject. The content generation module is configured to generate, by the large language model, at least one recommended content for the user based on the target subject and the words having the association relationship with the target subject, the recommended content being used to connect to an online service corresponding to the recommended content.

14. A computing device comprising a memory and a processor, wherein, The memory stores executable code, and the processor executes the executable code to implement the method in any one of claims 1-12.

Images