User intent recognition method based on multi-modal model

Abstract

The specification provides a user intention recognition method based on a multi-modal model, the method comprising: determining a keyword set of a user group according to historical description texts of a required service of the user group; inputting the keywords in the keyword set into a text encoder in a preset picture-text multi-modal model to obtain semantic feature vectors of the keywords output by the text encoder; clustering the keyword set according to the semantic feature vectors of the keywords in the keyword set, and processing each cluster obtained by clustering to obtain one or more high-quality clusters; the semantic similarity of the semantic feature vectors of the keywords in the high-quality cluster satisfies a preset similarity condition, and the distance distribution of the semantic feature vectors of the keywords and the cluster center satisfies a preset concentration condition; and determining a user intention of the user group for the required service according to the keywords in the high-quality cluster.

Description

Technical Field

[0001] This specification relates to the field of natural language processing technology, and in particular to a method for user intent recognition based on a multimodal model. Background Technology

[0002] In online consumption scenarios such as e-commerce or instant retail, service providers (platforms) can offer services to users. To improve service accuracy, service providers need to understand the true consumption needs and intentions of user groups. Currently, there are various technical approaches to analyzing user intent, such as statistical analysis based on users' historical purchase records, user profiling based on pre-defined tag systems, or collaborative filtering recommendations based on behavioral logs. Therefore, accurately identifying users' intent for the services they require has become a pressing technical problem that needs to be solved. Summary of the Invention

[0003] To overcome the problems existing in related technologies, this specification provides a user intent recognition method based on a multimodal model.

[0004] According to a first aspect of the embodiments of this specification, a user intent recognition method based on a multimodal model is provided, the method comprising:

[0005] Based on the historical descriptions of the required services by users in the user group to be identified, determine the keyword set of the user group;

[0006] The keywords in the keyword set are input into the text encoder of the preset image-text multimodal model to obtain the semantic feature vector of the keywords output by the text encoder;

[0007] Based on the semantic feature vectors of each keyword in the keyword set, the keyword set is clustered, and the clusters are processed to obtain one or more high-quality clusters; wherein, the semantic similarity of the semantic feature vectors of each keyword in the high-quality clusters satisfies a preset similarity condition, and the distance distribution between the semantic feature vectors of each keyword and the center of the cluster satisfies a preset concentration condition.

[0008] Based on the keywords in the high-quality clusters, determine the user intent of the user group for the required services.

[0009] The technical solutions provided in the embodiments of this specification may include the following beneficial effects:

[0010] In this embodiment, the technical approach for user intent recognition is as follows: based on the user's historical descriptions of the desired service, user intent is automatically and deeply identified from these texts that directly reflect user needs. For the service purchase scenario in this embodiment, a text encoder using a multimodal graph-text model is employed. This model can extract semantic feature vectors of keywords in a cross-modal semantic space, enabling the feature vectors to contain joint semantic information of the text and associated visual concepts. This allows subsequent clustering processing to move beyond surface-level text similarity and proceed within a unified semantic space that integrates visual scene information. For example, keywords such as "tent," "picnic mat," and "park" can be effectively clustered to accurately identify the deep-level scene intent of "outdoor picnic," overcoming the limitation of pure text models in associating with real-world visual scenes and significantly improving the depth and accuracy of intent recognition.

[0011] Furthermore, this embodiment designs a step of "processing the various clusters obtained from clustering" to ensure that they meet the "semantic similarity satisfying the preset similarity condition" and the "distance distribution satisfying the preset concentration condition," which optimizes the initial clustering results. Specifically, the "semantic similarity condition" ensures that members within high-quality clusters are highly related and thematically consistent across modalities; the "concentration condition based on distance distribution" effectively identifies and eliminates semantic outliers (noise) by constraining the dispersion of the distance from cluster members to the cluster center. The combined effect of these two conditions ensures that each high-quality cluster corresponds to a service intent that is as pure, focused, and thematically consistent as possible; ultimately, based on the keywords in the high-quality clusters, the user intent of a user group for the desired service can be accurately determined.

[0012] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Attached Figure Description

[0013] Figure 1 This is a flowchart illustrating a user intent recognition method based on a multimodal model according to an exemplary embodiment of this specification.

[0014] Figure 2A This is an application scenario diagram illustrating a user intent recognition method based on a multimodal model, according to an exemplary embodiment of this specification.

[0015] Figure 2B This is a flowchart illustrating a user intent recognition method based on a multimodal model according to an exemplary embodiment of this specification.

[0016] Figure 3This specification is a hardware structure diagram of a computer device containing a user intent recognition device based on a multimodal model, according to an exemplary embodiment.

[0017] Figure 4 This is a block diagram illustrating a user intent recognition device based on a multimodal model according to an exemplary embodiment of this specification. Detailed Implementation

[0018] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this specification as detailed in the appended claims.

[0019] The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of this specification. The singular forms “a,” “the,” and “the” as used in this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0020] It should be understood that although the terms first, second, third, etc., may be used in this specification to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this specification, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0021] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties. The collection, use and processing of related data shall comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals shall be provided for users to choose to authorize or refuse.

[0022] As mentioned above, existing technical approaches to analyzing user needs often focus on the results of user behaviors or static attributes, which are insufficient for mining massive amounts of unstructured text data (such as search terms, order notes, order reviews, etc.) that carry user initiative and contain rich contextual information. Therefore, this embodiment provides a new technical approach, namely, analyzing the historical description text of the services required by the user, aiming to automatically and deeply identify user intent from these texts that directly reflect user needs.

[0023] However, the technical approach based on historical descriptive text, employing pure text analysis techniques (such as text clustering based on language models like BERT), has the following inherent drawbacks: First, semantic understanding is limited to statistical patterns within the text context, lacking a connection to real-world visual scenes and objects, resulting in an inability to accurately grasp consumption scenarios such as "outdoor picnic" or "late-night party." Second, when directly clustering massive, sparse user short text data (such as search terms), it is highly susceptible to interference from high-frequency common words and the introduction of semantic noise, producing a large number of low-quality clusters with broad themes, internal inconsistencies, or outliers, leading to low accuracy in intent recognition.

[0024] Based on this, this embodiment provides a user intent recognition method based on a multimodal model, such as... Figure 1 As shown, Figure 1 This is a flowchart illustrating a user intent recognition method based on a multimodal model according to an exemplary embodiment, comprising the following steps:

[0025] Step 102: Determine the keyword set of the user group based on the historical description text of the required service by users in the user group to be identified.

[0026] Step 104: Input the keywords in the keyword set into the text encoder of the preset image-text multimodal model to obtain the semantic feature vector of the keywords output by the text encoder.

[0027] Step 106: Based on the semantic feature vectors of each keyword in the keyword set, perform clustering on the keyword set, and process the clusters obtained to obtain one or more high-quality clusters.

[0028] Among them, the semantic similarity of the semantic feature vectors of each keyword in the high-quality cluster meets the preset similarity condition, and the distance distribution between the semantic feature vectors of each keyword and the center of the cluster meets the preset concentration condition.

[0029] Step 108: Based on the keywords in the high-quality clusters, determine the user intent of the user group for the required service.

[0030] As an example, please see Figure 2A, Figure 2A This is a schematic diagram of an application scenario provided by an embodiment of this specification. The interactive system includes a server 20 and at least one client 21. Figure 2A (Taking three clients as an example), for instance, client 21 can access the server via the network to use the services provided by the server, including but not limited to instant delivery services, online shopping services, etc.

[0031] Server 20 can be a program installed in a background device to provide services to users. For example, such as... Figure 2A As shown, the backend device can be a server, which can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms. As an example, the user intent recognition method provided in this embodiment can be executed by the server 20.

[0032] Client 21 can be a program installed on a user's device to provide services to the user. Client 21 includes, but is not limited to, applications (APPs), web pages, mini-programs, plugins, or components. As shown in the figure, user devices include, but are not limited to, smartphones, personal digital assistants, tablets, personal computers, laptops, virtual reality terminal devices, and augmented reality terminal devices. As an example, server 20 can obtain data authorized by the user through client 21, such as the historical description text in step 102, which refers to the user's description text of the required service.

[0033] In practical applications, the user group to be identified refers to the group whose user needs need to be identified. This user group may include one or more users. The specific number of users is not limited in this embodiment. The user group to be identified can be flexibly set according to actual needs. This embodiment does not limit this.

[0034] For example, the server can maintain user data, which may include user identifiers and one or more dimensions of user information, including but not limited to the user's geographical location, user access time, user channel, and user lifecycle (such as a custom lifecycle for new or returning customers). User groups to be identified can be obtained based on actual needs using this dimensional information. For example, user groups to be identified can be selected from the entire user base based on defined dimensional information. Alternatively, instead of the entire user base, a specific range of users can be segmented into multiple user groups to be identified based on defined dimensional information. These user groups can also be called target groups, and user intent identification can be performed on each target group. For example, taking a specific range of users as a group with high purchase preferences, the definition of "high purchase preference" can be set according to actual needs, such as the total price of all orders within a preset time period exceeding a preset threshold, and / or the average price of orders within a preset time period exceeding another preset threshold, and / or the average price of services purchased within a preset time period exceeding another preset threshold, etc. This embodiment does not limit this. To analyze and compare the user intent of different dimensions within these user groups with high purchase preferences, multiple user groups to be identified can be segmented from these user groups with high purchase preferences.

[0035] For example, different dimensions of user information can be combined to identify user groups. As an example, using the dimensions "channel source" (such as A and B) and "user lifecycle" (new customers / returning customers), four user groups can be constructed for analysis, as shown in Table 1 below:

[0036]

[0037] The "new customers of channel A", "existing customers of channel A", "new customers of channel B", and "existing customers of channel B" in Table 1 can all be considered as user groups to be identified, and the above steps can be performed separately. Figure 1 The illustrated embodiment of the user intent recognition method obtains the user intent of each user group for the desired service. Optionally, further processing may be performed, such as comparing the user intent of multiple user groups for the desired service.

[0038] In step 102, the historical description text can be obtained by the client from the server and sent to the server; for example, the server can collect and store the historical description text of each user.

[0039] As an example, historical description text may include one or more of the following:

[0040] ① Historical search text of users for services obtained through the client;

[0041] For example, the client provides a service search function, such as a search box on the client's page. Users can enter a description of the service they want to purchase, such as "crayfish," "freshly grilled oysters," "iced beer," or "original flu medicine," reflecting the services the user needs. The client can retrieve this description and send it to the server, which can then store these historical descriptions.

[0042] ②The historical notes text of user service orders obtained through the client;

[0043] For example, users can purchase services through the client, thus creating a service order. The client can also provide a note function for this service order on the order page to meet the user's personalized needs. For example, when placing an order, users can fill in their requirements in the notes box, such as "no cilantro," "extra spicy," or "add an extra box of satay sauce," reflecting the user's personalized consumption preferences and their intention for the desired service. The client can obtain this description text and send it to the server, which can then store these historical description texts.

[0044] ③ Historical comment texts of users on service orders obtained through the client;

[0045] For example, after a service order is completed, the client's order page can also provide a comment function for that service order. Users can fill in comments in the comment box on the page, such as "The oysters were very fresh and plump" or "The satay sauce was delicious," which includes a direct description and feedback of the service consumed by the user, reflecting the service the user requested. The client can obtain this description text and send it to the server, which can then store these historical description texts.

[0046] In other examples, the historical description text may also include the service name of the user's historical purchases, such as the name of the purchased goods. In practical applications, this can be set as needed, and this embodiment does not limit it.

[0047] As an example, the historical description text can be preprocessed, such as through word segmentation or simplification, to obtain one or more keywords contained within it. Optionally, on the server side, each user's historical description text can be stored accordingly, or the server can preprocess each user's historical description text to obtain keywords and store the keywords for each user's historical description text.

[0048] In practical applications, the keyword set of the user group in this embodiment can be determined based on the keywords of each user in the user group; for example, different users can have the same keywords, and the keyword set of the entire user group to be identified can be obtained by deduplication. In other embodiments, the keywords of each user can also be processed to obtain the keyword set of the entire user group to be identified. The specific processing can be set as needed, and this embodiment does not limit it.

[0049] Considering the large number of users in the user group and the fact that each user has multiple historical description texts, meaning a large number of keywords can be identified from these historical description texts, this embodiment can further process the data to ultimately filter out the keyword set for the user group. As an example, this specification also provides the following two optional embodiments:

[0050] (1) In some cases, the set of keywords for the user group is identified, including:

[0051] Obtain the user group's group preference for keywords in historical descriptive text, and obtain the preset global basic group's global preference for keywords;

[0052] Based on group preference and global preference, keywords that indicate differences in user group preferences relative to the global basic group are selected from each historical descriptive text.

[0053] The goal of this embodiment is to identify the unique consumption tendencies or interests of the user group to be identified, which distinguish it from the general public, thereby avoiding the selection of "generic words" that are common to all groups but are high-frequency, and focusing on words that can define the differences between the group and the general public.

[0054] Here, "group preference" can refer to the relative frequency of a keyword appearing within the user group to be identified. For example, search frequency can be the proportion of searches for a keyword (such as "influenza original drug") within that user group. As an example, the calculation formula could be:

[0055] p_target(w) = C_target(w) / N_target

[0056] Where C_target(w): the total number of times the keyword w appears in all historical descriptive texts (such as search records) of the user group to be identified.

[0057] N_target: The total number of times all keywords appear in all historical descriptive texts generated by the user group to be identified.

[0058] Meaning of the calculation result: p_target(w) represents the percentage of frequency of keyword w within the target group.

[0059] Global preference can refer to the relative frequency of the same keyword (such as "influenza original drug") appearing in a preset global base group; where the global base group is set as needed, for example, it can be all users of the platform or a preset large user group, etc. The calculation formula can be:

[0060] p_baseline(w) = C_baseline(w) / N_baseline

[0061] C_baseline(w): The total number of times the keyword w appears in all historical descriptive texts of the preset global base group.

[0062] N_baseline: The total number of occurrences of all keywords in all historical descriptive texts generated by the global base population.

[0063] Meaning of the calculation result: p_baseline(w) represents the average frequency percentage of keyword w in the global base group, serving as the baseline.

[0064] The difference between group preference and global preference can be further obtained, for example, by comparing the two relative frequencies mentioned above; as an example, the Target Group Index (TGI) can be used. Its calculation formula can be:

[0065] TGI = (Percentage of keywords in the target audience / Percentage of keywords in the overall base audience) × 100; that is:

[0066] TGI(w) = [ p_target(w) / p_baseline(w) ]×100.

[0067] Based on this, this embodiment can filter out keywords that indicate a difference in preferences between the user group to be identified and the global basic group. For example, for a certain keyword, if the difference between the group preference and the global preference is small, it means that the user group to be identified and the global basic group have basically the same preference for the keyword. If the difference between the group preference and the global preference is large, it means that the user group to be identified and the global basic group have a large difference in preference for the keyword. Therefore, the keyword can represent the characteristic needs of the user group to be identified.

[0068] As an example, TGI-based filtering logic may include:

[0069] If the TGI is approximately 100, it means that the popularity of the keyword among the target user group (the user group to be identified) is consistent with the overall market, and there is no significant preference.

[0070] If the TGI is greater than 100 (e.g., greater than the preset threshold of 120), it indicates that the keyword is significantly more popular among the target user group than the overall market average. This is precisely the "characteristic demand" keyword that this embodiment aims to filter out for this group. For example, the TGI of "deep-sea salmon" in the user group to be identified may be much higher than 120, indicating that this is a characteristic interest of this group.

[0071] If TGI < 100: This means that the keyword receives less attention from the target audience than the general market, and may not be their core interest.

[0072] As can be seen from the above embodiments, this embodiment can filter out the group tendency characteristics of the user group to be identified, that is, keywords that represent the characteristic needs of the group.

[0073] (2) In some examples, identifying the set of keywords for a user group may include:

[0074] Obtain the intra-group saliency index of keywords in each historical descriptive text of a user group, and based on the intra-group saliency index, filter out keywords from each historical descriptive text whose intra-group saliency index is higher than a preset saliency threshold.

[0075] The goal of this embodiment is to select keywords that represent the core needs of the user group, and to avoid omitting those core need words that are important within the group but have a low TGI because they are also of interest to the global basic group (the majority of users).

[0076] In this embodiment, the intragroup saliency index used may include TF-IDF (termfrequency–inverse document frequency) value, which can measure the importance of a word to a specific document set (here, the set of all historical descriptive texts of the user group to be identified).

[0077] As an example, term frequency (TF) refers to the frequency with which a keyword appears within a group. The higher the frequency, the greater the importance is likely to be. For example, the term frequency of keyword w can be calculated as: "the total number of times keyword w appears in all historical descriptive text sets" divided by "the total number of times all words appear in all historical descriptive text sets".

[0078] The inverse document frequency IDF refers to the reciprocal of the universality of a word's appearance in the texts of all users (or all groups). The more common a word is (such as "的" or "了"), the lower its discrimination ability and the smaller its IDF value. As an example, the IDF of keyword w can be obtained by dividing the total number of historical description texts by the number of historical description texts containing the keyword w, and then taking the logarithm of the resulting quotient.

[0079] TF-IDF = TF×IDF: A high TF-IDF value for a word means that it appears frequently within the group and is relatively unique, not a common word that can be seen everywhere.

[0080] As an example, this embodiment can set a TF-IDF threshold and retain the keywords above this threshold. These words constitute the core keyword set of the user group to be recognized.

[0081] Based on this, this embodiment can screen out the words representing the core concerns within the group through the intra-group significance index. In practical applications, one of the above implementation methods can be selected according to needs, or both of the above implementation methods can be used simultaneously; for example, in the case of using both of the above implementation methods, after implementing them separately, the keyword sets obtained from the two embodiments can be de-duplicated to obtain the final keyword set of the user group. The above preprocessing step of constructing a high-quality keyword set through each historical description text before keyword vectorization can be screened from two different but complementary dimensions of inter-group comparison and intra-group focus to ensure that the obtained keyword set has both "distinctiveness" and "representativeness".

[0082] In step 104, this embodiment does not adopt the traditional idea of using language models such as BERT to vectorize the text. Language models such as BERT lack visual and scene associations. BERT is mainly trained based on large-scale text corpora, and the "semantics" it understands are more the co-occurrence relationships of text contexts rather than the associations with real-world visual concepts. Therefore, applying the traditional idea of using language models such as BERT to vectorize the text in scenarios such as e-commerce or instant retail cannot accurately mine users' needs. For example, in the semantic space of language models such as BERT, "小龙虾" and "皮皮虾" are very similar because they both belong to seafood in the text, but they cannot, like humans, naturally associate "小龙虾" with cross-modal life concepts such as "夜宵", "聚会", and "夏天", which are exactly the user needs to be mined in scenarios such as e-commerce or instant retail. However, the图文多模态模型 can realize the close connection of these cross-modal scene concepts.

[0083] Specifically, the image-text multimodal model is trained through contrastive learning on a massive number of image-text pairs. Its training objective determines that the vectors generated by its text encoder are aligned semantically with the visual features of the corresponding images. Therefore, even if the input is only text, the output feature vectors naturally contain visually relevant scene information. For example, in the CLIP semantic space, based on the image-text pair associations seen during training, the text vector for "crayfish" will be closer to the vectors of scene concepts such as "dinner party" and "beer," rather than just the text vector for "mantis shrimp," which is also a seafood category. This allows subsequent clustering based on such vectors to aggregate around real-world scenes, rather than simply the distribution of text words.

[0084] In addition, if models such as BERT are used for vectorization, the clustering results may be purely based on text co-occurrence statistics, which may easily group together general words that are not related to business scenarios. For example, it may not be able to distinguish words in different consumption scenarios under the same category, resulting in insufficient alignment between the clustering results and the real user intent that the business cares about (which is often strongly related to the scenario), and generating a large number of clusters with low business value.

[0085] Therefore, in step 104 of this embodiment, a preset image-text multimodal model is designed to vectorize keywords, including but not limited to various visual-language multimodal models, such as the CLIP model, etc. This embodiment does not limit this.

[0086] Among related technologies, there are applications based on CLIP (Contrastive Language-Image Pretraining) models, such as the Fashion CLIP model, a CLIP-based model specifically designed for the fashion industry and primarily used in e-commerce platforms. Its related applications include: Product representation and recommendation: By generating visual and textual representations of products, it provides richer information to the recommendation system, improving recommendation accuracy. Automatic product classification: Utilizing learned product features, it automatically classifies fashion products, replacing time-consuming and error-prone manual labeling. Search engine performance enhancement: It helps search engines more accurately understand user query intent and provide more relevant product search results.

[0087] Text-to-image semantic retrieval systems based on CLIP and Milvus can utilize the CLIP model to map text and images into a shared semantic space, and combine this with the Milvus vector database for efficient retrieval. These systems aim to address the semantic gap problem inherent in traditional keyword search by achieving cross-modal semantic search through understanding natural language descriptions.

[0088] In addition, another typical application scenario for CLIP is to perform zero-shot image classification (i.e., using text descriptions as classification labels) or text-based / image-based image search.

[0089] Therefore, this embodiment employs a text encoder from a text-image multimodal model to vectorize keywords, which is one of its innovations. Unlike BERT, the text-image multimodal model includes both a text encoder and an image encoder. The vectors produced by the text encoder in the text-image multimodal model are trained to align with the visual features of the image. This allows even text input to contain rich visual and scene information in its vectors. For example, the text vectors for "beach" and "sunshine" are closer to the scene concept of "vacation" in the CLIP space than in the BERT space.

[0090] Specifically, the core training objective of image-text multimodal models like CLIP is to align text and image representations in a shared semantic space through image-text contrastive learning, rather than learning co-occurrence relationships of texts solely based on a text corpus, as BERT does. The text encoder in an image-text multimodal model is forced to learn semantic representations associated with the visual scene because the training objective requires that the text description be aligned with the image content in the semantic space. For example, the output vector of the text encoder for "crayfish" must be semantically close to the visual feature vector of "picture of crayfish at a late-night snack party," rather than only close to the text vectors of similar seafood such as "mantis shrimp."

[0091] As an example, the keywords in the keyword set can be input into the text encoder of the text-image multimodal model. The text encoder can extract feature vectors from the input keywords and output them, which in this embodiment is called cross-modal semantic feature vectors. Thus, after step 104, each keyword in the keyword set has its own semantic feature vector.

[0092] As an example, in practical applications, the text-image multimodal model can be fine-tuned according to the actual service scenario. For example, in scenarios such as instant retail (e.g., food delivery), in order to make the CLIP model better adapt to the instant retail (food delivery) scenario, since the service may belong to a specific category, such as the food and other items provided in the food delivery scenario, the text-image multimodal model can be fine-tuned as needed.

[0093] The service scenario in this embodiment can connect "user group keywords / themes" with "specific service needs and marketing strategies"; for example, from keywords to scenarios: the solution uses multimodal clustering to aggregate and identify keywords such as "crayfish, beer, late night" as the service scenario of "late-night friends' dinner party", which can link from "what the user says" to "under what circumstances (service scenario) the user consumes".

[0094] From service scenarios to specific service needs (specific user requirements): Within the same scenario, user needs for goods exhibit distinct contextual characteristics specific to that service scenario, rather than simply being a list of product categories. For example, regarding "coffee," in an "office overtime" scenario, the core need is "quick and convenient alertness," with product formats leaning towards Americano, instant, and large quantities. In a "weekend gathering with friends" scenario, the core need might be "leisure and enjoyment, diverse flavors," with product formats likely leaning towards lattes, specialty blends, and pairings with desserts. Therefore, identifying the service scenario is equivalent to understanding the user's "implicit needs" for goods within that service scenario, making it more accurate than simply knowing that a user searched for "coffee."

[0095] From service scenarios to service recommendation strategies, a service recommendation strategy can refer to a service's strategy of recommending certain products to users. Therefore, in this embodiment, it is possible to "push the right product (service) to the right user in the right scenario." For example, for a service scenario of "late-night friends' dinner party" identified by keywords of a certain high-quality cluster of a certain user group, recommendations can be made based on multiple services (products) corresponding to this service scenario. For example, each service scenario can pre-set a corresponding set of products to be recommended, and optionally, discount rules corresponding to the set of products can also be set. Therefore, compared to recommendations based on a single product, this embodiment can recommend multiple products based on the service scenario.

[0096] As an example, a training dataset can be obtained, which includes data pairs consisting of images and text. Images can be from various services on the service provider's platform, such as product images. The text corresponding to the product images includes, but is not limited to, user descriptions, product names set by the store on the platform, or text descriptions of the service scenario to which the product image belongs. Furthermore, images representing different service scenarios can be collected or generated. Each service scenario can be pre-defined, including but not limited to working overtime in the office, watching TV in the living room, late-night dinners with friends, and weekend family gatherings. Text descriptions for each service scenario can be generated. Descriptive text can be generated for these images representing service scenarios. This generation can be done manually or through multimodal image-text models, such as BLIP. Images can be input into the image-text model, which generates text descriptions of the input images. The text descriptions output by the image-text model can be further analyzed to identify the associated service scenarios, ensuring that the text corresponding to the image representing a service scenario contains a description of the service scenario it represents. For example, the text description output by the image-text description model can be compared semantically with the descriptions of various preset service scenarios. Based on the semantic similarity, the text description of the target service scenario that matches the text description output by the image-text description model can be determined from the text descriptions of the preset service scenarios. Alternatively, a language model can be used to determine this. For example, the text description output by the image-text description model and the text descriptions of various preset service scenarios can be input into a language model, which can then determine the text description of the target service scenario that matches the text description output by the image-text description model from the text descriptions of the preset service scenarios.

[0097] Optionally, in the fine-tuning of the image multimodal model, a loss function can be set. The loss function can include image-text contrast loss, which is the loss between the image feature vector extracted by the image encoder in the image multimodal model from the images in the training data and the text feature vector extracted by the text encoder in the image multimodal model from the text in the training data.

[0098] Optionally, this embodiment can also add a "text-text scene alignment loss". For example, in this embodiment, each piece of training data (image I, text T) can be labeled or associated with a service scene label S. For example, the text of a product image on the platform has a service scene label; the text of an image representing a service scene also has a service scene label.

[0099] An in-batch text-to-text scene contrast loss L_s2s can be added to the standard image-to-text contrast loss L_i2t. For example, the training dataset could include:

[0100] Positive pairs: In the same training batch, texts T_i and T_j with the same scene label S constitute positive pairs. Even if they describe different specific items (such as "iced beer" and "spicy crayfish"), as long as they belong to the "late-night snack gathering" scene, their text vectors can be brought together.

[0101] Negative example pairs: Text pairs with different scene labels S constitute negative example pairs.

[0102] Calculation: Losses such as InfoNCE can be used. For a text vector t_i, the loss is to bring it closer to all text vectors in the same batch and the same scene, while simultaneously increasing its distance from all text vectors in different scenes. As an example, the final fine-tuning total loss L_total can be: L_total = L_i²t + λ × L_s²s, where λ is the weighting coefficient balancing the two losses, which can be customized as needed. That is, the image-text multimodal model can be optimized with this total loss as the objective. Specifically, optimization can be based on minimizing the "image-text" contrast loss and the "text-text scene contrast loss".

[0103] The "text-text scene contrast loss" includes the difference between the similarity between the anchor sample and the positive sample, and the sum of the similarities between the anchor sample and all samples. Here, the anchor refers to the text feature vector (anchor point) of each text currently input into the training data of the multimodal graph-text model. All samples refer to all other text samples in the current training batch (excluding itself). In other words, it allows text feature vectors from the same service scene to cluster together in the semantic space and distance themselves from text vectors from different service scenes.

[0104] Fine-tuning the data can cover different service / consumption scenarios, such as family dinners, friend gatherings, late-night snacks during overtime work, and other preset scenarios, enabling the CLIP text encoder to more accurately represent the association between items and service scenarios in various situations. The semantic feature vector output by the fine-tuned CLIP text encoder can effectively distinguish different service scenarios, providing higher-quality semantic representations for subsequent clustering.

[0105] In this embodiment, the keyword set can be clustered based on the semantic feature vectors of each keyword in the cross-modal semantic space.

[0106] In practical applications, algorithms such as K-Means or DBSCAN can be used to cluster cross-modal semantic feature vectors to obtain clustering results (multiple clusters). The number of clusters can be set according to actual needs, such as based on empirical values ​​or the number of keywords in the current user group to be identified. For example, the number of clusters is positively correlated with the number of keywords. It can also be set according to actual analysis requirements. This embodiment does not limit this.

[0107] In this embodiment, the clusters obtained by clustering the keyword set using a clustering algorithm are not high-quality clusters. This embodiment further processes the clustering results to obtain the high-quality clusters described in this embodiment. The purpose is to solve the problems that may exist when conventional clustering algorithms are directly applied to user text data, such as inconsistent clustering quality, noise, or broad topic coverage.

[0108] The high-quality class clusters in this embodiment may have the following characteristics:

[0109] ①The semantic similarity of the semantic feature vectors of each keyword within a high-quality cluster meets the preset similarity conditions;

[0110] Here, it refers to the overall similarity between the semantic feature vectors of all keywords within a cluster in the vector space reaching a preset standard. This ensures that members within the cluster are highly related or homogeneous at the semantic level.

[0111] As an example, for a cluster containing m keywords, the semantic similarity (cosine similarity) of the semantic feature vectors between each pair of keywords within the cluster is calculated. There are a total of m(m-1) / 2 pairs. The sum of all calculated cosine similarities is then divided by the total number of vector pairs, m(m-1) / 2, to obtain the "average cosine similarity (average semantic similarity)" of all keywords in the cluster. This metric reflects the overall consistency of all members within the cluster in the semantic space. A higher average value indicates that the keywords within the cluster are semantically closer and the themes are more concentrated.

[0112] As an example, in other examples, it is also optional to require that the semantic similarity of the semantic feature vectors between any pair of keywords within a high-quality cluster meets a preset similarity condition; or it is also optional that the proportion of the number of vector pairs that meet the "semantic similarity of the semantic feature vectors between a pair of keywords meets the preset similarity condition" is higher than a preset threshold within a high-quality cluster.

[0113] ② The distance distribution between the semantic feature vectors of each keyword and the center of the cluster satisfies the preset settling condition.

[0114] This refers to the distance from the semantic feature vectors of all keywords within a cluster to the cluster center (such as the mean center), and the overall dispersion of their distribution is lower than a preset standard. This ensures that members within the cluster are closely distributed around a core theme in the vector space, without any outliers that significantly deviate from the core.

[0115] In practical applications, after obtaining clustering results (multiple clusters) using clustering algorithms, one or more high-quality clusters with the above characteristics can be obtained through various methods.

[0116] For example, based on the two conditions mentioned above, multiple clusters obtained from clustering can be judged or screened out, keywords within clusters can be filtered, or the number of clusters can be adjusted and re-clustered to obtain high-quality clusters.

[0117] As an example, evaluation steps can be performed, such as calculating the internal semantic similarity measure (e.g., mean cosine similarity) and the concentration measure of member distance distribution (e.g., statistics based on distance variance or quantiles) for each cluster generated from the initial clustering.

[0118] It can perform judgment and optimization steps, such as comparing the above-mentioned metrics with preset threshold conditions. Based on the comparison results, corresponding optimization operations are performed on clusters that do not meet the standards. For example:

[0119] For clusters with low semantic similarity, internal consistency can be improved by adjusting their membership composition (such as removing irrelevant keywords) or re-dividing cluster boundaries.

[0120] For clusters with high dispersion in distance distribution, the concentration of the distribution can be improved by identifying and removing vectors that are too far from the center (i.e., semantic outliers).

[0121] For a broad category cluster that has multiple subtopics, it can be subdivided into several more focused sub-clusters that can each satisfy the above conditions.

[0122] After the above evaluation and optimization process, the remaining clusters that meet the two preset conditions constitute the "high-quality clusters" in this embodiment.

[0123] In some examples, the preset similarity conditions may include: the vector similarity between the semantic feature vectors of each keyword within a cluster is higher than a preset similarity threshold.

[0124] The vector similarity in the "preset similarity conditions" of this embodiment is a quantitative indicator that measures the directional similarity between semantic feature vectors of different keywords within the same cluster. In vector space, the closer the directions of two vectors are, the higher their similarity value, indicating that the semantic content they represent is more similar. In practical applications, calculation methods include, but are not limited to, mean cosine similarity.

[0125] In practical applications, the similarity threshold can be set to a specific value as needed; this embodiment does not impose any limitations on this. Setting a condition of "higher than the preset similarity threshold" ensures that high-quality clusters possess a high degree of internal semantic consistency. For example, when the average similarity between all member (keyword) vectors within a cluster reaches or exceeds this threshold, it means that the user needs expressed by these keywords are highly related or homogeneous at the cross-modal semantic level (especially in related scenarios). This effectively avoids forcibly aggregating keywords with weak semantic relevance, thereby ensuring that each cluster corresponds to a clear and pure semantic theme.

[0126] As an example, each cluster can be automatically labeled as a "strongly related cluster" (e.g., semantic similarity > 0.8), a "mediumly related cluster" (e.g., semantic similarity between 0.7 and 0.8), or a "weakly related cluster" (e.g., semantic similarity < 0.7) based on semantic similarity.

[0127] In some examples, the preset settling conditions may include: the dispersion of the distance from the semantic feature vector of each keyword within a cluster to the center of that cluster is lower than a preset dispersion threshold.

[0128] In this embodiment, the dispersion of distance in the "preset concentration condition" is a statistical measure that assesses the distribution of distance values ​​from the cross-modal semantic feature vectors of all keywords within the same cluster to the cluster center (such as the mean vector). The lower the dispersion, the more tightly all vector points are clustered around the center; conversely, the higher the dispersion, the more dispersed the points are, and there may be points far from the center.

[0129] In practical applications, the dispersion threshold can be set to a specific value as needed; this embodiment does not impose any limitation on this. Setting the condition of "below the preset dispersion threshold" can ensure that high-quality clusters have good structural compactness. For example, satisfying this condition means that the cluster has no significant semantic outliers in the vector space (i.e., keywords that are clearly inconsistent with the core topic). A cluster with compact distance distribution has all its members closely revolving around a common semantic core.

[0130] In some examples, the keyword set is clustered based on the semantic feature vectors of each keyword in the keyword set. The resulting clusters are then processed to obtain one or more high-quality clusters, which may include:

[0131] Based on the semantic feature vectors of each keyword in the keyword set, the keyword set is clustered to obtain multiple clusters;

[0132] Multiple clusters are selected as the clusters to be optimized, and the following optimization operations are performed to obtain high-quality clusters:

[0133] Calculate the semantic similarity of each type of cluster and obtain strongly related clusters with semantic similarity higher than a preset high similarity threshold;

[0134] Calculate the distance distribution for strongly correlated clusters, identify and remove semantic outliers that do not meet the preset settling conditions based on the distance distribution, and obtain high-quality clusters.

[0135] like Figure 2B The diagram shown is a flowchart illustrating a user intent recognition method based on a multimodal model according to an exemplary embodiment of this specification. As mentioned in the previous embodiments, a data source, namely historical descriptive text, can be obtained for the user group to be identified. Then, a keyword set is determined, and semantic feature vectors are extracted through a text-image multimodal model and then clustered. Next, in this embodiment, a cluster quality assessment is designed for the clusters obtained by clustering, that is, firstly, semantic similarity is calculated for screening (screening out strongly related clusters), and then semantic outliers are removed through distance distribution to obtain high-quality clusters.

[0136] In this embodiment, initial clustering can be performed first. That is, based on the cross-modal semantic feature vectors of each keyword in the keyword set, a clustering algorithm is used to cluster the keyword set to obtain multiple initial clusters.

[0137] Optimization process: Taking the above initial clusters as the objects to be optimized, optimization operations are performed to ultimately filter or transform out the "high-quality clusters" that meet the conditions:

[0138] Step 1: Preliminary screening and identification based on semantic similarity;

[0139] Calculate the semantic similarity metric for each initial cluster (e.g., the average cosine similarity between all pairs of vectors within the cluster). Clusters with semantic similarity metrics higher than a preset high relevance threshold are identified as "strongly related clusters." These clusters already possess a high degree of semantic consistency.

[0140] In practical applications, the specific value of the high correlation threshold can be set according to actual needs, and this embodiment does not limit it.

[0141] Step 2: Outlier cleanup based on distance distribution;

[0142] For each identified "strongly correlated cluster," the dispersion measure of the distance distribution of its internal vectors is further calculated. Based on this distance distribution, semantic outliers that do not meet the "preset concentration criteria" can be identified and removed. After this purification step, the clusters can be free of noise points, making the theme of the cluster purer.

[0143] In other words, keywords that meet the "preset concentration conditions" can be selected from strongly related clusters, and the selected keywords constitute high-quality clusters.

[0144] In some examples, identifying and removing semantic outliers that do not meet preset settling criteria based on distance distribution may include:

[0145] Calculate the distance from the semantic feature vector of each keyword in a strongly correlated cluster to the center of that cluster;

[0146] The outlier detection threshold is determined based on the interquartile range of distance.

[0147] Keywords whose distance exceeds the outlier determination threshold are identified as semantic outliers and removed.

[0148] As an example, this embodiment can perform interquartile range (IQR) detection on each strongly correlated cluster. That is, in the vector space of each strongly correlated cluster, the distance from each point (keyword) to the cluster center is calculated. Then, based on these distance distributions, the interquartile range method is used to determine the threshold for representing anomalies (i.e., the outlier determination threshold). In practical applications, the specific outlier determination threshold is determined based on the interquartile range, and may include, for example, Q3 + 1.5 * IQR.

[0149] As an example, a distance dataset can be obtained, which includes distance data that refers to the distance from the semantic feature vector of each keyword within a high-quality cluster to the center of that cluster; then, the distances in the distance dataset can be sorted in order (e.g., from smallest to largest or from largest to smallest).

[0150] Taking the sorting as ascending as an example, based on the distances after sorting, find three key quantile points, for example:

[0151] Q1: The 25th percentile, also known as the lower quartile;

[0152] Q2: The 50th percentile, also known as the median;

[0153] Q3: The 75th percentile, also known as the upper quartile.

[0154] Based on this, the interquartile range (IQR) can be calculated, where IQR = Q3 - Q1, which is the range of the middle 50% of the data.

[0155] Next, the outlier detection threshold can be calculated, for example, upper bound = Q3 + 1.5 × IQR;

[0156] Ultimately, the logic for determining semantic outliers is that a point is considered a semantic outlier if its distance is greater than the upper bound (the outlier determination threshold). In practical applications, a cluster may or may not contain semantic outliers. Points within a cluster are essentially keywords within that cluster.

[0157] As an example, suppose the current cluster contains N keywords, and the distances (such as Euclidean distance) from the vectors of these N keywords to the center of the cluster have already been calculated. The distance dataset is represented as: [d1, d2, d3, ..., dN].

[0158] Q3 (third quartile / upper quartile) refers to the distance value at the 75th percentile when these N distance values ​​are sorted from smallest to largest. It indicates that 75% of the keywords in this cluster are less than or equal to Q3 in distance to the center. Q3 marks the "upper edge" of the main data distribution.

[0159] IQR (Interquartile Range) = Q3 - Q1. Where Q1 is the distance value at the 25th percentile (first quartile). IQR represents the distribution range of the middle 50% of the data (the most core and densest part). A larger IQR indicates that the data in the middle is more dispersed; a smaller IQR indicates that the data in the middle is more concentrated.

[0160] In this embodiment, the outlier determination threshold can be determined based on the interquartile range (IQR), for example, Q3 + 1.5 × IQR. Here, 1.5 × IQR can be seen as a tolerance extension for the normal fluctuation range of the data. Q3 + 1.5 * IQR together define a "reasonable upper bound on distance." This upper bound considers not only the endpoint of the main data (Q3) but also the dispersion of the data itself (IQR).

[0161] Therefore, if a keyword's distance from the center exceeds this "reasonable upper bound," there is sufficient statistical reason to believe that it has significantly deviated from the core of the cluster, and thus it can be identified as a "semantic outlier" (noise word).

[0162] In this embodiment, interquartile range (ICM) is innovatively applied to the semantic vector space of CLIP to identify and filter out noise words that do not match the clustering topic.

[0163] The strongly correlated clusters obtained after the above two optimization operations of "preliminary screening" and "outlier purification" simultaneously satisfy the "preset similarity condition" and "preset concentration condition", thus constituting the high-quality clusters mentioned above in this embodiment.

[0164] In some examples, optimization operations may also include:

[0165] Obtain moderately related clusters whose semantic similarity falls between a preset high similarity threshold and a preset low similarity threshold;

[0166] The moderately related clusters are selected as the optimized clusters, and the optimization operation is performed again to obtain high-quality clusters; wherein, the preset high similarity threshold is higher than the preset low similarity threshold.

[0167] like Figure 2B As shown, this embodiment also incorporates a hierarchical refinement process for moderately relevant clusters, further subdividing them into high-quality clusters. In practical applications, the preset low similarity threshold can be set according to actual needs; this embodiment does not impose any limitations on this. In practical applications, clusters with semantic similarity below the preset low similarity threshold, i.e., low-relevance clusters, can be removed.

[0168] Therefore, this embodiment can identify clusters with moderate semantic similarity (such as semantic coherence indicators). Moderate semantic similarity indicates that the cross-modal semantic feature vectors of keywords within the cluster show a clear directional clustering trend in the vector space, but the clustering density is insufficient. This usually means that these keywords collectively point to a larger semantic concept or scenario category, but within this category, there may be multiple more concrete sub-concepts that are slightly different from each other.

[0169] For example, in the context of instant delivery, the scenario of users "liking spicy food" is quite broad and could be further broken down into sub-themes such as "Sichuan cuisine social" and "Hunan cuisine family".

[0170] For example, a moderately relevant cluster might contain keywords such as "hot pot," "barbecue," "spicy stir-fry," and "buffet." In the CLIP cross-modal semantic space, they all point to the broad scenario of "social hot food gatherings among multiple people," and thus can be initially grouped together. However, upon closer examination, "hot pot" and "spicy stir-fry" might be closer to "Chinese-style group meals around a table," while "barbecue" and "buffet" might be closer to "relaxed self-service gatherings." This difference in sub-topics prevents the overall semantic similarity of the cluster from reaching a "highly relevant" level. Therefore, a moderately relevant cluster can be considered a "parent topic" or "topic domain" that needs further subdivision.

[0171] The aforementioned optimization operations can be recursively performed on the keywords within these moderately relevant clusters. That is, taking the clusters as the optimization objects, the operations of first clustering, identifying high-quality clusters, and removing semantic outliers from high-quality clusters are performed again, so that moderately relevant clusters can be split into sub-clusters with finer granularity and more focused semantics.

[0172] Therefore, further optimization of the granularity and operability of user intent recognition will make the results of user intent recognition more refined and accurate.

[0173] From a business value perspective, identifying a broad "spicy scenario" makes it difficult to achieve more precise service recommendations, such as failing to guide specific product selection or event planning. However, segmenting it into "Sichuan cuisine social" (which can be associated with beer and late-night scenarios) and "Hunan cuisine family" (which can be associated with home-cooked meals and lunch scenarios) can enhance the targeting of service recommendation strategies, thereby achieving refined operations.

[0174] From a technical implementation perspective, without further refinement, the moderately related clusters will be the final output. The ambiguity of their internal structure will increase the difficulty and ambiguity of subsequent automatic or manual interpretation, and may also obscure the differences in needs among different subgroups. Through recursive optimization, it is possible to further decompose the clusters into higher-quality clusters as much as possible.

[0175] In practical applications, the number of times the optimization operation is repeated can be set according to actual needs. "Repeating the optimization operation" is usually a recursive or iterative process. Its termination condition can be set according to the technical objectives and can include the following two categories; termination occurs when either condition is met:

[0176] Condition 1: Quality meets the standard; after performing a new round of "clustering-optimization" on a moderately related cluster, all newly generated sub-clusters can simultaneously satisfy the "preset similarity condition" and the "preset clustering condition", that is, they have all become independent "high-quality clusters". At this point, it is considered that the refinement of this branch has achieved the goal, and there is no need to continue splitting.

[0177] Condition 2: Reaching preset limits; to prevent infinite subdivision or overfitting, some limitations can be set; for example, a quantity limit, i.e., stopping further subdivision when the number of keywords within a cluster falls below a certain minimum scale threshold. A depth limit: setting a maximum number of recursions or iterations, stopping after reaching this number. Alternatively, a similarity lower limit can be included: when the semantic similarity metric of the cluster to be split is lower than an acceptable subdivision lower limit threshold (such as the preset low similarity threshold mentioned above), it indicates that there may be no stable structure within it, and subdivision can be stopped.

[0178] In practical applications, the termination condition can be a combination of one or more of the above conditions, and this embodiment does not limit this.

[0179] Through the above processing, the keyword set corresponding to the user group to be identified in this embodiment can be used to determine one or more high-quality clusters. Each high-quality cluster consists of multiple keywords, which represent the user's demand for services. Therefore, the user needs of the user group can be determined through high-quality clusters.

[0180] The technical approach for user intent recognition is based on the user's historical descriptions of the desired service. From these texts that directly reflect user needs, user intent is automatically and deeply identified. In the service purchase scenario of this embodiment, a text encoder using a multimodal graph-text model extracts semantic feature vectors of keywords. This allows the vector representation to contain joint semantic information of the text and associated visual concepts. This enables subsequent clustering processing to move beyond surface-level text similarity and instead operate within a unified semantic space that integrates visual scene information. For example, keywords such as "tent," "picnic mat," and "park" can be effectively clustered to accurately identify the deep-level scene intent of "outdoor picnic." This overcomes the limitation of pure text models in associating with real-world visual scenes, significantly improving the depth and accuracy of intent recognition.

[0181] Furthermore, this embodiment designs a step of "processing the various clusters obtained from clustering" to ensure that they meet the "semantic similarity satisfying the preset similarity condition" and the "distance distribution satisfying the preset concentration condition," which optimizes the initial clustering results. Specifically, the "semantic similarity condition" ensures that members within high-quality clusters are highly related and thematically consistent across modalities; the "concentration condition based on distance distribution" effectively identifies and eliminates semantic outliers (noise) by constraining the dispersion of the distance from cluster members to the cluster center. The combined effect of these two conditions ensures that each high-quality cluster corresponds to a pure, focused, and internally highly consistent service intent; ultimately, based on the keywords in the high-quality clusters, the user intent of a user group for the required service can be accurately determined.

[0182] As an example, in this embodiment, the user intent of a user group for the required service is determined based on each keyword in the high-quality cluster. This can include the user group's consumption intent when consuming the service; or the determined user intent can include: determining the user group's demand theme when consuming the service. This user intent can characterize the user group's scenario and / or consumption motivation when consuming the service. This user intent can be used for operational planning and other purposes.

[0183] As an example, determining the user intent of a user group for the desired service based on keywords within high-quality clusters can include:

[0184] Input the first prompt word containing keywords from the high-quality cluster into the preset language model, and obtain the user intent topic description corresponding to the high-quality cluster output by the language model based on the first prompt word.

[0185] For example, keywords (including a list of representative keywords) from each high-quality cluster can be used as part of a prompt word and input into a large language model (LLM). The LLM is required to generate a concise, accurate, and business-context-appropriate intent topic name for each high-quality cluster; for example, "Healthy Light Food Focus" or "Exploring Regional Flavors".

[0186] In practical applications, in addition to including keywords from high-quality clusters, the first prompt can also include other prompt information as needed; for example, it can include instructions that direct the LLM to perform a "user intent topic summarization" task, including but not limited to:

[0187] Task definition information, which can instruct the model to output one or a set of topic names or phrases that can summarize the core commonalities of keywords in high-quality clusters;

[0188] Quality requirements information, which may include requirements for the user intent theme of the output, such as "concise", "accurate", and "in line with business context";

[0189] Optional guidance information, such as example information, etc.

[0190] As an example, a user intent description can be output to relevant personnel, such as data analysts or operations staff, for them to perform subsequent processing:

[0191] For example, services related to user intent topic descriptions can be obtained based on those descriptions, and these related services can be recommended to users within the user group.

[0192] Alternatively, if there are multiple user groups to be identified, and each user group has one or more corresponding user intent topic descriptions, the comparison results containing the user intent topic descriptions of each user group can be output for data analysts to refer to.

[0193] In some examples, this embodiment can also combine knowledge from the knowledge base to support the LLM in accurately outputting user intent topic descriptions. As an example, the knowledge base of this embodiment may include a multimodal scenario knowledge graph; the knowledge graph may include a graph structure, and the nodes in the graph structure may include: scenario nodes representing service scenarios, keyword nodes representing keywords, and user behavior nodes. For example, user behavior nodes may include user behaviors such as click-through rate or conversion rate of users towards services (such as products); the edges in the graph structure include: edges representing the association between "service scenarios and keywords" (the weight of the edge can represent the association strength between keywords and service scenarios), and edges representing the association between "service scenarios and user behavior" (used to record historical statistical information of user behavior under the service scenario, such as average click-through rate or average conversion rate).

[0194] As an example, in the construction of a knowledge base, the data source can include pre-trained data. For instance, the semantic feature vectors of the "text describing the service scenario" and the semantic feature vectors of the "keywords" can be extracted using the aforementioned multimodal graph model, and the semantic similarity between these two feature vectors can be calculated as the association strength between the aforementioned keywords and the service scenario. Based on the user's historical order behavior, various service scenario tags can be extracted from user orders, search records, and comments. For example, NLP can be used to identify scenario keywords such as "late at night" and "friends," or user-defined service scenario tags can be used.

[0195] Thus, for a high-quality cluster that needs to identify the user intent topic description, relevant knowledge can be retrieved from the knowledge base for the keyword set of the current high-quality cluster. The relevant knowledge here may include scene nodes related to the keyword set of the current high-quality cluster and their associated behavioral data.

[0196] As an example, during recall, the semantic feature vectors of keywords in the current high-quality clusters can be obtained, and the semantic feature vectors of each service scenario in the knowledge graph can be stored in advance. In this way, a preset number of target scenario nodes that are most similar to the semantic feature vectors of keywords in the current high-quality clusters can be recalled from the knowledge graph. Furthermore, for the recalled target scenario nodes, relevant historical behavior data of each user behavior node associated with them can be selected through the edges in the graph, such as the historical statistical information of user behavior mentioned above.

[0197] Based on this, in addition to the set of keywords of the current high-quality cluster, the first prompt word in this embodiment may also include the recall data, namely the target scene node mentioned above and the relevant historical behavior data of each user behavior node associated with it, so that LLM can refer to the recall data and output an accurate user intent topic description.

[0198] As an example, the method may also include:

[0199] The keywords of each high-quality cluster containing the user group and the second prompt word describing the user intent topic are input into the language model to obtain the service recommendation strategy corresponding to the user intent topic description output by the language model based on the second prompt word.

[0200] This embodiment can also help service providers design service recommendation strategies, which can be implemented based on the aforementioned user intent topic descriptions. In practical applications, in addition to including keywords from high-quality clusters and user intent topic descriptions, the second prompt can also include other prompt information as needed, such as instructions on how the LLM should generate a service recommendation strategy. This instruction information includes, but is not limited to, domain knowledge related to the service recommendation strategy; as an example, it can include domain knowledge related to "users (consumers), services, and scenarios" in the instant retail or e-commerce fields, thereby providing the LLM with the boundaries and direction of the service recommendation strategy.

[0201] As an example, there can be multiple high-quality clusters for this user group. For each high-quality cluster, a corresponding service recommendation strategy can be obtained. That is, each high-quality cluster corresponds to a service scenario. Therefore, for each keyword and user intent topic description of each high-quality cluster, a corresponding service recommendation strategy can be obtained.

[0202] As an example, the second prompt can also prompt the LLM to generate one or more actionable service recommendation strategies for each theme based on the extracted user intent theme description and combined with the preset "people (users), goods (services), place (scenarios)" operation framework. The service recommendation strategies can represent preliminary operation strategy suggestions, and these strategies can be output to the operation personnel for reference, thereby improving the efficiency of the operation personnel in setting service recommendation strategies.

[0203] As can be seen from the above embodiments, the disadvantages overcome by this embodiment include, but are not limited to:

[0204] Overcoming the shortcomings of pure text clustering: Using CLIP instead of BERT enables clustering to leap from "textual similarity" to "cross-modal semantic similarity", which can better understand the real-world scenarios and visual concepts behind user needs and gain deeper insights.

[0205] Overcoming the shortcomings of standard CLIP applications: This embodiment does not passively use CLIP for classification or retrieval, but actively utilizes its vector space for unsupervised intent discovery; it solves the problems of noise and business orientation through TGI screening; it solves the problem of uncontrollable clustering quality through dynamic process optimization; and it achieves systematic analysis of the population through a multi-dimensional comparison framework. These are all features that standard CLIP applications lack.

[0206] The technical effects achievable by this embodiment include, but are not limited to:

[0207] Improved depth and accuracy of insights: Clustering results better align with human intuitive understanding of scenes. For example, it can effectively cluster words such as "picnic mat," "sandwich," and "park," revealing the deeper scene of "outdoor outing," which is difficult for pure text models to achieve.

[0208] Significantly improved efficiency: The entire process from raw data to interpretable insight reports and preliminary strategic recommendations is highly automated, reducing the manual analysis time that previously took days or even weeks to hours.

[0209] Operational strategies are more concrete: Based on cross-modal understanding, the generated strategy suggestions are more contextual and actionable. For example, beer pairings can be recommended for the "late-night crayfish" scenario, and service images and set meal designs can be optimized for the "quality solo dining" scenario.

[0210] Highly implementable: The output is not just data reports, but strategic suggestions directly linked to operational actions (people, goods, and place), which service providers can directly refer to and implement.

[0211] The foregoing embodiments describe the user intent of the user group to be identified for the required services, that is, to identify the services preferred by the user group to be identified; in some examples, it is also possible to further identify the user's values, and after solving what the user group "wants", it is possible to further solve why these users want them.

[0212] In some examples, determining the user intent of a user group for the desired service based on keywords within high-quality clusters can include:

[0213] Obtain profile data for each user in the user group. The profile data includes one or more of the following: user attribute data, user historical order behavior data, historical order scenario data, and in-application interaction behavior data.

[0214] Input the user profile data and the third prompt words of each keyword in each high-quality cluster into the language model, and obtain the user behavior label set output by the language model based on the preset sociological domain knowledge and the third prompt words;

[0215] Input the fourth prompt word, which contains user profile data, behavioral tag set, and preset value concept, into the language model to obtain the value concept corresponding to the user's service order behavior output by the language model based on the fourth prompt word.

[0216] As an example, user profile data can be pre-integrated from multiple heterogeneous sources to build user profile data. Sources of user profile data include, but are not limited to:

[0217] Historical order data, including but not limited to: product category, brand, amount, time, frequency, and remarks.

[0218] In-app interaction data, including but not limited to: browsing, searching, favorites, and click patterns within the app.

[0219] Historical order scenario data refers to the scenario information when a user places an order, including but not limited to: geographical location (home / office / campus), weather, holidays, etc.

[0220] User attribute data, including but not limited to: demographic characteristics, life cycle stage, value stratification, etc.

[0221] As an example, this embodiment designs behavior decoding based on pre-set sociological knowledge. This sociological knowledge can be prepared in advance, including but not limited to various social science theories such as symbolic interactionism, ritual theory, and Maslow's hierarchy of needs. This knowledge can be constructed as a knowledge base for a third language model. The third language model in this embodiment can be the same as or different from the aforementioned first and second language models; this embodiment does not impose any limitations on this.

[0222] As an example, in symbolic interactionism, a third language model can be used to map specific consumer behaviors into identity symbols. For instance, conversion rules can be defined, such as IF (orders contain "light meals" or "salads") AND (users' social media shares contain the "fitness" tag) THEN, which is decoded into the symbol: "self-discipline persona construction".

[0223] As an example, ritual theory can utilize a third-language model to identify periodic and combinatorial consumer behaviors. For instance, the rule definition: IF (order time is Friday night) AND (order item combination is "pizza" + "beer") THEN decoded as the symbol: "weekend indulgence ritual".

[0224] As an example, for Maslow's hierarchy of needs, a third language model can be used to map consumer behavior to different levels of needs. For example, order (goods = "takeout from a Black Pearl restaurant") -> level of need: "esteem need".

[0225] Based on this, a set of behavior tags containing one or more behavior tags can be automatically generated for each user.

[0226] Next, this embodiment designs a value inference based on a large language model; this embodiment can construct a fourth prompt word containing a user's behavioral label set and preset value concepts; and construct a structured prompt word Prompt for each user containing the following parts to drive LLM for deep reasoning:

[0227] Character description: "You are an artificial intelligence that combines the abilities of a sociologist, psychologist, and market insight expert."

[0228] User profile summary: Inject the first-stage user holographic profile summary.

[0229] Decoded set of behavioral symbols: Inject the list of behavioral symbols generated in the second stage.

[0230] Values ​​Reference Framework: Provides pre-set values ​​(such as "efficiency first", "quality appreciation", "family care") for LLM reference.

[0231] Output format requirements: Instruct the LLM to output a structured JSON containing fields such as "Core Idea Label" and "Idea Description".

[0232] After receiving the structured Prompt, LLM performs a complex comprehensive reasoning task, analyzing the relationships between behavioral symbols, combining its world knowledge, and referring to pre-set values, to ultimately infer the value that best matches the user profile and behavioral symbols.

[0233] In some examples, the method of this embodiment may further include:

[0234] Vectorize the user's values ​​to obtain a value feature vector;

[0235] Based on the value characteristic vector, users in the user group are clustered to obtain subgroups of the user group.

[0236] As an example, in this embodiment, each user can obtain the value concept output by the language model. This value concept belongs to the text and can be vectorized using BERT or CLIP to obtain the value concept feature vector. Then, clustering can be performed to cluster each user in the user group based on the value concept feature vector to obtain the subgroups of the user group.

[0237] Corresponding to the aforementioned embodiments of the user intent recognition method based on multimodal models, this specification also provides embodiments of a user intent recognition device based on multimodal models and the computer equipment on which it is applied.

[0238] The embodiments of the user intent recognition device based on the multimodal model described in this specification can be applied to computer devices, such as servers or terminal devices. The device embodiments can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by its processor reading the corresponding computer program instructions from non-volatile memory into memory and executing them. From a hardware perspective, such as... Figure 3 The diagram shown is a hardware structure diagram of a computer device containing the user intent recognition device based on a multimodal model, as described in this specification. (Except for...) Figure 3 In addition to the processor 310, memory 330, network interface 320, and non-volatile memory 340 shown, the electronic device in which the user intent recognition device 331 based on the multimodal model is located in the embodiment may also include other hardware depending on the actual function of the electronic device, which will not be described in detail here.

[0239] like Figure 4 As shown, Figure 4 This is a block diagram illustrating a user intent recognition device based on a multimodal model according to an exemplary embodiment of this specification. The device includes:

[0240] Keyword determination module 41 is used to: determine the keyword set of the user group based on the historical description text of the user in the user group to be identified regarding the required service;

[0241] The vector acquisition module 42 is used to: input the keywords in the keyword set into the text encoder in the preset image-text multimodal model, and obtain the semantic feature vector of the keywords output by the text encoder;

[0242] The processing module 43 is used to: perform clustering processing on the keyword set according to the semantic feature vector of each keyword in the keyword set, and process the clusters obtained by clustering to obtain one or more high-quality clusters; wherein, the semantic similarity of the semantic feature vectors of each keyword in the high-quality clusters meets the preset similarity conditions, and the distance distribution between the semantic feature vectors of each keyword and the center of the cluster meets the preset concentration conditions.

[0243] The intent determination module 44 is used to: determine the user intent of the user group for the required service based on each keyword in the high-quality cluster.

[0244] In some examples, the keyword identification module 41 identifies a set of keywords for the user group, including:

[0245] Obtain the user group's group preference for keywords in historical descriptive text, and obtain the preset global basic group's global preference for keywords;

[0246] Based on group preference and global preference, keywords that indicate differences in user group preferences relative to the global basic group are selected from each historical descriptive text.

[0247] In some examples, the keyword identification module 41 identifies a set of keywords for the user group, including:

[0248] Obtain the intra-group saliency index of keywords in each historical descriptive text of a user group, and based on the intra-group saliency index, filter out keywords from each historical descriptive text whose intra-group saliency index is higher than a preset saliency threshold.

[0249] In some examples, preset similarity conditions include: the vector similarity between the semantic feature vectors of keywords within a cluster is higher than a preset similarity threshold;

[0250] And / or,

[0251] The preset concentration conditions include: the dispersion of the distance from the semantic feature vector of each keyword in the cluster to the center of the cluster is lower than the preset dispersion threshold.

[0252] In some examples, processing module 43 performs clustering on the keyword set based on the semantic feature vectors of each keyword in the keyword set, and processes the resulting clusters to obtain one or more high-quality clusters, including:

[0253] Based on the semantic feature vectors of each keyword in the keyword set, the keyword set is clustered to obtain multiple clusters;

[0254] Multiple clusters are selected as the clusters to be optimized, and the following optimization operations are performed to obtain high-quality clusters:

[0255] Calculate the semantic similarity of each type of cluster and obtain strongly related clusters with semantic similarity higher than a preset high similarity threshold;

[0256] Calculate the distance distribution for strongly correlated clusters, identify and remove semantic outliers that do not meet the preset settling conditions based on the distance distribution, and obtain high-quality clusters.

[0257] In some examples, the optimization operations in processing module 43 also include:

[0258] Obtain moderately related clusters whose semantic similarity falls between a preset high similarity threshold and a preset low similarity threshold;

[0259] The moderately related clusters are selected as the optimized clusters, and the optimization operation is performed again to obtain high-quality clusters; wherein, the preset high similarity threshold is higher than the preset low similarity threshold.

[0260] In some examples, processing module 43 identifies and removes semantic outliers that do not meet preset settling conditions based on distance distribution, including:

[0261] Calculate the distance from the semantic feature vector of each keyword in a strongly correlated cluster to the center of that cluster;

[0262] The outlier detection threshold is determined based on the interquartile range of distance.

[0263] Keywords whose distance exceeds the outlier determination threshold are identified as semantic outliers and removed.

[0264] In some examples, historical descriptive text includes one or more of the following:

[0265] Historical search text of users for services obtained through the client;

[0266] Historical notes on service orders obtained from the client;

[0267] Historical comment texts of users on service orders obtained through the client.

[0268] In some examples, the intent determination module 44 determines the user intent of a user group for the desired service based on keywords in the high-quality clusters, including:

[0269] Input the first prompt word containing keywords from the high-quality cluster into the language model, and obtain the user intent topic description corresponding to the high-quality cluster output by the language model based on the first prompt word.

[0270] In some examples, the device also includes a recommendation module for:

[0271] Input the second cue word, which contains the user intent topic description of the user group, into the language model, and obtain the service recommendation strategy corresponding to the user intent topic description output by the language model based on the second cue word.

[0272] In some examples, the intent determination module 44 determines the user intent of a user group for the desired service based on keywords in the high-quality clusters, including:

[0273] Obtain profile data for each user in the user group. The profile data includes one or more of the following: user attribute data, user historical order behavior data, historical order scenario data, and in-application interaction behavior data.

[0274] Input the user profile data and the third prompt words of each keyword in each high-quality cluster into the language model, and obtain the user behavior label set output by the language model based on the preset sociological domain knowledge and the third prompt words;

[0275] Input the fourth prompt word, which contains user profile data, behavioral tag set, and preset value concept, into the language model to obtain the value concept of the user's service order behavior output by the language model based on the fourth prompt word.

[0276] In some examples, the device also includes a clustering module for:

[0277] Vectorize the user's values ​​to obtain a value feature vector;

[0278] Based on the value characteristic vector, users in the user group are clustered to obtain subgroups of the user group.

[0279] The implementation process of the functions and roles of each module in the above-mentioned user intent recognition device based on the multimodal model is detailed in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0280] Accordingly, embodiments of this specification also provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the aforementioned user intent recognition method embodiment based on a multimodal model.

[0281] Accordingly, embodiments of this specification also provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of an embodiment of a user intent recognition method based on a multimodal model.

[0282] Accordingly, embodiments of this specification also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of an embodiment of a user intent recognition method based on a multimodal model.

[0283] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of the solution in this specification according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0284] The above embodiments can be applied to one or more computer devices. The computer device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. The hardware of the computer device includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0285] The computer device can be any electronic product that can interact with the user, such as a personal computer, tablet computer, smartphone, personal digital assistant (PDA), game console, interactive network television (IPTV), smart wearable device, etc.

[0286] The computer equipment may also include network equipment and / or user equipment. The network equipment includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0287] The network in which the computer device is located includes, but is not limited to, the Internet, wide area network, metropolitan area network, local area network, and virtual private network (VPN).

[0288] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0289] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this application.

[0290] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0291] The terms "specific example" or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with the embodiments or examples, which are included in at least one embodiment or example of this specification. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0292] Other embodiments of this specification will readily occur to those skilled in the art upon consideration of the specification and practice of the invention claimed herein. This specification is intended to cover any variations, uses, or adaptations that follow the general principles of this specification and include common knowledge or customary techniques in the art not claimed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this specification are indicated by the following claims.

[0293] It should be understood that this specification is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this specification is limited only by the appended claims.

[0294] The above description is merely a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of protection of this specification.

Claims

1. A user intent recognition method based on a multimodal model, characterized in that, The method includes: Based on the historical description text of the user group to be identified regarding the required service, a keyword set for the user group is determined; wherein, the keyword set includes keywords selected from each of the historical description texts that indicate a preference difference between the user group and a preset global base group, and / or keywords whose intra-group significance index is higher than a preset significance threshold. The keywords in the keyword set are input into the text encoder of the preset image-text multimodal model to obtain the semantic feature vector of the keyword output by the text encoder; wherein, the image-text multimodal model is trained by contrastive learning based on massive image-text pairs, the semantic feature vector output by the text encoder is aligned with the visual features of the corresponding image in the semantic space, and the semantic feature vector contains visually related scene information; Based on the semantic feature vectors of each keyword in the keyword set, the keyword set is clustered, and the clusters are processed to obtain one or more high-quality clusters; wherein, the semantic similarity of the semantic feature vectors of each keyword in the high-quality clusters satisfies a preset similarity condition, and the distance distribution between the semantic feature vectors of each keyword and the center of the cluster satisfies a preset concentration condition. Based on the keywords in the high-quality clusters, determine the contextualized user intent of the user group for the required services; Based on the semantic feature vectors of each keyword in the keyword set, the keyword set is clustered, and the resulting clusters are further processed to obtain one or more high-quality clusters, including: Based on the semantic feature vectors of each keyword in the keyword set, the keyword set is clustered to obtain multiple clusters; The aforementioned clusters are selected as the clusters to be optimized, and the following optimization operations are performed: Calculate the semantic similarity of each type of cluster, and obtain strongly related clusters whose semantic similarity is higher than a preset high similarity threshold; Calculate the distance distribution for the strongly correlated clusters, identify and remove semantic outliers that do not meet the preset settling conditions based on the distance distribution, and obtain the high-quality clusters; In addition, the algorithm obtains moderately related clusters whose semantic similarity is between the preset high similarity threshold and the preset low similarity threshold; uses the moderately related clusters as the optimized clusters, and performs the optimization operation again to obtain the high-quality clusters; wherein the preset high similarity threshold is higher than the preset low similarity threshold.

2. The method according to claim 1, characterized in that, The preference differences of the user group relative to the global base group are determined through the following operations, including: Obtain the user group's group preference for keywords in the historical description text, and obtain the preset global basic group's global preference for the keywords; Based on the group preference degree and the global preference degree, the preference difference of the user group relative to the global base group is determined.

3. The method according to claim 1, characterized in that, The preset similarity conditions include: the vector similarity between the semantic feature vectors of each keyword within a cluster is higher than a preset similarity threshold; And / or, The preset settling conditions include: the dispersion of the distance from the semantic feature vector of each keyword in the cluster to the center of the cluster is lower than a preset dispersion threshold.

4. The method according to claim 1, characterized in that, The step of identifying and removing semantic outliers that do not meet the preset settling conditions based on the distance distribution includes: Calculate the distance from the semantic feature vector of each keyword in the strongly related cluster to the center of the cluster; The outlier determination threshold is determined based on the interquartile range of the distance. Keywords whose distance exceeds the outlier determination threshold are identified as semantic outliers and removed.

5. The method according to claim 1, characterized in that, The historical description text includes one or more of the following: historical search text of the user for the service obtained through the client, historical remarks text of the user for the service order obtained through the client, and historical comment text of the user for the service order obtained through the client.

6. The method according to claim 1, characterized in that, The step of determining the user intent of the user group for the required service based on each keyword in the high-quality cluster includes: The first prompt word containing keywords from the high-quality cluster is input into the language model, and the user intent topic description corresponding to the high-quality cluster is obtained by the language model based on the first prompt word.

7. The method according to claim 6, characterized in that, The method further includes: The keywords of each high-quality cluster containing the user group and the second prompt word describing the user intent topic are input into the language model to obtain the service recommendation strategy corresponding to the user intent topic description output by the language model based on the second prompt word.

Images