A visualized reply content selection method based on expression and style decoupling

Abstract

The application discloses a kind of visual reply content selection methods based on expression and style decoupling, it is related to artificial intelligence technical field, including obtaining current multi-round conversation history and multiple candidate visual replies, respectively generating dialogue context representation and the basic visual semantic representation of candidate visual reply;The global matching score between each candidate visual reply and current conversation is calculated;The basic visual semantic representation of candidate visual reply is respectively mapped into expression feature component and style feature component, and the dialogue context representation is mapped into expression query vector and style query vector;Expression matching score is calculated;Style prototype representation is generated, and style prototype score is calculated;Global matching score, expression matching score and style prototype score are fused, to obtain the final score of candidate visual reply, and sorting is carried out.The application can improve the accuracy, naturalness and rationality of emoticon reply selection, and has good engineering compatibility.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a visual response content selection method based on the decoupling of expression and style. Background Technology

[0002] With the development of instant messaging platforms and social media, visual content such as emoticons, stickers, emojis, and internet memes has become an important form of expression in online communication. Compared with ordinary text replies, emoticons can usually express emotions, attitudes, and social tones in a more intuitive, exaggerated, and stylized way. In real chat scenarios, users send emoticons not only to supplement the meaning of text, but also often to soften the tone, express joking, convey intimacy, strengthen the emotional atmosphere, or achieve specific interactive goals. Therefore, emoticon reply selection is not simply an image retrieval problem, but a technical problem that simultaneously involves dialogue understanding, emotion perception, visual semantic understanding, and multimodal matching.

[0003] Existing emoji reply selection technologies typically use multi-turn dialogue context as input and select a suitable emoji from a set of candidate visual replies. Most existing solutions follow a technical approach of "dialogue context encoding, candidate visual reply encoding, cross-modal matching, and candidate ranking," and further incorporate additional information such as user preferences, emotion perception, or interaction intent. These solutions can, to some extent, select emojis that are semantically relevant to the current dialogue, emotionally appropriate, or consistent with the user's historical preferences. Existing emoji reply selection methods usually treat the matching between the dialogue context and candidate visual replies as an overall relevance matching problem, primarily judging whether the candidate visual replies generally match the semantics, emotion, or interaction intent of the current multi-turn dialogue. While this can filter out candidate visual replies that are semantically relevant or emotionally consistent to a certain extent, in real-world instant messaging scenarios, multiple candidate visual replies often have some degree of relevance, making it difficult to further determine which candidate visual reply is more suitable as the current reply based solely on overall semantic matching.

[0004] When candidate visual responses are modeled as a unified whole, it is difficult to explicitly distinguish the specific content and stable style attributes of emojis. Therefore, when faced with multiple candidates that are semantically reasonable but have different expressive details or style adaptability, inaccurate selection is likely to occur.

[0005] Therefore, the core technical problem this invention aims to solve is: in the task of selecting emoji replies, how to break through the candidate matching method based solely on overall semantic relevance and establish a structured emoji selection method that can simultaneously model the semantic relevance of the dialogue, the individual expressive differences of emojis, and the style group adaptability, so as to accurately select emojis that better fit the current dialogue context, expressive intent, and social style from multiple candidate visual replies with similar semantics or emotions. Specifically, this invention intends to solve the following technical problems:

[0006] First, existing methods typically model candidate visual responses as indivisible wholes, making it difficult to distinguish between the two distinct factors of "what is expressed" and "how it is expressed" in an emoji. For multiple semantically acceptable candidate visual responses, the model struggles to further determine whether their expressive intensity, attitudinal bias, emotional details, or visual presentation are truly suitable for the current conversation.

[0007] Second, emojis do not exist in isolation; many emojis naturally belong to a particular emoji series, visual theme, character type, or art style category. Emojis within the same style group typically share similar visual forms, interactive tones, and usage scenarios. Existing methods lack explicit modeling of these group-level style patterns, making it difficult for models to further determine whether the style type of a candidate visual response is suitable for the current social context of the conversation, even if they can determine that a candidate visual response is semantically relevant.

[0008] Third, in many real-world datasets, emojis lack manually labeled style categories, style tags, or clear style grouping information, making it difficult for models to directly perform style prior modeling. Relying entirely on manual annotation to construct style categories is not only costly but also hinders the migration of the method to new emoji libraries or application platforms. Summary of the Invention

[0009] Given that existing visual response content selection methods rely excessively on overall semantic matching, lack distinguishing modeling of expressive and stylistic features, and struggle to utilize style group priors, this invention provides a visual response content selection method based on expression and style decoupling. Addressing the issues of existing emoji response selection methods relying excessively on overall semantic matching, lacking distinguishing modeling of emoji expressive and stylistic features, and struggling to utilize style group priors, this invention provides a structured emoji response selection method based on the decoupling of expressive and stylistic features and style prototype priors. This improves the model's accuracy, contextual adaptability, and expressive rationality in scenarios with multiple candidates, similar semantics, and significant stylistic differences.

[0010] Therefore, the problem to be solved by this invention is how to provide a visual response content selection method based on the decoupling of expression and style.

[0011] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0012] In a first aspect, the present invention provides a method for selecting visual response content based on expression and style decoupling, which includes: obtaining the current multi-turn dialogue history and multiple candidate visual responses, and generating dialogue context representation and basic visual semantic representation of candidate visual responses respectively;

[0013] Based on the dialogue context representation and the basic visual semantic representation of the candidate visual responses, calculate the global matching score between each candidate visual response and the current dialogue;

[0014] The basic visual semantic representation of the candidate visual response is mapped to the expression feature component and style feature component, respectively, while the dialogue context representation is mapped to the expression query vector and style query vector;

[0015] The expression matching score is calculated by using the expression query vector and the expression feature components of the candidate visualization response to measure whether the specific expression content of the candidate visualization response meets the current dialogue requirements.

[0016] Style prototype representations are generated using style feature components and pre-constructed style group membership relationships, and style prototype scores of candidate visualization responses are calculated based on style query vectors and style prototype representations.

[0017] The global matching score, expression matching score, and style prototype score are combined to obtain the final score for each candidate visual response. The candidate visual responses are then sorted according to the final score, and the candidate visual response with the highest score is output as the target response emoji, or the top-ranked candidate visual responses are output.

[0018] As a preferred embodiment of the visual response content selection method based on expression and style decoupling described in this invention, wherein: the basic visual semantic representation for generating the dialogue context representation and candidate visual responses includes:

[0019] The dialogue context representation is obtained by processing the multi-turn dialogue history through a dialogue encoding function.

[0020] Multiple candidate visual responses are processed using an emoji encoding function to obtain the basic visual semantic representation of the candidate visual responses:

[0021] The dialogue encoding function includes a text encoder, a multimodal matching backbone network, and other context modeling networks; the emoji encoding function includes a visual Transformer, a convolutional neural network, and other image encoding networks.

[0022] As a preferred embodiment of the visual response content selection method based on expression and style decoupling described in this invention, wherein: calculating the global matching score between each candidate visual response and the current dialogue includes:

[0023] The dialogue context representation of the multi-turn dialogue history and the basic visual semantic representation of the candidate visual responses are input into the multimodal matching backbone network. Through cross-modal interaction, feature fusion and matching scoring, the global matching score corresponding to the candidate visual responses is obtained.

[0024] The multimodal matching backbone network includes multimodal BERT, cross-attention network, dual-tower matching network, and other multimodal fusion networks.

[0025] As a preferred embodiment of the visual response content selection method based on expression and style decoupling described in this invention, the step of generating a style prototype representation using style feature components and pre-constructed style group membership relationships includes:

[0026] Based on the visual style features of emoji images, weakly structured style cues are extracted using an offline image analysis model, and style group membership relationships are constructed based on these cues. There are two methods for obtaining styles: one is the offline image analysis model, and the other is to determine the initial style groups based on existing emoji series, character categories, visual themes, or sets in the emoji data, and then merge or re-cluster the initial style groups based on the similarity of visual features between groups.

[0027] For each style group, the style feature components corresponding to the members within the style group are aggregated to obtain the style prototype vector representation of the style group. The style prototype vector is used to reflect the shared visual style and stable social tone of the emoticons within the same style group.

[0028] As a preferred embodiment of the visual response content selection method based on expression and style decoupling described in this invention, the step of calculating the style prototype score of candidate visual responses based on style query vectors and style prototype representations includes:

[0029] By matching the style query vector corresponding to the current dialogue with the style prototype representation, the degree of fit between the current dialogue and different style prototypes is obtained. The style prototype score is represented as:

[0030]

[0031] in, Indicates the first Each candidate visual response is relative to the current conversation. The style prototype score, Indicates the first The vector representation of the style prototype to which each candidate visual response belongs. This represents the style query vector corresponding to the current dialogue. This represents the similarity function.

[0032] As a preferred embodiment of the visual response content selection method based on expression and style decoupling described in this invention, the step of sorting multiple candidate visual responses according to the final score includes:

[0033] Weighting coefficients are assigned to the expression matching score and style prototype score respectively, and the final score of the candidate visualization response is obtained by weighted summation, as follows:

[0034]

[0035] in, Indicates the first The final score of each candidate visual response. Indicates the first Candidate visual responses and the current conversation Global matching score between them Indicates the first Each candidate visual response is relative to the current conversation. The expression matching score, Indicates the first Each candidate visual response is relative to the current conversation. The style prototype score, and These represent the weighting coefficients for the expression matching score and the style prototype score, respectively.

[0036] The candidate visual responses are sorted according to the final score, and the candidate visual response with the highest score is selected as the current response, or several top-ranked candidates are output for the user to choose from.

[0037] As a preferred embodiment of the visual response content selection method based on expression and style decoupling described in this invention, the method further includes: training the model based on the output real response emoji packs, using the real response emoji packs as positive samples, and constructing negative samples. The negative samples include negative samples from the same group that belong to the same style prototype as the real response emoji packs but are not correct responses, and negative samples from different groups that belong to different style prototypes than the real response emoji packs.

[0038] Candidate score vectors are constructed based on the final scores of the three types of samples. Candidate matching loss is calculated based on the final scores of positive samples, negative samples in the same group, and negative samples across groups. Prototype supervision loss is calculated based on the adaptation scores of the current dialogue and all style prototypes. Expression ranking loss is calculated based on the expression matching scores of positive samples and negative samples in the same group. Orthogonal constraint loss is calculated based on the normalized inner product of expression feature components and style feature components.

[0039] The overall training objective of the model is constructed by combining candidate matching loss, prototype supervision loss, expression ranking loss, and orthogonality constraint loss.

[0040] After the model is trained, the input multi-turn dialogue history and candidate visual responses are encoded, matched, fused in score, and sorted for output.

[0041] In a second aspect, the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement a step of a visual response content selection method based on expression and style decoupling.

[0042] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements the steps of a visual response content selection method based on expression and style decoupling.

[0043] The beneficial effects of this invention are as follows: The emoji selection of this invention no longer relies solely on the overall matching score between candidate visual responses and the dialogue context, but further introduces expressive features, style features, and style prototype adaptation relationships. By dividing the candidate visual response representation into expressive feature components and style feature components, the model can understand "what" and "in what style" the candidate visual response "expresses," thereby improving its ability to distinguish subtle differences between semantically similar candidates.

[0044] This invention provides a better solution to the selection problem when multiple candidate visual responses are all somewhat relevant. In real-world instant messaging scenarios, it's often not just one candidate visual response that's relevant to the conversation, but multiple candidates that can express similar emotions or attitudes. By using expression matching scores to determine whether the specific content of the candidate visual responses meets the needs of the current conversation, the final selection result is not only semantically relevant but also more accurate in expressing its intended purpose.

[0045] This invention also leverages style grouping patterns among emojis to improve the naturalness and social relevance of responses. Actual emojis typically form style groups based on character imagery, visual themes, art style, or emoji series. Emojis within the same style group often share similar usage contexts and social tones. This invention extends the matching of individual candidates to style group-level adaptation judgment by constructing a style prototype library. This allows the model to determine whether the style of a candidate is suitable for the current conversational atmosphere, thereby reducing situations where "semantically related but stylistically jarring" or "emotionally similar but with unnatural tone."

[0046] This invention also demonstrates greater applicability in application scenarios lacking manual style labeling. Many emoji datasets do not contain explicit emoji series, style categories, or manual grouping information. Relying entirely on manual annotation to construct style tags increases data processing costs and hinders the method's transfer to new emoji libraries. This invention can construct weakly structured style cues based on the emoji image itself, such as the main object, object category, character identity, visual theme, or style description, and thereby form pseudo-style tags, style groups, or style prototypes. Therefore, this invention reduces reliance on manual style annotation and improves the method's transferability across different datasets, platforms, and emoji libraries.

[0047] The final ranking result of this invention is also more comprehensive. The global matching score ensures that the candidate visual response has a basic relevance to the current dialogue content; the expression matching score ensures that the specific expression function of the candidate meets the needs of the current dialogue; and the style prototype score ensures that the style group to which the candidate belongs is suitable for the tone of the current dialogue. The ranking criteria of this invention are more refined, simultaneously considering semantic relevance, expression suitability, and style suitability, thereby improving the accuracy, naturalness, and rationality of emoji response selection.

[0048] Furthermore, this invention can be extended on the existing emoji reply selection framework without completely rebuilding the entire system. Its core improvements lie primarily in the structured decomposition of candidate visual reply representations, the construction of a style prototype library, and the fusion and ranking of multiple score categories. Therefore, it can be used in conjunction with existing dialogue encoding modules, candidate image encoding modules, and basic matching modules. This gives the invention good engineering compatibility and deployment feasibility, making it suitable for both large-scale candidate ranking on the server side and reordering scenarios involving users' locally saved emojis. Attached Figure Description

[0049] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 A flowchart for a visual response content selection method based on the decoupling of expression and style. Detailed Implementation

[0051] To make the above-mentioned objects, features, and advantages of the present invention more readily understood, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0052] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0053] Secondly, the term "an embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single embodiment or an embodiment selectively excluded from other embodiments.

[0054] Reference Figure 1 This is the first embodiment of the present invention, which provides a visual response content selection method based on the decoupling of expression and style, including:

[0055] Step 100, Input Acquisition and Basic Encoding: Acquire the dialogue history and candidate visual responses, and generate dialogue context representation and basic visual semantic representation of candidate visual responses;

[0056] Step 200: Global Matching Calculation: Calculate the global matching score based on the dialogue context representation and the basic visual semantic representation of the candidate visual responses;

[0057] Step 300, Decoupling of Expression and Style: Map the basic visual semantic representation of the candidate visual response to expression feature components and style feature components, and at the same time map the dialogue context representation to expression query vector and style query vector;

[0058] Step 400: Expression matching calculation: Calculate the expression matching score based on the expression query vector and expression feature components;

[0059] Step 500: Prototype-level style adaptation calculation: Generate a style prototype using style feature components and style group membership relationships, and calculate the style prototype score;

[0060] Step 600, Score Fusion and Ranking Output: Fusion of global matching score, expression matching score and style prototype score, outputting the candidate visual response ranking results;

[0061] Step 700, Model Training: Jointly train the model using four types of loss.

[0062] Specifically, let the current multi-round dialogue history be... , No. The candidate visual responses are Candidate Visual Response Set Represented as:

[0063]

[0064] in, Indicates the first Candidate visual responses This indicates the number of candidate visual responses.

[0065] The dialogue encoding module and the emoji encoding module respectively obtain the basic visual semantic representation of the dialogue context and the candidate visual responses:

[0066]

[0067] in, This represents the contextual representation of the current multi-turn dialogue history. Indicates the first The basic visual semantic representation of each candidate visual response; For dialogue encoding functions, This function encodes emojis.

[0068] Dialogue encoding functions can be implemented by text encoders, multimodal matching backbone networks, or other context modeling networks; emoji encoding functions can be implemented by visual Transformers, convolutional neural networks, or other image encoding networks.

[0069] The dialogue context representation and the basic visual semantic representation of the candidate visual responses output in this step serve as inputs for both global matching calculations and subsequent decoupling of expression and style.

[0070] After obtaining the dialogue context representation and the basic visual semantic representation of the candidate visual responses, the overall relevance between each candidate visual response and the current dialogue is calculated.

[0071] Specifically, the dialogue context representation of the multi-turn dialogue history and the basic visual semantic representation of the candidate visual response are input into the multimodal matching backbone network. Through cross-modal interaction, feature fusion, and matching scoring, the global matching score corresponding to the candidate visual response is obtained, represented as:

[0072]

[0073] in, Indicates the first Candidate visual responses and the current conversation Global matching score between them This represents a multimodal matching function.

[0074] In one specific implementation, the multimodal matching backbone network adopts multimodal BERT, which combines dialogue text features and candidate visual response visual features into a joint input sequence, obtains a fused representation through multi-layer cross-modal interaction, and then uses a scoring layer to output a global matching score. Alternatively, a cross-attention network, a dual-tower matching network, or other multimodal fusion networks can be used to calculate this score.

[0075] The multimodal BERT backbone network and visual Transformer are one specific implementation of this invention. In practical applications, they can also be replaced with other multimodal matching networks and image coding networks.

[0076] The global matching score is primarily used to measure whether the candidate visual responses are generally relevant to the semantic content, emotional state, and response needs of the current multi-turn dialogue. This score is saved in this step and used in subsequent score fusion steps, together with the expression matching score and style prototype score, to calculate the final ranking score.

[0077] After obtaining the basic visual semantic representation of the candidate visual response, instead of directly using it as a single overall representation in all subsequent calculations, we utilize two different feature mapping branches to divide the candidate visual response representation into expressive feature components and style feature components, as follows:

[0078]

[0079]

[0080] in, Indicates the first The expressive feature components of each candidate visual response. Indicates the first Style feature components of each candidate visual response; To express the feature mapping function, These are style feature mapping functions, which can be implemented by linear mapping layers, multilayer perceptrons, or other neural network mapping modules.

[0081] A multi-layer perceptron, or MLP for short, is a nonlinear mapping module composed of multiple fully connected neural networks.

[0082] Expressive features are primarily used to characterize the specific content expressed by candidate visual responses in the current interaction, such as emotions, attitudes, or interactive functions like agreement, teasing, comfort, surprise, speechlessness, rejection, embarrassment, anger, and easing tension. Stylistic features are primarily used to characterize the relatively stable visual style and series attributes of candidate visual responses, such as character image, art style, visual theme, degree of exaggeration, humor, emoji series, or style groups.

[0083] Through the above division, the model can distinguish and model the two factors of "what the emoji expresses" and "what style the emoji expresses".

[0084] To map the dialogue context to expressive and stylistic information respectively, the dialogue context representation is further mapped to expressive query vectors and style query vectors, as follows:

[0085]

[0086]

[0087] in, This represents the expression query vector required for the current dialogue, used to measure whether the specific expression content of the candidate visual response meets the needs of the current dialogue; This represents the style query vector corresponding to the current dialogue, which is used for subsequent style prototype adaptation calculations. and It can be implemented using a linear mapping layer, a multilayer perceptron, or an attention mapping module.

[0088] An expression matching score is calculated using the expression query vector and the expression feature components of the candidate visualization response to measure whether the specific expression content of the candidate visualization response meets the current dialogue requirements. The expression matching score can be expressed as:

[0089]

[0090] in, Indicates the first Each candidate visual response is relative to the current conversation. The expression matching score, This represents the similarity function, which can be expressed as cosine similarity, as follows:

[0091]

[0092] in, This represents the L2 norm of the vector. The higher the score, the more the specific expressive function of the candidate visualization response fits the current dialogue context.

[0093] In other implementations, dot product similarity, bilinear matching functions, or learnable neural network scoring functions can also be used to calculate expression matching scores.

[0094] After obtaining the style feature components and style query vector, a style prototype is constructed by combining the style group membership relationship, and the style prototype score corresponding to the candidate visualization response is calculated.

[0095] The style prototype library includes relatively static style structure information and dynamically generated style prototype representations based on the model's current features. Static structure information may include style ID, member ID, neighbor ID, and the affiliation relationship between candidate visual responses and style groups. Style prototype representations are formed by aggregating the style feature components of emojis within the same style group.

[0096] For emoji data with clear grouping information, style groups can be constructed based on existing emoji series, character images, visual themes, emoji sets, or other grouping information on the platform. If the original grouping is too granular, or there is a high degree of visual similarity between different groups, the average visual features of each group can be calculated first, and then new style groups can be obtained through nearest neighbor retrieval, group merging, or clustering.

[0097] For data lacking human style labels, image analysis models can be used to perform offline analysis on individual emoji images to generate weakly structured style cues. These weakly structured style cues can include the main object, object category, visual style, and identity summary, forming pseudo-style tuples. , is represented as:

[0098]

[0099] in, Indicates the first The main object in an emoji. Indicates the object category, Indicates visual style, It represents an identity summary, role description, or stable visual identity.

[0100] Weakly structured style cues are primarily used to build style groups and style prototype libraries, rather than directly replacing the model's final matching judgment of candidate visual responses.

[0101] In one specific implementation, style grouping relationships can be constructed based on pseudo-style tuples. For example, fine-grained style groups can be formed first based on the combination of "main object, object category, visual style"; if the number of members in the fine-grained style group is insufficient, it can revert to the combination of "object category, visual style" to form a coarse-grained style group; if the coarse-grained style group is still insufficient, the emoji can be treated as a separate style group, or it can be merged into the closest style group based on the proximity of visual features. Through this fine-to-coarse grouping method, even in scenarios lacking manual style labels, a weakly supervised structure that can be used for prototype-level style modeling can be constructed from the emoji image itself.

[0102] Let the first The members of each style group are... The style feature components corresponding to each member's emoji pack are: . No. A style prototype representation can be obtained by aggregating the style feature components of the group members:

[0103]

[0104] in, Indicates the first Vector representation of a style prototype The aggregation function can be represented by average pooling, weighted average, attention aggregation, or other feature aggregation methods.

[0105] In this way, style prototypes can reflect the shared visual style and stable social tone of emojis within the same style group.

[0106] During the candidate ranking phase, the style query vector corresponding to the current dialogue is matched with the style prototype representation to obtain the degree of fit between the current dialogue and different style prototypes. For the first... There are 10 candidate visual responses, if their corresponding style prototype number is 1. Then its style prototype score can be expressed as:

[0107]

[0108] in, Indicates the first Each candidate visual response is relative to the current conversation. The style prototype score, Indicates the first The vector representation of the style prototype to which each candidate visual response belongs. The similarity function can be represented by cosine similarity, dot product similarity, or other learnable matching functions.

[0109] This score measures whether the style group to which the candidate visual response belongs matches the current conversational context, interaction relationship, and social tone as a whole.

[0110] Following the aforementioned steps, for each candidate visualization response, we obtain: a global matching score, an expression matching score, and a style prototype score. These three scores are then fused to obtain the final score of the candidate visualization response, expressed as:

[0111]

[0112] in, Indicates the first The final score of each candidate visual response. and These represent the weighting coefficients for the expression matching score and the style prototype score, respectively.

[0113] In practice, in addition to weighted summation, a learnable fusion network can also be used to fuse the global matching score, expression matching score, and style prototype score.

[0114] The candidate visual responses are sorted according to the final score, and the candidate visual response with the highest score is selected as the current response, or several top-ranked candidates are output for the user to choose from.

[0115] During model training, real response emojis are used as positive samples, and negative samples are constructed to enhance the candidate discrimination ability.

[0116] Specifically, two types of negative samples can be constructed: one type consists of negative samples from the same group that belong to the same style prototype as the actual reply emoji but are not the correct reply; the other type consists of negative samples from different groups that belong to different style prototypes. Negative samples from the same group are used to enhance the model's ability to distinguish different expressions within the same style group, preventing the model from prioritizing incorrect candidates based solely on style similarity. Negative samples from different groups are used to enhance the model's ability to identify differences between different style groups, enabling the model to determine whether a particular style is generally suitable for the current dialogue.

[0117] Let the real reply emoji be The negative samples in the same group are Cross-group negative samples are Then, a candidate score vector is constructed based on the final scores of the three. :

[0118]

[0119] Candidate Matching Loss Represented as:

[0120]

[0121] in, Represents cross-entropy loss, This represents the label position of the genuine reply emoji in the candidate score vector. The candidate matching loss can be calculated based on the final scores of positive samples, negative samples within the same group, and negative samples across groups, so that the final comprehensive score of the genuine reply emoji is higher than that of the negative samples.

[0122] Prototype Supervision Loss This is used to enhance the model's ability to identify the correct style prototype, so that the model not only learns which individual emoji is more appropriate, but also learns which style prototype is more suitable for the current dialogue as a whole.

[0123] Calculate the fit score between the style query vector of the current dialogue and all style prototypes, and form a prototype score vector. :

[0124]

[0125] Let the style prototype number of the real reply emoji be . Then the prototype supervision loss Represented as:

[0126]

[0127] The expression ranking loss is used to constrain the true response emoji to score higher than the negative samples in the same group in expression matching, thereby improving the model's ability to distinguish expression differences between candidates of the same style, and is expressed as:

[0128]

[0129] in, This represents the loss for ranking. This indicates a genuine reply using emojis. This indicates negative samples that belong to the same style archetype as real reply emojis. The preset interval parameter is used. This loss parameter enables the model to further focus on whether the content of the expression truly matches the current dialogue among candidates with similar styles.

[0130] To avoid highly overlapping information learned by the expressive and style feature components, this invention introduces orthogonal constraint loss during training, ensuring that the expressive and style feature components of the same candidate visualization response maintain as much difference as possible. Assume a training batch contains... If there are 10 candidate visual responses, then the orthogonal constraint loss can be expressed as:

[0131]

[0132] in, This represents the orthogonal constraint loss. and They represent the first The loss function assigns expressive and stylistic features to each candidate visualization response. It reduces information aliasing between expressive and stylistic features by constraining their normalized inner product to be as close to zero as possible. This makes the expressive branch focus more on the specific content of the candidate visualization response, and the stylistic branch focus more on the stable visual style and group attributes of the candidate visualization response. In practical calculations, to avoid a zero denominator, a very small constant can be added to the norm term.

[0133] Combining candidate matching loss, prototype supervision loss, representation ranking loss, and orthogonality constraint loss, the overall training objective of this invention is... Represented as:

[0134]

[0135] in, , and These are the weighting coefficients for the corresponding loss terms.

[0136] Through the joint training described above, the model can simultaneously learn the overall relevance between candidate visual responses and the dialogue context, the expressive differences within candidate visual responses, the adaptation relationship between the candidate's style group and the dialogue context, and the complementary relationship between expressive features and style features.

[0137] After the model has been trained, during the actual use phase, loss calculation and parameter updates are no longer performed. Instead, the input multi-turn dialogue history and candidate visual responses are encoded, matched, score fused, and sorted according to the aforementioned process.

[0138] In summary, the main advantage of this invention lies in the fact that emoji selection no longer relies solely on the overall matching score between candidate visual responses and the dialogue context, but further introduces expressive features, style features, and style prototype adaptation relationships. Existing methods, even if they can determine that a candidate visual response is semantically or emotionally relevant to the current dialogue, struggle to further determine whether its expression is appropriate, its visual style natural, or its social tone suitable for the current interaction scenario. This invention, by dividing the candidate visual response representation into expressive feature components and style feature components, enables the model to understand "what" and "in what style" the candidate visual response "expresses," thereby improving its ability to distinguish subtle differences between semantically similar candidates.

[0139] This invention better addresses the selection problem when multiple candidate visual responses are all somewhat relevant. In real-world instant messaging scenarios, candidate visual responses are often not just one relevant to the conversation, but multiple candidates that can express similar emotions or attitudes. Existing overall matching methods tend to remain at the level of "relevance ranking," while this invention further utilizes expression matching scores to determine whether the specific content of the candidate visual response meets the needs of the current conversation, such as whether it is more suitable for expressing teasing, comfort, agreement, silence, or easing the atmosphere. This makes the final selection result not only semantically relevant but also more accurate in expressing its intended purpose.

[0140] This invention also leverages style grouping patterns among emojis to improve the naturalness and social adaptability of responses. Actual emojis typically form style groups based on character imagery, visual themes, art style, or emoji series. Emojis within the same style group often share similar usage atmospheres and social tones. Existing methods often focus on matching individual candidate visual responses with the context, while this invention extends the matching of individual candidates to style group-level adaptability judgment by constructing a style prototype library. This allows the model to determine whether the style of a candidate is suitable for the current conversation atmosphere, thereby reducing situations where "semantically related but stylistically abrupt" or "emotionally similar but with unnatural tone."

[0141] This invention also demonstrates greater applicability in application scenarios lacking manual style labeling. Many emoji datasets do not contain explicit emoji series, style categories, or manual grouping information. Relying entirely on manual annotation to construct style tags increases data processing costs and hinders the method's transfer to new emoji libraries. This invention can construct weakly structured style cues based on the emoji image itself, such as the main object, object category, character identity, visual theme, or style description, and thereby form pseudo-style tags, style groups, or style prototypes. Therefore, this invention reduces reliance on manual style annotation and improves the method's transferability across different datasets, platforms, and emoji libraries.

[0142] The final ranking result of this invention is also more comprehensive. The global matching score ensures that the candidate visual responses are fundamentally relevant to the current dialogue content; the expression matching score ensures that the specific expressive function of the candidate meets the needs of the current dialogue; and the style prototype score ensures that the style group to which the candidate belongs is compatible with the tone of the current dialogue. Compared to existing methods that rely solely on a single overall matching score, the ranking criteria of this invention are more refined, simultaneously considering semantic relevance, expression suitability, and style suitability, thereby improving the accuracy, naturalness, and rationality of emoji response selection.

[0143] Furthermore, this invention can be extended on the existing emoji reply selection framework without completely rebuilding the entire system. Its core improvements lie primarily in the structured decomposition of candidate visual reply representations, the construction of a style prototype library, and the fusion and ranking of multiple score categories. Therefore, it can be used in conjunction with existing dialogue encoding modules, candidate image encoding modules, and basic matching modules. This gives the invention good engineering compatibility and deployment feasibility, making it suitable for both large-scale candidate ranking on the server side and reordering scenarios involving users' locally saved emojis.

[0144] This invention verifies its feasibility and effectiveness through experiments on publicly available datasets. The experiments were conducted on two publicly available emoji response selection datasets: DSTC10-MOD and StickerChat. Both datasets employ a candidate ranking method for evaluation, meaning the system needs to rank the actual response emojis as high as possible within a given set of candidate visual responses. DSTC10-MOD uses a ranking setting of 10 candidate visual responses, therefore R10@1, R10@2, R10@5, and MRR are used as evaluation metrics; StickerChat uses a ranking setting of 20 candidate visual responses, therefore R20@1, R20@2, R20@5, R20@10, and MRR are used as evaluation metrics.

[0145] R10@K indicates whether the authentic response emoji appears in the top K positions of the model's ranking among 10 candidate visual responses. For example, R10@1 represents the proportion of authentic response emojis ranked first among 10 candidates, and R10@5 represents the proportion of authentic response emojis appearing in the top 5. R20@K has a similar meaning, except the candidate set size is 20 emojis. For example, R20@1 represents the proportion of authentic response emojis ranked first among 20 candidates, and R20@10 represents the proportion of authentic response emojis appearing in the top 10. MRR, or Mean Reverse Rank, measures the average ranking of authentic response emojis in the ranking list; the higher the ranking of the authentic response emoji, the higher the MRR. All the above metrics are reported as percentages; higher values ​​indicate that the model is more accurate in ranking authentic response emojis at the top of the candidate list.

[0146] Regarding experimental data, DSTC10-MOD, after preprocessing, contains 211,575 training samples and 3,542 evaluation samples; StickerChat contains 320,168 training samples, 10,000 validation samples, and 10,000 test samples, and includes 174,695 emojis. These two datasets respectively cover a smaller-scale candidate visual response scenario and a larger-scale emoji library scenario, and can be used to verify the effectiveness of the present invention in different emoji response selection scenarios.

[0147] Table 1. Comparison of experimental results of our method with existing representative methods on the DSTC10-MOD dataset.

[0148]

[0149] The results above demonstrate that our proposed method achieves state-of-the-art performance on the DSTC10-MOD dataset. Compared to the robust EIGML method, this invention improves performance by 3.1 percentage points in both R10@1 and R10@2, by 1.7 percentage points in R10@5, and by 2.5 percentage points in MRR. These results indicate that, when multiple candidate visual responses possess a certain degree of semantic relevance, this invention, through decoupling expressive features from style features and prior modeling of style prototypes, can more accurately prioritize realistic response emojis in the candidate list.

[0150] Table 2 compares the experimental results of our method with those of existing representative methods on the StickerChat dataset.

[0151]

[0152] The results above show that this method still achieves good experimental results on the larger StickerChat dataset, which has a greater number of emojis and a more complex style structure. This invention achieves optimal results on R20@1, R20@2, and MRR, and reaches the same level as the optimal result on R20@10. Although R20@5 is slightly lower than PBR, this invention has advantages on R20@1 and the overall ranking quality metric MRR, which focus more on the quality of the first-order ranking, indicating that this invention has a practical effect on improving the ranking ability of real-world reply emojis.

[0153] To further verify the effectiveness of each technical component of this invention, ablation experiments were also conducted. Taking the DSTC10-MOD dataset as an example, the key ablation results are as follows:

[0154] Table 3 Ablation results of this method on the DSTC10-MOD dataset

[0155]

[0156] Ablation analysis results show that, compared to the model using only the basic backbone network, this method improves R10@1 from 47.04 to 53.13, R10@2 from 67.65 to 73.49, and MRR from 65.11 to 69.98. Removing the style prototype prior, expression matching score, or prototype supervision loss all resulted in varying degrees of performance degradation, indicating that the expression-style decoupling, prototype-level style adaptation, and prototype supervision mechanisms in this invention all contribute to the final results.

[0157] Furthermore, the analysis results show that the complete model can improve the matching score between positive sample dialogues and emojis, and improve the separability of positive and negative samples in the representation space. The parameter sensitivity experiment also shows that expression branch weights and style branch weights can both bring performance improvements within an appropriate range, indicating that expression matching and style prototype matching are effective complementary signals.

[0158] In summary, the structured emoji response selection method proposed in this invention, based on the decoupling of expressive features and style features and style prototype prior, is feasible and effective. This method can further improve the emoji response selection effect on the basis of existing multimodal matching frameworks, especially improving the fine-grained discrimination ability when multiple candidate visual responses are semantically similar, and increasing the ranking position of the actual response emojis, thus possessing good practical application value.

[0159] This embodiment also provides a computer device applicable to a visual response content selection method based on expression and style decoupling, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement all or part of the steps of the method described in the above embodiments of the present invention.

[0160] This embodiment also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, it performs the method in any optional implementation of the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0161] The storage medium proposed in this embodiment and the data storage method proposed in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0162] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A visual response content selection method based on expression and style decoupling, characterized in that: include: Obtain the current multi-turn dialogue history and multiple candidate visual responses, and generate dialogue context representation and basic visual semantic representation of candidate visual responses respectively; Based on the dialogue context representation and the basic visual semantic representation of the candidate visual responses, calculate the global matching score between each candidate visual response and the current dialogue; The basic visual semantic representation of the candidate visual response is mapped to the expression feature component and style feature component, respectively, while the dialogue context representation is mapped to the expression query vector and style query vector; The expression matching score is calculated by using the expression query vector and the expression feature components of the candidate visualization response to measure whether the specific expression content of the candidate visualization response meets the current dialogue requirements. Style prototype representations are generated using style feature components and pre-constructed style group membership relationships, and style prototype scores of candidate visualization responses are calculated based on style query vectors and style prototype representations. The global matching score, expression matching score, and style prototype score are combined to obtain the final score for each candidate visual response. The candidate visual responses are then sorted according to the final score, and the candidate visual response with the highest score is output as the target response emoji, or the top-ranked candidate visual responses are output.

2. The visual response content selection method based on expression and style decoupling as described in claim 1, characterized in that: The underlying visual semantic representations for generating the dialogue context representation and candidate visual responses include: The dialogue context representation is obtained by processing the multi-turn dialogue history through a dialogue encoding function. Multiple candidate visual responses are processed using an emoji encoding function to obtain the basic visual semantic representation of the candidate visual responses: The dialogue encoding function includes a text encoder, a multimodal matching backbone network, and other context modeling networks; the emoji encoding function includes a visual Transformer, a convolutional neural network, and other image encoding networks.

3. The visual response content selection method based on expression and style decoupling as described in claim 1, characterized in that: The calculation of the global matching score between each candidate visual response and the current dialogue includes: The dialogue context representation of the multi-turn dialogue history and the basic visual semantic representation of the candidate visual responses are input into the multimodal matching backbone network. Through cross-modal interaction, feature fusion and matching scoring, the global matching score corresponding to the candidate visual responses is obtained. The multimodal matching backbone network includes multimodal BERT, cross-attention network, dual-tower matching network, and other multimodal fusion networks.

4. The visual response content selection method based on expression and style decoupling as described in claim 1, characterized in that: The generation of style prototype representations using style feature components and pre-constructed style group membership relationships includes: Based on the visual style features of emoji images, weakly structured style cues are extracted using an offline image analysis model, and style group membership relationships are constructed based on these cues. There are two methods for obtaining styles: one is the offline image analysis model, and the other is to determine the initial style groups based on existing emoji series, character categories, visual themes, or sets in the emoji data, and then merge or re-cluster the initial style groups based on the similarity of visual features between groups. For each style group, the style feature components corresponding to the members within the style group are aggregated to obtain the style prototype vector representation of the style group. The style prototype vector is used to reflect the shared visual style and stable social tone of the emoticons within the same style group.

5. The visual response content selection method based on expression and style decoupling as described in claim 1, characterized in that: The calculation of the style prototype score for candidate visual responses based on style query vectors and style prototype representations includes: By matching the style query vector corresponding to the current dialogue with the style prototype representation, the degree of fit between the current dialogue and different style prototypes is obtained. The style prototype score is represented as: ； in, Indicates the first Each candidate visual response is relative to the current conversation. The style prototype score, Indicates the first The vector representation of the style prototype to which each candidate visual response belongs. This represents the style query vector corresponding to the current dialogue. This represents the similarity function.

6. The visual response content selection method based on expression and style decoupling as described in claim 1, characterized in that: The process of ranking multiple candidate visual responses based on the final score includes: Weighting coefficients are assigned to the expression matching score and style prototype score respectively, and the final score of the candidate visualization response is obtained by weighted summation, as follows: ； in, Indicates the first The final score of each candidate visual response. Indicates the first Candidate visual responses and the current conversation Global matching score between them Indicates the first Each candidate visual response is relative to the current conversation. The expression matching score, Indicates the first Each candidate visual response is relative to the current conversation. The style prototype score, and These represent the weighting coefficients for the expression matching score and the style prototype score, respectively. The candidate visual responses are sorted according to the final score, and the candidate visual response with the highest score is selected as the current response, or several top-ranked candidates are output for the user to choose from.

7. The visual response content selection method based on expression and style decoupling as described in claim 1, characterized in that, Also includes: The model is trained based on the real response emojis after output. The real response emojis are used as positive samples, and negative samples are constructed. Negative samples include negative samples in the same group that belong to the same style prototype as the real response emojis but are not correct replies, and negative samples in different groups that belong to different style prototypes than the real response emojis. Candidate score vectors are constructed based on the final scores of the three types of samples. Candidate matching loss is calculated based on the final scores of positive samples, negative samples in the same group, and negative samples across groups. Prototype supervision loss is calculated based on the adaptation scores of the current dialogue and all style prototypes. Expression ranking loss is calculated based on the expression matching scores of positive samples and negative samples in the same group. Orthogonal constraint loss is calculated based on the normalized inner product of expression feature components and style feature components. The overall training objective of the model is constructed by combining candidate matching loss, prototype supervision loss, expression ranking loss, and orthogonality constraint loss. After the model is trained, the input multi-turn dialogue history and candidate visual responses are encoded, matched, fused in score, and sorted for output.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the visualization response content selection method based on expression and style decoupling as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the visualization response content selection method based on expression and style decoupling as described in any one of claims 1 to 7.

Images