Emoticon package retrieval method, electronic device and computer-readable storage medium

Abstract

The present application discloses an emoticon package retrieval method, an electronic device, and a computer-readable storage medium. The method includes: obtaining a conversation text, splicing a plurality of relationship supplementary texts for the conversation text, and obtaining a spliced ​​text; wherein each relationship supplementary text corresponds to a preset common sense relationship; based on the spliced ​​text, determining the intention information of the conversation text and the intention fusion text feature corresponding to the conversation text; wherein the intention fusion text feature is fused with the intention information; obtaining a target visual attribute that matches the intention fusion text feature, and obtaining a target emoticon package that matches the intention fusion text feature from an emoticon package collection corresponding to the target visual attribute; wherein the target visual attribute is selected from a plurality of candidate visual attributes, and the candidate visual attribute is determined based on the description text of a plurality of emoticon packages, and each candidate visual attribute corresponds to a respective emoticon package collection. The above scheme can improve the accuracy of emoticon package retrieval.

Description

Technical Field

[0001] The present application relates to the technical field of emoticon package retrieval systems, and in particular to an emoticon package retrieval method, an electronic device, and a computer-readable storage medium. Background Art

[0002] Emoji retrieval involves searching for and providing relevant emojis from a pre-prepared emoji database based on the context of a conversation. This task is typically used in chat applications, social media platforms, or smart assistants. However, current emoji retrieval systems are limited by model performance, resulting in low emoji retrieval accuracy. Therefore, improving emoji retrieval accuracy has become a pressing issue. Summary of the Invention

[0003] The main technical problem solved by this application is to provide an emoticon package retrieval method, an electronic device and a computer-readable storage medium, which can improve the accuracy of emoticon package retrieval.

[0004] In order to solve the above technical problems, the first aspect of the present application provides an emoticon package retrieval method, comprising: obtaining a dialogue text, splicing a plurality of relationship supplementary texts for the dialogue text, and obtaining a spliced ​​text; wherein each of the relationship supplementary texts corresponds to a preset common sense relationship; based on the spliced ​​text, determining the intention information of the dialogue text and the intention fusion text feature corresponding to the dialogue text; wherein the intention fusion text feature is fused with the intention information; obtaining a target visual attribute that matches the intention fusion text feature, and obtaining a target emoticon package that matches the intention fusion text feature from an emoticon package collection corresponding to the target visual attribute; wherein the target visual attribute is selected from a plurality of candidate visual attributes, and the candidate visual attributes are determined based on the description texts of a plurality of emoticon packages, and each of the candidate visual attributes corresponds to its own emoticon package collection.

[0005] In order to solve the above technical problems, the second aspect of this application provides an electronic device, comprising: a memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the emoticon package retrieval method described in the first aspect above.

[0006] In order to solve the above technical problems, the third aspect of the present application provides a computer-readable storage medium on which program instructions are stored. When the program instructions are executed by a processor, the emoticon package retrieval method described in the first aspect is implemented.

[0007] The above scheme obtains the dialogue text and obtains a plurality of relationship supplementary texts based on the dialogue text, each relationship supplementary text corresponds to a preset common sense relationship, and splices the plurality of relationship supplementary texts with the dialogue text to obtain a spliced ​​text. Based on the spliced ​​text, the intention information of the dialogue text and the intention fusion text features fused with the intention information are determined. By combining a plurality of common sense relationships, the intention information of the dialogue text can be inferred more accurately, and the target visual attribute that matches the intention fusion text feature is selected from a plurality of candidate visual attributes, and the target emoticon package that matches the intention fusion text feature is obtained from the emoticon package collection corresponding to the target visual attribute. The emoticon package retrieval is performed by combining the intention information of the dialogue text and the cross-modal relationship between the visual attributes of the emoticon package, thereby improving the accuracy of the emoticon package retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without inventive efforts. Among them:

[0009] Figure 1 This is a flow chart of an embodiment of the emoticon package retrieval method of the present application;

[0010] Figure 2 This is a flowchart of an implementation method corresponding to part of the content in step S101 of this application;

[0011] Figure 3 It is a structural diagram of an implementation method of the retrieval model of the present application;

[0012] Figure 4 This is a flowchart of an implementation method of the training process of the retrieval model of the present application;

[0013] Figure 5 This is a schematic structural diagram of an embodiment of the electronic device of the present application;

[0014] Figure 6 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0015] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them, and different implementation methods can be adaptively combined. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0016] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of the emoticon package retrieval method of the present application, which includes:

[0017] S101: Acquire a conversation text, and splice a plurality of relationship supplementary texts to the conversation text to obtain a spliced ​​text; wherein each relationship supplementary text corresponds to a preset common sense relationship.

[0018] Specifically, after obtaining the dialogue text, a plurality of relationship supplementary texts are obtained based on the dialogue text, each relationship supplementary text corresponds to a preset common sense relationship, and the plurality of relationship supplementary texts are spliced ​​with the dialogue text to obtain a spliced ​​text.

[0019] In one application method, after obtaining the dialogue text, the trained COMET model is used to generate corresponding sentences describing these common sense relationships based on multiple preset common sense relationships as multiple relationship supplementary texts, and the multiple relationship supplementary texts are spliced ​​with the dialogue text to obtain spliced ​​text.

[0020] In another application method, after obtaining the dialogue text, the keywords and phrases in the dialogue text are identified according to preset rules, and the preset common sense relationships are introduced based on these keywords and phrases to obtain corresponding multiple relationship supplementary texts, and the multiple relationship supplementary texts and the dialogue text are filled into the corresponding positions of the preset template or framework to obtain the spliced ​​text.

[0021] It should be noted that this application uses ATOMIC2020 as a common knowledge base, which contains knowledge that is not easy to obtain in pre-trained language models, and can generate accurate and representative knowledge for unseen entities and events. For each entity and event, 5 common sense relationships can usually be inferred, including: intent reasoning (xIntent), premise reasoning (xNeed), demand reasoning (xWant), impact reasoning (xEffect), and reaction reasoning (xReact).

[0022] S102: Based on the concatenated text, determine the intention information of the dialogue text and the intention-fused text features corresponding to the dialogue text; wherein the intention-fused text features are fused with the intention information.

[0023] Specifically, the intention information of the dialogue text and the intention-fused text features corresponding to the dialogue text that are fused with the intention information are determined based on the spliced ​​text.

[0024] In one application, a concatenated text feature of the concatenated text is obtained, intention information of the conversation text is determined based on the concatenated text feature, and the intention information is fused with the concatenated text to obtain an intention fused text feature.

[0025] In another application, a concatenated text feature of the concatenated text is obtained, intention information of the dialogue text is determined based on the concatenated text feature, and the intention information is fused with the dialogue text to obtain an intention fused text feature.

[0026] S103: Obtain a target visual attribute that matches the intended fusion text feature, and obtain a target emoticon package that matches the intended fusion text feature from the emoticon package collection corresponding to the target visual attribute; wherein the target visual attribute is selected from multiple candidate visual attributes, the candidate visual attributes are determined based on the description texts of multiple emoticon packages, and each candidate visual attribute corresponds to its own emoticon package collection.

[0027] Specifically, a target visual attribute that matches the intended fusion text feature is selected from multiple candidate visual attributes, and a target emoticon package that matches the intended fusion text feature is obtained from a collection of emoticon packages corresponding to the target visual attribute.

[0028] In one application method, the similarity between the intention-fusion text feature and multiple candidate visual attributes is calculated, and the candidate visual attributes that meet the preset conditions are used as target visual attributes that match the intention-fusion text feature. From the collection of emoticon packages corresponding to the target visual attributes, the emoticon package that meets the preset conditions with the intention-fusion text feature is selected as the target emoticon package.

[0029] In another application method, the correlation between the intention fusion text feature and multiple candidate visual attributes is calculated, and the candidate visual attributes with high correlation are used as target visual attributes that match the intention fusion text feature. From the collection of emoticon packages corresponding to the target visual attributes, emoticon packages with high correlation with the intention fusion text feature are selected as target emoticon packages.

[0030] It should be noted that the emoticon collection provided in this application is a new emoticon multi-label dataset obtained by pre-collection and annotation, called StickerInt. The specific construction process is as follows: a dataset is obtained from an internal test chat and dating application software, which includes a large number of emoticons that can be used for private and group chats. These emoticons have identifiers different from typical images. The chat content in the dataset is divided into multiple conversations, and in order to ensure that the dataset is truly multimodal (containing text and stickers), each emoticon in the chat history is systematically traversed and its related context is captured. This ensures that the context is within the scope of the same conversation, and the emoticon and its corresponding text also come from the same object. Then, the coarse-grained and fine-grained intent labels of the emoticon are marked according to the conversation.

[0031] The above scheme obtains the dialogue text and obtains a plurality of relationship supplementary texts based on the dialogue text, each relationship supplementary text corresponds to a preset common sense relationship, and splices the plurality of relationship supplementary texts with the dialogue text to obtain a spliced ​​text. Based on the spliced ​​text, the intention information of the dialogue text and the intention fusion text features fused with the intention information are determined. By combining a plurality of common sense relationships, the intention information of the dialogue text can be inferred more accurately, and the target visual attribute that matches the intention fusion text feature is selected from a plurality of candidate visual attributes, and the target emoticon package that matches the intention fusion text feature is obtained from the emoticon package collection corresponding to the target visual attribute. The emoticon package retrieval is performed by combining the intention information of the dialogue text and the cross-modal relationship between the visual attributes of the emoticon package, thereby improving the accuracy of the emoticon package retrieval.

[0032] In one embodiment, see Figure 2 , Figure 2 This is a flowchart of an implementation method corresponding to part of step S101 of this application. Each common sense relationship corresponds to its own text collection, and the text collection includes multiple candidate texts. Step S101 specifically includes:

[0033] S201: Acquire the conversation text, and extract some candidate texts for the conversation text from the text collection corresponding to each common sense relationship as relationship supplementary texts.

[0034] Specifically, after obtaining the conversation text, representative partial candidate texts related to the conversation text are extracted from the text collection corresponding to each common sense relationship as the relationship supplementary text.

[0035] In a specific application scenario, there are five types of common sense relationships: intent reasoning (xIntent), premise reasoning (xNeed), requirement reasoning (xWant), impact reasoning (xEffect), and reaction reasoning (xReact). Each common sense relationship corresponds to its own text collection, which is usually generated using the common knowledge base ATOMIC2020. The BART version of the COMET model trained on this knowledge base can be used to extract representative candidate text related to the conversation text as the relationship supplementary text. The process is as follows:

[0036] C r =COMET(T,r)(1)

[0037]

[0038] Among them, C know represents the relation supplementary text, T represents the conversation text, and r represents each common sense relation, r∈R, R∈{xIntent, xNeed, xWant, xEffect, xReact}.

[0039] S202: Splicing the dialogue text and the relationship supplementary text to obtain a spliced ​​text.

[0040] Specifically, the conversation text is spliced ​​with the relationship supplementary texts corresponding to the five common sense relationships to obtain a spliced ​​text.

[0041] In one embodiment, see Figure 3 , Figure 3 It is a structural diagram of an embodiment of the retrieval model of the present application. The target emoticon package is obtained using the retrieval model. The retrieval model includes a first branch for processing text and a second branch for processing the emoticon package. The retrieval model is trained using training text and multiple emoticon packages, and the training text is spliced ​​with multiple relationship supplementary texts. In step S102, based on the spliced ​​text, the intention information of the dialogue text and the intention fusion text features corresponding to the dialogue text are determined, specifically including: inputting the spliced ​​text into the first text encoding module of the first branch to obtain the spliced ​​text features; inputting the spliced ​​text features into the classification module of the first branch to obtain the intention information of the dialogue text; inputting the intention information into the second text encoding module of the first branch to obtain the intention fusion text features.

[0042] Specifically, after the spliced ​​text is input into the first text encoding module of the first branch, the text features of the spliced ​​text are extracted to obtain the spliced ​​text features, and then the spliced ​​text features are input into the classification module to obtain the intention information of the dialogue text, and then the intention information is input into the second text encoding module to obtain the intention fusion text features.

[0043] In a specific application scenario, the first and second text encoding modules are pre-trained Multilingual BERT (M-BERT) models, and the classification module is a softmax classifier. The concatenated text is input into Multilingual BERT to obtain concatenated text features. The process is as follows:

[0044]

[0045] Among them, H T Represents concatenated text features.

[0046] In other application scenarios, other models may be used instead of Multilingual BERT, such as Roberta, etc., and this application does not limit this.

[0047] Furthermore, the concatenated text feature H T Input the softmax classifier to obtain the intent information of the conversation text. The process is as follows:

[0048] ρ=softmax(WH T +b)(4)

[0049]

[0050] Among them, ρ represents the use of the softmax layer to convert the neural network output into a probability distribution, W represents the weight matrix, and b represents the bias term. Indicates intention information.

[0051] Furthermore, the intent information is input into Multilingual BERT to obtain the intent-fused text features. The process is as follows:

[0052]

[0053] Among them, H Y Intent fusion text features.

[0054] In one implementation scenario, the candidate visual attributes to which the emoticon package belongs are determined during the training process of the retrieval model, and the visual features of the emoticon package are obtained during the training process and divided into the corresponding emoticon package collection. In step S103, the target visual attribute that matches the intention fusion text feature is obtained, and the target emoticon package that matches the intention fusion text feature is obtained from the emoticon package collection corresponding to the target visual attribute. Specifically, the method includes: determining the target visual attribute that matches the intention fusion text feature based on the similarity between the intention fusion text feature and the attribute features of multiple candidate visual attributes; and determining the target emoticon package that matches the intention fusion text feature based on the similarity between the intention fusion text feature and the visual features in the emoticon package collection corresponding to the target visual attribute.

[0055] Specifically, the similarity between the intent-fusion text feature and the attribute features of multiple candidate visual attributes is calculated, and the candidate visual attributes that meet the preset conditions are determined as target visual attributes that match the intent-fusion text feature, wherein the preset condition is the candidate visual attribute with the highest similarity value. Then, the similarity between the intent-fusion text feature and the visual features in the emoticon package collection corresponding to the target visual attribute is calculated, and the emoticon package that meets the preset condition is determined as the target emoticon package that matches the intent-fusion text feature. The preset condition can be that the K emoticon packages with the highest similarity values ​​are used as target emoticon packages, or the preset condition can be that all emoticon packages with similarities greater than a preset similarity threshold are used as target emoticon packages.

[0056] In a specific application scenario, cosine similarity is used to calculate the similarity between the intended fusion text features and the attribute features of multiple candidate visual attributes. The process is as follows:

[0057] CA=cos(H Y ,HR )(7)

[0058] In one embodiment, the process of constructing candidate visual attributes includes: obtaining an emoticon package, inputting the emoticon package into a large language model to obtain a description text, inputting the emoticon package into the visual encoding module of the second branch to obtain the visual features of the emoticon package; inputting the description text into the third text encoding module of the second branch to obtain the description text features of the emoticon package; based on the visual features and description text features of the emoticon package, obtaining the candidate visual attributes belonging to the emoticon package; wherein the candidate visual attributes include at least gestures, postures, facial expressions and language.

[0059] Specifically, the acquired emoticon package is input into the large language model and visual encoding module of the second branch respectively, the large language model is used to generate the description text corresponding to the emoticon package, and the visual encoding module is used to obtain the visual features of the emoticon package.

[0060] In a specific application scenario, in order to extract key expression information from the expression package and reduce unnecessary interference from irrelevant information, this application designs a gesture L G , Posture L P 、Facial expression L F and language L V Four visual attributes are used to construct prompts for emoticons. The Owen-VL (Qwen Large Vision Language Model) multimodal large language model (Multimodal Large Language Model) generates description text based on the above four visual attributes. The process is as follows:

[0061] {A G ,A P ,A F ,A V}=MLLM({L G ,L P ,L F ,L V})(8)

[0062] Among them, {A G ,A P ,A F ,A V} represents the description text.

[0063] In other application scenarios, other multimodal large language models can also be used instead of Owen-VL, such as MiniGPT4, LlaVA, etc., and this application does not limit this.

[0064] In other application scenarios, other visual attributes may also be designed, such as actions, expressions, etc., which are not limited in this application.

[0065] Furthermore, in order to learn the visual information of the emoticon package, the visual encoding module can use the ViT model pre-trained by CLIP. The emoticon package is input into the ViT model pre-trained by CLIP to obtain the visual features of the emoticon package. The process is as follows:

[0066] H I =ViT(v)(9)

[0067] Among them, H I Represents the visual characteristics of the emoticon.

[0068] In other application scenarios, other models can also be used instead of the CLIP-based pre-trained ViT model, such as VGG16, ResNet50, MobileNet, VGG19, etc.

[0069] Furthermore, after the description text is input into the third text encoding module, the description text features of the emoticon package are generated using the third text encoding module.

[0070] In a specific application scenario, the third text encoding module is a pre-trained Multilingual BERT (M-BERT) (multilingual BERT model), which describes the text {A G ,A P ,A F ,A V Input the pre-trained Multilingual BERT (M-BERT) model to obtain the description text features of the emoticon package. The process is as follows:

[0071]

[0072] Among them, H A Represents the descriptive text features of the emoticon package.

[0073] Furthermore, candidate visual attributes of the emoticon package, including at least gestures, postures, facial expressions and language, are obtained based on the visual features and description text features of the emoticon package.

[0074] It is understandable that candidate visual attributes may also include actions, expressions, etc., which are not limited in this application.

[0075] In a specific application scenario, the visual feature H I and description text features H A Specifically, two fully connected layers f are used to highlight the important regions in the emoji. vis and f des To transform the visual feature H I and description text features H AProjecting into the same dimension d, the process is as follows:

[0076]

[0077] Among them, j∈{G,P,F,V}, G represents gesture, P represents posture, F represents facial expression, and V represents language.

[0078] Furthermore, the cross-modal attention mechanism is used to calculate h j A With h I The relationship between them is as follows:

[0079]

[0080] in, represents the relationship between hjA and hI, is a randomly initialized projection matrix, Denote the entity, and set dq, dk, dv = d / h for each parallel attention layer, where h represents the number of heads in each multi-head attention layer. Next, perform the maximum pooling operation on Mj, i.e. let Represents the relationship score between hjA and hI, and finally for each visual feature hI and relationship score Perform multiplication to obtain the candidate visual attributes of the emoticon package. The process is as follows:

[0081]

[0082] Among them, H R represents candidate visual attributes.

[0083] In one embodiment, the process of constructing the emoticon package collection includes: adding emoticons of the same candidate visual attribute and their corresponding visual features to the emoticon package collection of the corresponding candidate visual attribute.

[0084] Specifically, emoticon packages of candidate visual attributes of the same type and their corresponding visual features are added together to a collection of emoticon packages of the corresponding candidate visual attributes.

[0085] In one implementation scenario, the emoticon package retrieval method provided in the present application also includes: in response to obtaining a new emoticon package outside the training process, obtaining the visual features and descriptive text features of the new emoticon package; based on the visual features and descriptive text features of the new emoticon package, adding the new emoticon package and its corresponding visual features to the emoticon package collection of the corresponding candidate visual attributes.

[0086] Specifically, when a new emoticon package is obtained outside the training process of the retrieval model, the visual features and descriptive text features of the new emoticon package are obtained respectively. The specific acquisition process can refer to the above steps and will not be repeated here.

[0087] Furthermore, based on the similarity between the visual features of the newly added emoticon package and the descriptive text features, the newly added emoticon package and its corresponding visual features are added to the corresponding emoticon package collection of matching candidate visual attributes. By updating the emoticon package collection, the restrictions on emoticon package retrieval can be reduced to a certain extent, thereby improving the accuracy of emoticon package retrieval.

[0088] In one embodiment, see Figure 4 , Figure 4 This is a flowchart of an embodiment of the training process of the retrieval model of the present application. The training process of the retrieval model includes:

[0089] S401: Input the training text into the first branch to obtain the predicted intent information and the training intent fusion text features, and input the emoticon package into the second branch to obtain the candidate visual attributes; wherein the training text includes the true intent label.

[0090] Specifically, the training text including the true intent label is input into the first encoding module, the second encoding module and the classification module of the first branch to obtain the predicted intent information and the training intent fusion text features, and the emoticon package is input into the large language model, the third text encoding module and the visual encoding module of the second branch respectively to obtain the descriptive text features and visual features of the emoticon package, and the candidate visual attributes are obtained based on the descriptive text features and visual features.

[0091] S402: Determine the intent prediction loss based on the predicted intent information and the true intent label, and determine the retrieval loss based on the training intent fusion text features and candidate visual attributes.

[0092] Specifically, based on the predicted intent information and the true intent label, the intent prediction loss is determined as follows:

[0093]

[0094] in, Indicates the intention to predict loss, y int represents the true intent label, and n represents the number of samples.

[0095] In one implementation scenario, step S402 determines the retrieval loss based on the training intent fusion text features and the candidate visual attributes, specifically including: determining the predicted visual attributes matched by the training intent fusion text features from multiple candidate visual attributes; wherein the emoticon package collection corresponding to the predicted visual attributes is a positive emoticon package, and the emoticon package collection corresponding to the remaining candidate visual attributes is a negative emoticon package; determining the retrieval loss based on the similarity between the training intent fusion text features and the visual features of the positive emoticon package, and the similarity between the training intent fusion text features and the visual features of the negative emoticon package.

[0096] Specifically, by using the cross-modal attention mechanism, by calculating the cosine similarity between multiple candidate visual attributes and the training intent fusion text features, the candidate visual attributes whose calculated similarity is higher than the preset similarity threshold are determined as the predicted visual attributes that match the training intent fusion text features, wherein the emoji collection corresponding to the predicted visual attributes is a positive emoji collection, and the emoji collection corresponding to the remaining candidate visual attributes is a negative emoji collection. The retrieval loss is determined based on the similarity between the training intent fusion text features and the visual features of the positive emoji collection, as well as the similarity between the training intent fusion text features and the visual features of the negative emoji collection. The process is as follows:

[0097]

[0098] in, represents the retrieval loss, ρ neg Represents the similarity between the training intention fusion text features and the visual features of the negative emoticon package, ρ pos It represents the similarity between the training intention fusion text features and the visual features of the positive emoticon package, and margin represents the hyperparameter of the marginal readjustment.

[0099] S403: Based on the intent prediction loss and the retrieval loss, the parameters of the retrieval model are adjusted until the convergence conditions are met, thereby obtaining a trained retrieval model.

[0100] Specifically, based on the intent prediction loss and retrieval loss, the parameters of the retrieval model are adjusted until the preset convergence conditions are met, thereby obtaining the trained retrieval model. The process is as follows:

[0101]

[0102] Among them, λ1 and λ2 are hyperparameters that act as scaling factors.

[0103] See also Figure 5 , Figure 5: This is a structural diagram of an embodiment of an electronic device of the present application. The electronic device 50 includes a memory 501 and a processor 502 coupled to each other, wherein the memory 501 stores program data (not shown), and the processor 502 calls the program data to implement the method in any of the above embodiments. For the description of the relevant content, please refer to the detailed description of the above method embodiments, which will not be repeated here. Specifically, the electronic device 50 includes but is not limited to: a desktop computer, a laptop computer, a tablet computer, a server, etc., which are not limited here. In addition, the processor 502 can also be called a CPU (Center Processing Unit). The processor 502 may be an integrated circuit chip with signal processing capabilities. The processor 502 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. In addition, the processor 502 can be implemented by an integrated circuit chip.

[0104] See also Figure 6 , Figure 6 This is a structural diagram of an embodiment of a computer-readable storage medium of the present application. The computer-readable storage medium 60 stores program data 600. When the program data 600 is executed by the processor, the method in any of the above embodiments is implemented. For an explanation of the relevant content, please refer to the detailed description of the above method embodiments, which will not be repeated here.

[0105] It should be noted that the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of this embodiment.

[0106] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0107] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0108] The above description is only an implementation method of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the description and drawings of this application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for searching emoticons, characterized in that: include: Acquire a conversation text, and splice a plurality of relationship supplementary texts to the conversation text to obtain a spliced ​​text; wherein each relationship supplementary text corresponds to a preset common sense relationship; Based on the concatenated text, determining the intention information of the dialogue text and the intention fusion text feature corresponding to the dialogue text; wherein the intention fusion text feature is fused with the intention information; Obtaining a target visual attribute that matches the intended fusion text feature, and obtaining a target emoticon package that matches the intended fusion text feature from the emoticon package collection corresponding to the target visual attribute; wherein the target visual attribute is selected from a plurality of candidate visual attributes, the candidate visual attributes being determined based on description texts of a plurality of emoticon packages, and each candidate visual attribute corresponding to a respective emoticon package collection; The target emoticon package is obtained by using a retrieval model, the retrieval model includes a first branch for processing text and a second branch for processing the emoticon package, the retrieval model is trained using training text and multiple emoticon packages, and the training text is spliced ​​with multiple relationship supplementary texts; The determining, based on the concatenated text, the intention information of the dialogue text and the intention fusion text features corresponding to the dialogue text includes: Inputting the concatenated text into the first text encoding module of the first branch to obtain concatenated text features; Inputting the concatenated text features into the classification module of the first branch to obtain the intention information of the conversation text; Inputting the intent information into the second text encoding module of the first branch to obtain the intent fusion text feature; The candidate visual attributes of the emoticon package are determined during the training process of the retrieval model, and the visual features of the emoticon package are obtained during the training process and divided into corresponding emoticon package collections; The step of obtaining a target visual attribute that matches the intended fusion text feature, and obtaining a target emoticon package that matches the intended fusion text feature from an emoticon package collection corresponding to the target visual attribute, includes: Determining a target visual attribute that matches the intention-fused text feature based on similarities between the intention-fused text feature and attribute features of the plurality of candidate visual attributes; Based on the similarity between the intention-fusion text feature and the visual features in the emoticon package collection corresponding to the target visual attribute, a target emoticon package that matches the intention-fusion text feature is determined.

2. The method according to claim 1, characterized in that Each common sense relationship corresponds to a respective text collection, and the text collection includes a plurality of candidate texts; The step of obtaining the dialogue text and splicing the dialogue text with multiple related supplementary texts to obtain a spliced ​​text includes: Obtaining the conversation text, and extracting some candidate texts for the conversation text from the text collection corresponding to each common sense relationship as supplementary texts for the relationship; The dialogue text and the relationship supplementary text are spliced ​​together to obtain the spliced ​​text.

3. The method according to claim 1, characterized in that The process of constructing the candidate visual attributes includes: Obtaining an emoticon package, inputting the emoticon package into the large language model of the second branch to obtain the description text, and inputting the emoticon package into the visual encoding module of the second branch to obtain the visual features of the emoticon package; Inputting the description text into the third text encoding module of the second branch to obtain the description text features of the emoticon package; Based on the visual features and descriptive text features of the emoticon package, the candidate visual attributes to which the emoticon package belongs are obtained; wherein the candidate visual attributes include at least gestures, postures, facial expressions and languages; The process of constructing the emoticon package collection includes: The emoticon package of the same candidate visual attribute and its corresponding visual features are added to the corresponding emoticon package collection of the candidate visual attribute.

4. The method according to claim 3, characterized in that Also includes: In response to obtaining a new emoticon package outside the training process, obtaining visual features and descriptive text features of the new emoticon package; Based on the visual features and descriptive text features of the newly added emoticon package, the newly added emoticon package and its corresponding visual features are added to the corresponding collection of emoticon packages of the candidate visual attributes.

5. The method according to claim 1, wherein The training process of the retrieval model includes: Input the training text into the first branch to obtain predicted intent information and training intent fusion text features, and input the emoticon package into the second branch to obtain the candidate visual attributes; wherein the training text includes the true intent label; Determining an intent prediction loss based on the predicted intent information and the true intent label, and determining a retrieval loss based on the training intent fusion text features and the candidate visual attributes; Based on the intent prediction loss and the retrieval loss, the parameters of the retrieval model are adjusted until the convergence condition is met, thereby obtaining the trained retrieval model.

6. The method according to claim 5, characterized in that The fusing of the text features and the candidate visual attributes based on the training intent to determine the retrieval loss includes: Determining, from the plurality of candidate visual attributes, a predicted visual attribute that matches the training intent fusion text feature; wherein the set of emoticons corresponding to the predicted visual attribute is a positive emoticon, and the set of emoticons corresponding to the remaining candidate visual attributes is a negative emoticon; The retrieval loss is determined based on the similarity between the training intention fusion text features and the visual features of the positive emoticon package, and the similarity between the training intention fusion text features and the visual features of the negative emoticon package.

7. An electronic device, characterized in that: include: A memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the emoticon package retrieval method according to any one of claims 1 to 6.

8. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the emoticon package retrieval method according to any one of claims 1 to 6 is implemented.

Images