Emoji prediction method, apparatus, device, and storage medium

Abstract

Embodiments of the present application provide an emoji prediction method and device, equipment and storage medium, relating to the technical field of artificial intelligence, the method comprising: obtaining a target image and target text information, the target image being an image with labeled information, the labeled information including the target text information and an emoji; determining at least one predicted emoji of the labeled information according to the target image, the target text information and a pre-trained emoji prediction model, the emoji prediction model being trained according to a plurality of training samples, each training sample including a sample image and labeled information of the sample image, the labeled information of the sample image including text labeled information and a labeled emoji; and displaying the at least one predicted emoji of the labeled information. Thus, the emoji contained in the labeled information of the image can be predicted, ensuring the integrity of the transcription of the labeled information of the image.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an emoji prediction method, apparatus, device, and storage medium. Background Technology

[0002] The internet has not only changed the speed and quality of human information dissemination, but has also greatly enriched the ways in which humans express themselves, forming a unique online language characterized by a large number of nonverbal emoticons (emojis). Emojis are used to vividly present and depict nonverbal information in everyday face-to-face communication, making both parties feel as if they can hear each other's voices and see each other. With the increasing richness of multimedia content, more and more emoticons appear in videos and images in social media and other scenarios, enabling richer expression; different emoticons can influence semantic and contextual expression.

[0003] Currently, users are supported in editing images, such as adding annotations. The added annotation information can include text and emojis. However, for images with annotations that include both text and emojis, the transcription of these annotations is problematic because emojis are diverse and come in many versions, making it difficult to output the corresponding emojis via keyboard input. Current technologies only transcribe the text information from the image, ignoring emojis, resulting in incomplete transcription. Summary of the Invention

[0004] This application provides an emoji prediction method, apparatus, device, and storage medium, which can predict emojis contained in the annotation information of an image, ensuring the integrity of the transcription of the image annotation information.

[0005] In a first aspect, embodiments of this application provide an emoji prediction method, including:

[0006] Acquire target image and target text information, wherein the target image is an image with annotation information, and the annotation information includes the target text information and emoticons;

[0007] Based on the target image, the target text information, and a pre-trained emoji prediction model, at least one predicted emoji is determined from the labeled information. The emoji prediction model is trained based on multiple training samples, each of which includes a sample image and the labeled information of the sample image. The labeled information of the sample image includes text labeled information and labeled emojis.

[0008] At least one predicted emoji is displayed to show the labeled information.

[0009] Optionally, the method further includes:

[0010] In response to a user's operation of selecting a target emoji from at least one predicted emoji in the annotation information, the target text information and the target emoji are merged according to the distribution order of the annotation information to obtain target annotation information;

[0011] Display the target annotation information.

[0012] Optionally, acquiring the target image and target text information includes:

[0013] Receive the input target image and target text information.

[0014] Optionally, determining at least one predicted emoji based on the target image, the target text information, and a pre-trained emoji prediction model includes:

[0015] The target image and the target text information are combined to form a prompt paradigm, which is then input into the emoji prediction model. The model outputs at least one predicted emoji based on the labeled information. The content to be filled in the prompt paradigm structure is the emoji to be predicted.

[0016] Optionally, the emoji prediction model includes an image feature extraction model and a text feature extraction model. The step of inputting the target image and the target text information into the prompt paradigm and outputting at least one predicted emoji based on the labeled information includes:

[0017] The target image is input into the image feature extraction model, and the model outputs the feature vector of the target image.

[0018] The target text information and each emoji in the emoji library are sequentially merged according to the prompt paradigm structure to obtain candidate information;

[0019] Each candidate information is sequentially input into the text feature extraction model, and the feature vector of the candidate information is output.

[0020] Calculate the cosine similarity between the feature vector of the target image and the feature vector of the candidate information respectively. Select the candidate information corresponding to the first K cosine similarities in descending order of the cosine similarity. Output the emojis in the K candidate information as at least one predicted emoji of the labeled information. K is a positive integer.

[0021] Optionally, the emoji prediction model is trained based on the N training samples in the following manner:

[0022] For each of the N training samples, the sample image, the text annotation information of the sample image, and the annotated emojis of the sample image are used as inputs to the emoji prediction model to obtain the feature vector of the sample image and the feature vector of the annotation information of the sample image.

[0023] A loss function is constructed based on the cosine similarity between the feature vectors of the N sample images and the feature vectors of the labeled information of the N sample images;

[0024] The parameters of the emoji prediction model are adjusted according to the loss function to obtain a trained emoji prediction model.

[0025] Optionally, the emoji prediction model includes an image feature extraction model and a text feature extraction model. The step of using sample images from the training samples and the annotation information of the sample images as input to the emoji prediction model to obtain the feature vectors of the sample images and the feature vectors of the annotation information of the sample images includes:

[0026] The image feature extraction model takes the sample images from the training samples as input and outputs the feature vectors of the sample images.

[0027] The text feature extraction model takes the annotation information of the sample image as input and outputs the feature vector of the annotation information of the sample image.

[0028] Secondly, embodiments of this application provide an emoji prediction device, comprising:

[0029] The acquisition module is used to acquire a target image and target text information, wherein the target image is an image with annotation information, and the annotation information includes the target text information and emoticons;

[0030] The determining module is configured to determine at least one predicted emoji based on the target image, the target text information, and a pre-trained emoji prediction model, wherein the emoji prediction model is trained based on multiple training samples, each training sample including a sample image and the annotation information of the sample image, the annotation information of the sample image including text annotation information and labeled emojis;

[0031] A display module is used to display at least one predicted emoji of the labeled information.

[0032] Optionally, the display module is further configured to:

[0033] In response to a user's operation of selecting a target emoji from at least one predicted emoji in the annotation information, the target text information and the target emoji are merged according to the distribution order of the annotation information to obtain target annotation information;

[0034] Display the target annotation information.

[0035] Optionally, the acquisition module is used for:

[0036] Receive the input target image and target text information.

[0037] Optionally, the determining module is used to:

[0038] The target image and the target text information are combined to form a prompt paradigm, which is then input into the emoji prediction model. The model outputs at least one predicted emoji based on the labeled information. The content to be filled in the prompt paradigm structure is the emoji to be predicted.

[0039] Optionally, the emoji prediction model includes an image feature extraction model and a text feature extraction model, and the determining module is used for:

[0040] The target image is input into the image feature extraction model, and the model outputs the feature vector of the target image.

[0041] The target text information and each emoji in the emoji library are sequentially merged according to the prompt paradigm structure to obtain candidate information;

[0042] Each candidate information is sequentially input into the text feature extraction model, and the feature vector of the candidate information is output.

[0043] Calculate the cosine similarity between the feature vector of the target image and the feature vector of the candidate information respectively. Select the candidate information corresponding to the first K cosine similarities in descending order of the cosine similarity. Output the emojis in the K candidate information as at least one predicted emoji of the labeled information. K is a positive integer.

[0044] Optionally, the emoji prediction model is trained based on the N training samples in the following manner:

[0045] For each of the N training samples, the sample image, the text annotation information of the sample image, and the annotated emojis of the sample image are used as inputs to the emoji prediction model to obtain the feature vector of the sample image and the feature vector of the annotation information of the sample image.

[0046] A loss function is constructed based on the cosine similarity between the feature vectors of the N sample images and the feature vectors of the labeled information of the N sample images;

[0047] The parameters of the emoji prediction model are adjusted according to the loss function to obtain a trained emoji prediction model.

[0048] Optionally, the emoji prediction model includes an image feature extraction model and a text feature extraction model. The step of using sample images from the training samples and the annotation information of the sample images as input to the emoji prediction model to obtain the feature vectors of the sample images and the feature vectors of the annotation information of the sample images includes:

[0049] The image feature extraction model takes the sample images from the training samples as input and outputs the feature vectors of the sample images.

[0050] The text feature extraction model takes the annotation information of the sample image as input and outputs the feature vector of the annotation information of the sample image.

[0051] Thirdly, embodiments of this application provide an electronic device, including: a processor and a memory, the memory being used to store a computer program, and the processor being used to call and run the computer program stored in the memory to perform the methods as described in the first aspect or its various implementations.

[0052] Fourthly, embodiments of this application provide a computer-readable storage medium for storing a computer program that causes a computer to perform the methods described in the first aspect or its various implementations.

[0053] Fifthly, embodiments of this application provide a computer program product including computer program instructions that cause a computer to perform the methods as described in the first aspect or its various implementations.

[0054] In summary, in this embodiment, an emoji prediction model is trained based on multiple training samples. Each training sample includes a sample image and its annotation information. The annotation information includes text annotations and labeled emojis. The emoji prediction model can learn the correspondence between an image, its text annotations, and the labeled emojis. Therefore, after obtaining an image and its text annotations, at least one predicted emoji can be determined based on the image, its text annotations, and the pre-trained emoji prediction model, and displayed. This facilitates users selecting and typing the corresponding emoji from the predicted emojis when transcribing image annotations (e.g., copying or cutting and pasting a displayed predicted emoji), solving the problem of emojis being difficult to type using a keyboard. Thus, it enables the prediction of emojis contained in the image annotations, ensuring the integrity of the image annotation transcription.

[0055] Furthermore, in the method of this application embodiment, in response to the user's operation of selecting a target emoji from at least one predicted emoji in the annotation information, the target text information and the target emoji are merged according to the distribution order of the annotation information to obtain the target annotation information and display it. For images whose annotation information includes emojis, the annotation information of such images can be directly and efficiently transcribed completely, solving the problem that emojis are difficult to correctly type using a keyboard input method and realizing the usability of emoji input. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 This is a schematic diagram illustrating an application scenario of an emoji prediction method provided in an embodiment of this application.

[0058] Figure 2 A flowchart illustrating an emoji prediction method provided in this application embodiment;

[0059] Figure 3 A flowchart illustrating a method for training an emoji prediction model provided in this application embodiment;

[0060] Figure 4 This is a schematic diagram of an emoji prediction interface provided in an embodiment of this application;

[0061] Figure 5This is a schematic diagram of the structure of an emoji prediction device provided in an embodiment of this application;

[0062] Figure 6 This is a schematic block diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0063] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0064] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0065] Before introducing the technical solution of this application, the following is a brief introduction to relevant knowledge about this application:

[0066] 1. Artificial Intelligence (AI) is the theory, methods, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making functions. AI technology is a comprehensive discipline involving a wide range of fields, including both hardware and software technologies. Basic AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, and mechatronics. AI software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning. The technical solutions provided in this application mainly involve natural language processing technology and machine learning / deep learning in artificial intelligence.

[0067] 2. Computer Vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and further processes the images to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0068] In related technologies, for images whose annotation information includes text information and emoticons, when transcribing the annotation information of such images, due to the rich variety and numerous versions of emoticons, it is difficult to output the corresponding emoticons through keyboard input methods. Often, only the text information in the image is transcribed, ignoring the emoticons, resulting in incomplete transcribed information.

[0069] To address this technical problem, this application embodiment trains an emoji prediction model based on multiple training samples. Each training sample includes a sample image and its annotation information. The annotation information includes text annotations and labeled emojis. The emoji prediction model learns the correspondence between an image, its text annotations, and the labeled emojis. Therefore, after obtaining an image and its text annotations, at least one predicted emoji can be determined based on the image, its text annotations, and the pre-trained emoji prediction model, and displayed. This facilitates users selecting and typing the corresponding emoji from the predicted emojis when transcribing image annotations (e.g., copying or cutting and pasting a displayed predicted emoji), solving the problem of difficulty in typing emojis using a keyboard. Thus, it enables the prediction of emojis contained in the image annotations, ensuring the integrity of the image annotation transcription.

[0070] Furthermore, in the method of this embodiment, in response to the user's operation of selecting a target emoji from at least one predicted emoji in the annotation information, the target text information and the target emoji are merged according to the distribution order of the annotation information to obtain the target annotation information and display it. For images whose annotation information includes emojis, the annotation information of such images can be directly and efficiently transcribed completely, solving the problem that emojis are difficult to type correctly using a keyboard input method and realizing the usability of emoji input.

[0071] It should be understood that the technical solution of this application can be applied to the following scenarios, but is not limited to:

[0072] For example, Figure 1 This is a schematic diagram illustrating an application scenario of an emoji prediction method provided in an embodiment of this application, such as... Figure 1 As shown, this application scenario involves a terminal device 110 and a server 120, and the terminal device 110 can communicate with the server 120.

[0073] In some possible ways, Figure 1 The application scenarios shown can also include: base stations, core network side equipment, etc., in addition, Figure 1 An exemplary terminal device and a server are shown, but in practice, other numbers of terminal devices and servers may be included, and this application does not limit this.

[0074] In some possible ways, Figure 1The server 120 in this application can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. This application does not impose any limitations on these aspects.

[0075] In some possible implementations, such as Figure 1 The terminal device 110 shown can have an application client installed. When the application client runs on the terminal device, it can interact with the server 120. Specifically, the client may include, for example, an in-vehicle client, a smart home client, a game client, a multimedia client (such as a video client), a social networking client, and an information client (such as a news client).

[0076] Optionally, in this embodiment, the terminal device 110 can be a device with rich human-computer interaction methods, internet access capabilities, typically running various operating systems, and possessing strong processing capabilities. The terminal device 110 can be a smartphone, smart TV, tablet computer, vehicle terminal, etc., but is not limited to these.

[0077] In one possible implementation, the server 120 and the terminal device 110 can interactively execute the emoji prediction method provided in the embodiments of this application, or the terminal device 110 can execute the emoji prediction method provided in the embodiments of this application.

[0078] During the training phase of the emoji prediction model, the server 120 or the terminal device 110 uses the training method provided in the embodiments of this application to train the emoji prediction model.

[0079] In the emoji prediction stage, in one feasible manner, the user can upload a target image and type target text information from the annotation information of the target image through a client, browser client or instant messaging client installed on the terminal device 110. After the terminal device obtains the target image and target text information, it determines at least one predicted emoji from the annotation information based on the target image, target text information and pre-trained emoji prediction model, and displays at least one predicted emoji from the annotation information.

[0080] In another possible implementation, the emoji prediction method provided in this application embodiment can also be executed interactively between the terminal device 110 and the server 120. For example, a user can upload a target image and type target text information from the annotation information of the target image through a client, browser client, or instant messaging client installed on the terminal device 110. After the terminal device 110 obtains the target image and target text information, it sends the target image and target text information to the server 120. The server 120 determines at least one predicted emoji from the annotation information based on the target image, target text information, and a pre-trained emoji prediction model, and sends the at least one predicted emoji from the annotation information to the terminal device for display.

[0081] It should be noted that the training and prediction processes of the emoji prediction model can be completed on a server or on a terminal device. Optionally, the pre-trained model file (model file) can be ported to the terminal device. If emoji prediction is required, the target image and target text information are input into the pre-trained model file (model file), and running the model file will yield at least one predicted emoji based on the annotation information of the target image.

[0082] The emoji prediction method provided in this application can be applied to the transcription of complete information of images whose annotation information includes emojis. It solves the problem that emojis are difficult to type correctly using a keyboard input method when transcribing information, and realizes the usability of emoji input.

[0083] The technical solution of this application will be described in detail below:

[0084] Figure 2 A flowchart illustrating an emoji prediction method provided in this application embodiment is provided. This method can be, for example, derived from... Figure 1 The terminal device 110 shown executes the commands, but is not limited to this, such as... Figure 2 As shown, the method may include the following steps:

[0085] S101. Obtain the target image and target text information. The target image is an image with annotation information, which includes target text information and emoticons.

[0086] Specifically, in the embodiments of the present application, the target image is an image with annotation information. The annotation information can be, for example, the annotation information edited by the user on the image, which not only includes the target text information but also includes emoji. Among them, emoji are the emoji used by current Internet users and are a unique Internet language. Emoji can be used to express love, gratitude, congratulations, etc. Emoji are mostly used in instant messaging conversations. Emoji are used to vividly present and depict non-verbal information in daily face-to-face communication, making both parties seem to hear the voice and see the person. For example, emoji such as "Come on - ", "Cute - ", and "Covering the face - ", etc.

[0087] Exemplarily, for example, the annotation information edited by the user on a landscape image is "So beautiful! ", and for another example, the annotation information edited by the user on an image of a dancer dancing is "This dance ", or the annotation information is "This dance ". It can be seen that the annotation information "This dance " and "This dance " express different semantics.

[0088] In an implementable manner, obtaining the target image and the target text information can be specifically:

[0089] Receiving the input target image and target text information.

[0090] Among them, the target image can be uploaded by the user after entering the image address, and the target text information can be typed by the user.

[0091] S102. According to the target image, the target text information, and a pre-trained emoji prediction model, determine at least one predicted emoji of the annotation information. The emoji prediction model is trained according to multiple training samples. Each training sample includes a sample image and the annotation information of the sample image. The annotation information of the sample image includes text annotation information and annotated emoji.

[0092] Specifically, the emoji prediction model is trained according to multiple training samples. Each training sample includes a sample image and the annotation information of the sample image. The annotation information of the sample image includes text annotation information and annotated emoji. Through model training, the emoji prediction model can learn the correspondence between the image, the text annotation information of the image, and the annotated emoji of the image. Thus, after obtaining the image and the text annotation information of the image, at least one predicted emoji of the annotation information of the image can be determined according to the image, the text annotation information of the image, and the pre-trained emoji prediction model.

[0093] Optionally, the emoji prediction model in this application embodiment can be a transform model based on the prompt paradigm, or it can be a model using other machine learning algorithms, such as a hidden Markov model (HMM).

[0094] Optionally, based on the target image, target text information, and a pre-trained emoji prediction model, at least one predicted emoji from the labeled information can be determined. Specifically, this can be:

[0095] The target image and target text information are combined to form a prompt paradigm input to the emoji prediction model, and the output is at least one predicted emoji with annotation information. The content to be filled in the prompt paradigm structure is the emoji to be predicted.

[0096] Specifically, Prompt essentially transforms task data, converting the original target and labels and incorporating them into the data. This transformation allows the task's objectives to be achieved through natural self-supervised learning. In this embodiment, it specifically transforms the input target image and target text information.

[0097] For example, the target image and target text information are combined to form a prompt paradigm. The prompt paradigm can be: target image + target text information + the emoji to be transcribed here, which may be [emoji], where the content in [] is the content to be filled. In this embodiment, the content to be filled is the emoji to be predicted.

[0098] Optionally, the prompt paradigm in this embodiment can also be: target image + target text information + the emoji to be transcribed here, which may be [emoji 1], [emoji 2], or [emoji 3]. Other forms are also possible, and this embodiment does not limit them.

[0099] In one feasible approach, the emoji prediction model includes an image feature extraction model and a text feature extraction model. The target image and target text information are input into the emoji prediction model in a prompt paradigm, and the model outputs at least one predicted emoji with annotation information. Specifically, this can be:

[0100] S1021. Input the target image into the image feature extraction model and output the feature vector of the target image.

[0101] Optionally, the image feature extraction model can be a Vision Transformer (vit) or a deep residual network (resnet50). The image feature extraction model is used to extract feature vectors from the target image.

[0102] S1022. The target text information and each emoji in the emoji library are merged sequentially according to the prompt paradigm structure to obtain candidate information.

[0103] Specifically, the target text information and each emoji in the emoji library are sequentially merged according to the prompt paradigm structure to obtain candidate information. The resulting candidate information can be: "target text information + the emoji to be transcribed here may be [candidate emoji]", where the candidate emoji is any emoji in the emoji library. For example, if there are 200 emojis in the emoji library, then the candidate information obtained through S1022 will be 200.

[0104] S1023. Input each candidate information into the text feature extraction model in sequence, and output the feature vector of the candidate information.

[0105] The text feature extraction model can be a Bidirectional Encoder Representations from Transformer (BERT) network model, which is a pre-trained language representation model. Specifically, after the candidate information is input into the text feature extraction model, it undergoes word segmentation and then the text feature extraction model extracts text feature vectors to obtain the feature vectors of the candidate information. Optionally, emoticons can be used as a word segment after word segmentation.

[0106] S1024. Calculate the cosine similarity between the feature vector of the target image and the feature vector of the candidate information respectively. Select the candidate information corresponding to the first K cosine similarities in descending order of cosine similarity. Output the emojis in the K candidate information as at least one predicted emoji of the annotation information. K is a positive integer.

[0107] Specifically, based on the cosine similarity scores from highest to lowest, the top K candidate information points with the highest cosine similarity scores are selected, and the emojis from these K candidate information points are output as at least one predicted emoji in the annotation information. K can be 1 or a value greater than 1.

[0108] S103, Display at least one predicted emoji with annotation information.

[0109] Furthermore, after S103, the method of this embodiment may further include:

[0110] S104. In response to the user's operation of selecting a target emoji from at least one predicted emoji in the annotation information, the target text information and the target emoji are merged according to the distribution order of the annotation information to obtain the target annotation information.

[0111] Specifically, the distribution order of the annotation information refers to the distribution order of the target text information and the emoji.

[0112] S105, Display target annotation information.

[0113] The emoji prediction method provided in this embodiment trains an emoji prediction model based on multiple training samples. Each training sample includes a sample image and its annotation information. The annotation information includes text annotations and labeled emojis. The emoji prediction model learns the correspondence between the image, the image's text annotations, and the labeled emojis. Therefore, after obtaining the image and its text annotations, at least one predicted emoji can be determined based on the image, its text annotations, and the pre-trained emoji prediction model, and then displayed. This facilitates users selecting and typing the corresponding emoji from the predicted emojis when transcribing image annotations (e.g., copying or cutting and pasting a displayed predicted emoji), solving the problem of emojis being difficult to type using a keyboard. Thus, it enables the prediction of emojis contained in the image's annotation information, ensuring the integrity of the image annotation transcription.

[0114] Figure 2 In the illustrated embodiment, the emoji prediction model is pre-trained based on multiple training samples. The following section will combine... Figure 3 This application illustrates a training method for an emoji prediction model. In this embodiment, the emoji prediction method can be trained by… Figure 3 The method shown is used for training.

[0115] Figure 3 A flowchart illustrating a method for training an emoji prediction model provided in this application embodiment. This method can be, for example, by... Figure 1 The terminal device 110 or server 120 shown may perform this action, but it is not limited to this, such as... Figure 3 As shown, the method may include the following steps:

[0116] S201. Obtain N training samples. Each training sample includes a sample image and its annotation information. The annotation information of the sample image includes text annotation information and annotation of emoticons.

[0117] Specifically, the annotation information for the sample images can be image-related annotation information, including text annotation information and annotations of emoticons. The annotation information can be text containing emoticons selected from a large amount of online text. This embodiment does not impose any limitations on this. The sample images can be online pictures or images taken by users; this embodiment does not impose any limitations on this either.

[0118] S202. For each training sample among the N training samples, take the sample image, the text annotation information of the sample image, and the annotated emojis of the sample image as input to the emoji prediction model to obtain the feature vector of the sample image and the feature vector of the annotation information of the sample image.

[0119] Specifically, for each of the N training samples, the feature vector of the sample image and the feature vector of the annotation information of the sample image are obtained through S202, and finally the feature vectors of the N sample images and the feature vectors of the annotation information of the N sample images are obtained.

[0120] Optionally, the emoji prediction model includes an image feature extraction model and a text feature extraction model. In step S202, the sample images and their annotation information from the training samples are used as inputs to the emoji prediction model to obtain the feature vectors of the sample images and the feature vectors of the annotation information. Specifically, this can be:

[0121] S2021. The image feature extraction model takes the sample images in the training samples as input and outputs the feature vector of the sample images.

[0122] Optionally, the image feature extraction model can be a Vision Transformer (vit) or a deep residual network (resnet50). The image feature extraction model is used to extract feature vectors from sample images.

[0123] S2022. The text feature extraction model takes the annotation information of the sample image as input and outputs the feature vector of the annotation information of the sample image.

[0124] Optionally, the text feature extraction model can be a Bidirectional Encoder Representations from Transformer (BERT) network model, which is a pre-trained language representation model. The text feature extraction model is used to extract feature vectors from the labeled information of sample images.

[0125] S203. Construct a loss function based on the cosine similarity between the feature vectors of N sample images and the feature vectors of the labeled information of N sample images.

[0126] For example, in this embodiment, the N training samples include N sample image-annotation pairs, where each annotation includes an emoji. For instance, the output of the image feature extraction model for the N sample images is (I1, I2, ..., I...). N ), I NLet be the feature vector of the sample image, with dimensions (N, d). i The annotation information of N sample images is output by the text feature extraction model as (T1, T2, ..., T...). N ), T N The feature vector of the labeled information of the sample image has dimensions (N, d). t ), resulting in (I1, I2, ..., I N ) and (T1, T2, ..., T N In this model, the annotation information of each sample image is in a one-to-one correspondence. For example, I1 corresponds to T1 and belongs to the same training sample; I2 corresponds to T2 and belongs to the same training sample; and so on. N With T N Correspondingly, they belong to the same training sample. However, I1 and T2 do not belong to the same training sample. Thus, there are N pairs of feature vectors and labeled feature vectors belonging to the same training sample, and N 2 -N feature vector pairs of sample images that do not belong to the same training sample and the feature vector pairs of the annotation information.

[0127] Calculate the cosine similarity between the feature vectors of N sample images and the feature vectors of the labeled information of N sample images, i.e., calculate I. i With T j The cosine similarity between them, according to I i With T j The loss function is constructed using the cosine similarity between the two values.

[0128] S204. Adjust the parameters of the emoji prediction model according to the loss function to obtain the trained emoji prediction model.

[0129] Specifically, the parameters of the emoji prediction model are adjusted according to the loss function to obtain an optimized loss function. The optimized loss function maximizes the cosine similarity between the feature vectors of N sample images belonging to the same training sample and the feature vectors of the labeled information, and minimizes N. 2 - The cosine similarity between the feature vectors of N sample images that do not belong to the same training sample and the feature vectors of the labeled information.

[0130] For example, the optimized loss function can be represented by the following formula:

[0131]

[0132] In the above formula, I i (i = 1…N) are the feature vectors of the sample images, T j (j=1….N) is the feature vector of the annotation information of the sample image, I i ·T j For Ii and T j The cosine similarity between them, as expressed in the formula above, is N. 2 The goal is to minimize the sum of the cosine similarities between the feature vectors of N sample images that do not belong to the same training sample and the feature vectors of their labeled information, minus the sum of the cosine similarities between the feature vectors of N sample images that belong to the same training sample and the feature vectors of their labeled information. This ensures that the optimized loss function maximizes the cosine similarity between the feature vectors of N sample images that belong to the same training sample and the feature vectors of their labeled information, while minimizing N. 2 - The cosine similarity between the feature vectors of N sample images that do not belong to the same training sample and the feature vectors of the labeled information.

[0133] The emoji prediction model training method provided in this embodiment trains the emoji prediction model based on multiple training samples. Each training sample includes a sample image and its annotation information. The annotation information of the sample image includes text annotation information and labeled emojis. The emoji prediction model can learn the correspondence between the image, the text annotation information of the image, and the labeled emojis of the image. Thus, the trained emoji prediction model can predict the emojis contained in the annotation information of the image, which facilitates the integrity of the annotation information transcription when transcribing the annotation information.

[0134] The technical solution of the present application embodiment will be described in detail below with reference to a specific example.

[0135] Figure 4 This is a schematic diagram of an emoji prediction interface provided in an embodiment of this application. Users can upload a target image and type in the target text information from the annotation information of the target image through a client, browser client, or instant messaging client installed on the terminal device 110. Before uploading the target image and typing in the target text information from the annotation information of the target image, the user opens this emoji prediction interface through the terminal device. Combined with... Figure 4 The emoji prediction method provided in this embodiment may include:

[0136] S301. Receive the input target image and target text information. The target image is an image with annotation information, which includes target text information and emoticons.

[0137] Specifically, the execution subject of the method in this embodiment can be a terminal device, and the user can access it through... Figure 4 The emoji prediction interface, as shown, displays the target image's address upon inputting it. Clicking the "Show" button will then display the emoji on the current screen. Figure 4 The target image shown has the annotation "this jump". The annotation includes the target text "this jump" and an emoji. Users can Figure 4 Enter the target text "This jump" in the text input box shown, and then click the emoji recommendation button. The terminal device will then receive the entered target image and target text information.

[0138] S302. Input the target image into the image feature extraction model and output the feature vector of the target image.

[0139] S303. The target text information and each emoji in the emoji library are merged sequentially according to the prompt paradigm structure to obtain candidate information.

[0140] S304. Input each candidate information into the text feature extraction model in sequence, and output the feature vector of the candidate information.

[0141] S305. Calculate the cosine similarity between the feature vector of the target image and the feature vector of the candidate information respectively. Select the candidate information corresponding to the first K cosine similarities in descending order of cosine similarity, and output the emojis in the K candidate information as the predicted emojis of the labeled information.

[0142] like Figure 4 As shown, the predicted emojis with the annotation information are 5.

[0143] S306. Display predicted emojis with annotation information.

[0144] S307. In response to the user's operation of selecting a target emoji from at least one predicted emoji in the annotation information, the target text information and the target emoji are merged according to the distribution order of the annotation information to obtain the target annotation information.

[0145] Specifically, such as Figure 4 As shown, the displayed annotation information predicts five emojis, one of which is an emoji. The target emoji is selected by the user. The target text information and the target emoji are merged according to the distribution order of the annotation information, resulting in the target annotation information "This jump...". This allows for the complete transcription of the image's annotation information.

[0146] S308, Display target annotation information.

[0147] Figure 5 This is a schematic diagram of the structure of an emoji prediction device provided in an embodiment of this application, as shown below. Figure 5 As shown, the emoji prediction device may include: an acquisition module 11, a determination module 12, and a display module 13.

[0148] The acquisition module 11 is used to acquire target image and target text information. The target image is an image with annotation information, which includes target text information and emoticons.

[0149] The determining module 12 is used to determine at least one predicted emoji based on the target image, target text information and a pre-trained emoji prediction model, wherein the emoji prediction model is trained based on multiple training samples, each training sample includes a sample image and the annotation information of the sample image, the annotation information of the sample image includes text annotation information and labeled emoji, and N is a positive integer.

[0150] Display module 13 is used to display at least one predicted emoji with annotation information.

[0151] Optionally, the display module 13 is also used for:

[0152] In response to a user's action of selecting a target emoji from at least one predicted emoji in the annotation information, the target text information and the target emoji are merged according to the distribution order of the annotation information to obtain the target annotation information;

[0153] Display target annotation information.

[0154] Optionally, module 11 is used for:

[0155] Receive the input target image and target text information.

[0156] Optionally, module 12 is used to:

[0157] The target image and target text information are combined to form a prompt paradigm input to the emoji prediction model, and the output is at least one predicted emoji with annotation information. The content to be filled in the prompt paradigm structure is the emoji to be predicted.

[0158] Optionally, the emoji prediction model includes an image feature extraction model and a text feature extraction model, and the determination module 12 is used for:

[0159] The target image is input into the image feature extraction model, which outputs the feature vector of the target image.

[0160] The target text information and each emoji in the emoji library are merged sequentially according to the prompt paradigm structure to obtain candidate information;

[0161] Each candidate information is sequentially input into the text feature extraction model, which outputs the feature vector of the candidate information.

[0162] Calculate the cosine similarity between the feature vector of the target image and the feature vector of the candidate information respectively. Select the candidate information corresponding to the first K cosine similarity values ​​in descending order of cosine similarity. Output the emojis in the K candidate information as at least one predicted emoji in the annotation information, where K is a positive integer.

[0163] Optionally, the emoji prediction model is trained using N training samples in the following manner:

[0164] For each of the N training samples, the sample image, the text annotation information of the sample image, and the annotated emojis of the sample image are used as inputs to the emoji prediction model to obtain the feature vector of the sample image and the feature vector of the annotation information of the sample image.

[0165] A loss function is constructed based on the cosine similarity between the feature vectors of N sample images and the feature vectors of the labeled information of N sample images;

[0166] The parameters of the emoji prediction model are adjusted according to the loss function to obtain the trained emoji prediction model.

[0167] Optionally, the emoji prediction model includes an image feature extraction model and a text feature extraction model. The model takes sample images and their annotations from the training samples as input and outputs feature vectors of the sample images and their annotations, including:

[0168] The image feature extraction model takes the sample images from the training samples as input and outputs the feature vectors of the sample images.

[0169] The text feature extraction model takes the annotation information of the sample image as input and outputs the feature vector of the annotation information of the sample image.

[0170] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details will not be provided here. Specifically, Figure 5 The emoji prediction device shown can perform... Figure 2 The corresponding method embodiments, and the foregoing and other operations and / or functions of each module in the emoji prediction device are respectively implemented to achieve Figure 2 For the sake of brevity, the corresponding processes in the method embodiments will not be described in detail here.

[0171] The emoji prediction model training device of this application embodiment has been described above from the perspective of functional modules with reference to the accompanying drawings. It should be understood that this functional module can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the method embodiments in this application can be completed by the integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the method disclosed in this application embodiment can be directly manifested as being executed by a hardware encoding processor, or by a combination of hardware and software modules in the encoding processor. Optionally, the software module can be located in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps in the above method embodiments.

[0172] Figure 6 This is a schematic block diagram of an electronic device provided in an embodiment of this application. The electronic device may be a server as described in the above method embodiments.

[0173] like Figure 6 As shown, the electronic device may include:

[0174] The system includes a memory 210 and a processor 220. The memory 210 stores computer programs and transfers the program code to the processor 220. In other words, the processor 220 can retrieve and run the computer program from the memory 210 to implement the methods described in the embodiments of this application.

[0175] For example, the processor 220 can be used to execute the above-described method embodiments according to instructions in the computer program.

[0176] In some embodiments of this application, the processor 220 may include, but is not limited to:

[0177] General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0178] In some embodiments of this application, the memory 210 includes, but is not limited to:

[0179] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0180] In some embodiments of this application, the computer program may be divided into one or more modules, which are stored in the memory 210 and executed by the processor 220 to perform the method provided in this application. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device.

[0181] like Figure 6 As shown, the electronic device may also include:

[0182] Transceiver 230, which can be connected to processor 220 or memory 210.

[0183] The processor 220 can control the transceiver 230 to communicate with other devices; specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 230 may include a transmitter and a receiver. The transceiver 230 may further include antennas, and the number of antennas may be one or more.

[0184] It should be understood that the various components in the electronic device are connected through a bus system, which includes a data bus, a power bus, a control bus, and a status signal bus.

[0185] This application also provides a computer storage medium storing a computer program thereon, which, when executed by a computer, enables the computer to perform the methods of the above-described method embodiments. Alternatively, embodiments of this application also provide a computer program product containing instructions that, when executed by a computer, cause the computer to perform the methods of the above-described method embodiments.

[0186] When implemented using software, it can be implemented entirely or partially as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0187] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0188] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0189] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0190] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for predicting emojis, characterized in that, include: Acquire target image and target text information, wherein the target image is an image with annotation information, and the annotation information includes the target text information and emoticons; Based on the target image, the target text information, and a pre-trained emoji prediction model, at least one predicted emoji is determined from the labeled information. The emoji prediction model is trained based on multiple training samples, each of which includes a sample image and the labeled information of the sample image. The labeled information of the sample image includes text labeled information and labeled emojis. Display at least one predicted emoji of the labeled information; The step of determining at least one predicted emoji based on the labeled information according to the target image, the target text information, and a pre-trained emoji prediction model includes: The target image and the target text information are combined into a prompt paradigm and input into the emoji prediction model. The model outputs at least one predicted emoji based on the labeled information. The content to be filled in the prompt paradigm structure is the emoji to be predicted. The emoji prediction model includes an image feature extraction model and a text feature extraction model. The target image and the target text information are combined into a prompt paradigm and input into the emoji prediction model. The model outputs at least one predicted emoji based on the labeled information, including: The target image is input into the image feature extraction model, and the model outputs the feature vector of the target image. The target text information and each emoji in the emoji library are sequentially merged according to the prompt paradigm structure to obtain candidate information; Each candidate information is sequentially input into the text feature extraction model, and the feature vector of the candidate information is output. Calculate the cosine similarity between the feature vector of the target image and the feature vector of the candidate information respectively. Select the candidate information corresponding to the first K cosine similarities in descending order of the cosine similarity. Output the emojis in the K candidate information as at least one predicted emoji of the labeled information, where K is a positive integer.

2. The method according to claim 1, characterized in that, The method further includes: In response to a user's operation of selecting a target emoji from at least one predicted emoji in the annotation information, the target text information and the target emoji are merged according to the distribution order of the annotation information to obtain target annotation information; Display the target annotation information.

3. The method according to claim 1, characterized in that, The acquisition of target image and target text information includes: Receive the input target image and target text information.

4. The method according to any one of claims 1-3, characterized in that, The emoji prediction model is trained using N training samples in the following manner: For each of the N training samples, the sample image, the text annotation information of the sample image, and the annotated emojis of the sample image are used as inputs to the emoji prediction model to obtain the feature vector of the sample image and the feature vector of the annotation information of the sample image. A loss function is constructed based on the cosine similarity between the feature vectors of N sample images and the feature vectors of the labeled information of the N sample images; The parameters of the emoji prediction model are adjusted according to the loss function to obtain a trained emoji prediction model.

5. The method according to claim 4, characterized in that, The emoji prediction model includes an image feature extraction model and a text feature extraction model. The step of using sample images from the training samples and their annotation information as input to the emoji prediction model to obtain feature vectors of the sample images and feature vectors of the annotation information includes: The image feature extraction model takes the sample images from the training samples as input and outputs the feature vectors of the sample images. The text feature extraction model takes the annotation information of the sample image as input and outputs the feature vector of the annotation information of the sample image.

6. An emoji prediction device, characterized in that, include: The acquisition module is used to acquire a target image and target text information, wherein the target image is an image with annotation information, and the annotation information includes the target text information and emoticons; The determining module is configured to determine at least one predicted emoji based on the target image, the target text information, and a pre-trained emoji prediction model, wherein the emoji prediction model is trained based on multiple training samples, each training sample including a sample image and the annotation information of the sample image, the annotation information of the sample image including text annotation information and labeled emojis; A display module is used to display at least one predicted emoji of the annotation information; The determining module is specifically used to input the target image and the target text information into the emoji prediction model in a prompt paradigm, and output at least one predicted emoji of the labeled information, wherein the content to be filled in the prompt paradigm structure is the emoji to be predicted. The emoji prediction model includes an image feature extraction model and a text feature extraction model; The determining module is further specifically used to input the target image into the image feature extraction model and output the feature vector of the target image; sequentially merge the target text information and each emoji in the emoji library according to the prompt paradigm structure to obtain candidate information; sequentially input each candidate information into the text feature extraction model and output the feature vector of the candidate information; calculate the cosine similarity between the feature vector of the target image and the feature vector of the candidate information respectively; select the candidate information corresponding to the first K cosine similarities in descending order of the cosine similarity; and output the emojis in the K candidate information as at least one predicted emoji of the labeled information, where K is a positive integer.

7. An electronic device, characterized in that, include: A processor and a memory, the memory being used to store a computer program, the processor being used to invoke and run the computer program stored in the memory to perform the method of any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, Includes instructions that, when executed on a computer program, cause the computer to perform the method as described in any one of claims 1 to 5.

Images