Emoticon package selection and text generation method based on prompt learning and multi-modal model

Abstract

The application discloses an emoticon package selection and text generation method based on prompt learning and a multi-modal model, and comprises the following steps: S1, intercepting a big model agent and a user's recent round of dialogue, and compressing the dialogue into a dialogue prompt mark; S2, arranging an emoticon package set into a candidate emoticon package input set graph and pairing the candidate emoticon package input set graph into an emoticon package reasoning sample; S3, accessing an improved RT-DETR model to extract and fuse multi-scale visual features; S4, an object query group organization structure is truncated, independently detected, and inquired into an entry, and is grouped into a query group; S5, a Transformer decoder with a query group cross-layer inheritance structure forms a candidate emoticon package to undertake a query; S6, a prediction head reads out a path constraint candidate reply text generation; and S7, after dialogue closure verification, emoticon package images and reply texts are output. The application relates to the technical field of artificial intelligence multi-modal content processing, and realizes emoticon package selection and reply text generation collaborative processing.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence multimodal content processing technology, and in particular to a method for emoji selection and text generation based on cue learning and multimodal models. Background Technology

[0002] Existing intelligent interactive devices typically generate text responses using large-scale model agents and select emojis or emoticons through rule matching, emotion classification, or tool invocation. While this approach can enhance the interactive experience, the selection of emoticons largely relies on emotion tags or image-text similarity, making it difficult to uniformly handle the relationship between the user's expected response, the tone of the reply, the emoticon image area, and the generated text. This can easily lead to a disconnect between the emoticon and the reply text.

[0003] In the scenario of intelligent AI toys with screen display capabilities, if a single large model is used to handle both chat replies and emoji selection, the emoji display will consume additional context and reasoning steps, increasing response latency. If tool calls or ordinary multimodal retrieval methods are used, emojis are easily treated as independent candidate results, lacking a stable connection between emoji selection, text generation, and image-text closure verification.

[0004] Furthermore, existing object detection or multimodal models typically feed visual queries as independent detection objects into the decoder and directly read out the category and bounding box at the prediction head. They do not organize queries within the same candidate emoji within a group or inherit them across layers, making it difficult to utilize query location, region boundaries, and inter-layer update state constraints to generate response text.

[0005] Therefore, how to provide emoji selection and text generation methods based on cue learning and multimodal models is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] One objective of this invention is to propose a method for emoji selection and text generation based on cue learning and a multimodal model. This invention combines response cue markers with an improved RT-DETR model. It compresses the content of the most recent dialogue rounds to form the response object and reply tone, and organizes the emoji set into a candidate emoji input set graph with corresponding positions and numbers within the graph. Based on this, it introduces an object query group organization structure and a Transformer decoder with a cross-layer inheritance structure for query groups. Independent detection query sets are rewritten into query groups within the same candidate emoji region, maintaining the arrangement and update status within each group across decoder layers, forming candidate emoji acceptance queries. Furthermore, it truncates the ordinary detection box readout path by predicting the head-front readout path, aligning the candidate emoji acceptance queries with the response cue markers and constraining the generation of candidate reply text. Finally, it filters the emoji image and reply text combination through response closure verification. This method has the advantages of accurate emoji selection, stable image-text integration, low response latency, and suitability for real-time emoji display on intelligent interactive devices.

[0007] The emoji selection and text generation method based on cue learning and multimodal models according to embodiments of the present invention includes the following steps: S1. Extract the most recent rounds of dialogue from the current dialogue between the large model agent and the user, and compress them into dialogue prompts with the target of the conversation and the tone of the reply. S2. Organize the emoji set into a candidate emoji input set graph, and pair the candidate emoji input set graph with the reply prompts to form emoji inference samples; S3. An improved RT-DETR model is used to access emoji inference samples. A convolutional backbone network and an efficient hybrid encoder extract and fuse multi-scale visual features from the candidate emoji input set graph. S4, IoU perception query selection extracts object queries from multi-scale visual features. The object query group organization structure truncates the independent detection query tiling entry and groups the object queries in the same candidate emoji pack into query groups according to the adjacent arrangement relationship in the candidate emoji pack input set graph. S5. The Transformer decoder with query group cross-layer inheritance structure receives query groups. The object queries in each group output by each decoder layer are fed back to the next decoder layer according to the original group position, and candidate emojis are formed to accept queries along the inter-layer inheritance group arrangement relationship and update status. S6. The candidate emoji pack is truncated before the reading path of the prediction head. The query enters the reading path of the classification prediction head and the bounding regression head. The final layer position, region boundary and inter-layer update status of the query group are preserved. The emoji pack is registered with the answer prompt mark to constrain the generation of candidate reply text. S7. Perform a closed-loop verification on the candidate reply text and candidate emojis, block candidate combinations with broken loops, retain closed candidate combinations, and output emoji images and reply text.

[0008] Optionally, S1 specifically includes: S11. Extract the most recent rounds of dialogue from the end of the current dialogue between the large model agent and the user, remove system control statements, and attach the remaining speech to the large model agent side and the user side to form a role-aligned dialogue segment. S12. Align the speaking order of the dialogue segment along the role and compare the user-side terminal speech with the adjacent speech on the large model agent side. Subtract the semantic content that has been taken over by the adjacent speech on the large model agent side to form the response segment. S13. Based on the core predicate in the response segment, retain the object component that is directly connected to the core predicate, and delete the modifier component that does not change the direction of the response to form the response object. S14. Merge polar expressions and sentence-final intonation in the response segment, and use sentence-final intonation to correct the direction of the response of polar expressions, forming the response tone; S15. Merge and compress the respondent and the reply tone in the order of respondent first and tone second to form a response prompt mark.

[0009] Optionally, S2 specifically includes: S21. Using the theme affiliation in the emoji set as the sorting entry point, group emoji images under the same theme into theme candidate image groups; S22. Crops the emoji images in the candidate image group of the theme to the display boundary and arranges them in the grid area according to the display order of the set to form a theme input set map with the position in the map; S23. Align the grid areas in the topic input set diagram with the serial number markers one by one to form a candidate emoji input set diagram with the correspondence between the position and serial number in the diagram. S24. Pair the candidate emoji input set graph with the answer prompt markers in the same round, retain the position and sequence number markers in the candidate emoji input set graph, and compress them into emoji inference samples.

[0010] Optionally, S3 specifically includes: S31, The improved RT-DETR model includes a convolutional backbone network, an efficient hybrid encoder, an IoU-aware query selection, an object query group organization structure, a Transformer decoder with a cross-layer inheritance structure of query groups, a prediction head readout path, and a prediction head; S32. The candidate emoji input set graph in the emoji inference sample is connected to the convolutional backbone network. The convolutional backbone network performs progressive convolution and downsampling on the candidate emoji input set graph, and retains the correspondence between the position in the graph and the serial number label in each level of feature to form multi-scale candidate visual features. S33. The high-efficiency hybrid encoder performs same-scale feature interaction on multi-scale candidate visual features, compressing the boundaries of emoji images, grid region boundaries and local semantic content within the same scale into same-scale encoded features. S34. The efficient hybrid encoder performs cross-scale fusion of the same-scale encoded features, aligns the shallow boundary features and deep semantic features along the in-graph position of the candidate emoji input set graph, and forms multi-scale visual features for IoU-aware query selection to extract object queries.

[0011] Optionally, S4 specifically includes: S41. IoU perceptual query selection performs candidate quality readout on the encoded feature positions in multi-scale visual features, selects encoded feature positions and region boundaries in order of candidate quality, and forms a set of object queries to be organized. S42. The object queries query group organization structure is truncated. The set of object queries to be organized is directly flattened into the entry point of the Transformer decoder. The position of the encoded feature carried by each object query is matched with the position and sequence number of the candidate emoji input set graph to form position-matched object queries. S43. The object queries query group organization structure verifies the location of object queries within the image. Object queries falling within the same candidate emoji image range are grouped into the same candidate emoji internal set. Object queries that cross adjacent image ranges are back-matched along the region boundary to the candidate emoji internal set with the largest boundary overlap area. S44. Object queries that do not fall within the range of candidate emoji images are matched back to the corresponding candidate emoji set along the grid region with the smallest positional distance in the image. If the positional distances in the image are the same, they are matched back along the grid region with the earlier sequence number. S45. If the image position of the missing object query is in the same candidate emoji set, retrieve the encoded feature position of the corresponding image position from the multi-scale visual features to generate a placeholder query. S46. The object queries group organization structure organizes the object queries and placeholder queries within the same candidate emoji set according to the adjacent arrangement relationship in the candidate emoji input set graph, and retains the graph position and sequence number label corresponding to each object query and placeholder query. S47. Group the sorted object queries and placeholder queries according to their position within the same candidate emoji image to form query groups.

[0012] Optionally, S5 specifically includes: S51. The Transformer decoder with query group cross-layer inheritance structure receives the query group at the input of the first decoder layer and establishes the original group position arrangement along the graph position order of object queries and placeholder queries in the query group. S52. The first decoder layer performs self-attention interaction along the original group position. Object queries in the same query group exchange query content, graph position and region boundary to form the first layer group update query. S53. The first decoder layer performs cross-attention interaction between the update query within the first layer group and the multi-scale visual features, and fills the corresponding object query with the encoded features of the corresponding position in the image to form the first layer output object queries. S54. The cross-layer inheritance structure of query groups truncates the input path of independent object queries between adjacent decoder layers, and feeds back the object queries within the group output by the previous decoder layer to the input of the next decoder layer according to the original group position, while retaining the query content, region boundaries and update status formed by the previous decoder layer. S55. The next decoder layer continues to perform self-attention interaction and cross-attention interaction along the original group bit arrangement after the reconnection. When the original group bit is missing an object query, it is filled in by the update state of the adjacent original group bit in the same query group. When the adjacent original group bit does not have an update state, it is taken over by the placeholder query in the query group along the original group bit. S56. The query group cross-layer inheritance structure performs a disconnection check on the original group bit arrangement of the output of adjacent decoder layers. When the original group bit arrangement is continuous, the current decoder layer update state is retained. When the original group bit arrangement is disconnected, the current decoder layer output is corrected by the original group bit arrangement of the previous decoder layer. S57. The object queries within the group output by the last decoder layer are arranged and merged along the original group position, and the query content, region boundary and inter-layer update status corresponding to each object query are retained to form candidate emoji packs to carry out the query.

[0013] Optionally, S6 specifically includes: S61. The path before the prediction head is read out to receive the candidate emojis and receive the query, and the object queries are arranged along the final layer of the query group to form the query arrangement before the prediction head. S62. Before the prediction head readout path enters the classification prediction head and bounding box regression head, the direct readout path of classification score and bounding box coordinates is truncated, and the query content, region boundary and inter-layer update status of each object query are retained as the query status before readout. S63. The pre-read path of the prediction head aligns the region boundary in the pre-read query state with the position in the graph of the candidate emoji input set graph. When the region boundary goes beyond the range of the corresponding candidate emoji graph, it is cropped along the range of the corresponding candidate emoji graph to form a boundary calibration query state. S64. Predict the reading path along the final layer group position of the query group, calibrate the inter-layer update state in the query state, retain the continuously updated object queries in the same query group as text constraint queries, and replace the object queries with interrupted updates with placeholder queries for the corresponding group positions. S65. The predicted head-read path registers the text constraint query with the response prompt mark, aligns the response object with the query content, and aligns the response tone with the inter-layer update status to form candidate text constraint queries. S66. Candidate text constraint query constraint candidate response text generation. When the generated text content is separated from the respondent, it will return to the query content. When the tone deviates from the response tone, it will return to the inter-layer update status. S67. Arrange the constrained terms into candidate response texts and retain the correspondence between the candidate response texts and the candidate text constrained queries.

[0014] Optionally, S7 specifically includes: S71. Following the correspondence between candidate response texts and candidate emojis in the query, the candidate response texts are aligned with the query location and area boundaries to form candidate combinations to be verified. S72. Verify that the response object in the candidate combination to be verified is consistent with the response object in the call prompt mark. If the response object is not aligned, the corresponding candidate combination to be verified is blocked. S73. Verify the tone of the candidate combination to be verified with the tone of the reply prompt in the call prompt mark. If the tone conflicts, the corresponding candidate combination to be verified is blocked. S74. Verify the candidate response text in the candidate candidate combination to be verified with the boundary of the area reserved for candidate emoticons in the query with consistent tone. If the text exceeds the boundary of the area, the corresponding candidate candidate combination to be verified is blocked. S75. Retain the unblocked candidate combinations to be verified as closed candidate combinations, and extract the emoji images and reply text from the closed candidate combinations.

[0015] The beneficial effects of this invention are: This invention, through the combined processing of response prompts and a candidate emoji input set graph, can compress the most recent rounds of dialogue between a large model agent and the user into response objects and reply tones, and organize the emoji set into visual input objects with corresponding positions and numbers within the graph. Therefore, emoji selection no longer relies solely on emotion tags or image-text similarity, but can establish a stable connection around the content to be responded to in the current dialogue, reducing the disconnect between emojis and reply text.

[0016] Furthermore, this invention employs an improved RT-DETR model to process the candidate emoji input set graph. It truncates independent detection query tiled entry points through an object query group organization structure and utilizes a Transformer decoder with a cross-layer inheritance structure for query groups to maintain the arrangement and update state within groups. This allows queries within the same candidate emoji region to proceed simultaneously across multiple decoding layers. Compared to conventional object detection readout methods, this invention preserves query positions, region boundaries, and inter-layer update states, providing more stable visual constraints for candidate response text generation.

[0017] This invention further truncates the direct readout paths of the classification prediction head and the bounding box regression head by pre-readout path truncating, enabling candidate emoticons to be registered with query and response prompt tags, and filtering out candidate combinations with broken responses through response closure verification. This ensures the matching between emoticon images and response text while reducing the response latency caused by tool-invoking emoticon display, thus improving the real-time performance, accuracy, and naturalness of emoticon display in intelligent interactive devices. Attached Figure Description

[0018] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is an overall flowchart of the emoji selection and text generation method based on cue learning and multimodal model proposed in this invention; Figure 2 This is a schematic diagram of the improved RT-DETR model for the emoji selection and text generation method based on cue learning and multimodal models proposed in this invention. Figure 3 This diagram illustrates the object queries group organization structure and cross-level inheritance structure of the emoji selection and text generation method based on cue learning and multimodal models proposed in this invention. Detailed Implementation

[0019] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0020] refer to Figures 1-3 The emoji selection and text generation method based on cue learning and multimodal models includes the following steps: S1. Extract the most recent rounds of dialogue from the current dialogue between the large model agent and the user, and compress them into dialogue prompts with the target of the conversation and the tone of the reply. S2. Organize the emoji set into a candidate emoji input set graph, and pair the candidate emoji input set graph with the reply prompts to form emoji inference samples; S3. An improved RT-DETR model is used to access emoji inference samples. A convolutional backbone network and an efficient hybrid encoder extract and fuse multi-scale visual features from the candidate emoji input set graph. S4, IoU perception query selection extracts object queries from multi-scale visual features. The object query group organization structure truncates the independent detection query tiling entry and groups the object queries in the same candidate emoji pack into query groups according to the adjacent arrangement relationship in the candidate emoji pack input set graph. S5. The Transformer decoder with query group cross-layer inheritance structure receives query groups. The object queries in each group output by each decoder layer are fed back to the next decoder layer according to the original group position, and candidate emojis are formed to accept queries along the inter-layer inheritance group arrangement relationship and update status. S6. The candidate emoji pack is truncated before the reading path of the prediction head. The query enters the reading path of the classification prediction head and the bounding regression head. The final layer position, region boundary and inter-layer update status of the query group are preserved. The emoji pack is registered with the answer prompt mark to constrain the generation of candidate reply text. S7. Perform a closed-loop verification on the candidate reply text and candidate emojis, block candidate combinations with broken loops, retain closed candidate combinations, and output emoji images and reply text.

[0021] In this embodiment, S1 specifically includes: The conversation prompt marker is generated by the current dialogue end between the large model agent and the user. The current dialogue end extracts continuous messages through a round window, which is determined by the device's computing power, context length, and emoticon refresh rate. In one implementation, it extracts the most recent 6 rounds of continuous messages, with each round including one user message and one large model agent reply. If the current dialogue is shorter than the round window length, all continuous messages are extracted. System control statements include role settings, tool invocation commands, output format constraints, and security prompts. System control statements are identified by source fields, control markers, and fixed control phrases, and are then removed from continuous messages. The remaining messages retain their original time sequence. Messages with the source field "assistant" or "agent" are linked to the large model agent side, and messages with the source field "user" are linked to the user side. If the source field is missing, messages containing tool invocation markers or model reply formats are linked to the large model agent side, and the remaining messages are linked to the user side, forming a role-aligned dialogue segment.

[0022] After the role-aligned dialogue segment is formed, the user-side terminal speech serves as the starting point for the current response, and the adjacent speech on the large model agent side is the reply from the agent closest to the user-side terminal speech. The system performs word segmentation, part-of-speech tagging, and dependency relation analysis on the user-side terminal speech and the adjacent speech on the large model agent side, extracting predicates, object components, negation words, and sentiment words. Semantic content already inherited by the adjacent speech on the large model agent side is judged by predicate consistency and object overlap; when the predicates are completely identical, or the predicates are grouped into the same category by the semantic merging table, object overlap is calculated. The semantic merging table is formed by organizing commonly used question-and-answer words, evaluation words, and interactive action words from the product dialogue corpus. The object overlap ratio is obtained by dividing the number of common object words by the number of duplicates after merging object words on both sides. In one implementation, when the object overlap ratio reaches 0.6 or higher, the corresponding semantic content in the user-side terminal speech is deducted. The remaining requests, questions, evaluations, or follow-up questions after deduction form the response segment; if the text after deduction is empty, the last evaluation or question sentence in the user's terminal speech is taken as the response segment to ensure that the response processing continues.

[0023] The core predicate is determined from the pending response segment. The system prioritizes verbs or adjectives that directly connect to the user's intent. When both verbs and adjectives exist, words with objects or complements are prioritized. When multiple candidate core predicates exist simultaneously, the word closest to the end of the sentence and connected to the object component is selected. The object component includes noun phrases directly connected to the core predicate, pronoun references, and omitted subject restoration objects. The pronoun reference object is retrieved from the role-aligned dialogue segment, with the nearest similar noun phrase retrieved backwards. Similar noun phrases are determined by the cosine similarity of word vectors, which are trained from the dialogue corpus or output by a trained language model. In one implementation, the similarity threshold is set to 0.75. If no similar noun phrase is found, the original pronoun is retained. The omitted subject restoration object is extracted from the previous user's statement or the large model agent's reply and added. Adverbs of degree, time supplements, tone trailings, and additional descriptions that do not change the direction of the response are deleted as modifiers. The core predicate and the object component are merged into the respondent. When the respondent exceeds 32 Chinese characters, the core predicate, the complete object component, and 8 Chinese characters before and after the core predicate are retained.

[0024] Polarity expressions are derived by merging affirmative words, negative words, transition words, and emotional words. Affirmative words and positive emotional words are categorized as positive, negative words and negative emotional words as negative, and expressions following transition words cover those preceding them. Emotional words are derived from an emotional word list compiled from product interaction corpora, with each word in the list marked as positive, negative, or soothing. Sentence-final tone is determined by sentence-final punctuation, modal particles, and rhetorical questions. Question marks or rhetorical questions are categorized as follow-up questions, exclamation marks and intensifying modal particles as strengthening modal particles, and periods or words without obvious modal particles as neutral modal particles. When modifying polarity expressions with sentence-final tone, rhetorical questions correct literal positive evaluations to irony, strengthening modal particles retain the current polarity and increase the intensity of the tone, and neutral modal particles retain the current polarity. The response tone can be positive, negative, ironic, probing, comforting, or neutral; when the polarity expression is empty, the response tone is neutral; when the following paragraph contains words of comfort or low mood, the response tone is comforting.

[0025] The response prompt marker is formed by merging and compressing the response object and the reply tone, arranged in the order of response object first, followed by reply tone. The system deletes duplicate words and stop words, retaining the core predicate, object components, polarity direction, and tone type, and records the boundary position between the response object and the reply tone. Stop words include function words, non-directive titles, and continuously repeated tone words. Before deletion, it is verified whether the stop word is connected to the core predicate; words connected to the core predicate are retained. The response prompt marker output is a text marker string, the length of which is controlled within 64 Chinese characters in one implementation; if it exceeds 64 Chinese characters, the core predicate and object components in the response object are retained first, then the reply tone is retained, and modifying components that are not connected to the core predicate and do not change the polarity direction are deleted. The formed response prompt marker serves as the text-side receiving object when matching emoticon input set graphs.

[0026] In this embodiment, S2 specifically includes: The candidate emoji input set is generated from the emoji collection. The emoji collection includes emoji images that can be displayed on the dual screens of the smart AI toy in the production system. The theme attribution is derived from the theme name, emotion tag, or display package name bound when the emoji is added to the database. During the organization process, the theme attribution of each emoji image is first read, and emoji images with the same theme attribution are grouped into the same theme candidate image group. When the same image has multiple theme attributions, the theme attribution is matched against the response tone in the current conversation prompt marker; if a match fails, it is retained in the general theme candidate image group.

[0027] After the candidate image group for a theme is formed, the display boundary is cropped for the emoji images within the group. Display boundary cropping first identifies the outer boundaries of non-transparent or non-background pixels in the emoji image, then retains a margin outside the outer boundaries. In one implementation, the margin is taken as 5% of the smaller of the image width and height. After cropping, the image is scaled to a unified display unit, and any insufficient short sides are filled with transparent pixels to prevent the emoji from being stretched. The display order of the collection is determined by the order in which the images within the candidate image group were added to the database, their usage frequency, and the matching results of the response tone. Images with successfully matched response tones are placed before images of the same theme, and usage frequency is only considered in the sorting when the response tone is consistent.

[0028] The grid area is a regular display unit in the topic input set graph, and each grid area holds only one cropped emoji image. The position within the image is determined by the row and column order of the grid area. In one embodiment, the topic input set graph is arranged in 4 columns. When the number of images exceeds the capacity of a single set graph, it is split into multiple topic input set graphs, while retaining the consecutive page order under the same topic. The serial number is written in the fixed corner of the corresponding grid area. The serial number is matched with the position within the image one by one. The matching process records the correspondence between the serial number, row order, column order, and emoji image file index, forming a candidate emoji input set graph with the correspondence between the position within the image and the serial number. When an image is missing or fails to load, a blank placeholder image is filled in the corresponding grid area and the unselectable state is recorded, without changing the position within the image of other grid areas.

[0029] The emoji inference samples are obtained by pairing the candidate emoji input set graph with the response cue tags in the same round. Same-round pairing means binding the candidate emoji input set graph with the response cue tags generated at the end of the same dialogue session. During binding, the response cue tags retain the respondent, the tone of the reply, and the in-graph position and sequence number in the candidate emoji input set graph. When compressing to form the emoji inference samples, the candidate emoji input set graph is saved on the image side, the response cue tags are saved on the text side, and the in-graph position and sequence number correspondence is saved on the index side. The emoji inference samples serve as the input objects for the improved RT-DETR model.

[0030] In this embodiment, S3 specifically includes: The improved RT-DETR model takes into account the candidate emoji input set graph in emoji inference samples. The original RT-DETR model is designed for end-to-end object detection, typically using a convolutional backbone network to extract multi-scale features, an efficient hybrid encoder to complete intra-scale feature interaction and cross-scale fusion, IoU-aware query selection to extract object queries from the encoded features, a Transformer decoder to update object queries layer by layer, and a prediction head to read out classification results and bounding box regression results. The original RT-DETR model can detect image regions, but in the emoji scenario, the original object queries enter the decoder as independent object queries. The main body region, text region, and writable region within the same candidate emoji are easily separated during the decoding stage, making it difficult to retain stable region connections for the generation of reply text.

[0031] The improved RT-DETR model includes a convolutional backbone network, an efficient hybrid encoder, an IoU-aware query selection mechanism, an object query group organization structure, a Transformer decoder with a cross-layer inheritance structure for query groups, a pre-readout path for the prediction head, and a prediction head. The convolutional backbone network extracts multi-scale candidate visual features from the input image of candidate emojis; the efficient hybrid encoder performs same-scale feature interaction and cross-scale fusion on the multi-scale candidate visual features; the IoU-aware query selection extracts object queries from the multi-scale visual features; the object query group organization structure rewrites independent detection query entry points into query group entry points; the Transformer decoder with a cross-layer inheritance structure for query groups maintains the arrangement relationship within query groups and the inter-layer update state; the pre-readout path for the prediction head preserves the query position, region boundaries, and inter-layer update state of the candidate emojis; and the prediction head retains region classification and boundary readout capabilities and assists in calibrating region boundaries.

[0032] The learnable parameters for improving the RT-DETR model include the convolutional kernel weights and normalization parameters in the convolutional backbone network; the linear projection matrix, attention query matrix, key matrix, value matrix, and feedforward network weights in the efficient hybrid encoder; the candidate quality projection parameters and boundary regression projection parameters in IoU-aware query selection; the self-attention parameters, cross-attention parameters, and feedforward network parameters in the Transformer decoder; and the classification projection parameters and bounding box regression parameters in the prediction head. The intra-graph attribution threshold, matching rules, and low-confidence values ​​for placeholder queries in the object queries query group organization structure; the original group position break threshold and missing query completion rules in the cross-layer inheritance structure with query groups; and the object matching threshold, tone matching threshold, and text boundary crossing handling threshold in the prediction head's pre-readout path are engineering judgment parameters, determined through calibration samples before deployment. The calibration samples consist of a set graph of numbered emoji inputs, response prompt markers, manually confirmed emoji numbers, and reply text. In one implementation, during the training phase, the convolutional backbone network, efficient hybrid encoder, IoU-aware query selection, Transformer decoder, and prediction head are first trained using existing emoji detection annotations. Then, the projection parameters involved in registration in the IoU-aware query selection, Transformer decoder, and prediction head readout path are fine-tuned using emoji inference samples. The engineering decision parameters are fixed on the validation samples.

[0033] The convolutional backbone network consists of input convolutional layers, hierarchical convolutional blocks, downsampling layers, and feature output layers. The input convolutional layers map the pixel channels of the candidate emoji input map to the model channel space; the hierarchical convolutional blocks extract features of emoji boundaries, grid region boundaries, image subjects, and text regions at different spatial resolutions; the downsampling layers compress the feature map space size and expand the image range corresponding to individual feature locations; and the feature output layers extract candidate visual features from different layers while preserving the correspondence between the in-image positions and index labels in the candidate emoji input map.

[0034] When processing the candidate emoji input set image, the convolutional backbone network first scales the candidate emoji input set image to the model input size. In one embodiment, the input size is 640 pixels by 640 pixels, and the image pixels are normalized to the range of 0 to 1. The convolutional backbone network performs convolution, normalization, nonlinear compression, and downsampling in stages. At each convolution position, it reads neighboring pixels or features from the previous stage. The convolution kernel weights are multiplied by the corresponding channel features one by one. All multiplications are summed and the bias is added. Then, nonlinear compression is performed to obtain the output features at the current position. In the downsampling stage, adjacent local regions are summarized into a lower resolution feature map. Shallow features retain boundaries and textures, while deep features retain the main body of the expression and the semantics of the image. In one embodiment, the convolutional backbone network outputs candidate visual features at three scales, with 128, 256, and 512 channels at each scale.

[0035] The correspondence between the position and the index label in the image is preserved in each level of the convolutional backbone network's features. Before scaling, each grid region in the candidate emoji input set image records the coordinates of its top-left corner, bottom-right corner, row order, column order, and index label; after scaling, the coordinates are synchronously transformed to the model input size along the same scaling ratio. Each time the convolutional backbone network performs downsampling, the grid region coordinates are synchronously transformed to the current feature map coordinate system along the downsampling ratio; if the transformed coordinates fall outside the feature map boundary, they are cropped along the feature map boundary; if the width or height of a grid region in a low-resolution feature map is less than one feature position, it is taken over by the feature position corresponding to the center point of the grid region. Therefore, candidate emojis in each level of features can be traced back to their position and index label in the candidate emoji input set image.

[0036] The efficient hybrid encoder comprises a same-scale feature interaction unit, a cross-scale feature fusion unit, a channel projection unit, and an encoded feature output unit. The same-scale feature interaction unit is responsible for exchanging grid region boundaries, emoji image boundaries, and local semantic content within the same scale; the cross-scale feature fusion unit is responsible for aligning shallow boundary features with deep semantic features along the in-image position; the channel projection unit is responsible for compressing candidate visual features of different scales and different numbers of channels to a unified channel dimension; and the encoded feature output unit is responsible for outputting multi-scale visual features for IoU perceptual query selection and retrieval.

[0037] In the efficient hybrid encoder, same-scale feature interaction is performed within each scale. Candidate visual features at each scale are first converted to a unified channel dimension by a channel projection unit; in one implementation, the unified channel dimension is 256. Channel projection reads all channel values ​​at a single feature location, multiplies each channel value by the corresponding weight in the projection matrix, accumulates the multiplications, and then adds the bias to form a unified channel feature. Subsequently, the same-scale feature interaction unit reads feature locations within the same grid region and adjacent grid regions at the same scale, calculating the query vector, key vector, and value vector respectively. The query vector and key vector are multiplied channel by channel and accumulated to obtain the attention response. The attention response is normalized and used as a weight to multiply the value vector. The weighted value vector is then added to the original feature location to form the same-scale encoded feature. Emoji image boundaries, grid region boundaries, and local semantic content enter the same channel space in the same-scale encoded feature.

[0038] Cross-scale fusion in the efficient hybrid encoder is performed between encoded features of the same scale. Shallow boundary features are adjusted to the spatial size of deep features through stride convolution or pooling, while deep semantic features are adjusted to the spatial size of shallow features through interpolation or linear projection. The adjusted features are aligned along the in-image position of the candidate emoji input set graph. Shallow boundary features and deep semantic features corresponding to the same in-image position are concatenated along the channel direction and then compressed to a unified channel dimension through the channel projection unit. Each output channel of the channel projection reads all the concatenated input channels, multiplies and accumulates them channel by channel, and adds the bias to form the fused channel value. The fused channel value is then non-linearly compressed and enters the encoded feature output unit to form the multi-scale visual features required for IoU-aware query selection to extract object queries.

[0039] In this embodiment, S4 specifically includes: IoU-aware query selection takes multi-scale visual features and extracts a set of object queries to be organized. IoU-aware query selection includes a candidate quality readout unit, a region boundary readout unit, and an object query extraction unit. The candidate quality readout unit is responsible for determining whether each encoded feature location is suitable as a source of object queries. The region boundary readout unit is responsible for reading the corresponding image region for each encoded feature location. The object query extraction unit is responsible for organizing the channel features, region boundaries, and candidate quality of the encoded feature locations into a set of object queries to be organized. In one embodiment, each encoded feature location carries 256-dimensional channel features. The candidate quality readout unit reads the 256-dimensional channel features, multiplies each feature value by the corresponding weight in the candidate quality projection parameters, sums all multiplications, adds the bias, and then compresses it to the range of 0 to 1 using a sigmoid function to form a classification confidence value. The region boundary readout unit reads the same 256-dimensional channel features, obtains the center coordinates, width, and height through boundary regression projection, and converts the center coordinates, width, and height into region boundaries.

[0040] Candidate quality is jointly formed by classification confidence value and boundary fit quality. Boundary fit quality is determined by the degree of overlap between the region boundary and the grid region in the candidate emoji input set graph. The calculation first takes the overlapping area between the region boundary and the corresponding grid region, then takes the combined area of ​​the region boundary area and the corresponding grid region area. Dividing the overlapping area by the combined area yields the boundary fit quality. The candidate quality is obtained by multiplying the classification confidence value by the boundary fit quality. In one implementation, encoded feature positions with a candidate quality below 0.35 are not included in the object queries set to be organized. Encoded feature positions with a candidate quality of 0.35 or higher are sorted from highest to lowest quality. Within the same candidate emoji graph, a maximum of several encoded feature positions with the highest candidate quality are retained. The number of sorted positions is jointly determined by the grid capacity of the candidate emoji input set graph and the input capacity of the Transformer decoder. When there are insufficient encoded feature positions, placeholder query generation processing is initiated.

[0041] The object queries group organization structure includes a query entry control unit, a location registration unit, an intra-graph attribution verification unit, a back-matching and completion unit, and a query group arrangement unit. The query entry control unit is located between the IoU-aware query selection and the Transformer decoder. After receiving the set of object queries to be organized, it disables independent query detection and directly flattens the data into the input entry of the Transformer decoder, transferring the set of object queries to be organized into the query group organization cache. The query group organization cache records the encoded feature location, region boundary, candidate quality, intra-graph location, and sequence number of the object query. The location registration unit is responsible for converting the encoded feature location to the coordinate system of the candidate emoji input set graph. The intra-graph attribution verification unit is responsible for determining which candidate emoji graph range the object query belongs to. The back-matching and completion unit is responsible for handling cross-boundary, miss, and missing locations. The query group arrangement unit is responsible for arranging object queries within the same candidate emoji into query groups.

[0042] When processing each object query, the location registration unit first reads the scale number and feature map coordinates of the encoded feature location, and then reads the coordinate conversion ratio formed when the candidate emoji input set map is scaled to the model input size. The horizontal coordinate of the encoded feature location is multiplied by the stride of the current scale relative to the model input size to obtain the horizontal center in the model input coordinate system; the vertical coordinate is multiplied by the corresponding stride to obtain the vertical center in the model input coordinate system. The model input coordinates are then divided by the scaling ratio of the candidate emoji input set map to return to the original coordinate system of the candidate emoji input set map. When the center point after returning to the original coordinate system falls within a grid area, the location registration unit writes the in-map position and sequence number of the grid area into the object query, forming location registration object queries; when the center point falls near a grid line, the area boundary is retained as the basis for attribution verification.

[0043] The in-image attribution verification unit performs in-image range verification on the location registration object queries. The in-image range of the candidate emoji is determined by the coordinates of the upper left corner, the lower right corner, and the boundary margin of the grid region. In one embodiment, the boundary margin is 5% of the shorter side length of the grid region. When the center point and the region boundary of the location registration object query both fall within the same candidate emoji's in-image range, the object query is assigned to the corresponding candidate emoji's internal set. When the region boundary crosses adjacent in-image ranges, the overlapping area between the region boundary and the adjacent in-image range is calculated, and the object query is reassigned to the candidate emoji's internal set with the largest overlapping area. If the overlapping areas are the same, the candidate emoji's internal set with the closer center point is selected.

[0044] Object queries that do not fall within the candidate emoji image range are processed by the matching and completion unit. The matching and completion unit first calculates the in-image position distance from the center point of the object query to the center point of each grid region. This in-image position distance is obtained by adding the absolute values ​​of the horizontal and vertical pixel differences. The object query is then matched to the candidate emoji set corresponding to the grid region with the smallest in-image position distance. When multiple grid regions have the same in-image position distance, the matching is performed to the grid region with the earlier sequence number. After matching, the object query retains its original region boundaries and candidate quality, and the matched in-image position and sequence number are written to it, ensuring that missed queries continue to be processed within the candidate emoji set.

[0045] For in-image locations lacking object queries, placeholder queries are generated by the matching and completion unit. A missing object query refers to a location within a grid region of the candidate emoji set that does not have any object query included. The matching and completion unit retrieves the encoded feature locations near the center point of the corresponding in-image location from multi-scale visual features, prioritizing the encoded feature locations with the highest candidate quality. If no encoded feature location reaches the candidate quality threshold for the corresponding in-image location, the channel features of the feature location where the center point of the in-image location is located are read to generate a placeholder query. The placeholder query retains the in-image location and sequence number, the region boundary is taken from the corresponding grid region boundary, and the candidate quality is written as a low confidence value. In one implementation, the low confidence value is half of the candidate quality threshold; when the candidate quality threshold is 0.35, the low confidence value is 0.175. The low confidence value only participates in subsequent attention response scaling and disconnection verification and does not overwrite the candidate quality of existing object queries.

[0046] The query group arrangement unit organizes the sets within the same candidate emoji pack according to the adjacent arrangement relationships in the candidate emoji pack input set graph. The adjacent arrangement relationship is determined by the row order and column order in the topic input set graph, first arranged in ascending order of row order, and then arranged in ascending order of column order within the same row. When multiple object queries exist at the same position in the graph, the object query with the highest candidate quality is placed at the main query position in the graph, and the remaining object queries are arranged after the same position in the graph according to their candidate quality order. Placeholder queries only fill in missing positions in the graph and do not change the arrangement of objects already assigned to them. The query group arrangement unit groups the organized object queries and placeholder queries together according to their positions within the same candidate emoji pack in the graph, forming query groups. Each query group retains its position in the graph, sequence number, query content, region boundary, and candidate quality, serving as the input object for the Transformer decoder with a cross-layer inheritance structure for query groups.

[0047] Compared to the original RT-DETR model, the improvement in this implementation lies in rewriting the object queries entry structure between the IoU-aware query selection and the Transformer decoder. In the original RT-DETR model, the object queries output by the IoU-aware query selection enter the Transformer decoder as independent detection query sets, with each object query primarily updating around a single detection target. This implementation adds an object queries group organization structure between the IoU-aware query selection output and the Transformer decoder input, changing the direct tiling entry of independent detection query sets to an entry for query groups within candidate emoji packs. This structural modification occurs before the Transformer decoder receives object queries and is not part of post-processing of detection results. It allows queries within the same candidate emoji pack to enter the decoder in an in-group arrangement, providing stable input for cross-layer inheritance of query groups and preserving a traceable source of visual constraints for candidate response text generation.

[0048] In this embodiment, S5 specifically includes: The Transformer decoder with a cross-layer inheritance structure for query groups takes over query groups. In a standard RT-DETR decoder, after receiving object queries, these queries are passed layer by layer between decoder layers as independent sets of detection queries. Each object query is primarily updated around a single detection target, and the inter-layer transmission does not preserve the intra-group arrangement relationships between regions within the same candidate emoji. The Transformer decoder with a cross-layer inheritance structure for query groups changes the input path between decoder layers, preserving the original group arrangement of query groups at the input of each decoder layer. This allows object queries and placeholder queries within the same candidate emoji to advance sequentially within the same group during multi-layer decoding.

[0049] The Transformer decoder with a query group cross-layer inheritance structure includes a layer input sorting unit, an intra-group self-attention unit, a cross-attention unit, a feedforward network unit, an original group position reconnection unit, a disconnection verification unit, and an output merging unit. The layer input sorting unit establishes the original group position arrangement; the intra-group self-attention unit exchanges query content, in-image positions, and region boundaries within the same query group; the cross-attention unit supplements the encoded features corresponding to the in-image positions from multi-scale visual features; the feedforward network unit performs a non-linear transformation on the query content after attention output; the original group position reconnection unit sends the intra-group object queries output from the previous decoder layer to the next decoder layer according to their original group positions; the disconnection verification unit identifies and corrects disconnections in the original group position arrangement; and the output merging unit forms candidate emojis to accommodate queries. The self-attention projection parameters, cross-attention projection parameters, feedforward network weights, and normalization parameters in the above units are obtained through backpropagation of training samples. The original group position disconnection threshold, region boundary pruning margin, and missing query completion rules are calibrated using validation samples before deployment.

[0050] The original group position arrangement is established within the layer input sorting unit. The layer input sorting unit reads the in-graph position, sequence number, region boundary, and query content carried by each object query and placeholder query in the query group. It first arranges them according to the row order within the in-graph position, then according to the column order within the same row. When multiple object queries exist at the same in-graph position, the object query with the highest candidate quality becomes the primary query, and the remaining object queries follow the primary query in descending order of candidate quality; placeholder queries only fill in missing in-graph positions. After the original group position arrangement is formed, each query position is written with the original group position sequence number, query content vector, in-graph position encoding, region boundary vector, and query type. Query types include object queries and placeholder queries. Placeholder queries participate in the calculation at the input of the first decoder layer and retain a low-confidence state.

[0051] The decoder input vector is formed after the original group position is established. In one embodiment, the query content vector is 256-dimensional, the in-image position encoding is 256-dimensional, and the region boundary vector includes four values: center x-coordinate, center y-coordinate, width, and height. The region boundary vector enters the boundary linear projection. Each output channel of the boundary linear projection reads the four boundary values, multiplies each of the four boundary values ​​by the corresponding weights in the projection matrix, accumulates the multiplications, and then adds the bias to form a 256-dimensional boundary encoding. The query content vector, in-image position encoding, and boundary encoding are added dimension by dimension, and then normalized layer by layer to form the decoder input vector. The layer normalization reads the 256-dimensional summation result, calculates the 256-dimensional mean and variance, subtracts the mean from each dimension, divides by the square root of the variance, multiplies by the learnable scaling parameter, and then adds the learnable bias to form the normalized input.

[0052] Intra-group self-attention interactions are performed within the intra-group self-attention unit. The intra-group self-attention unit generates a query vector, key vector, and value vector for each decoder input vector. The generation process reads the 256-dimensional input vector, multiplies each input value by its corresponding projection weight, accumulates all multiplications, and then adds the bias to form 256-dimensional query vectors, 256-dimensional key vectors, and 256-dimensional value vectors, respectively. The query vector of any original group position is multiplied dimension-wise by the key vectors of other original group positions within the same query group and accumulated. The accumulated value is divided by the square root of the channel dimension to form the attention response. The attention response is normalized using Softmax to form attention weights, which are then multiplied by the corresponding value vectors and accumulated to obtain the intra-group attention output. The intra-group attention output is added to the decoder input vector residual and normalized again to form the first-layer intra-group update query. When placeholder queries participate in attention calculation, the attention response corresponding to a low-confidence state is multiplied by a scaling factor less than 1; in one implementation, the scaling factor is 0.3.

[0053] Cross-attention interaction is performed within the cross-attention unit. The cross-attention unit reads the in-image location and region boundary of the first-layer intra-group update query, and extracts the encoded feature locations within the same in-image location range from multi-scale visual features. When the encoded feature location spans multiple scales, it first reads the scale feature closest to the region boundary area, then reads features from adjacent scales to supplement boundary information; when the region boundary falls outside the candidate emoji image's in-image range, it is cropped along the candidate emoji image's in-image range before reading the encoded feature location. The first-layer intra-group update query generates a cross-attention query vector, and the multi-scale visual features generate cross-attention key and value vectors, calculated in the same way as the intra-group self-attention interaction. The cross-attention output is added to and normalized with the first-layer intra-group update query residual before entering the feedforward network unit. The feedforward network unit first linearly expands the 256-dimensional vector to 1024 dimensions, multiplying, accumulating, and superimposing biases dimension by dimension during expansion. After GELU nonlinear compression, it is linearly compressed back to 256 dimensions to form the first-layer output object queries.

[0054] The original group bit reconnection is performed within the original group bit reconnection unit. This unit truncates the input paths of independent object queries layer by layer between adjacent decoder layers, preserving the original group bit sequence number, query content, region boundaries, and update status from the output of the previous decoder layer. The update status includes changes in query content and region boundaries; the change in query content is obtained by subtracting the input query content from the output query content of the current decoder layer, and the change in region boundaries is obtained by subtracting the input boundary vector from the boundary vector read by the current decoder layer. The input of the next decoder layer receives the object queries within the group along the original group bit sequence number preserved by the previous decoder layer, and writes the changes in query content and region boundaries into the input record of the next decoder layer, enabling the next decoder layer to continue performing intra-group self-attention and cross-attention interactions along the original group bit sequence.

[0055] Missing original group positions are filled in within the original group position reconnection unit. When the input of the next decoder layer detects a missing object query in an original group position, it first reads the update status of adjacent original group positions within the same query group. If adjacent original group positions have update status, the change in the query content of the adjacent original group positions is added to the query content of the previous layer of the missing original group position to form the filled query content. The change in the region boundary of the adjacent original group positions is added to the region boundary of the previous layer of the missing original group position to form the filled region boundary. If adjacent original group positions do not have update status, the placeholder query in the query group continues along the missing original group position. The placeholder query retains the position and grid region boundary in the original image, and the query content takes the channel features of the corresponding multi-scale visual feature center position. The filled object query continues to participate in the self-attention interaction and cross-attention interaction of the current decoder layer and is not deleted from the query group.

[0056] Disconnect verification is performed within the disconnect verification unit. The disconnect verification unit reads the original group position number, the arrangement of adjacent original group positions, and the region boundary between each adjacent decoder layer. It verifies whether the original group position number is complete, whether adjacent original group positions within the same query group remain adjacent, and whether the region boundary of the same original group position is still within the corresponding candidate emoji image. When all three verifications are true, the current decoder layer update state is retained. If the original group position number is missing, the adjacent arrangement is misaligned, or the region boundary exceeds the candidate emoji image, the disconnect verification unit corrects the current decoder layer output using the original group position arrangement of the previous decoder layer. During correction, the query content retains the current decoder layer output, the original group position number and the image position are written back to the previous decoder layer record, and the region boundary is cropped along the corresponding candidate emoji image's range. If the width or height of the cropped region boundary is less than a feature position, the grid region boundary of the corresponding image position is used as the replacement region boundary.

[0057] Candidate emoji pick-up queries are formed within the output merging unit. The output merging unit reads the in-group object queries output from the last decoder layer, arranging and merging the final query content, corresponding in-graph position, final region boundary, and inter-layer update status of each object query along the original group position. The inter-layer update status is formed by accumulating the query content change and region boundary change layer by layer; in one embodiment, the query content change is limited to a range of -3 to 3 for each dimension, and the boundary value is taken when it exceeds the limit; the region boundary change is limited to within the range of the candidate emoji graph, and is cropped along the graph range when it exceeds the graph range. The output merging unit organizes the final query content, final region boundary, and accumulated update status of all original group positions within the same query group into candidate emoji pick-up queries, which serve as the pick-up objects for the pre-readout path of the prediction head.

[0058] Compared to the original RT-DETR model, the improvement of this implementation lies in rewriting the inter-layer advancement structure of object queries between adjacent layers of the Transformer decoder. In the original RT-DETR model, object queries are passed layer by layer between decoder layers as independent query sets. Each object query is mainly updated around a single detection target, and the inter-layer transmission does not retain the intra-group arrangement constraints of queries within the same candidate emoji. This implementation adds a cross-layer inheritance structure of query groups in the Transformer decoder, so that the intra-group object queries output by the previous decoder layer are fed back to the next decoder layer in their original group positions, while retaining the query content, region boundaries, and update state between layers. The structural modification occurs in the inter-layer input path within the Transformer decoder and is not part of the rule processing after the prediction head is read out. This allows queries within the same candidate emoji to advance together in the multi-layer decoding process, avoiding the separation and updating of queries corresponding to the main body region, text region, and writable region, and providing a continuous source of inter-layer states for candidate emojis to receive queries.

[0059] In this embodiment, S6 specifically includes: The readout path before the prediction head inherits the candidate emoji query. In a typical RT-DETR model, the object queries output from the final layer of the Transformer decoder usually go directly into the classification prediction head and the bounding box regression head. The classification prediction head projects the query content into class scores, and the bounding box regression head projects the query content into bounding box coordinates. This readout method is suitable for object detection tasks, but in emoji selection and text generation scenarios, if the candidate emoji query is directly collapsed into classification scores and bounding box coordinates, the inheritance relationship between the final layer position of the query group, the inter-layer update state, and the response prompt markers will be lost, making it difficult to constrain the generation of candidate response text.

[0060] The pre-readout path of the prediction head includes a query arrangement unit, a readout truncation unit, a boundary calibration unit, an update state organization unit, a cue registration unit, a text constraint unit, and a candidate text arrangement unit. The query arrangement unit is responsible for arranging object queries along the final layer group position of the query group; the readout truncation unit is responsible for truncating the query content, region boundaries, and inter-layer update states before the classification prediction head and bounding box regression head; the boundary calibration unit is responsible for aligning region boundaries to the range of the candidate emoji image; the update state organization unit is responsible for handling continuous updates and update breaks; the cue registration unit is responsible for aligning the respondent and reply tone to the query content and inter-layer update states respectively; the text constraint unit is responsible for constraining the generation of candidate reply text; and the candidate text arrangement unit is responsible for forming candidate reply text and preserving the corresponding relationships.

[0061] The pre-prediction query arrangement is formed within the query arrangement unit. The query arrangement unit reads the final layer group position of the query group in the candidate emoji packet inheriting query, the final query content of each object query, the region boundary, and the inter-layer update state. The final layer group position of the query group comes from the original group position arrangement retained at the output of the last decoder layer. The query arrangement unit first arranges them in ascending order of the original group position number, then places the main query at the same position in the graph at the front, and low-confidence placeholder queries at the corresponding missing positions. The formed pre-prediction query arrangement records the final layer group position, query content vector, region boundary vector, and inter-layer update state, serving as the input object for the readout truncation unit.

[0062] The pre-readout query state is formed within the readout truncation unit. Located between the final output of the Transformer decoder and the classification prediction head and bounding box regression head, the readout truncation unit receives the pre-prediction query arrangement but does not directly write the query content into the classification prediction head and bounding box regression head. The calculation process of the ordinary classification prediction head involves reading a 256-dimensional query content vector, multiplying each dimension's query content value by the classification projection parameter, summing all multiplications, and then adding the classification bias to form the category score. The ordinary bounding box regression head reads the same query content vector, multiplies each dimension's query content value by the bounding box projection parameter, sums them, adds the bounding box bias, and then compresses them using a sigmoid function to form the bounding box coordinates. The readout truncation unit blocks both projection inputs, transferring the 256-dimensional query content vector, region boundary vector, and inter-layer update state of each object query into the pre-readout query state. The classification prediction head and bounding box regression head retain their parameters, but in the pre-prediction head readout path, only the boundary calibration unit is allowed to read the region boundaries; ordinary detection results are not directly output.

[0063] The boundary calibration query state is formed within the boundary calibration unit. The boundary calibration unit reads the region boundary from the query state before reading out and reads the in-image position range corresponding to the final layer group position in the candidate emoji input set graph. The region boundary is represented by the center x-coordinate, center y-coordinate, width, and height. It is first converted to the upper left and lower right corner coordinates, and then overlapped with the in-image range of the candidate emoji. The region boundary remains unchanged when it is within the in-image range; when the region boundary exceeds the in-image range, the upper left or lower right corner coordinates of the exceeding region are cropped along the edge of the in-image range. In one embodiment, the boundary calibration unit retains a 5% boundary margin within the in-image range. When the width or height after cropping is less than a feature position, the grid region boundary corresponding to the in-image position is used as the replacement boundary. The query content after boundary calibration, the cropped region boundary, and the original inter-layer update state together form the boundary calibration query state.

[0064] Text constraint queries are formed within the update state organization unit. This unit reads the inter-layer update state in the query state along the boundary calibration of the final layer group position of the query group. The inter-layer update state includes the change in query content and the change in region boundaries. The change in query content is obtained by subtracting the query content output from the adjacent decoder layer, and the change in region boundaries is obtained by subtracting the region boundaries output from the adjacent decoder layer. The judgment for continuous updates is whether the same final layer group position has both the change in query content and the change in region boundaries in adjacent decoder layers. When continuous updates are established, the corresponding object query is retained as a text constraint query. When an update is disconnected, the update state organization unit reads the placeholder query for the corresponding group position, replaces the graph position and region boundary of the disconnected object query with the graph position and region boundary of the placeholder query, and retains the output of the last layer before the disconnection, forming a receptive text constraint query. If the placeholder query does not exist, the inter-layer update state of the adjacent final layer group positions within the same query group is read and added.

[0065] Candidate text constraint queries are formed within the prompt registration unit. The prompt registration unit reads the query content, region boundaries, and inter-layer update status from the text constraint query, and simultaneously reads the responding object and response tone from the response prompt marker. When registering the responding object with the query content, the responding object is first converted into a text vector; in one embodiment, a trained text encoder outputs a 256-dimensional text vector. The 256-dimensional query content in the text constraint query is multiplied dimension-by-dimensional with the text vector and accumulated to obtain the object matching response. The object matching response is compressed to the range of 0 to 1 using a Sigmoid function. If it is below 0.4, the prompt registration unit marks the text constraint query as a weak object registration and requires the text constraint unit to prioritize the use of the core predicates and object components in the responding object. When registering the response tone with the inter-layer update status, the response tone is mapped to a 256-dimensional tone vector. The tone vector is multiplied dimension-by-dimensional with the change in query content in the inter-layer update status and accumulated to obtain the tone matching response. If the tone matching response is below 0.4, the text constraint unit prioritizes the word range corresponding to the response tone. The text constraint query, object matching response, and tone matching response together form the candidate text constraint query.

[0066] The constraints for generating candidate response text are executed within the text constraint unit. The text constraint unit reads the candidate text constraint query, the respondent, and the response tone, and also reads candidate lexical units from the small model's vocabulary. Candidate lexical units are first read by the language model to obtain a base probability, which comes from the softmax normalization result of the vocabulary by the language model's output layer. The tone category of candidate lexical units is jointly determined by tone annotations and the sentiment vocabulary in the training corpus. Tone categories include positive, negative, ironic, probing, appeasing, and neutral; candidate lexical units that do not match tone annotations or the sentiment vocabulary are classified as neutral. When the generated text content deviates from the respondent, the deviation judgment is completed by the similarity between the candidate lexical unit and the respondent's text vector. In one implementation, candidate lexical units with a similarity below 0.3 have their weights reduced; candidate lexical units connected to the core predicate or object component in the respondent retain their original weights. When the tone deviates from the response tone, the deviation judgment is completed by whether the tone category of the candidate lexical unit is consistent with the response tone; inconsistent candidate lexical units have their weights reduced. When reducing the weight, the basic probability of the candidate word element is multiplied by 0.5; when the similarity between the candidate word element and the text vector of the respondent reaches 0.6 or above and the tone category is consistent with the tone of the reply, the basic probability of the candidate word element is multiplied by 1.2, and the corrected probability is normalized again to form a word element selection range constrained by the candidate text query.

[0067] The candidate text arrangement unit forms the candidate response text. The candidate text arrangement unit selects candidate lexical units one by one according to the generation order of the language model. In one implementation, the length of a single candidate response text is controlled between 3 and 24 Chinese characters. When the generated length is less than 3 Chinese characters, core object words are added from the respondent; when the generated length exceeds 24 Chinese characters, low-probability modifiers are deleted from the end of the sentence. For each lexical unit generated, the candidate text arrangement unit writes the object matching response and tone matching response between the lexical unit and the candidate text constraint query into the correspondence record. After the candidate response text is completed, the system retains the candidate response text, candidate text constraint query, object matching response, tone matching response, and region boundary as input objects for the response closure verification.

[0068] Compared to the original RT-DETR model, the improvement in this implementation lies in rewriting the readout path from the Transformer decoder output to the prediction head. In the original RT-DETR model, the object queries output from the final layer of the decoder directly enter the classification prediction head and the bounding box regression head. The output results are mainly class scores and bounding box coordinates. The final layer group positions and inter-layer update states of the query group no longer participate in text generation after detection readout. This implementation adds a readout path before the prediction head between the Transformer decoder output and the prediction head, truncating the direct readout path of classification scores and bounding box coordinates, and retaining the candidate emoji as a text constraint object carrying query content, region boundaries, and inter-layer update states. The structural modification occurs before the prediction head readout and is not part of the post-processing of detection results. It can transform the group position arrangement and inter-layer update states formed inside the RT-DETR decoder into candidate response text generation constraints, ensuring that the emoji image region and the response text are consistent in terms of the respondent and the tone of the response.

[0069] In this embodiment, S7 specifically includes: The dialogue closure verification verifies the correspondence between candidate response texts, candidate text constraint queries, and candidate emoji reception queries. Dialogue closure verification includes a combination fitting unit, a response object verification unit, a tone direction verification unit, a region boundary fitting unit, and a result retrieval unit. The combination fitting unit is responsible for associating candidate response texts with the query position and region boundary of the candidate emoji reception query; the response object verification unit is responsible for determining whether the candidate response text responds to the response object marked in the dialogue prompt; the tone direction verification unit is responsible for determining whether the candidate response text maintains the response tone; the region boundary fitting unit is responsible for determining whether the candidate response text falls within the region boundary retained by the candidate emoji reception query; and the result retrieval unit is responsible for outputting the emoji image and response text.

[0070] The candidate combinations to be verified are formed within the combination and fitting unit. The combination and fitting unit reads the correspondence between the candidate response text and the candidate text constraint query, and also reads the candidate emoji inheritance query from which the candidate text constraint query originates. The candidate emoji inheritance query records the query position, region boundary, in-figure position, and sequence number. The combination and fitting unit attaches the candidate response text to the emoji image corresponding to the same sequence number and places the display starting point of the candidate response text at the query position. When the query position is outside the region boundary, the display starting point is moved back to the center of the region boundary; when the region boundary is missing, the grid region boundary at the corresponding in-figure position in the candidate emoji input set graph is used as a temporary region boundary. After fitting is completed, the candidate combinations to be verified are formed.

[0071] Response object verification is performed within the response object verification unit. This unit first extracts the response object from the candidate response text, using a method consistent with the formation method of the response object in the answer prompt markers. Specifically, it first determines the core predicate in the candidate response text and then retains the object components directly connected to the core predicate. When verifying the response object and the response object, it first compares whether the object words are completely identical. If they are not completely identical, the response object and the response object are converted into text vectors respectively, and the cosine similarity between the two text vectors is calculated. In one implementation, if the cosine similarity reaches 0.65 or higher, the response objects are considered aligned; if it is lower than 0.65 and there is no synonym merging relationship, the candidate combinations to be verified are written into the mask set. The mask set only records candidate combinations that fail verification and does not participate in the closed candidate combination output.

[0072] The tone direction verification is performed within the tone direction verification unit. This unit reads polar expressions, sentence-final intonation, and modal particles from the candidate response text and merges them to form the tone direction. The merging rules for polar expressions are consistent with the response tone formation rules in the response prompt markers: affirmative words and words with positive emotions are categorized as positive, negative words and words with negative emotions are categorized as negative, and rhetorical questions correct literal positive expressions to an ironic direction. When the tone direction matches the response tone, the candidate combination to be verified enters the region boundary fitting verification; when the tone direction is opposite to the response tone, the candidate combination to be verified is written into the shield set. When the tone direction is neutral and the response tone is soothing, candidate responses containing comforting or conciliatory words are retained; otherwise, they are written into the shield set.

[0073] The region boundary alignment verification is performed within the region boundary alignment unit. The region boundary alignment unit reads the number of characters, display font size, and region boundary width of the candidate reply text. In one embodiment, the display width of Chinese characters is the font size in pixels, the display width of English characters and numbers is half the font size in pixels, and the display width of punctuation is half the font size in pixels. The display width of the candidate reply text is obtained by summing the display widths of each character, and the display height is obtained by multiplying the number of lines by the font size in pixels. When the display width does not exceed the region boundary width and the display height does not exceed the region boundary height, the text is determined to be aligned with the region boundary. When the display width exceeds the region boundary width, the modifiers with the lowest generation probability that do not connect to the responding object are first deleted, and then the display width is recalculated. If it still exceeds the region boundary after recalculation, the candidate combination to be verified is written into the mask set. When the display height exceeds the region boundary height, the repeated interjections at the end of the sentence are first deleted, and the display height is recalculated. If it still exceeds the region boundary after recalculation, the candidate combination to be verified is written into the mask set.

[0074] Closed candidate combinations are formed by the result extraction unit. Candidate combinations that pass the response object verification, tone direction verification, and region boundary fitting verification are retained as closed candidate combinations. When multiple closed candidate combinations exist, the result extraction unit reads the object matching response, tone matching response, and region fitting margin respectively; the region fitting margin is obtained by subtracting the display width of the candidate response text from the region boundary width. The result extraction unit normalizes the object matching response, tone matching response, and region fitting margin to the range of 0 to 1, sums them, and selects the closed candidate combination with the largest sum; if the sums are the same, the closed candidate combination with the earlier sequence number is selected. The result extraction unit reads the emoji image corresponding to the sequence number from the closed candidate combinations, and reads the candidate response text from the closed candidate combinations as the response text, outputting the emoji image and the response text.

[0075] Example 1: To verify the feasibility of this invention in practice, it was applied to a dialogue system for a smart AI toy with dual screens. When the device interacts continuously with the user, it needs to simultaneously output reply text and on-screen emoticons. Existing solutions typically rely on a large model agent to handle chat replies, then select emoticons through tool calls or emotion classification. However, emoticon output often lags behind text replies. Furthermore, when users express questions, playful teasing, complaints, or requests for comfort, simply relying on emotion tags can lead to the selection of emoticons with similar tones but different recipients, resulting in a disconnect between on-screen emoticons and reply text. Especially in scenarios involving children and casual conversation, users often initiate dialogue using short sentences, colloquial expressions, and omit subjects. Ordinary emotion classification struggles to identify the true recipient, leading to overly exaggerated or off-key on-screen emoticons that negatively impact the naturalness of the interaction.

[0076] In application, the main agent model remains responsible for generating the main dialogue, while the expression inference process is handled by a bypass model. The system first extracts dialogue content from the last six rounds, removes system control statements, and generates response prompts. Then, it crops and arranges emoji images under the same theme into numbered grid regions, forming a candidate emoji input set graph. After pairing the candidate emoji input set graph with the response prompts, it connects to an improved RT-DETR model. The model uses an object query group organization structure and a Transformer decoder with a cross-layer inheritance structure for query groups to preserve the arrangement and update status of queries within the same candidate emoji. The prediction head's readout path does not directly output ordinary detection boxes; instead, it registers the candidate emoji's query with the response prompts, constrains the generation of candidate response text, and finally outputs the emoji image and response text through dialogue closure verification. The bypass model runs in parallel with the main dialogue chain, eliminating the need to wait for the tool to complete its generation. The range of candidate emojis is no longer limited by the number of parameters in a single tool call, allowing for the simultaneous coverage of more display options within the same theme set graph.

[0077] To verify the feasibility of implementation, a test set was composed of multi-round real interaction records from the intelligent AI toy dialogue system and manually constructed boundary tone samples. The test set covers lightweight interaction scenarios such as daily chat, child companionship, emotional soothing, responding to questions, joking and teasing, and knowledge quizzes, containing a total of 3,000 dialogue samples with no more than six rounds, and candidate emojis covering 28 display options. The method of this invention was compared with a single large model tool invocation method, a pure text emotion classification method, and a general multimodal retrieval method. The emojis were manually reviewed to ensure they fit the respondent, the tone of the reply, and the generated text. The statistical results are shown in Table 1 below. Table 1. Comparison of Smart AI Toy Emoji Selection and Text Generation Effects

[0078] As can be seen from the data in Table 1 above, the method of the present invention achieves good results in terms of expression output delay, expression selection accuracy, response text matching pass rate, image-text closure pass rate, and average number of selectable expressions. The average expression output delay of the single large model tool call method is 1680ms. The main reason is that when the expression display is used as a tool call to access the main dialogue link, additional call content needs to be generated and the tool needs to be waited for to return, resulting in a significant time difference between the screen expression and the text response. The average delay of the pure text emotion classification method is reduced to 430ms, but the expression selection accuracy is only 80.9%, indicating that it is difficult to distinguish the respondent and the tone of the reply based solely on the emotion category. The ordinary multimodal retrieval method improves the expression selection accuracy to 84.6% and the average number of selectable expressions to 24, but the image-text closure pass rate is still 79.4%, indicating that image similarity or semantic similarity does not necessarily guarantee a natural connection between candidate expressions and generated text. The average output latency of the emoticon method of this invention is 510ms, which is slightly higher than that of the pure text emotion classification method, but significantly lower than that of the single large model tool calling method. At the same time, the accuracy of emoticon selection reaches 91.8%, the success rate of reply text matching reaches 90.6%, the success rate of image-text closure reaches 88.9%, and the average number of selectable emoticons reaches 28. This shows that the present invention improves the matching stability between emoticon images, reply content and reply text while maintaining real-time performance.

[0079] This embodiment avoids having a single large model agent simultaneously handle chat replies and emoji display tasks by bypassing the emoji inference process. Main dialogue generation and emoji inference are executed in parallel, reducing response wait times caused by tool calls. Simultaneously, the content of the last six rounds of dialogue is compressed into response prompts, preserving the user's current target audience and tone, reducing misselections caused by relying solely on emotion tags. Regarding emoji processing, this embodiment arranges emojis under the same topic into a numbered candidate emoji input set graph, enabling the multimodal model to read multiple candidate emojis at once. The improved RT-DETR model further utilizes an object query group organization structure and a Transformer decoder with a cross-layer inheritance structure for query groups. This maintains the region queries within the same candidate emoji as query groups and updates their state across layers, preventing ordinary detection models from breaking down the internal regions of an emoji into independent detection objects. The direct readout path of the prediction head is truncated before the prediction head readout path of the classification prediction head and the bounding box regression head. The candidate emoji is registered with the query and the response prompt mark, which constrains the generation of candidate response text. The response closure verification further filters the response object, response tone and region boundary, so that the final output emoji image and response text are more consistent in semantics, tone and image-text fit, which is suitable for the real-time interactive display scenario of smart AI toys.

[0080] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for emoji selection and text generation based on cue learning and multimodal models, characterized in that, Includes the following steps: S1. Extract the most recent rounds of dialogue from the current dialogue between the large model agent and the user, and compress them into dialogue prompts with the target of the conversation and the tone of the reply. S2. Organize the emoji set into a candidate emoji input set graph, and pair the candidate emoji input set graph with the reply prompts to form emoji inference samples; S3. An improved RT-DETR model is used to access emoji inference samples. A convolutional backbone network and an efficient hybrid encoder extract and fuse multi-scale visual features from the candidate emoji input set graph. S4, IoU perception query selection extracts object queries from multi-scale visual features. The object query group organization structure truncates the independent detection query tiling entry and groups the object queries in the same candidate emoji pack into query groups according to the adjacent arrangement relationship in the candidate emoji pack input set graph. S5. The Transformer decoder with query group cross-layer inheritance structure receives query groups. The object queries in each group output by each decoder layer are fed back to the next decoder layer according to the original group position, and candidate emojis are formed to accept queries along the inter-layer inheritance group arrangement relationship and update status. S6. The candidate emoji pack is truncated before the reading path of the prediction head. The query enters the reading path of the classification prediction head and the bounding regression head. The final layer position, region boundary and inter-layer update status of the query group are preserved. The emoji pack is registered with the answer prompt mark to constrain the generation of candidate reply text. S7. Perform a closed-loop verification on the candidate reply text and candidate emojis, block candidate combinations with broken loops, retain closed candidate combinations, and output emoji images and reply text.

2. The emoji selection and text generation method based on cue learning and multimodal models according to claim 1, characterized in that, S1 specifically includes: S11. Extract the most recent rounds of dialogue from the end of the current dialogue between the large model agent and the user, remove system control statements, and attach the remaining speech to the large model agent side and the user side to form a role-aligned dialogue segment. S12. Align the speaking order of the dialogue segment along the role and compare the user-side terminal speech with the adjacent speech on the large model agent side. Subtract the semantic content that has been taken over by the adjacent speech on the large model agent side to form the response segment. S13. Based on the core predicate in the response segment, retain the object component that is directly connected to the core predicate, and delete the modifier component that does not change the direction of the response to form the response object. S14. Merge polar expressions and sentence-final intonation in the response segment, and use sentence-final intonation to correct the direction of the response of polar expressions, forming the response tone; S15. Merge and compress the respondent and the reply tone in the order of respondent first and tone second to form a response prompt mark.

3. The emoji selection and text generation method based on cue learning and multimodal models according to claim 1, characterized in that, S2 specifically includes: S21. Using the theme affiliation in the emoji set as the sorting entry point, group emoji images under the same theme into theme candidate image groups; S22. Crops the emoji images in the candidate image group of the theme to the display boundary and arranges them in the grid area according to the display order of the set to form a theme input set map with the position in the map; S23. Align the grid areas in the topic input set diagram with the serial number markers one by one to form a candidate emoji input set diagram with the correspondence between the position and serial number in the diagram. S24. Pair the candidate emoji input set graph with the answer prompt markers in the same round, retain the position and sequence number markers in the candidate emoji input set graph, and compress them into emoji inference samples.

4. The emoji selection and text generation method based on cue learning and multimodal models according to claim 1, characterized in that, S3 specifically includes: S31, The improved RT-DETR model includes a convolutional backbone network, an efficient hybrid encoder, an IoU-aware query selection, an object query group organization structure, a Transformer decoder with a cross-layer inheritance structure of query groups, a prediction head readout path, and a prediction head; S32. The candidate emoji input set graph in the emoji inference sample is connected to the convolutional backbone network. The convolutional backbone network performs progressive convolution and downsampling on the candidate emoji input set graph, and retains the correspondence between the position in the graph and the serial number label in each level of feature to form multi-scale candidate visual features. S33. The high-efficiency hybrid encoder performs same-scale feature interaction on multi-scale candidate visual features, compressing the boundaries of emoji images, grid region boundaries and local semantic content within the same scale into same-scale encoded features. S34. The efficient hybrid encoder performs cross-scale fusion of the same-scale encoded features, aligns the shallow boundary features and deep semantic features along the in-graph position of the candidate emoji input set graph, and forms multi-scale visual features for IoU-aware query selection to extract object queries.

5. The emoji selection and text generation method based on cue learning and multimodal models according to claim 1, characterized in that, S4 specifically includes: S41. IoU perceptual query selection performs candidate quality readout on the encoded feature positions in multi-scale visual features, selects encoded feature positions and region boundaries in order of candidate quality, and forms a set of object queries to be organized. S42. The object queries query group organization structure is truncated. The set of object queries to be organized is directly flattened into the entry point of the Transformer decoder. The position of the encoded feature carried by each object query is matched with the position and sequence number of the candidate emoji input set graph to form position-matched object queries. S43. The object queries query group organization structure verifies the location of object queries within the image. Object queries falling within the same candidate emoji image range are grouped into the same candidate emoji internal set. Object queries that cross adjacent image ranges are back-matched along the region boundary to the candidate emoji internal set with the largest boundary overlap area. S44. Object queries that do not fall within the range of candidate emoji images are matched back to the corresponding candidate emoji set along the grid region with the smallest positional distance in the image. If the positional distances in the image are the same, they are matched back along the grid region with the earlier sequence number. S45. If the image position of the missing object query is in the same candidate emoji set, retrieve the encoded feature position of the corresponding image position from the multi-scale visual features to generate a placeholder query. S46. The object queries group organization structure organizes the object queries and placeholder queries within the same candidate emoji set according to the adjacent arrangement relationship in the candidate emoji input set graph, and retains the graph position and sequence number label corresponding to each object query and placeholder query. S47. Group the sorted object queries and placeholder queries according to their position within the same candidate emoji image to form query groups.

6. The emoji selection and text generation method based on cue learning and multimodal model according to claim 1, characterized in that, S5 specifically includes: S51. The Transformer decoder with query group cross-layer inheritance structure receives the query group at the input of the first decoder layer and establishes the original group position arrangement along the graph position order of object queries and placeholder queries in the query group. S52. The first decoder layer performs self-attention interaction along the original group position arrangement. Object queries within the same query group exchange query content, graph position and region boundary to form the first layer group update query. S53. The first decoder layer performs cross-attention interaction between the update query within the first layer group and the multi-scale visual features, and fills the corresponding object query with the encoded features of the corresponding position in the image to form the first layer output object queries. S54. The cross-layer inheritance structure of query groups truncates the input path of independent object queries between adjacent decoder layers, and feeds back the object queries within the group output by the previous decoder layer to the input of the next decoder layer according to the original group position, while retaining the query content, region boundaries and update status formed by the previous decoder layer. S55. The next decoder layer continues to perform self-attention interaction and cross-attention interaction along the original group bit arrangement after the reconnection. When the original group bit is missing an object query, it is filled in by the update state of the adjacent original group bit in the same query group. When the adjacent original group bit does not have an update state, it is taken over by the placeholder query in the query group along the original group bit. S56. The query group cross-layer inheritance structure performs a disconnection check on the original group bit arrangement of the output of adjacent decoder layers. When the original group bit arrangement is continuous, the current decoder layer update state is retained. When the original group bit arrangement is disconnected, the current decoder layer output is corrected by the original group bit arrangement of the previous decoder layer. S57. The object queries within the group output by the last decoder layer are arranged and merged along the original group position, and the query content, region boundary and inter-layer update status corresponding to each object query are retained to form candidate emoji packs to carry out the query.

7. The emoji selection and text generation method based on cue learning and multimodal model according to claim 1, characterized in that, S6 specifically includes: S61. The path before the prediction head is read out to receive the candidate emojis and receive the query, and the object queries are arranged along the final layer of the query group to form the query arrangement before the prediction head. S62. Before the prediction head readout path enters the classification prediction head and bounding box regression head, the direct readout path of classification score and bounding box coordinates is truncated, and the query content, region boundary and inter-layer update status of each object query are retained as the query status before readout. S63. The pre-read path of the prediction head aligns the region boundary in the pre-read query state with the position in the graph of the candidate emoji input set graph. When the region boundary goes beyond the range of the corresponding candidate emoji graph, it is cropped along the range of the corresponding candidate emoji graph to form a boundary calibration query state. S64. Predict the reading path along the final layer group position of the query group, calibrate the inter-layer update state in the query state, retain the continuously updated object queries in the same query group as text constraint queries, and replace the object queries with interrupted updates with placeholder queries for the corresponding group positions. S65. The predicted head-read path registers the text constraint query with the response prompt mark, aligns the response object with the query content, and aligns the response tone with the inter-layer update status to form candidate text constraint queries. S66. Candidate text constraint query constraint candidate response text generation. When the generated text content is separated from the respondent, it will return to the query content. When the tone deviates from the response tone, it will return to the inter-layer update status. S67. Arrange the constrained terms into candidate response texts and retain the correspondence between the candidate response texts and the candidate text constrained queries.

8. The emoji selection and text generation method based on cue learning and multimodal model according to claim 1, characterized in that, Specifically, S7 includes: S71. Following the correspondence between candidate response texts and candidate emojis in the query, the candidate response texts are aligned with the query location and area boundaries to form candidate combinations to be verified. S72. Verify that the response object in the candidate combination to be verified is consistent with the response object in the call prompt mark. If the response object is not aligned, the corresponding candidate combination to be verified is blocked. S73. Verify the tone of the candidate combination to be verified with the tone of the reply prompt in the call prompt mark. If the tone conflicts, the corresponding candidate combination to be verified is blocked. S74. Verify the candidate response text in the candidate candidate combination to be verified with the boundary of the area reserved for candidate emoticons in the query with consistent tone. If the text exceeds the boundary of the area, the corresponding candidate candidate combination to be verified is blocked. S75. Retain the unblocked candidate combinations to be verified as closed candidate combinations, and extract the emoji images and reply text from the closed candidate combinations.

Images