Text interaction method and device for assisting social software interaction, equipment and medium

Abstract

The invention relates to a text interaction method and device for assisting social software interaction, equipment and a medium. The method comprises the steps of obtaining a screen image of current social software in response to an obtained auxiliary interaction instruction; the auxiliary interaction instruction is used for instructing the intelligent terminal to capture a screen image according to a corresponding screenshot mode and feed back the screen image in a standard format form; preprocessing the screen image to obtain a to-be-recognized image; performing text recognition on the to-be-recognized image to obtain structured text data; the structured text data comprises text content and role affiliation information; performing semantic analysis on the structured text data to obtain a formatted prompt template; and inputting the formatted prompt template into a pre-trained auxiliary social text interaction large model to obtain an auxiliary interaction text. By adopting the method, timely and accurate interaction support can be provided for the user when the user chats on the social software, and the time for thinking, language organization and searching of the user is saved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and particularly relates to a text interaction method, device, equipment and medium for assisting social software interaction. Background Art

[0002] With the development of artificial intelligence and natural language technology, intelligent text interaction technology has emerged. With powerful semantic understanding and text generation capabilities, it can provide reasonable and coherent dialogue suggestions according to the context and is widely used in scenarios such as intelligent customer service.

[0003] In the process of social software interaction, users often need to quickly and accurately reply to the messages of others. Usually, users manually edit the dialogue text based on personal thinking and diction organization ability. However, when encountering complex problems, users need to spend a lot of time thinking about appropriate replies; some users have difficulties in language expression and are prone to problems such as improper diction and unclear information expression. To solve the above problems, some social applications have introduced intelligent recommended reply functions. Commonly, it is to perform associative matching of preset phrases based on the keywords input by the user, or to pre-define fixed response modes and perform automatic replies after detecting specific trigger conditions.

[0004] However, the above methods usually cannot understand the complete dialogue logic and can only perform matching based on single sentences, making it difficult to handle complex conversations, resulting in the recommended replies lacking coherence, having a single answer content and lacking diversity. Automatic replies based on templates are difficult to generate personalized answers and cannot face open-ended conversations, emotional expressions or long text exchanges, making the interaction appear mechanical. In addition, the embedded intelligent recommended reply function cannot cover multiple social software and has limitations in application. Summary of the Invention

[0005] Based on this, it is necessary to provide a text interaction method, device, equipment and medium for assisting social software interaction that can provide high-quality and logical assisted replies through semantic analysis and intention classification combined with intelligent dialogue generation and warning optimization for the above technical problems.

[0006] In a first aspect, the present application provides a text interaction method for assisting social software interaction, including:

[0007] In response to the obtained auxiliary interaction instruction, obtain the screen image of the current social software; the auxiliary interaction instruction is used to instruct the intelligent terminal to capture the screen image according to the corresponding screenshot mode and feedback it in the form of a standard format;

[0008] Preprocess the screen image to obtain an image to be recognized;

[0009] Perform text recognition on the image to be recognized to obtain structured text data; the structured text data includes text content and role attribution information;

[0010] Perform semantic parsing on the structured text data to obtain a formatted prompt template;

[0011] Input the formatted prompt template into a pre-trained auxiliary social text interaction large model to obtain auxiliary interaction text.

[0012] In one embodiment, performing text recognition on the image to be recognized to obtain structured text data includes:

[0013] Adopt a text region segmentation algorithm to detect text regions in the image to be recognized, and obtain each text region and its corresponding text box coordinates;

[0014] Arrange the text box coordinates corresponding to each text region in a preset order to obtain a time series;

[0015] Based on a text recognition cascade model, perform text content recognition on each text region and perform optimization processing to obtain text content;

[0016] Analyze the element correspondence relationship between each text region and a preset UI control to determine the role attribution information corresponding to each text content;

[0017] Arrange each text content and its corresponding role attribution information in a time series to obtain structured text data.

[0018] In one embodiment, based on a text recognition cascade model, performing text content recognition on each text region and performing optimization processing to obtain text content includes:

[0019] Perform character recognition on each text region to obtain corresponding initial text and confidence scores;

[0020] If the confidence score exceeds a preset threshold, perform content cleaning on the initial text to obtain text content;

[0021] If the confidence score is lower than the preset threshold, re-recognize the corresponding text region until the confidence score exceeds the preset threshold, and perform content cleaning on the initial text corresponding to the confidence score exceeding the preset threshold to obtain the initial text.

[0022] In one embodiment, performing semantic parsing on the structured text data to obtain a formatted prompt template includes:

[0023] Perform semantic parsing on the text content according to the role attribution information to obtain an intention classification and a tone determination result;

[0024] Perform context association on the text content using the context window to obtain the context content;

[0025] Construct a formatted prompt template based on the intent classification, tone determination result, and context content.

[0026] In one embodiment, the pre-trained auxiliary social text interaction large model obtains the auxiliary interaction text through the following method:

[0027] Parse the formatted prompt template and generate the dialogue text according to the corpus;

[0028] Optimize the grammar variants of the dialogue text to obtain the structured dialogue text;

[0029] Filter sensitive content from the structured dialogue text to obtain the auxiliary interaction text.

[0030] In one embodiment, capture the screen image according to the corresponding screenshot mode and feedback it in the form of a standard format, including:

[0031] Identify the interaction type corresponding to the auxiliary interaction instruction; the interaction type includes short press operation and long press operation;

[0032] Determine the target screenshot type according to the preset mapping relationship between the interaction type and the screenshot mode; the target screenshot type includes short screenshot and long screenshot;

[0033] Capture the screen content of the intelligent terminal according to the target screenshot type and output the image in the standard format to obtain the screen image;

[0034] Among them, when the target screenshot type is a short screenshot, capture the screen image within the current visible area;

[0035] When the target screenshot type is a long screenshot, capture and splice the continuous screen content beyond the current visible area within the duration corresponding to the long press operation.

[0036] In one embodiment, the text interaction method for auxiliary social software interaction further includes: in response to the obtained regeneration instruction, insert random grammar variant markers into the formatted prompt template to obtain a new formatted prompt template;

[0037] Input the new formatted prompt template into the pre-trained auxiliary social text interaction large model to obtain a new auxiliary interaction text.

[0038] In a second aspect, the present application also provides a text interaction device for auxiliary social software interaction, including:

[0039] A data acquisition module, configured to obtain the screen image of the current social software in response to the obtained auxiliary interaction instruction;

[0040] A data processing module for preprocessing a screen image to obtain an image to be recognized;

[0041] A text recognition module for performing text recognition on the image to be recognized to obtain structured text data;

[0042] A semantic parsing module for performing semantic parsing on the structured text data to obtain a formatted prompt template;

[0043] A model calling module for inputting the formatted prompt template into a pre-trained large auxiliary social text interaction model to obtain auxiliary interaction text.

[0044] Thirdly, the present application also provides a computer device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps of any one of the above text interaction methods for assisting social software interaction are implemented.

[0045] Fourthly, the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any one of the above text interaction methods for assisting social software interaction are implemented.

[0046] The above text interaction method, device, equipment and medium for assisting social software interaction capture a screen image through a user instruction to obtain key information of the current social software interface, so as to facilitate the user to call the assistance for social software interaction at any time. The unified standard format enables data to be transmitted and circulated in different modules, reducing the error risk caused by format incompatibility. By preprocessing the screen image, the burden of text recognition is reduced, and the recognition speed and accuracy are increased. The text content and role attribution information are structurally integrated, facilitating subsequent semantic parsing and processing. Clearly defining the role attribution of each text content can help understand the context and intention of the conversation. Semantic parsing of the structured text data accurately grasps the user's needs and emotional tendencies, providing more standardized and targeted input for the pre-trained large auxiliary social text interaction model, and improving the quality and relevance of the auxiliary interaction text. Using the pre-trained large model can reduce the time for technology research and development, and can also mobilize the corpus, as well as the language understanding and generation capabilities originally possessed by the large model. Through step-by-step circulation and technical association, without the need for the user to input sentence by sentence, timely and accurate interaction support can be provided for the user during social software chatting, saving the user's time for thinking, organizing language and searching. Especially when the user needs to reply quickly or lacks inspiration, the auxiliary interaction text can help the user participate in communication and interaction activities more efficiently. Description of the Drawings

[0047] To more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the related art. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0048] Figure 1 Schematic flowchart of the text interaction method for assisting social software interaction according to the present invention;

[0049] Figure 2 Schematic sub-step flowchart of step S103;

[0050] Figure 3 Schematic sub-step flowchart of step S104;

[0051] Figure 4 Composition structure diagram of the text interaction device for assisting social software interaction according to the present invention. Detailed implementation manners

[0052] In order to make the objectives, technical solutions and advantages of the present application clearer, the following further details the present application in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0053] In one embodiment, as Figure 1 shown, a text interaction method for assisting social software interaction is provided. In this embodiment, it is exemplified that the method is applied to a terminal. It can be understood that the method can also be applied to a server, or to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0054] S101. In response to the obtained auxiliary interaction instruction, obtain the screen image of the current social software; the auxiliary interaction instruction is used to instruct the intelligent terminal to capture the screen image in the corresponding screenshot mode and feedback it in the form of a standard format.

[0055] Schematically, the operating system installed on the terminal provides an event handling mechanism for capturing various interaction behaviors of the user on the screen of the terminal device to constantly monitor the user operations. In a common graphical interface environment of a mobile device or a PC (Personal Computer), the user operations are mainly monitored by listening for touch events or mouse events. Exemplarily, a floating window is designed in the graphical environment interface. The floating window always floats above the application interface or the system interface for the user to wake up the text interaction for assisting social software interaction at any time. Specifically, when the user clicks or operates on the floating window, according to the design logic of the floating window, a specific event handler will be triggered, that is, an auxiliary interaction instruction is generated and sent. The auxiliary interaction instruction is used to instruct the intelligent terminal to capture the screen image in the corresponding screenshot mode and feedback it in the form of a standard format. The system obtains the screen image transmitted by the intelligent terminal and stores it for subsequent operations such as image recognition and semantic analysis to provide the user with an auxiliary social interaction function.

[0056] S102. Preprocess the screen image to obtain an image to be recognized.

[0057] The chat interface of social software has rich colors, including various chat backgrounds, emoji colors, text colors, etc. The diverse colors will increase the complexity of the image data, which may cause the character recognition algorithm to misjudge the character boundaries. Similarly, when there are noise, blurring, etc. in the image, it may cause errors in the character recognition algorithm during recognition, misidentifying the noise as a character or missing real characters, affecting the recognition accuracy.

[0058] Schematically, after receiving the screen image data uploaded by the front end through HTTPS (Hypertext Transfer Protocol Secure), the cloud server decodes the data, restores it to the original pixel matrix, and restores the original pixel information of the image. Further, for the accuracy of subsequent image recognition, the screen image is preprocessed. Specifically, in order to reduce the interference of the image color on subsequent text recognition, the weighted average method is used to convert the color image into a grayscale image. Exemplarily, according to the formula Y = 0.299R + 0.587G + 0.114B, the red (R), green (G), and blue (B) components of each pixel are calculated to obtain the corresponding grayscale value, thereby converting the color image into an image with only grayscale information and outputting a grayscale image matrix.

[0059] Perform binaryzation processing on the grayscale image using an adaptive threshold or a fixed threshold method to avoid interference from the chat background or watermarks of social software. Exemplarily, perform binaryzation processing on the grayscale image using an adaptive threshold method. The adaptive threshold dynamically adjusts the threshold according to the local features of the image. By comparing the grayscale value with the threshold, the pixel points in the image are divided into foreground and background. Specifically, set the text area to white with a value of 255 and the non-text area to black with a value of 0, thereby outputting a binary image to highlight the text area.

[0060] Perform denoising processing on the binaryzation image using Gaussian blur or median filtering. Exemplarily, perform denoising processing using Gaussian blur. Gaussian blur performs weighted averaging on the image pixels and determines the weights according to the Gaussian function, making the image smoother and thus removing noise.

[0061] S103. Perform text recognition on the image to be recognized to obtain structured text data; the structured text data includes text content and role attribution information.

[0062] Exemplarily, use the Optical Character Recognition (OCR) algorithm for text recognition. OCR can extract text information from the image and convert it into computer-readable character data. Exemplarily, in a screenshot of the chat interface of a social software, OCR can recognize the text content in the chat bubble and store it in text format for subsequent semantic analysis. Exemplarily, based on the Differentiable Binarization Network (DBNet), locate the text area of the image to be recognized; use a self-developed OCR model with the Convolutional Recurrent Neural Network (CRNN) architecture to convert the text in the text area into editable text; reconstruct the editable text to optimize the text output quality to obtain the text content. Among them, the text content is the text information displayed in the chat record, which may include the user's speech, the other party's reply, timestamp, emoticons, etc. The role attribution information refers to the identity of the speaker in the chat conversation, determining whether the message is sent by the user himself or the other party, so as to add corresponding role tags to the text content and provide a basis for subsequent emotion and intention analysis.

[0063] S104. Perform semantic parsing on the structured text data to obtain a formatted prompt template.

[0064] Semantic parsing refers to the in-depth understanding of the text content of structured text data, mining information such as meaning, intention, emotion, etc., so as to transform the text from a simple combination of characters into semantic information that can be understood and processed by a computer. Schematically, in the chat scenario of social software, semantic parsing needs to figure out the theme, purpose, emotional tendency of the chat content, as well as the logical relationship between different sentences, etc. Optionally, by means of deep neural networks such as recurrent neural network (RNN), long short-term memory network (LSTM), gated recurrent unit (GRU), and Transformer (neural network) architecture, the text is modeled and analyzed. The deep neural network can automatically learn the semantic features and context information of the text. Exemplarily, using the Transformer architecture for semantic parsing, the attention mechanism is used to focus on the associations between words in different positions, so as to more accurately understand the semantics of the text. Specifically, by analyzing the keywords, sentence patterns, and context information in the text, the intention category of the text can be judged to clarify the core purpose of the text, whether the other party is asking for information, making suggestions, expressing opinions, or making jokes, etc. Further, judge the emotional tendency expressed by the text, whether it is positive, negative, or neutral, so as to generate a reply that better conforms to the emotional atmosphere and improve the user experience. Considering the position of the text in the whole conversation and the association with the context before and after, understanding the theme and logical relationship of the conversation can avoid understanding each sentence in isolation and make the generated reply more coherent and accurate. Exemplarily, in a multi-round conversation, a certain event is mentioned earlier, and the subsequent reply needs to respond in combination with this event.

[0065] According to the information obtained from semantic parsing, a formatted prompt template is constructed in a specific format, which can include key contents such as intention category, emotional tendency, context information, etc., and is presented to the auxiliary social text interaction large model in a structured manner. The auxiliary large model can better understand the user's needs and conversation scenarios according to the information in the template, and generate more appropriate auxiliary interaction text.

[0066] S105. Input the formatted prompt template into the pre-trained auxiliary social text interaction large model to obtain auxiliary interaction text.

[0067] Pre-trained auxiliary social text interaction models, such as DeepSeek and ChatGPT, are trained on large-scale multi-domain data. The models have a strong data reserve for knowledge in different fields, so they have a strong text generation capability and can generate coherent and logically reasonable text based on the input content. Schematically, the formatting prompt template contains key information such as intent classification, tone judgment results, contextual content, etc. After receiving the template, the auxiliary social text interaction model matches and integrates the information with its own learned knowledge. For example, if the template indicates that the intention is to inquire and the context revolves around tourism, the model will retrieve tourism-related information from its knowledge reserve and generate auxiliary interaction text with reference value.

[0068] In the above-mentioned text interaction method for assisting social software interaction, by directly capturing the screen image, the more comprehensive and contextual text of social interaction can be obtained without the need for user word-by-word input. Through high-precision text recognition and structured data processing, the problem that traditional OCR can only extract pure text and lose context structure is avoided, and the ability to understand context is improved. By distinguishing the identity of the dialogue object, the response generated by the large model can be ensured to be consistent with the identity and context. The semantic intent, tone and context of semantic parsing recognition enable the subsequent large model output to conform to social habits and improve the level of intelligence. The pre-trained large model for assisting social text interaction can generate more accurate, comprehensive and in-depth responses with its rich knowledge reserves and powerful generation capabilities. Compared with simple rule matching responses, the text generated by the large model is more in line with human language habits and can better meet user needs. The generalization and understanding ability of the large model enable it to adapt to the complexity and variability of social scenarios and cope with various complex situations. Whether it is a vague question, implicit intention or complex logic in multiple rounds of dialogue, it can generate appropriate auxiliary interaction text to improve user experience.

[0069] In one embodiment, if Figure 2 As shown, text recognition is performed on the image to be recognized to obtain structured text data, including:

[0070] S201 , using a text region segmentation algorithm to perform text region detection on an image to be recognized, and obtaining each text region and corresponding text box coordinates.

[0071] Schematically, a text region segmentation algorithm based on the DBNet deep network is used to locate the text region. The DBNet deep network includes a backbone network, a feature pyramid network (FPN), and a differentiable binarization module (DB Head). Among them, the backbone network can adopt pre-trained convolutional neural networks such as ResNet (residual network) and ResNeXt (cardinal residual network) to extract the features of the image to be recognized. Exemplarily, ResNet is used for feature extraction. Through a series of convolutional layers and pooling layers, the input image is gradually downsampled, and features at different levels are extracted simultaneously. These features include the strokes of the text and the shapes of the characters. Further, the FPN in the feature pyramid fuses the features at different levels of the backbone network. Through upsampling and lateral connection operations, feature maps with different resolutions are generated. Among them, the low-resolution feature map contains global information and is suitable for detecting large-size text; the high-resolution feature map contains more detailed information and is suitable for detecting small-size text. Furthermore, the differentiable binarization module receives the feature map from the feature pyramid, processes the feature map through convolutional layers, obtains the probability map and threshold map of the text region, and uses the differentiable binarization function to convert the probability map and threshold map into the final binarized text region prediction result, obtaining the predicted text region and the binarized threshold map, that is, determining which regions in the image contain text and accurately obtaining the bounding box coordinates of each text region, marking the position of the text in the form of a rectangular box, and outputting each text region and the corresponding text box coordinates.

[0072] S202. Arrange the text box coordinates corresponding to each text region in a preset order to obtain a time series.

[0073] Schematically, in the chat text of the image to be recognized, the messages that appear first are usually the basis for subsequent conversations. Subsequent messages will respond to, supplement, or expand the discussion based on the previous content. If the text boxes are not arranged in the "top-down" order, it may lead to confusion in the conversation logic and make it difficult to accurately understand the complete meaning of the conversation. Furthermore, according to the text reading habit in the language environment, the text forms a paragraph in the "left-to-right" order. Therefore, the y-coordinates (ordinates) of the text boxes are arranged in descending order according to the "top-down" principle, that is, the later the text appears, the lower its position; for the text boxes in the same row, they are arranged in ascending order according to the x-coordinate according to the "left-to-right" principle, so as to obtain a time series that conforms to people's reading habits and facilitate subsequent processing of the text content in order.

[0074] S203. Recognize the text content of each text region based on the text recognition cascade model and perform optimization processing to obtain the text content.

[0075] Schematically, the text recognition cascade model is composed of a CRNN model and a CTC (Connectionist Temporal Classification) decoding structure. Among them, the CRNN model includes convolutional layers, recurrent layers, and fully connected layers. The convolutional layer is mainly responsible for extracting the features of the text area image. It performs convolution operations by sliding the convolution kernel on the image and automatically learns the local features in the image, such as the strokes, contours, corners and other character features of the characters; the recurrent layer is used to process information with sequence characteristics. In text recognition, each character can be regarded as an element in the sequence. Usually, a bidirectional LSTM layer (Long Short-Term Memory) is used to model the character sequence. LSTM contains an input gate, a forget gate, and an output gate, which can effectively avoid the gradient vanishing problem, enabling it to process longer text sequences, remember the important information that appeared in the front, and apply it in subsequent recognition. For example, to recognize the text "The weather is very good today", the forward LSTM starts with the character "今", and processes each Chinese character one by one. When processing the character "今", it will learn the characteristic information of the character "今" and memorize it; when processing the character "天", it will combine the information of the character "今" and the characteristics of the character "天" itself to further understand this sequence. The reverse LSTM processes from the last character "好" forward, so that the bidirectional LSTM can make full use of the information of the context. For example, when the forward LSTM processes the character "气", it knows that the previous character is "今日天", and it can be inferred that there is a high probability that the content here is related to the weather; when the reverse LSTM processes the character "气", it can combine the information of the following "真的好" to better understand the meaning of the character "气" in this context. In this way, the bidirectional LSTM learns the contextual relationship between characters, captures the complex contextual dependencies between Chinese characters or English in a sentence, and improves the understanding of Chinese semantics; the fully connected layer maps the feature vector output by the recurrent layer to the character category space, obtains the probability distribution of each character belonging to different categories, and integrates all the previous features, outputs the same number of nodes as the character categories, and each node corresponds to the prediction score of a character category, and then converts the prediction score into a probability value through the Softmax (activation function) function, thereby determining the most likely character for each position. For example, the fully connected layer calculates the possible probabilities of all Chinese characters such as "金", "今", "进", etc. for the first position based on the features output by the previous LSTM, and finally selects "今" with the highest probability as the prediction result.

[0076] The CTC algorithm is applied to segment characters. By allowing the model to directly predict the entire text sequence without the need to know the exact boundaries of each character in advance, even if there are adhesions or overlaps between characters, CTC can determine the category of each character through the analysis of the entire sequence. Further, the CTC algorithm introduces a special alignment mechanism to align the character sequence predicted by the model with the true label. Specifically, it introduces a blank label in the predicted sequence. When the model predicts the blank label, it means that there is no valid character at the current position. In this way, CTC can handle the situation where the lengths of the predicted sequence and the true label are inconsistent, find the best alignment between the predicted sequence and the true label, calculate the loss between the two, and thus guide the training of the model. Exemplarily, in the predicted sequence "apple <blank> <blank>”Among them, the CTC will remove the blank tags according to the probability information to obtain the final recognition result "apple".

[0077] S204. Analyze the corresponding relationship between each text area and the elements of the preset UI control to determine the role attribution information corresponding to each text content.

[0078] The preset UI (User Interface) control includes the layout rules of the chat interface of common social software. Schematically, the distribution rules of the commonly used UI controls in social software include that the user's speech text is close to the right side of the picture area, and the other party's speech text is close to the right side of the picture area, etc. Exemplarily, it can be judged according to the x coordinate (abscissa) of the text box. If the x coordinate of the center point of the text box is in the right half of the screen, that is, it occupies more than 50% of the screen width, then it is judged that the text is the user's speech; if the x coordinate is in the left half of the screen, then it is judged as the other party's speech. In addition, the color feature can be further combined for auxiliary judgment. Sample the pixel color within 5px - 10px around the text box. If the color belongs to the "own message color" preset in the UI, then it belongs to the user; otherwise, it belongs to the other party. Specifically, if the color of the coordinate occupying the right half of the screen is green, then green belongs to the user's color, and if the color of the coordinate occupying the left half of the screen is white, then white belongs to the other party's color. For some social software, the avatar can also be combined for judgment. The avatar is generally close to the user's speech. By detecting whether there is an avatar area on the left or right side of the chat bubble, the correctness of the role attribution can be further verified.

[0079] S205. Arrange each text content and the corresponding role attribution information in time series to obtain structured text data.

[0080] Arrange each text content and the corresponding role attribution information according to the previously generated time series to form structured text data, so as to associate the context information later. Each text content and its role attribution information are integrated together, which is convenient for subsequent semantic analysis to clearly understand the order and source of the text and provide support for accurately parsing the conversation content.

[0081] The above method, through the text area segmentation algorithm, accurately locates the area where the text is located in the image to be recognized, can effectively separate the text area from the complex background, obtain the text box coordinates corresponding to each text area, represent the text area in a digital way, and further form a time series using the text box coordinates, so that when processing continuous conversations, the context information before and after can be accurately associated according to the time series, which helps to better understand the semantics and intentions of the text. The text recognition cascade model can adapt to different text display fonts set by the user system for effective recognition, enhancing the generality and applicability. Clearly defining the role attribution information can support in-depth analysis of the interaction between different roles.

[0082] In one embodiment, the text content of each text region is recognized based on a text recognition cascade model and optimized to obtain the text content, including:

[0083] S31. Character recognition is performed on each text region to obtain the corresponding initial text and confidence score.

[0084] The text region image is input into the CRNN model. The convolutional layer extracts image features, and the bidirectional LSTM performs sequence modeling on the feature map to learn the context relationship between characters and outputs the feature vector at each time step. The fully connected layer maps the feature vector to the character category space to obtain the probability distribution of each character category, forming the predicted character sequence. The CTC algorithm decodes the predicted character sequence to find the most likely true character sequence and generates a confidence score according to the probability. Schematically, the confidence score is used to measure the reliability of the recognition result. If the prediction probability of a certain character in all possible character categories is significantly higher than that of other characters, then its corresponding confidence score will also be higher.

[0085] S32. If the confidence score exceeds the preset threshold, the initial text is cleaned to obtain the text content.

[0086] The preset threshold can be a reference value determined according to actual application requirements and a large number of experiments to balance recognition efficiency and accuracy on the premise of ensuring a high recognition accuracy. When the confidence score of a character exceeds the preset threshold, it indicates that the recognition result of the model for this character is relatively reliable, but there may still be some noise or unreasonable characters. Content cleaning is to further process the initial text, including removing interference information, text planning methods, and spelling and grammar correction. Schematically, filtering emoji, etc. to prevent interference with semantic parsing, converting colloquial expressions into normative text, or correcting format errors that occur during text recognition. By cleaning the initial text, noise and incorrect characters are removed, thereby obtaining more accurate and standardized text content.

[0087] S33. If the confidence score is lower than the preset threshold, the corresponding text region is recognized again until the confidence score exceeds the preset threshold, and the initial text corresponding to the confidence score exceeding the preset threshold is cleaned to obtain the initial text.

[0088] If the confidence score of a certain character is lower than the preset threshold, the corresponding text region is recognized again. Schematically, the accuracy of recognition can be improved by adjusting the parameters of the CRNN model, increasing training data, or using ensemble learning, etc. The recognition result is cleaned to remove noise characters, correct typos, etc., and finally accurate text content is obtained.

[0089] In the above method, the confidence score serves as an important reference indicator, providing a basis for subsequent content cleaning and re-identification operations, helping to distinguish which recognition results are relatively reliable and which may have errors, so as to handle them targeted and improve the overall processing efficiency and accuracy. Content cleaning further improves the quality of the text content, making the finally output text more accurate, standardized and easy to understand.

[0090] In one embodiment, as Figure 3 shown, semantic parsing is performed on the structured text data to obtain a formatted prompt template, including:

[0091] S301. Perform semantic parsing on the text content according to the role attribution information to obtain the intent classification and tone determination results.

[0092] Illustratively, according to the role attribution information in the structured text data, a semantic classification model of a text classifier based on Transformer is used to perform semantic parsing on the text content. The role attribution information clarifies whether the text content is from the user himself or the chat partner. The speech purposes and emotional expressions of different roles are often different. In the chat scenario of social software, knowing that a certain piece of text is a question sent by the user, compared with it being a reply from the chat partner, the direction of semantic parsing will be very different. The semantic classification model learns the intent features expressed by different sentence patterns through training on a large amount of text data. The intent classification includes inquiry, suggestion and teasing. For texts containing interrogative words such as "how", "what", etc., the model will tend to classify their intent as inquiry; when words such as "suggestion", "might as well", "can be considered" appear in the text, it may be determined as a suggestion intent.

[0093] Illustratively, an emotion analysis model is used to judge the tone of the text. The model will comprehensively consider information such as the words, grammar structure and punctuation marks in the text. Positive words such as "great", "so happy", "like it very much", etc. usually indicate a positive tone; negative words such as "annoyed to death", "too bad", "hate it really" reflect a negative tone; while texts with neutral words and objective expressions have a neutral tone determined, such as "The weather is nice today". Some special punctuation marks, such as exclamation marks may enhance the emotional expression, and question marks may imply a doubtful tone. Through these analyses, determine whether the tone of the text is positive, negative, neutral or humorous, etc., to help understand the emotional color behind the text.

[0094] S302. Use the context window to perform context association on the text content to obtain the context content.

[0095] Within the context window, the system extracts key information such as mentioned people, places, events, etc., and associates them with the current text content. By analyzing factors such as lexical repetition and semantic coherence, a complete context is constructed. Schematically, a context corpus is constructed using the context window, and context association is performed in combination with the current text content. By analyzing the key information, lexical usage, semantic logic, etc. in the context corpus, the context content related to the current text is extracted, making the semantic analysis more accurate and comprehensive, taking into account the coherence and logic of the conversation. Exemplarily, if it is mentioned earlier that "I plan to travel to Beijing", and then it is said that "There are many scenic spots there", the system can understand through context association that "there" refers to Beijing, and can also consider the logical relationship of the conversation, such as causal relationship, turning relationship, etc. If it is said earlier that "I didn't bring an umbrella", and then it is said that "So I can only take a taxi home", the system can recognize this causal connection, thus understanding the text meaning more accurately, obtaining rich context content, and constructing a comprehensive language scenario.

[0096] S303. Construct a formatted prompt template according to the intention classification, tone determination result and context content.

[0097] According to the intention classification, tone determination result and context content, construct a formatted prompt template according to the predefined format, integrate the information together, form input data with a specific structure and content, and provide accurate and rich information for subsequent input into the pre-trained auxiliary social text interaction large model, guiding the model to generate more context-compliant answers.

[0098] The above method brings significant beneficial effects to the construction of a formatted prompt template through multi-dimensional analysis of the text content. Analyzing the semantics according to the role attribution information can accurately judge the intention classification and tone, and help understand the user's needs and emotional tendencies. Using the context window to associate content can make the text meaning more complete and the logic more coherent. Constructing a formatted prompt template based on the above results provides comprehensive and accurate input for subsequent model processing, improves the accuracy of text understanding, and can also make the generated auxiliary interactive text more in line with the actual scenario, greatly optimizing the user interaction experience and enhancing the practicality and intelligence of the system in social text processing.

[0099] In one of the embodiments, the pre-trained auxiliary social text interaction large model obtains the auxiliary interactive text through the following method:

[0100] S41. Analyze the formatted prompt template and generate conversation text according to the corpus.

[0101] After receiving the formatted prompt template, large pre-trained auxiliary social text interaction models such as DeepSeek and ChatGPT parse it. Schematically, the template contains key information such as intent classification, tone determination results, context content, and the original text. The model understands its meaning through an internal language understanding mechanism. Exemplarily, when the intent classification in the template is "question", the model realizes that it needs to generate a text to answer the question; if the tone determination is "positive", the generated text should use a positive and friendly language style. Further, the model constructs a corpus based on a large amount of text data learned during training, understands the input intent, tone, and context information, and generates corresponding dialogue text, initially giving an answer content. Schematically, according to the parsed template information, the model retrieves relevant knowledge and language expressions from the corpus. If the context content involves tourism, the model will look for tourism-related information in the corpus, such as scenic spot introductions, food recommendations, travel guides, etc. The model will combine this information and generate preliminary dialogue text according to certain logical and language generation rules. For the question "What are the delicious foods in Beijing?", the model may extract food information such as Peking duck, Mongolian hot pot, and noodles with soybean paste from the corpus and organize it into a dialogue text like "There are many delicious foods in Beijing, such as Peking duck, with crispy skin and tender meat; Mongolian hot pot, with a delicious taste; and noodles with soybean paste, with chewy noodles and rich sauce."

[0102] S42. Optimize the grammatical variants of the dialogue text to obtain a structured dialogue text.

[0103] To increase the diversity and naturalness of the generated text and avoid the generated answers being too monotonous and mechanical, it is necessary to optimize the grammatical variants of the initially generated dialogue text to make the dialogue text more in line with human language habits and improve the user experience. Schematically, the grammatical variant optimization methods include adjusting sentence structures, replacing words, changing sentence patterns, etc. Exemplarily, for the sentence "There are many delicious foods in Beijing, such as Peking duck, with crispy skin and tender meat; Mongolian hot pot, with a delicious taste; and noodles with soybean paste, with chewy noodles and rich sauce.", the sentence structure can be adjusted to "There are quite a few delicious foods in Beijing, like Peking duck with crispy skin and tender meat, Mongolian hot pot with a delicious taste, and noodles with soybean paste with chewy noodles and rich sauce." By replacing words, changing "many" to "quite a few" and "such as" to "like", the text expression becomes more rich. The sentence pattern can also be changed, changing the declarative sentence to a rhetorical question, such as "Isn't there a lot of delicious food in Beijing? Like Peking duck, with crispy skin and tender meat; Mongolian hot pot, with a delicious taste; and noodles with soybean paste, with chewy noodles and rich sauce." Through these methods, the dialogue text is optimized to obtain a structured dialogue text, making its expression more diverse and natural and fluent while maintaining the same semantics.

[0104] S43. Filter sensitive content from the structured dialogue text to obtain an auxiliary interaction text.

[0105] In a social interaction scenario, in order to ensure that the generated auxiliary interaction text complies with ethical norms, laws and regulations, as well as the privacy requirements of users, it is necessary to filter sensitive content from structured dialogue text. Sensitive content includes, but is not limited to, sensitive words such as swear words and insulting words, personal privacy information, and information that may cause controversy or adverse effects. Illustratively, a pre-set sensitive word library can be used for matching detection. The sensitive word library contains various sensitive words and phrases. When the words in the structured dialogue text match the words in the sensitive word library, the system will process them. The processing method can be to replace sensitive words, replace swear words with civilized expressions; or delete sensitive content. If personal privacy information is detected, the relevant text is directly deleted. The filtering system can also combine semantic analysis technology to understand the semantics of the text to detect content that is sensitive in meaning although no sensitive words are directly used. After filtering sensitive content, an auxiliary interaction text that meets the requirements is obtained to ensure the security and appropriateness of the information transmitted in social interactions.

[0106] In one embodiment, a screen image is captured according to the corresponding screenshot mode and fed back in a standard format, including:

[0107] S51. Identify the interaction type corresponding to the auxiliary interaction instruction; the interaction type includes a short press operation and a long press operation.

[0108] Illustratively, when the user's finger touches the floating window, the system starts to record the touch time. If the touch time is short and does not exceed the preset threshold, it is determined as a short press operation; if the touch time is long and exceeds the preset threshold, it is determined as a long press operation. Exemplarily, the preset threshold is 500 milliseconds.

[0109] S52. Determine the target screenshot type according to the preset mapping relationship between the interaction type and the screenshot mode; the target screenshot type includes a short screenshot and a long screenshot.

[0110] Illustratively, after the system identifies the interaction type, it quickly determines the target screenshot type according to the mapping relationship. If it is a short press operation, it corresponds to the short screenshot mode; a long press operation corresponds to the long screenshot mode to prepare for subsequent screenshot operations.

[0111] S53. Capture the screen content of the intelligent terminal according to the target screenshot type and output the image in a standard format to obtain a screen image.

[0112] Whether it is a short screenshot or a long screenshot, the captured image needs to be converted into a standard format. Schematically, use Canvas (an HTML5 tag for constructing website content) or corresponding tools to convert the captured image into a base64 string or binary stream in PNG (Portable Network Graphics) or JPEG (Joint Photographic Experts Group) format, for more convenient transmission and subsequent processing in the network, and finally obtain screen image data that meets the requirements.

[0113] S54. When the target screenshot type is a short screenshot, capture the screen image within the current visible area.

[0114] Schematically, in the short screenshot mode, the PC side or the Web side (browser) calls platform interfaces such as getDisplayMedia (a web application interface), and the mobile side calls platform interfaces such as MediaProjection API (a web application interface) to directly capture the screen image within the current visible area.

[0115] S55. When the target screenshot type is a long screenshot, splice and capture the continuous screen content that exceeds the current visible area during the duration corresponding to the long press operation.

[0116] In the long screenshot mode, during the long press of the user, the system continuously calls the corresponding interface to obtain continuous screen frames, so as to capture the screen content that exceeds the current visible area, that is, the newly appeared screen content below the current visible area. Specifically, set a sliding window to continuously capture the screen to obtain multiple continuous screen frames, and use image stitching technology to merge the multiple continuous screen frames into a complete long picture. Exemplarily, use image stitching algorithms based on feature point matching such as SIFT (Scale-Invariant Feature Transform) or ORB (Oriented FAST and Rotated BRIEF) for stitching processing, detect the feature points of the image, find the overlapping area between adjacent images, and perform alignment and fusion processing to obtain a stitched long picture without breaks or misalignments.

[0117] In one embodiment, the method further includes:

[0118] S61. In response to the obtained regenerate instruction, insert random syntax variant markers into the formatted prompt template to obtain a new formatted prompt template;

[0119] S62. Input the new formatted prompt template into the pre-trained auxiliary social text interaction large model to obtain a new auxiliary interaction text.

[0120] Schematically, when the user clicks the "Regenerate" button to send a regeneration instruction, and the system receives the regeneration instruction, it will randomly insert syntax variant markers at appropriate positions in the formatted prompt template. Schematically, at the location of a word or phrase with multiple expressions, one or more appropriate words are randomly selected and marked. Exemplarily, for the formatted prompt template "[Question]: What are the delicious foods in Beijing?, [Neutral], [Previously mentioned planning to travel to Beijing and currently discussing information related to Beijing]", a marker can be inserted at "delicious foods" to obtain a new formatted prompt template "[Question]: What are the [VAR:delicious foods] in Beijing?, [Neutral], [Previously mentioned planning to travel to Beijing and currently discussing information related to Beijing]". Then, the new formatted prompt template is input into the pre-trained large auxiliary social text interaction model to re-output the auxiliary interaction text, and it is ensured that the auxiliary interaction texts generated multiple times are different.

[0121] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are sequentially shown according to the indication of the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear indication in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same moment, but can be executed at different moments. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0122] Based on the same inventive concept, the embodiments of the present application also provide an auxiliary social software interaction text interaction device for implementing the text interaction method for auxiliary social software interaction described above. The implementation solutions for solving problems provided by this device are similar to the implementation solutions described in the above method. Therefore, the specific limitations in one or more embodiments of the following auxiliary social software interaction text interaction device can refer to the limitations on the text interaction method for auxiliary social software interaction in the above text, and will not be repeated here.

[0123] In an exemplary embodiment, as Figure 4 shown, an auxiliary social software interaction text interaction device is provided, including:

[0124] A data acquisition module, configured to acquire the screen image of the current social software in response to the acquired auxiliary interaction instruction;

[0125] A data processing module, configured to preprocess the screen image to obtain an image to be recognized;

[0126] A text recognition module, which is used to perform text recognition on the image to be recognized to obtain structured text data;

[0127] A semantic parsing module, which is used to perform semantic parsing on the structured text data to obtain a formatted prompt template;

[0128] A model calling module, which is used to input the formatted prompt template into a pre-trained large auxiliary social text interaction model to obtain auxiliary interaction text.

[0129] In one embodiment, it further includes a text region extraction module, a timing module, a text region recognition module, and a role discrimination module;

[0130] The text region extraction module is used to detect text regions in the image to be recognized by using a text region segmentation algorithm, and obtain each text region and the corresponding text box coordinates;

[0131] The timing module is used to arrange the text box coordinates corresponding to each text region in a preset order to obtain a time series;

[0132] The text region recognition module is used to recognize the text content of each text region based on a text recognition cascade model, and perform optimization processing to obtain the text content;

[0133] The role discrimination module is used to analyze the element correspondence relationship between each text region and a preset UI control to determine the role attribution information corresponding to each text content.

[0134] In one embodiment, the text region recognition module is further used to perform character recognition on each text region to obtain the corresponding initial text and confidence score.

[0135] In one embodiment, the semantic parsing module is further used to perform semantic parsing on the text content according to the role attribution information to obtain an intention classification and a tone determination result;

[0136] The semantic parsing module is further used to perform context association on the text content by using a context window to obtain context content.

[0137] In one embodiment, it further includes a decoding module, an optimization module, and a filtering module;

[0138] The decoding module is used to parse the formatted prompt template and generate dialogue text according to a corpus;

[0139] The optimization module is used to optimize the grammar variants of the dialogue text to obtain structured dialogue text;

[0140] The filtering module is used to filter sensitive content from the structured dialogue text to obtain auxiliary interaction text.

[0141] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps in the above method embodiments are implemented.

[0142] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0143] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial descriptions of the method embodiments. The device embodiments described above are only illustrative. The components described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present disclosure solution. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0144] The above embodiments only represent several implementation manners of the embodiments of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the embodiments of the application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the embodiments of the present application, several deformations and improvements can still be made, and these all belong to the protection scope of the embodiments of the present application.< / blank> < / blank>

Claims

1. A text interaction method for assisting social software interaction, characterized in that, The method includes: In response to the acquired auxiliary interaction instruction, obtaining a screen image of the current social software; the auxiliary interaction instruction is used to instruct the intelligent terminal to capture the screen image according to the corresponding screenshot mode and feedback it in the form of a standard format; Preprocessing the screen image to obtain an image to be recognized; Performing text recognition on the image to be recognized to obtain structured text data; the structured text data includes text content and role attribution information; Performing semantic parsing on the structured text data to obtain a formatted prompt template; Inputting the formatted prompt template into a pre-trained large auxiliary social text interaction model to obtain auxiliary interaction text.

2. The method according to claim 1, wherein The performing text recognition on the image to be recognized to obtain structured text data includes: Using a text region segmentation algorithm to detect text regions in the image to be recognized, obtaining each text region and the corresponding text box coordinates; Arranging the text box coordinates corresponding to each text region in a preset order to obtain a time series; Performing text content recognition on each text region based on a text recognition cascade model and performing optimization processing to obtain the text content; Analyzing the element correspondence relationship between each text region and a preset UI control to determine the role attribution information corresponding to each text content; Arranging each text content and the corresponding role attribution information in the time series to obtain the structured text data.

3. The method according to claim 2, wherein The performing text content recognition on each text region based on a text recognition cascade model and performing optimization processing to obtain the text content includes: Performing character recognition on each text region to obtain a corresponding initial text and a confidence score; If the confidence score exceeds a preset threshold, cleaning the content of the initial text to obtain the text content; If the confidence score is lower than the preset threshold, re-recognizing the corresponding text region until the confidence score exceeds the preset threshold, and cleaning the content of the initial text corresponding to the confidence score exceeding the preset threshold to obtain the initial text.

4. The method according to claim 1, wherein The performing semantic parsing on the structured text data to obtain a formatted prompt template includes: Performing semantic parsing on the text content according to the role attribution information to obtain an intention classification and a tone determination result; Using a context window to perform context association on the text content to obtain context content; Constructing a formatted prompt template according to the intention classification, the tone determination result, and the context content.

5. The method according to claim 1, wherein The pre-trained large auxiliary social text interaction model obtains auxiliary interaction text through the following method: Parsing the formatted prompt template and generating a dialogue text according to a corpus; Performing grammar variant optimization on the dialogue text to obtain a structured dialogue text; Performing sensitive content filtering on the structured dialogue text to obtain the auxiliary interaction text.

6. The method according to claim 1, characterized in that, The capturing the screen image according to the corresponding screenshot mode and feedbacking it in the form of a standard format includes: Identifying the interaction type corresponding to the auxiliary interaction instruction; the interaction type includes a short press operation and a long press operation; Determine a target screenshot type according to a preset mapping relationship between the interaction type and the screenshot mode; the target screenshot type includes short screenshots and long screenshots; Capture the screen content of the intelligent terminal according to the target screenshot type and output the image in a standard format to obtain a screen image; Wherein, when the target screenshot type is a short screenshot, capture the screen image within the currently visible area; When the target screenshot type is a long screenshot, perform stitching capture on continuous screen content that exceeds the currently visible area within the duration corresponding to the long-press operation.

7. The method according to any one of claims 1 to 6, characterized in that , the method further includes: In response to the obtained regeneration instruction, insert random syntax variant markers into the formatted prompt template to obtain the new formatted prompt template; Input the new formatted prompt template into a pre-trained auxiliary social text interaction large model to obtain a new auxiliary interaction text.

8. A text interaction device for assisting social software interaction, characterized in that, The device includes: A data acquisition module, configured to acquire a screen image of the current social software in response to the obtained auxiliary interaction instruction; A data processing module, configured to preprocess the screen image to obtain an image to be recognized; A text recognition module, configured to perform text recognition on the image to be recognized to obtain structured text data; A semantic parsing module, configured to perform semantic parsing on the structured text data to obtain a formatted prompt template; A model call module, configured to input the formatted prompt template into a pre-trained auxiliary social text interaction large model to obtain an auxiliary interaction text.

9. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images