Information processing method and device, electronic equipment, storage medium and program product

Abstract

Embodiments of the present application provide an information processing method and device, electronic equipment, storage medium and program product. The method responds to receiving question information containing a picture, extracts a plurality of text segments contained in the picture, extracts the image block corresponding to each text segment on the picture, and the image block corresponding to each text segment on the picture represents the spatial layout of the text segment in the picture. The spatial layout can more comprehensively and detailedly describe the content in the picture. Furthermore, based on the plurality of text segments and the image block of each text segment, the corresponding purpose intention of the question information can be accurately identified, the purpose reply information corresponding to the purpose intention is filtered in a preset reply information library, and more accurate purpose reply information is output, thereby improving the response efficiency of the user's question information and improving the user experience.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an information processing method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] With the development of online dialogue service technology, dialogue service platforms now support responding to user inquiries based on images. In related technologies, after receiving a question containing an image, the dialogue service platform identifies the text contained in the question, including the text within the image. Based on the identified text, it determines the intent of the question and then outputs a response accordingly.

[0003] However, since questions contain not only text but also images, determining the intent solely based on the text ignores the information presented by the images, making it impossible to accurately and completely determine the intent of the question. This usually requires human intervention to continue the dialogue with the user to accurately determine the intent, resulting in a longer overall response time for user questions and a poor user experience. Summary of the Invention

[0004] To overcome the problems existing in related technologies, embodiments of this application provide an information processing method, apparatus, electronic device, storage medium, and program product.

[0005] According to a first aspect of the embodiments of this application, an information processing method is provided, the method comprising: In response to receiving a query message containing an image, extract multiple text segments contained in the image and the image blocks corresponding to each text segment on the image; Based on the multiple text segments and image blocks of each text segment, the purpose and intent corresponding to the question information are identified; Filter the target response information corresponding to the stated target intent from the preset response information database, and output the target response information.

[0006] According to a second aspect of the embodiments of this application, an information processing apparatus is provided, the apparatus comprising: The extraction module is used to extract multiple text segments contained in the image and the corresponding image blocks of each text segment in the image in response to receiving a query information containing an image; The recognition module is used to identify the purpose and intent corresponding to the question information based on the multiple text segments and image blocks of each text segment; The output module is used to filter the target response information corresponding to the target intent from the preset response information database and output the target response information.

[0007] According to a third aspect of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, the computer program being executed by the processor to cause the electronic device to perform the method as described in the first aspect.

[0008] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided having a computer program stored thereon, the program being executed by a processor to implement the method as described in the first aspect.

[0009] According to a fifth aspect of the present application, a computer program product is provided, including instructions that, when executed on a computer, cause the computer to perform the method as described in the first aspect.

[0010] The technical solutions provided in this application embodiment may include the following beneficial effects: The information processing method in this embodiment, in response to receiving a question containing an image, extracts multiple text segments contained in the image, and then extracts the corresponding image blocks on the image for each text segment. These image blocks represent the spatial layout of the text segments within the image, providing a more comprehensive and detailed description of the image's content. Furthermore, based on the multiple text segments and their corresponding image blocks, the intended purpose of the question can be accurately identified. The method then filters relevant response information from a preset response information database, outputting more accurate responses and improving the efficiency of responding to user questions, thus enhancing the user experience.

[0011] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly described below. It should be understood that those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0013] Figure 1 This is a schematic diagram of a scenario corresponding to the information processing method provided in the embodiments of this application; Figure 2 This is a schematic diagram of the server in the scenario corresponding to the information processing method provided in the embodiments of this application; Figure 3 An exemplary flowchart corresponding to the information processing method provided in the embodiments of this application; Figure 4 An exemplary schematic diagram of the information processing apparatus provided in the embodiments of this application; Figure 5 This is an exemplary schematic diagram of an information processing apparatus provided in an embodiment of this application. Detailed Implementation

[0014] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings.

[0015] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments and is not intended to limit the technical solutions of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise.

[0016] It should also be understood that although the terms "first," "second," etc., may be used in the following embodiments to describe a certain type of object, the object should not be limited to these terms. These terms are used to distinguish the specific implementation objects of that type of object.

[0017] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0018] The technical scenarios of the embodiments of this application are described below.

[0019] Online dialogue services refer to a service model that leverages internet technology to provide real-time answers or remote support for users' various inquiries and questions. In fields such as e-commerce and after-sales service, online dialogue services have become one of the important channels for user communication. In the process of providing services, online dialogue services typically receive user inquiries that include images. For inquiries containing images, the online dialogue service needs to determine the user's intent based on the image, generate a response based on that intent, and then output the response.

[0020] Determining the intent associated with an image typically involves extracting text from the image and then basing the intent on that text. However, this method relies solely on the text, which cannot accurately convey the image's content. Therefore, the accuracy of intent determined from the text is low, and there are instances where the intent cannot be determined at all. Furthermore, even after extracting the text from the image, it may still be impossible to determine the intent based on the text alone. In such cases, a human intervention in the online chat service is usually required to engage in multiple rounds of dialogue with the user, determining the intent based on the user's responses. This results in a prolonged overall response time for user queries.

[0021] In view of this, embodiments of this application propose an information processing method that can be applied to an information processing system. For example... Figure 1 The image shows an information processing system that implements an information processing method. For example... Figure 1 As shown, the information processing system includes a client 101 and a server 102, which are connected via a network. The client 101 can be hardware, such as a mobile phone, personal computer, tablet computer, wearable device, or other electronic device, or it can be an application (APP) or software module deployed on an electronic device. The client 101 serves as an online service portal for users. Users can log in to this portal via electronic devices or applications and send queries to the server 102. The server 102 can be an application, service, instance, software module, virtual machine, container, or cloud server, or a hardware device or chip with data processing capabilities. After receiving the query, the server 102 processes it to generate a corresponding response.

[0022] In some embodiments, such as Figure 2 As shown, the server 102 may further include a multimodal understanding unit 1021, a retrieval and matching unit 1022, a database 1023, and a decision-making unit 1024. The multimodal understanding unit 1021 is used to understand the question information, and can parse the question information by combining text, images, context, and user profiles, achieving multimodal parsing of text and image information, which improves the accuracy of understanding the intent of the question information to a certain extent. The retrieval and matching unit 1022 is used to retrieve matching historical question-and-response pairs from the database 1023 based on the question information, obtaining relatively similar historical questions and responses. The decision-making unit 1024 is used to generate target response information based on the historical response information matched in the database 1023 by the multimodal understanding unit 1021 and the retrieval and matching unit 1022.

[0023] In some embodiments, the client 101 and the server 102 can also interact in the following ways: The user sends a question to the server 102 through the client 101, the question including an image. The server 102 calls the multimodal understanding unit 1021 to parse the question based on context and user profile, determining the intended purpose of the question. The parsing process can utilize large language models, image recognition models, etc., to identify the question and obtain the target intent. In determining the target intent, the server 102's multimodal understanding unit 1021 comprehensively employs various technical means to ensure the accuracy and comprehensiveness of intent recognition.

[0024] Specifically, the multimodal understanding unit 1021 first utilizes text extraction technology to extract text from the images in the question information, converting the text information in the images into a processable text format. Simultaneously, it employs image recognition models and other technologies to analyze the image content, identifying the image blocks corresponding to each text segment, thereby obtaining the image's visual feature information. Secondly, the multimodal understanding unit 1021 combines this text information with visual feature information, incorporating the question's context and user profile data for multi-dimensional analysis. Through a large language model, it performs semantic parsing on the fused information to accurately determine the question's intent.

[0025] After determining the purpose and intent of the question, the server 102 invokes the retrieval and matching unit 1022 to retrieve matching historical question-and-response pairs from the database 1023 based on the target intent parsed by the multimodal understanding unit 1021. The database 1023 stores a large amount of historical interaction data; through indexing and search algorithms, the retrieval and matching unit 1022 can locate historical questions similar to the current question and their corresponding responses.

[0026] Finally, the decision-making unit 1024 integrates the parsing results of the multimodal understanding unit 1021 and the historical response information found by the retrieval and matching unit 1022 to generate the target response information. The decision-making unit 1024 not only considers the accuracy and relevance of historical responses, but also adjusts and optimizes the response information based on the specific context of the current question and the user's needs to ensure the accuracy of the generated target response information.

[0027] Of course, the units in the server 102 described above are merely illustrative examples. Each unit can also be implemented through various types of intelligent functional modules. The target response module is determined and output by calling each intelligent module. This application embodiment does not limit this. In addition to the units described above, the server 102 also includes a human service unit / interface, so that relevant staff can use the human service unit / interface to process the response if the server 102 fails to generate a response or the user does not agree with the generated response information.

[0028] It should be understood that Figure 1 and Figure 2 The schematic diagram of the application scenario is only a schematic representation of the information processing system involved in the embodiments of this application, and does not constitute a limitation on the technical solutions of the embodiments of this application.

[0029] In the above operating environment, the embodiments of this application provide as follows: Figure 3 The information processing method shown extracts image blocks corresponding to each text segment. Based on text recognition, it combines this with the spatial layout of the corresponding image blocks within the image to achieve a more detailed understanding of the image. This allows for a more accurate determination of the intended message and improves the accuracy of the target response. Please refer to [reference needed]. Figure 3 This information processing method is applied to, for example Figure 1 or Figure 2 The corresponding server is 102. This information processing method includes the following steps: In step S301, in response to receiving a query message containing an image, multiple text segments contained in the image and the corresponding image blocks on the image for each text segment are extracted.

[0030] For example, a text segment refers to an independent, semantically separable block of text, such as a word, a phrase, a sentence, or all the text within a text box; an image block refers to a local area in an image that corresponds one-to-one with any text segment in spatial location and surrounds that text segment. An image block not only contains the text itself, but also the visual features of the text area, such as font, color, background, and position in the image.

[0031] In the implementation process, an image text recognition model can be called to recognize text segments in the image. Specifically, all regions containing text in the image can be detected to locate image blocks; then, the text content in each image block is detected to obtain the text segment corresponding to the image block. Compared with the traditional implementation method that only extracts text, the implementation method of this application initially converts unstructured image information into multiple structured text segments and the image block coordinate information corresponding to each text segment, providing basic data for subsequent analysis.

[0032] After obtaining each text segment and its corresponding image block, a mapping relationship between the text block and the image block can be established. Specifically, image blocks can be marked in the image, and a unique identifier can be assigned to each image block. This identifier can include information such as the relative position of the image block in the image, font, color, and background, establishing a unique mapping relationship between the image block identifier and the text segment. This achieves fine-grained alignment between text segments and image blocks in the image, which helps to understand the true intent expressed by the image.

[0033] In step S302, the purpose and intent corresponding to the question information are identified based on multiple text segments and image blocks of each text segment.

[0034] For example, the target intent refers to the specific task problem that the user hopes to solve or the help they need in the question message containing the image, sent in the user's context. For example, the return process was blocked due to an incorrect physical order number, and the order number needs to be corrected; the qualification review failed because the copy of the business license was blurry, and a clear version of the business license image needs to be uploaded again, etc.

[0035] Before intent confirmation, multiple text segments need to be fused with their corresponding image blocks. Step S301 implements the mapping of identifiers between each text segment and its corresponding image block. During intent recognition, the features included in the image block identifiers are fused with the text segments. These features include the relative position of the image block within the image, font, color, and background information. The background can be whether the image block is a pop-up, button, or table, etc. This information, along with the text segments, forms a unified semantic vector, which is then used for intent recognition.

[0036] For example, the relevant intent recognition model can be invoked to retrieve the same or similar services corresponding to the semantic vector based on the tasks provided by the corresponding online dialogue services stored in the server, and the purpose intent can be determined based on the service.

[0037] In step S303, target response information corresponding to the target intent is filtered from the preset response information database, and the target response information is output.

[0038] For example, the pre-defined response information database refers to the standardized operating procedure (SOP) for each intent response. Each SOP corresponds to a tag, which can be the intent of the problem that the SOP is intended to address. The intent of the problem can be used to match the target intent. The purpose response information refers to the answer content ultimately generated by the online dialogue service in response to this user's question, which can be directly used to reply to the user. It can be a structured, executable decision package, typically including a diagnosis of the problem, specific operational steps, possible script templates, links to relevant tools or pages, etc.

[0039] After determining the purpose and intent, candidate responses corresponding to that purpose and intent can be matched from a preset response information database. Specifically, question intents with a high degree of similarity to the purpose and intent can be matched from the preset response information database, and the SOP corresponding to that question intent can be used as the purpose and response information.

[0040] As can be seen, by adopting the embodiments of this application, in response to receiving a question containing an image, multiple text segments contained in the image are extracted, and then the corresponding image blocks on the image are extracted for each text segment. The image blocks corresponding to each text segment represent the spatial layout of the text segment in the image, and the spatial layout can more comprehensively and in detail describe the content in the image. Furthermore, based on multiple text segments and the image blocks of each text segment, the intended purpose of the question can be accurately identified. The intended purpose of the question can be filtered from a preset response information database, and a more accurate response can be output, improving the response efficiency to user questions and enhancing the user experience. Specifically, by extracting multiple text segments and their corresponding image blocks from the image and establishing a mapping relationship between text segments and image blocks, the structured transformation of unstructured image information is achieved. This fine-grained alignment helps to understand image content more accurately, avoiding semantic loss caused by extracting only text and ignoring other information in the image, as is the case with traditional methods. By fusing the semantic information of text segments with the visual features of image blocks (such as position, font, color, background, etc.) into a unified semantic vector, the dual textual and visual information of the image is utilized, improving the accuracy of intent recognition. Based on a pre-set response information database, standardized responses corresponding to the intended intent are matched to ensure the standardization and executability of the responses.

[0041] In some embodiments, extracting multiple text segments contained in an image and the corresponding image blocks on the image for each text segment includes: calling a structural recognition model to identify at least one image block containing text content in the image; determining the positional relationship of at least one image block containing text content in the image; and extracting text segments from at least one image block containing text content.

[0042] For example, a structure recognition model typically refers to a model that can identify the layout, type, and interrelationships of different elements (such as text, tables, images, titles, etc.) in an image. It can be a general object detection model from the YOLO series or a specialized scene text detection model.

[0043] The structural recognition model is invoked to process the received query image. The processing involves scanning and analyzing the image to identify all possible local regions containing text, and generating a bounding box for each identified region. This bounding box defines the image patch containing the text content. The model's output can be a list of bounding boxes, each labeled with its coordinates, color, background, and other features. This completes the initial perception of text in the image, structurally extracting all independent regions carrying text information from the pixel-level image, laying the spatial foundation for subsequent text recognition. This solves the problem that traditional methods may use the entire image as input and fail to distinguish different text regions.

[0044] After obtaining the bounding boxes of all text segment regions, the positional relationships of these bounding boxes are resolved. These positional relationships can include the actual positions of the bounding boxes within the image and the relative positions between the bounding boxes.

[0045] The calculation of the true location can be based on the coordinates of the bounding box, which can calculate the precise location and area of ​​each image patch in the original image.

[0046] Relative position calculations can be performed by comparing the relative coordinates of different bounding boxes. Based on these relationships, the layout structure between image blocks can be determined. For example, it can be determined whether image blocks are arranged horizontally or vertically, or whether they belong to a table structure, or whether there are containment relationships, such as a large title box containing multiple smaller paragraph boxes. Relative positional relationships allow for a more accurate understanding of the information and semantics contained within the image. Discrete text segments can be transformed into a structured representation with spatial layout through the relationships between image blocks.

[0047] Building upon the above embodiments, to further improve the accuracy of intent in determining the intent corresponding to the question information, in addition to the image, reference response information related to the reference information of the question information can be further combined. Therefore, in some embodiments, the above-mentioned identification of the purpose intent corresponding to the question information based on multiple text segments and image blocks of each text segment includes: for any text segment, combining the text segment with an image block containing the text segment to obtain the information to be identified; for any information to be identified, filtering reference information matching the information to be identified in a pre-maintained information database; the reference information includes text segment type and reference response information; and determining the purpose intent based on the information to be identified, the text segment type, and the reference response information.

[0048] For example, the information to be identified is composed of a text segment and the image block containing that text segment. This includes both the semantic content of the text and its visual presentation, such as font, color, background, actual position within the image, relative position, and style. The pre-maintained knowledge base refers to an image case library, which stores a database of historical service experiences. Each record may contain a consultation image, a processed multimodal vector, annotated question intent, and a validated, effective solution—that is, historical response information. Text segment type refers to the classification label of the information to be identified according to its function or attributes. Examples include order numbers, button text, error codes, qualification text, interface titles, and operation prompts. Classifying text segment types helps in understanding the role of the text segment in the task scenario.

[0049] After extracting text segments and locating image blocks, an alignment relationship is established for each text segment and its corresponding image block, combining them into information to be identified. This alignment relationship can be the identification mapping between text segments and image blocks mentioned above, where the image block identifier includes information such as real position, relative position, font, color, and background. Alternatively, a unified semantic feature can be established by combining the semantic features of the text segments and the identifier features of the image blocks. For example, a feature matrix can be generated based on the content {text segment, real position, relative position, font, color, ..., background}, and this feature matrix can be used as the information to be identified. This application embodiment does not limit the generation method and representation form of the above-mentioned information to be identified; those skilled in the art can set it according to the actual situation. This feature recognition information is a vector representation of the same dimension obtained by fusing the semantic features of the text and the visual features of the image blocks.

[0050] After obtaining the information to be identified, which clearly represents the information in the image, the information database stores a large number of labeled "standard image-text patterns" and their corresponding knowledge. Therefore, the database can be used to match the same or similar reference images based on the information to be identified, and the corresponding response information is the reference response information. For example, the database may contain combinations of "red exclamation mark icon + 'error' text" and their type label "error message," as well as general preliminary processing suggestions. By calculating the similarity between the vector of the information to be identified and the standard information vectors in the database, one or more of the most matching reference information are selected. It can quickly and accurately identify and associate knowledge with each independent image-text unit in the image. It not only identifies the text content but also combines its visual representation to understand its function (whether it is a button, title, error message, or data field) and associate it with general explanations or common response frameworks related to the pattern.

[0051] After determining the reference response information, analyze the combination, positional relationship, and logical connection between the information to be identified, the text segment type, and the reference response information. For example, the image in the question information contains an information to be identified of type "serious error pop-up," which in turn contains an information to be identified of type "specific error code," combined with information such as the current operation interface type in the image.

[0052] The information in the image is processed based on the text segment types and reference response information in the matched reference information. The text segment type represents the level of focus or importance of a corresponding text segment in the image. For example, if two different text segments correspond to different response messages, the target intent can be determined based on the difference in the importance of the text segments. Alternatively, if only the text segment corresponding to a certain text segment type is clear, while other text segments in the image are unclear, the target intent can be directly generated based on the content of that text segment because it is of higher importance.

[0053] Building upon the above embodiments, to further improve the accuracy of intent in determining the intent corresponding to the question information, in addition to the image, the context of the question information and user profiles can be further considered. Therefore, in some embodiments, relevant large language models and other technologies can be used to determine the intended intent. Specifically, this includes: filling the information to be identified, the text segment type, and the reference response information into the corresponding positions of a preset prompt information template to obtain target prompt words; and calling an intent recognition model to determine the intended intent based on the target prompt words.

[0054] For example, the preset prompt template can be a structured text framework that defines multiple placeholders. Each placeholder can correspond to the information to be identified, the text segment type, and the reference response information. Of course, to further improve the accuracy of the target intent, corresponding placeholders can also be designed for the context of the question and user profile information. In addition to placeholders for various types of information, the logical structure between these types of information also needs to be defined. For example, this logical structure might specify that the determination of the target intent requires using the response process in the reference response information as an initial response benchmark, and adjustments are made based on this benchmark. Or, for example, the logical structure might specify at what step in the process of determining the target intent the context of the question and the user profile are referenced, or under what circumstances they are not needed.

[0055] The system identifies the information to be recognized in the image, along with text segment types and reference response information. Following the logical structure defined in the prompt information template, it fills the corresponding placeholders to generate target prompt words. This target prompt information integrates natural language or structured text from the image's text segment content, reference responses, and contextual information. It consolidates fragmented, multimodal information into a unified, complete input text that conforms to the processing habits of large language models. This provides high-quality context for calling the language model for intent reasoning, solving the problem of comprehension bias or information loss caused by directly feeding the model raw images or messy information.

[0056] After generating target prompts, these prompts are used as prompts for the intent recognition model, which then generates the target intent. This intent recognition model can be a large language model that receives the structured target prompts as input. Since the prompts already contain key text extracted from the image, the text segment types of these texts, and relevant reference responses, the model can comprehensively analyze and reason based on its natural language understanding and logical reasoning capabilities. It infers the possible task request of the user who uploaded the image. Finally, it outputs one or more structured intent judgments, typically including a classification description of the purpose intent, confidence level, and possible root cause analysis.

[0057] After determining the intended purpose of the image based on the above embodiments, the intended response information can be further determined. In some embodiments, the preset response information database includes multiple sets of response information corresponding to multiple intent types. Each set of response information includes at least one response information corresponding to the intent type. Correspondingly, filtering the intended response information corresponding to the intended purpose in the preset response information database includes: matching at least one response information corresponding to the type of the intended purpose in the preset response information database; determining the service scenario indicated by the intended purpose, where the service scenario is related to the processing stage of the pending order; and determining the intended response information corresponding to the service scenario from the at least one response information. For example, intent type is a standardized task classification based on the purpose of a user's question. In a merchant service scenario, common intent types might include order issues, logistics issues, product issues, after-sales requests, qualification verification, and operational consultations, used for initial screening in a database. A response information set refers to the collection of all relevant and historically valid specific solutions aggregated for each intent type within a pre-defined response information database. Each set contains multiple response messages. A service scenario is a more detailed description of the specific task stage or context in which the problem occurs under a given intent type. This service scenario is related to the processing stage of the order to be served and can include pre-order consultation, post-order payment, pre-shipment modifications, post-shipment logistics tracking, and after-sales service after a successful transaction. A service scenario is more refined than an intent type; it combines the specific order status and the dynamic context of user operation nodes. A service scenario can also be called an application of a specific scenario within an intent type.

[0058] After determining the purpose and intent in the previous step, before matching at least one response message corresponding to the type of purpose and intent in the preset response information database, the purpose and intent can be further transformed into structured data. This structured data can be directly matched with the relevant tags and other content of the response messages in the preset response information database.

[0059] First, based on the intended purpose, the intent type corresponding to that purpose can be determined. For example, if the intended purpose is to change the shipping address, the intent type is "order issue" or "logistics issue." Next, a first-round search is performed in a pre-defined response database. The key to this search is the intent type; it retrieves a set of responses in the database that perfectly match the identified intent type. This set contains all effective solutions to this type of problem. This quickly narrows the search scope from the entire knowledge base to a pool of specific solutions highly relevant to the macro-type of the current problem, improving the efficiency and targeting of subsequent searches.

[0060] After obtaining the set of response information corresponding to the intent type, response information that is randomly selected or highly similar to the intended intent can be chosen as the intended response information. Of course, to improve the accuracy of the intended response information, it is necessary to further analyze the specific service scenario in which the intended intent occurs. This service scenario can be determined by parsing the current session context, which can be real-time status information related to the order to be served. For example, for the intended intent of modifying the shipping address, the service scenario might be that the order has been paid but not shipped, or the order has been shipped but not signed for. Different scenarios correspond to different processing flows, permissions, scripts, and operability.

[0061] Finally, among the "at least one response message" belonging to the same intent type selected in the first round, a service scenario-based matching is performed. The applicable scenario metadata associated with each candidate response message is compared, and one or more response messages with the highest matching degree with the currently inferred service scenario tag are selected.

[0062] If only one response is received based on the service scenario, that response should be used directly as the intended response.

[0063] When only multiple response messages are obtained based on the service scenario, the historical dialogue information of the session in which the image question is located is obtained; based on the similarity between the intent of the historical dialogue information and the intent corresponding to each response message, a preset number of response messages are output in descending order of similarity.

[0064] For example, the history of the conversation in which the image question is located refers to all the interaction records in the current user's current continuous conversation before sending the current question containing the image, which may include the text, images and related replies sent by the user before.

[0065] From the conversation database, all historical dialogue information of the current user in the current session, prior to the current question about the image, is extracted. Then, the dialogue intent of this continuous historical dialogue is identified, which can also be determined using an intent recognition model. The historical dialogue information is sequentially arranged and input into the intent recognition model. The output is a context vector representing the overall semantics and intent evolution trajectory of this historical dialogue. The context vector includes the topics discussed, the stage of the conversation, the user's underlying emotions, and unresolved issues. This transforms unstructured, temporal dialogue information into structured semantic features. The intent recognition model then generates the intent of this historical dialogue based on these semantic features.

[0066] It is possible to calculate the similarity between the intent of any response and the intent of historical dialogue information, specifically through methods such as cosine similarity.

[0067] Based on the similarity score calculated in the previous step, all matched responses are sorted from highest to lowest similarity. Then, a predetermined number of responses are selected from the top of the sorted list. These selected responses represent the solutions most relevant to the current user's overall question context. Finally, these responses are output sequentially according to their similarity scores. This embodiment does not limit the specific number of output options; those skilled in the art can set the appropriate number based on actual circumstances.

[0068] Other embodiments allow storing responses with high similarity scores, or responses that have resolved issues based on user feedback, in a pre-maintained database. This enables dynamic maintenance of the database, allowing for the generation of responses based on cases in the database when similar questions are received subsequently. The response information stored in the database may also include images, text segment types of images, and other relevant information.

[0069] As can be seen, this application's embodiments comprehensively utilize text segments and image block information from images, combined with reference information (including text segment types and reference response information), and further consider the context of the question and user profiles. This fusion of multi-dimensional information allows the identification of question intent to move beyond a single information source, enabling a multi-faceted understanding of user needs and improving the accuracy of intent recognition. For example, in complex task scenarios, relying solely on text in an image may not accurately determine the user's intent, but by combining the font and color of the text segment within the image block with reference response information, the core issue the user wants to express can be grasped more precisely.

[0070] Furthermore, a large language model is employed to integrate the information to be identified, text segment type, and reference response information into target prompt words, and intent inference is performed based on these prompt words. The powerful natural language understanding and logical reasoning capabilities of the large language model enable it to comprehensively analyze and judge complex information, infer the possible needs behind the user's uploaded images, and provide users with more comprehensive and accurate intent recognition results.

[0071] The pre-defined response database categorizes responses according to intent type. Once the target intent is determined, the database can quickly match the corresponding set of responses. This categorized storage and retrieval method improves the efficiency of response filtering, avoids blindly searching through a vast knowledge base, and rapidly narrows the search scope to responses relevant to the current question, laying the foundation for subsequently identifying the target response.

[0072] After obtaining the set of response information corresponding to the intent type, further matching is performed based on the specific service scenario in which the intended intent occurs. The service scenario, as a more detailed description of the problem's stage, enables the identification of response information suitable for the current situation. Different service scenarios have different processing flows, permissions, wording, and operability. By considering the service scenario, it is ensured that the generated intended response information closely matches the actual problem, possessing greater relevance and practicality.

[0073] When multiple responses are received based on the service scenario, the system retrieves historical dialogue information from the session containing the image-based question and analyzes the similarity between the question's intent and the intent of each response. A preset number of responses are then output in descending order of similarity. By considering the context of the user's question, the system better understands the position and evolution of the user's intent within the entire dialogue, thus providing responses that better meet the user's overall needs and improving the quality of the responses and user satisfaction.

[0074] The above implementation steps can also be implemented through software modules to achieve the corresponding functions. Corresponding to the above information processing method, the embodiments of this application can also provide an information processing device.

[0075] like Figure 4 As shown, an information processing device is provided, which is applied to a target device. The device may include: an extraction module 41, an identification module 42, and an output module 43.

[0076] For example: Extraction module 41 is used to extract multiple text segments contained in the image and the image blocks corresponding to each text segment on the image in response to receiving a query information containing an image; The recognition module 42 is used to identify the purpose and intent corresponding to the question information based on the multiple text segments and image blocks of each text segment; The output module 43 is used to filter the target response information corresponding to the target intent in the preset response information database and output the target response information.

[0077] Optionally, the extraction module is further configured to: invoke a structure recognition model to identify at least one image block containing text content contained in the image; Determine the positional relationship of the at least one image block containing text content within the image; Extract the text segments from each of the at least one image block containing text content.

[0078] Optionally, the identification module is further configured to: For any text segment, the text segment is combined with an image block containing the text segment to obtain the information to be identified; For any piece of information to be identified, reference information matching the information to be identified is filtered from a pre-maintained information database; the reference information includes text segment types and reference response information; The intended purpose is determined based on the information to be identified, the text segment type, and the reference response information.

[0079] Optionally, the identification module is further configured to: The information to be identified, the text segment type, and the reference response information are filled into the corresponding positions of the preset prompt information template to obtain the target prompt words; Invoke the intent recognition model to determine the target intent based on the target prompt word.

[0080] Optionally, the preset response information database includes multiple response information sets corresponding to intent types, and any response information set includes at least one response information for the corresponding intent type; the output module is further configured to: In the preset response information database, at least one response information corresponding to the type of the stated intention is matched; Determine the service scenario indicated by the stated purpose; the service scenario is related to the processing stage of the order to be served; The target response information corresponding to the service scenario is determined from the at least one response information.

[0081] Optionally, if the target response information includes multiple response information, the output module is further configured to: obtain the historical dialogue information of the session in which the image question information is located; Based on the similarity between the intent of the historical dialogue information and the intent corresponding to each reply, a preset number of reply messages are output in descending order of similarity.

[0082] Optionally, the device is further configured to: store the image, the text segment type of the image, and the target response information in a pre-maintained database.

[0083] As can be seen, by adopting the embodiments of this application, in response to receiving a question containing an image, multiple text segments contained in the image are extracted, and then the corresponding image blocks on the image are extracted for each text segment. The image blocks corresponding to each text segment represent the spatial layout of the text segment in the image. The spatial layout can more comprehensively and in detail describe the content in the image. Furthermore, based on the multiple text segments and the image blocks of each text segment, the purpose and intent corresponding to the question can be accurately identified. The purpose and intent corresponding to the question can be filtered in a preset response information database, and a more accurate purpose and intent response information is output, thereby improving the response efficiency to user questions and enhancing the user experience.

[0084] Understandable Figure 4 The division of the various modules is merely a logical functional division. In actual implementation, the functions of these modules can be integrated into the hardware entity of the electronic device.

[0085] Please refer to Figure 5 , Figure 5 An electronic device is provided, and this disclosure also provides an electronic device for performing some or all of the steps of the above-described information processing method. Please refer to... Figure 5 The electronic device includes a processor 500, a memory 501, a bus 502, and a communication interface 503, wherein the processor 500, the communication interface 503, and the memory 501 are connected via the bus 502; the memory 501 stores a computer program that can run on the processor 500, and the processor 500 can be equivalent to a CPU, executing the aforementioned provisions of this disclosure when running the computer program. Figure 3 The information processing method provided in any of the embodiments, and in the process of executing the aforementioned method, interacts with the corresponding client through the communication interface 503 to exchange data and instructions.

[0086] The memory 501 may include high-speed random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 503 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc.

[0087] Bus 502 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. Memory 501 is used to store programs, and the processor 500 executes the programs after receiving execution instructions. Figure 2 The illustrated embodiments reveal that the information processing method can be applied to or implemented by the processor 500.

[0088] The processor 500 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 500 or by instructions in software form. The processor 500 may be a general-purpose processor, including a CPU, a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this disclosure. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this disclosure can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 501. The processor 500 reads the information in memory 501 and, in conjunction with its hardware, completes the steps of the above method.

[0089] The electronic devices and information processing methods provided in this disclosure are based on the same inventive concept and have the same beneficial effects as the methods they employ, operate, or implement.

[0090] This application also provides a computer-readable storage medium storing instructions for information processing, which, when executed on a computer, cause the computer to perform some or all of the steps in the methods described in the foregoing embodiments.

[0091] This application also provides a computer program product including instructions for information processing, which, when run on a computer, causes the computer to perform some or all of the steps in the method described in the foregoing embodiments.

[0092] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0093] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0094] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0095] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0096] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, smartphone, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0097] Although alternative embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make further changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0098] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above description is only a specific embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of this application should be included within the scope of protection of this invention.

Claims

1. An information processing method, characterized in that, The method includes: In response to receiving a query message containing an image, extract multiple text segments contained in the image and the image blocks corresponding to each text segment on the image; Based on the multiple text segments and image blocks of each text segment, the purpose and intent corresponding to the question information are identified; Filter the target response information corresponding to the stated target intent from the preset response information database, and output the target response information.

2. The method according to claim 1, characterized in that, The step of extracting multiple text segments contained in the image and the corresponding image blocks of each text segment on the image includes: The structural recognition model is invoked to identify at least one image patch containing text content within the image; Determine the positional relationship of the at least one image block containing text content within the image; Extract the text segments from each of the at least one image block containing text content.

3. The method according to claim 2, characterized in that, The step of identifying the intended purpose corresponding to the question information based on the multiple text segments and image patches of each text segment includes: For any text segment, the text segment is combined with an image block containing the text segment to obtain the information to be identified; For any piece of information to be identified, reference information matching the information to be identified is filtered from a pre-maintained information database; the reference information includes text segment types and reference response information; The intended purpose is determined based on the information to be identified, the text segment type, and the reference response information.

4. The method according to claim 3, characterized in that, Determining the intended purpose based on the information to be identified, the text segment type, and the reference response information includes: The information to be identified, the text segment type, and the reference response information are filled into the corresponding positions of the preset prompt information template to obtain the target prompt words; Invoke the intent recognition model to determine the target intent based on the target prompt word.

5. The method according to any one of claims 1-4, characterized in that, The preset response information database includes multiple sets of response information corresponding to multiple intent types, and each set of response information includes at least one response information corresponding to the intent type. The step of filtering target response information corresponding to the target intent from a preset response information database includes: In the preset response information database, at least one response information corresponding to the type of the stated intention is matched; Determine the service scenario indicated by the stated purpose; the service scenario is related to the processing stage of the order to be served; The target response information corresponding to the service scenario is determined from the at least one response information.

6. The method according to claim 5, characterized in that, When the target response information includes multiple response information, the output of the target response information includes: Obtain the historical dialogue information of the session in which the image question information is located; Based on the similarity between the intent of the historical dialogue information and the intent corresponding to each reply, a preset number of reply messages are output in descending order of similarity.

7. The method according to claim 3, characterized in that, The method further includes: The image, the text segment type of the image, and the target response information are stored in a pre-maintained database.

8. An information processing device, characterized in that, The device includes: The extraction module is used to extract multiple text segments contained in the image and the corresponding image blocks of each text segment in the image in response to receiving a query information containing an image; The recognition module is used to identify the purpose and intent corresponding to the question information based on the multiple text segments and image blocks of each text segment; The output module is used to filter the target response information corresponding to the target intent from the preset response information database and output the target response information.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method of any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by a processor to implement the method as described in any one of claims 1-7.

11. A computer program product, characterized in that, Includes instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1-7.

Images