Image information processing method in multimodal-based knowledge question answering system and electronic device

Abstract

The present disclosure provides an image information processing method in a multi-modal knowledge question answering system and an electronic device. The method comprises: obtaining a target image from a to-be-processed file, the to-be-processed file comprising text data and an image, the target image comprising text information; extracting the text information in the target image as first text information, and extracting context information of the target image in the to-be-processed file according to the text data; selecting a retained target image according to the first text information and the context information; determining retained second text information in the first text information of the retained target image according to the context information, the second text information comprising a paragraph; and selecting third text information corresponding to a plurality of preset questions in the second text information according to a matching result of the second text information and the plurality of preset questions, the third text information comprising a sentence. The present disclosure can improve the processing efficiency of multi-modal information and the proportion of effective information in the information base.

Description

Technical Field

[0001] This disclosure relates to the field of multimodal information processing technology, and more specifically, to an image information processing method and electronic device based on a multimodal knowledge question answering system. Background Technology

[0002] A multimodal knowledge question-answering system is an intelligent system that integrates various information modalities (such as text, images, voice, and video) and can comprehensively process and analyze this multi-source heterogeneous information to accurately answer user questions. In a multimodal knowledge question-answering system, users can ask questions through text, voice, images, etc. The system analyzes the question information and provides response information based on the system's information database.

[0003] To build an information database, it is typically necessary to collect various types of information related to the knowledge-based question-answering system. In related technologies, relevant information is usually organized and collected based on text. However, in many cases, the textual information related to the knowledge-based question-answering system is insufficient to provide adequate information. Due to the limited availability of relevant information, multimodal knowledge-based question-answering systems that rely solely on textual information to generate responses still have room for improvement in user satisfaction.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this disclosure is to provide an image information processing method and electronic device in a multimodal knowledge question answering system, which can provide accurate and rich reference information for the database of the question answering system.

[0006] According to a first aspect of the present disclosure, an image information processing method is provided in a multimodal knowledge question answering system, comprising: obtaining a target image from a file to be processed, the file to be processed including text data and an image, the target image including text information; extracting the text information in the target image as first text information, and extracting context information of the target image in the file to be processed based on the text data; selecting a retained target image based on the first text information and the context information; determining a retained second text information from the retained first text information of the target image based on the context information, the second text information including paragraphs; and selecting third text information corresponding to the multiple preset questions from the second text information based on the matching results of the second text information and multiple preset questions, the third text information including sentences.

[0007] According to a second aspect of this disclosure, an electronic device is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to perform the method as described in any one of the preceding methods based on instructions stored in the memory.

[0008] According to a third aspect of this disclosure, a computer-readable storage medium is provided having a program stored thereon that, when executed by a processor, implements the image information processing method in a multimodal knowledge question-answering system as described in any of the preceding claims.

[0009] According to a fourth aspect of this disclosure, a computer program product is provided, comprising a computer program, characterized in that, when executed by a processor, the computer program implements the steps of any of the preceding methods.

[0010] This embodiment of the disclosure, through image extraction from a file containing text data and images in a multimodal knowledge question-answering system, obtains target images with text information. The target images are then filtered based on contextual information within the file being processed. Furthermore, second text information related to the contextual information is extracted from the filtered target images, and third text information related to a preset question is retained within the second text information. This allows for the full utilization of multimodal information, including images and text, to form a multimodal knowledge question-answering system's information base during system construction. It also effectively avoids irrelevant or redundant information from reducing the quality of answer generation, providing rich and accurate reference information for the multimodal knowledge question-answering system and improving the quality and efficiency of question-answering system construction.

[0011] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0012] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0013] Figure 1 This is a flowchart of an image information processing method in a multimodal knowledge question-answering system according to an exemplary embodiment of this disclosure.

[0014] Figure 2 This is a sub-flowchart of step S3 in an exemplary embodiment of this disclosure.

[0015] Figure 3 This is a sub-flowchart of step S4 in an exemplary embodiment of this disclosure.

[0016] Figure 4 This is a sub-flowchart of step S5 in an exemplary embodiment of this disclosure.

[0017] Figure 5 This is a schematic diagram illustrating the effect of image processing in an exemplary embodiment of this disclosure.

[0018] Figure 6 This is a schematic diagram of the workflow of the question-and-answer system in an exemplary embodiment of this disclosure.

[0019] Figure 7 This is a block diagram of an image information processing apparatus in a multimodal knowledge question-answering system according to an exemplary embodiment of the present disclosure.

[0020] Figure 8 This is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0021] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0022] Furthermore, the accompanying drawings are merely illustrative of this disclosure, and the same reference numerals in the drawings denote the same or similar parts, thus repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0023] The exemplary embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0024] Figure 1 This is a flowchart of an image information processing method in a multimodal knowledge question-answering system according to an exemplary embodiment of this disclosure.

[0025] refer to Figure 1 The image information processing method 100 in a multimodal knowledge question answering system may include:

[0026] Step S1: Obtain the target image from the file to be processed, wherein the file to be processed includes text data and an image, and the target image includes text information;

[0027] Step S2: Extract the text information from the target image as the first text information, and extract the context information of the target image in the file to be processed based on the text data;

[0028] Step S3: Select the target image to be retained based on the first text information and the context information;

[0029] Step S4: Based on the context information, determine the second text information to be retained from the first text information of the retained target image, wherein the second text information includes paragraphs;

[0030] Step S5: Based on the matching results between the second text information and multiple preset questions, select the third text information corresponding to the multiple preset questions from the second text information. The third text information includes sentences.

[0031] This embodiment of the disclosure, through image extraction from a file containing text data and images in a multimodal knowledge question-answering system, obtains target images with text information. The target images are then filtered based on contextual information within the file being processed. Furthermore, second text information related to the contextual information is extracted from the filtered target images, and third text information related to a preset question is retained within the second text information. This allows for the full utilization of multimodal information, including images and text, to form a multimodal knowledge question-answering system's information base during system construction. It also effectively avoids irrelevant or redundant information from reducing the quality of answer generation, providing rich and accurate reference information for the multimodal knowledge question-answering system and improving the quality and efficiency of question-answering system construction.

[0032] The following is a detailed description of each step of the image information processing method 100 in a multimodal knowledge question answering system.

[0033] In step S1, a target image is obtained from a file to be processed, the file to be processed including text data and an image, and the target image including text information.

[0034] The method described in this disclosure can be used to construct a database for a question-and-answer system. A question-and-answer system is, for example, a system built by an enterprise or industry specifically to answer one or more types of questions. It can provide more accurate, targeted, and professional answers based on a specialized database. Some question-and-answer systems with limited public access can even provide users with internal information to answer their questions.

[0035] In building a question-answering system, the scope of information involved can be determined first, such as the type and source of the information. Then, information within this scope is collected through various methods and stored as reference information for subsequent question-answering. Generally, textual information is the most common source of reference information; however, in some cases, files containing images, or even simply image files, may also involve the knowledge needed to build the information base. These files can be referred to as multimodal information. Therefore, building a question-answering system involves integrating information from different modalities (such as text and images) to improve the performance of the question-answering and retrieval systems.

[0036] Because images come from various sources and are of diverse types, multimodal information often includes webpage screenshots, mobile phone screenshots, and other similar data. The conventional approach is to use OCR to parse the images and extract text information, which is then used as part of an information database. However, images often contain a lot of noise, and the text extracted solely by OCR technology can be extremely lengthy. This not only fails to help understand the original knowledge fragments but may also dilute useful information, negatively impacting the recall process and easily introducing invalid information into the response process.

[0037] Therefore, the application of multimodal files sometimes not only fails to improve response accuracy, but may even reduce response accuracy.

[0038] Based on the discovered problem, the inventors of this disclosure have provided an image information processing method specifically for multimodal files to overcome the problem of decreased response accuracy caused by the introduction of multimodal files.

[0039] In this embodiment of the disclosure, the file to be processed may include files from multiple sources used to build the information base of the question-and-answer system, and its content may include both text information and images.

[0040] In the exemplary embodiment, only one file to be processed is used as an example for illustration. In practical applications, the embodiments of this disclosure are applicable to batch processing of massive files to be processed, so as to build and enrich the information database of the question-and-answer system through massive files to be processed.

[0041] In step S1, images can be extracted from the file to be processed, and then it can be determined whether the image can provide information that is helpful to the answer. Images that may be helpful to the answer are identified as target images.

[0042] In an exemplary embodiment, the category of each image in the image of the file to be processed can be determined, and then the image corresponding to the preset category can be determined as the target image.

[0043] When determining the category of an image, it can be based on the characteristics corresponding to different image categories. Image categories include, for example, QR codes, photographs, screenshots, flowcharts, uncertain types (with text information), and uncertain types (without text information). A QR code is a small black-and-white square pattern containing coded information, which can be read by a scanning device to obtain data such as links and text. A photograph is created by taking pictures of actual scenes, people, or objects, and unlike other synthetic or generated types, it has a realistic feel. A flowchart is a visual representation that includes a sequence of steps, decision points, and relationships between processes, typically used to show process flow and logical order. A screenshot is, for example, a screenshot of a webpage or application. There can be multiple image categories; those skilled in the art can set categories and define the characteristics of each category according to the actual information range to determine the image category.

[0044] Images can be initially categorized into those with text information and those without. For example, categories with text information might include flowcharts, screenshots, and undefined types (with text information); categories without text information might include QR codes, photos, and undefined types (without text information).

[0045] In the initial screening, images corresponding to categories without text information can be removed first, and images corresponding to categories with text information can be identified as target images. In some embodiments, images can also be directly classified into categories with and without text information for classification and screening. By performing initial screening on images in the file to be processed, noise and redundant information can be reduced, improving the efficiency of subsequent information processing and the accuracy of information extraction.

[0046] The process of extracting, classifying, and filtering images described above can be performed using either a pre-trained classification model or a large-scale visual-language model.

[0047] Next, information is extracted from the target image.

[0048] In step S2, the text information in the target image is extracted as the first text information, and the context information of the target image in the file to be processed is extracted based on the text data.

[0049] In images containing text information, the text information can be extracted first, and the collection of text information in the target image is called the first text information. Extracting text information from a target image can use OCR (Optical Character Recognition) technology or other recognition technologies. Text information refers to text information including but not limited to characters, numbers, letters, etc.

[0050] Next, contextual information corresponding to the target image is extracted from the text data of the document to be processed. This contextual information can include, for example, the text information of paragraphs within a preset distance range before and after the target image (e.g., the 5 paragraphs before and after the target image), or it can include all the text information of the document to be processed. If there are multiple target images in the document, a set of contextual information can be determined for each target image, and subsequent analysis will be based on this contextual information.

[0051] In an exemplary embodiment, both textual information extraction and contextual information extraction from the target image can be performed using a large visual-language model to improve the intelligence and accuracy of the process.

[0052] Since not all images are helpful to the text information or can provide effective information, this embodiment of the disclosure sets up a method to further filter the target images in order to select the target images that can provide effective information.

[0053] In step S3, the target image to be retained is selected based on the first text information and the context information.

[0054] Figure 2 This is a sub-flowchart of step S3 in an exemplary embodiment of this disclosure.

[0055] refer to Figure 2 In an exemplary embodiment, step S3 may include:

[0056] Step S31: Determine the degree of relevance between the first text information and the context information;

[0057] Step S32: When the relevance is greater than a first preset value, determine the relevance type between the first text information and the context information. The relevance type includes at least explanatory information and supplementary information.

[0058] Step S33: When the relevant type is explanatory information and / or supplementary information, retain the target image.

[0059] The relevance between primary text information and its context can be determined through semantic analysis, keyword matching, topic modeling analysis, and contextual understanding. Semantic analysis involves a deep understanding and analysis of the text's semantics to determine the semantic connection between the primary text and its context. For example, analyzing the subject-verb-object structure of sentences and the semantic roles of words helps determine if they align with the semantic logic of the context. Keyword matching involves finding whether keywords in the primary text appear in the context, and their frequency and location. Frequent and closely related keywords suggest a high degree of relevance. Topic modeling analysis uses algorithms such as Latent Dirichlet Allocation (LDA) to analyze the topics covered in the primary text and its context. If they belong to the same or similar topic categories, they are considered highly relevant. Contextual understanding considers the contextual information, including the text's background, purpose, and target audience. A comprehensive understanding of these factors determines the relevance of the primary text within that context. For example, in an article about technological development, specific technological products or innovations mentioned are relevant to the overall context, while content unrelated to technology is less relevant.

[0060] The method described above for analyzing the relevance between the first text information and the context information can be implemented through various natural language processing methods, such as through a pre-trained neural network model. In an exemplary embodiment, it can be implemented through a large vision-language model to improve accuracy and efficiency.

[0061] If the correlation between the primary text information and the context information of a target image is low (not greater than a first preset value), it indicates that the information is noise or redundant. The target image can be discarded, and the process can continue to the next target image or the next file to be processed (if there is no next target image). If the correlation between the primary text information and the context information of a target image is high (greater than a first preset value), it indicates that the target image can provide information related to the context, most likely supplementing, explaining, or interpreting the context. In this case, the target image is temporarily retained for further processing. Different methods of determining the correlation result in different first preset values. When using a large model to determine the correlation, the first preset value can also be automatically set by the large model.

[0062] If a target image is temporarily retained, it is further determined whether its corresponding first text information is explanatory or supplementary information of the context. If so, the target image is retained; otherwise, the target image is discarded, and the next target image or the next file to be processed is continued (if there is no next target image).

[0063] Among them, the explanation information that determines the first text information as context information can be achieved, for example, through semantic similarity analysis, keyword coverage analysis, text structure analysis, referential resolution verification, logical coherence verification, positional relationship analysis, and other methods.

[0064] Semantic similarity analysis can be performed by extracting paragraph vectors using pre-trained language models (such as BERT and RoBERTa), calculating cosine similarity or Jaccard similarity, and determining whether the semantic scope of the first text information covers the semantic scope of the context information through vector space projection when the similarity is higher than a threshold (such as cosine similarity ≥ 0.7). If so, the two are considered semantically similar.

[0065] Keyword coverage analysis refers to extracting keywords from contextual information (e.g., through TF-IDF, TextRank, etc.), verifying whether the first text information contains keywords from the contextual information and introducing new related terms. If the coverage rate of related terms from the contextual information in the first text information is greater than a certain value (e.g., 80%) and new terms related to the topic of the contextual information are added (e.g., the proportion of new terms is ≥30%), then it is determined that the first text information may be explanatory information of the contextual information.

[0066] Text structure analysis refers to detecting whether the first text information contains explanatory language patterns (such as markers like "because," "for example," and "that is") and analyzing whether its sentence structure exhibits explanatory features (e.g., the context information often presents concise conclusions, while the first text information often contains causal chains, examples, or detailed explanations). If the probability of explanatory markers appearing in the first text information is ≥50%, and its average sentence length is significantly higher than that of the context information (e.g., the average sentence length of the context information is ≤15 words, while that of the first text information is ≥20 words), then it further supports the judgment that it is explanatory information.

[0067] Referential resolution verification involves identifying pronouns or vague references (such as "this," "this method," etc.) in the contextual information and checking whether the first textual information replaces them with specific nouns or explicit statements. If all referential components in the contextual information are specified in the first textual information, the credibility of the interpretation relationship can be enhanced.

[0068] Logical coherence verification refers to analyzing whether the first textual information provides logical support for the contextual information. For example, it can be done by constructing a text dependency graph (such as using Stanford CoreNLP) to verify whether the first textual information contains reasoning chains that directly support the contextual information (such as causality, definition, examples, etc.). If there is at least one explicit logical support relationship, it meets the characteristics of explanatory information.

[0069] Positional relationship analysis refers to examining the physical positional relationship between the first text information and the context information (such as whether they are adjacent, whether there are indentation or numbering marks). If the first text information immediately follows the context information without any gaps or has obvious explanatory layout features (such as indentation or dashes), it can be used as an auxiliary basis for judgment.

[0070] If multiple indicators meet preset thresholds (e.g., semantic similarity ≥ 0.7, keyword coverage ≥ 80%, clear logical support) using one or more of the above methods, then the first text information can be determined as explanatory information for the context. In practical applications, weighted scoring (e.g., setting weights for each indicator and then weighting the scores of each indicator to obtain a weighted sum) or machine learning models (e.g., XGBoost) can be used for automated decision-making. Alternatively, a large model can be used directly to implement the above judgment to improve the accuracy and adaptability of the judgment.

[0071] Determining that the first text information is supplementary to the context information can be achieved, for example, through methods such as semantic expansion analysis, information increment detection, topic consistency verification, context coherence analysis, and redundancy assessment.

[0072] Semantic expansion analysis refers to verifying whether the first textual information expands upon the contextual information with new semantics. For example, using pre-trained language models (such as BERT or GPT) to extract semantic vectors from two texts, the analysis examines whether the first textual information introduces new topics or refines a certain aspect of the contextual information. Then, the information entropy difference is calculated. If the entropy value of the first textual information is significantly higher than that of the contextual information (e.g., an increase of more than 20%), it may contain supplementary content. If the semantic similarity between the first textual information and the contextual information is between 0.5 and 0.8 (neither completely identical nor unrelated), and the first textual information contains at least 30% new entities or concepts (e.g., new terms, examples, or data), then the first textual information may be supplementary information to the contextual information.

[0073] Information increment detection refers to identifying whether the first textual information provides new content not covered by the context. For example, comparing two texts using Named Entity Recognition (NER) and keyword extraction, if the first textual information contains specific details not mentioned in the context (such as time, location, or data support), it is considered supplementary. If the proportion of newly added entities is ≥40% (e.g., the context mentions "climate change," while the first text adds "the Arctic ice cap melts at an average annual rate of 12%)) and the newly added content is strongly related to the context topic (the topic relevance is verified to be ≥70% through LDA topic modeling), then the first textual information is likely supplementary to the context information.

[0074] Topic consistency verification refers to ensuring that the first text information shares the core topic with the context information but provides additional perspectives. For example, using topic modeling (such as LDA or BERTopic) to analyze the topic distribution of two texts, if the main topic overlap is ≥70%, but the first text information introduces sub-topic differences (e.g., the context discusses "technological advantages," while the first text supplements "potential ethical risks"), then it meets the complementarity requirement.

[0075] Contextual cohesion analysis refers to detecting whether the initial textual information naturally expands upon the surrounding context through conjunctions or logical relationships. For example, it involves identifying conjunctions (such as "in addition," "on the other hand," "notably") and analyzing whether the sentence structure presents a progressive or parallel structure (such as "not only that,..." "specifically,..."). If the initial textual information contains multiple conjunctions and forms a logical expansion with the context (such as supplementary comparison or detailed explanation), then its supplementary nature is supported.

[0076] Redundancy assessment excludes cases where the primary text information highly overlaps with the context information. For example, it calculates the text repetition rate (using Rouge-L or edit distance). If the repetition rate is less than 20%, it checks whether the new content is implemented through restatement, expansion, or illustrative examples (e.g., expanding "efficient algorithm" to "gradient descent-based optimization algorithm"). If the core information repetition rate between two texts is ≤30% and the new content accounts for ≥40%, it is considered a valid supplement.

[0077] The above indicators can be weighted and scored. For example, weights can be assigned to each indicator (e.g., semantic expansion 25%, information increment 30%, redundancy 20%). When the total score is ≥0.7, the first text information is considered to be supplementary information to the context information. In addition, if the first text information meets the three core conditions of "theme consistency + information increment + low redundancy" for the context information, then the first text information is directly considered to be supplementary information to the context information.

[0078] If the first text information is explanatory or supplementary information of the context, the target image is retained; otherwise, the target image is discarded, and the process continues with the next target image or the next file to be processed (if there is no next target image).

[0079] The types of relationships between the first text information and the context information, besides being explanatory or supplementary information, can also be descriptive relationships, corresponding relationships, emphasizing relationships, and so on.

[0080] Descriptive relationships refer to the fact that contextual information can be information that describes the target image. For example, an image shows a city gate and a plaque on it. The first extracted textual information is the plaque, while the contextual information is a description of the city gate. In this case, the first textual information may not be able to explain or supplement the contextual information in words, and therefore does not belong to supplementary or explanatory information of the contextual information.

[0081] A correspondence relationship refers to a one-to-one correspondence between contextual information and elements in the target image, as well as the primary textual information. For example, on a map, the primary textual information is the name of each city on the map, while the contextual information is a description of the map. In this case, the primary textual information may not be able to explain or supplement the contextual information in words, and therefore does not belong to supplementary or explanatory information of the contextual information.

[0082] Emphasis refers to the primary textual information serving to highlight key information or important parts of an image, thereby attracting the reader's attention. For example, in a product promotional image, eye-catching text emphasizes the product's core features or advantages. This primary textual information serves to emphasize the contextual information, but it may not provide any textual explanation or supplement to the contextual information; therefore, it does not fall under the category of supplementary or explanatory information.

[0083] The determination of the degree of relevance and correlation can be achieved through machine learning models, such as fine-tuning pre-trained models (e.g., BERT) or training classifiers (e.g., SVM), automatically classifying based on feature vectors (similarity, number of new entities, density of connecting words, etc.). Alternatively, a large model can be used directly to perform the determination, thereby improving the accuracy and adaptability of the judgment.

[0084] By determining the correlation between the target image and the context information based on the text information corresponding to the target image (i.e., the first text information), target images that are strongly correlated with the context information can be filtered out, thereby improving the efficiency and accuracy of subsequent processing.

[0085] However, directly extracting text information from the target image is prone to errors such as typos and omissions due to the accuracy issues of character recognition technologies like OCR. Furthermore, step S3 is based on the overall judgment of the first text information of the target image. Since images typically carry a massive amount of information, the first text information often contains redundant or irrelevant information. Even if the first text information includes explanatory or supplementary information about the context, it may still contain some information useless to the context. For example, the first text information may also include decorative information (such as text on billboards in city photos) and other information unrelated to the context. Such information may not be filtered out during the overall relevance judgment in step S31 (provided that most of the content in the first text information is relevant to the context).

[0086] Therefore, after determining the target image to be retained, this embodiment of the disclosure sets up further filtering and simplification of the first text information of the target image.

[0087] In step S4, based on the context information, a second text information to be retained is determined from the first text information of the retained target image, wherein the second text information includes paragraphs.

[0088] The paragraph can be a natural paragraph. In other embodiments of this disclosure, the second text information can be a paragraph, a sentence, or a word.

[0089] Figure 3 This is a sub-flowchart of step S4 in an exemplary embodiment of this disclosure.

[0090] refer to Figure 3 In an exemplary embodiment, step S4 may include:

[0091] Step S41: The first text information is segmented to form multiple text units;

[0092] Step S42: Determine the type of association between each text unit and the context information;

[0093] Step S43: Save the text unit whose related category is the explanatory information and / or the supplementary information as the second text information.

[0094] In an exemplary embodiment, the first text information can be finely screened using sentences or paragraphs as text units. This removes text units that are less relevant to or provide little supplementation to the context information, while retaining text units that interpret or supplement the context information, thus greatly improving the effectiveness of the retained information. In some embodiments, the removed text units may also include redundant information with a greater than a threshold (e.g., greater than 90%) of overlap with the context information. The division of text units for a given set of first text information can be the same or different. For example, each paragraph can be set as a text unit, meaning all text units are paragraphs; or each sentence can be set as a text unit, meaning all text units are sentences; or, based on the semantic coherence between sentences, one or more sentences with a semantic coherence greater than a threshold can be set as a text unit. In this case, the text unit may be one sentence, multiple sentences, one paragraph, multiple paragraphs, or multiple sentences spanning paragraphs. The method for segmenting text units can refer to natural language processing schemes, and this disclosure does not impose any special limitations on it.

[0095] In addition, before determining the relevant category in step S42, the degree of relevance between the text unit and the context information can be determined first. The relevant category is determined only when the degree of relevance is greater than the threshold, thereby eliminating decorative information such as billboard text in the city photo in the above example that is irrelevant to the context information.

[0096] The methods for determining the types of correlation between text units and contextual information can be the same as those mentioned above, and can also be implemented through large models, so they will not be elaborated here.

[0097] Because the filtering is based on text units, even if there are some typos or omissions due to character recognition, the impact on the final filtering is minimal. This increases the probability of valid information being saved and reduces the probability of valid information being accidentally deleted. Valid information here refers to information that can explain or supplement the context.

[0098] After filtering out one or more text units that are relevant to explanatory information and / or supplementary information, the text information set consisting of the one or more text units is called the second text information.

[0099] By further compressing and refining the first text information to obtain the second text information, the effective information ratio of the information database can be greatly improved, and invalid and redundant information can be reduced.

[0100] In some embodiments, candidate information can be directly generated based on the second text information and its corresponding context information, and saved to an information database to form reference information for the question-answering system. Furthermore, text processing such as typo correction, omission correction, semantic fusion, and grammatical correction can be performed based on the second text information and context information to generate logically clear and well-organized text (this can be achieved using a large model), which is then saved as candidate information to improve the accuracy of the reference information in the database. Candidate information does not need to be categorized; it only needs to be complete.

[0101] Therefore, the question-answering system can respond to user questions, obtain question information, retrieve relevant information from at least one candidate information, and generate question-answer information corresponding to the question information based on the relevant information.

[0102] By generating candidate information based on second text information and context information, multiple candidate information can be generated based on a single document to be processed, thereby improving the processing efficiency of the document, supplementing the deficiencies of the text information in the document, and providing more accurate reference information for the construction of the information database.

[0103] In an exemplary embodiment, in order to further improve the accuracy of the reference information, the second text information (paragraph fragments) can be further extracted and filtered based on the functions of the question-and-answer system, so as to further retain effective information and remove redundant information, filter and retain high-quality sentences, and improve the information accuracy of the information database of the question-and-answer system.

[0104] In step S5, based on the matching results between the second text information and multiple preset questions, third text information corresponding to the multiple preset questions is selected from the second text information, and the third text information includes sentences.

[0105] Here, a sentence refers to a string that ends with a period. In this embodiment of the disclosure, the third text information may also be certain words or combinations of words from a complete sentence.

[0106] Figure 4 This is a sub-flowchart of step S5 in an exemplary embodiment of this disclosure.

[0107] refer to Figure 4 In an exemplary embodiment, step S5 may include:

[0108] Step S51: Determine the textual relevance between the second textual information and the preset question;

[0109] Step S52: When the text relevance is greater than the second preset value, the second text information is determined as the third text information corresponding to the preset question.

[0110] Preset questions can be determined based on the goals set by the question-and-answer system, or based on users' historical questions in the system's usage records. When initially building the question-and-answer system, multiple possible user questions can be generated based on the system's goals (e.g., solving database retrieval problems), and these possible questions can be saved as preset questions. During subsequent maintenance and updates to the system, the most frequently asked questions or topics can be summarized based on users' historical questions, and multiple preset questions can be generated based on these topics, or the most frequently asked questions can be directly set as preset questions. This process can be achieved by building a question library corresponding to the question-and-answer model. Building the question library can be achieved by collecting a large number of user questions (ensuring coverage of various question formats). Therefore, in step S5, the questions in the question library can be processed as preset questions.

[0111] exist Figure 5In the illustrated embodiment, information related to preset questions in the question bank (text relevance greater than a second preset value) can be further filtered from the second text information and saved as third information to form reference information corresponding to the question bank. In step S5, candidate information can also be formed based on the second text information and its corresponding context information, and then the relevance of the candidate information to the preset question text (text relevance greater than a second preset value) can be determined, thereby filtering out candidate information related to the preset question.

[0112] In some embodiments, the validity of the information can be determined by calculating the mean similarity between the aforementioned information (second text information or candidate information) and a preset question using a text matching model (such as BERT). If the mean similarity is greater than a set threshold (0.5), the information is considered likely to be frequently asked and is considered valid information, and is retained; otherwise, it is considered invalid information and is removed.

[0113] After determining the third text information, in order to avoid problems with the fluency of the third text information obtained from filtering based on the original information, which would affect subsequent responses, the refined text can be further polished to enhance its fluency. This ensures that the generated text can be used for high-quality responses, thereby improving the accuracy and relevance of the question-and-answer system.

[0114] All of the above processes can be completed using large models to improve efficiency and accuracy.

[0115] After obtaining the simplified image-related third or second text information, the target image can be replaced in the document to be processed using the third or second text information. The document to be processed is then segmented according to semantic coherence and other segmentation principles and stored in the information database of the question-and-answer system.

[0116] By further filtering the second text information to extract third text information related to the preset questions of the question-and-answer system, the effective information ratio of the reference information of the question-and-answer system can be increased, the storage size of the information database can be reduced, and the relevance and accuracy of the information database can be improved.

[0117] Figure 5 This is a schematic diagram illustrating the effect of image processing in an exemplary embodiment of this disclosure.

[0118] refer to Figure 5 The initially selected target image, as shown in A, includes various image elements and text information. The first text information extracted from the target image, as shown in B, is fragmented, redundant, and may contain errors or omissions. The final third text information, as shown in C, is more concise and highlights key information.

[0119] Therefore, this embodiment of the present disclosure employs a triple information compression method to streamline and filter images in the document to be processed. First, image filtering is performed to determine whether the image contains information useful for understanding the segment. Then, paragraph filtering is performed to summarize the image and retain effective paragraph information. Finally, text filtering is performed by comparing and filtering paragraphs using text compression technology to retain high-quality sentences for generating responses. This can completely solve the problem of introducing a large amount of noise caused by images in the process of multimodal document processing.

[0120] In an exemplary embodiment, after determining the third text information, candidate answer information can be generated in advance based on one or more third text information corresponding to the preset question. Therefore, when a user subsequently asks a question through the question-and-answer system, a preset question related to the user's question can be determined based on the user's question, and then an answer to the user's question can be quickly and accurately formed based on the candidate answer information corresponding to the preset question, greatly improving the response efficiency and accuracy of the question-and-answer system.

[0121] In practical applications, the methods of this disclosure can be integrated into information retrieval and question-answering systems. By performing triple filtering on the multimodal information (documents to be processed) input by the user, the system can quickly identify and extract useful information, thereby generating more accurate and relevant answers. For example, in a swarm intelligence question-answering system, screenshots uploaded by users can be processed using the methods of this disclosure, enabling the system to quickly extract key information from the screenshots and provide effective responses.

[0122] Figure 6 This is a schematic diagram of the workflow of the question-and-answer system in an exemplary embodiment of this disclosure.

[0123] refer to Figure 6 In step S601, the document to be processed is obtained from the multimodal information database. The document to be processed includes image and text information.

[0124] In step S602, the images are classified, and images of the preset category are retained as target images.

[0125] In step S603, first text information and context information of the target image are extracted.

[0126] In step S604, image filtering is performed. The target image is filtered according to the degree of relevance and the type of relevance between the first text information and the context information to obtain a target image that interprets or supplements the context information.

[0127] In step S605, paragraph filtering is performed to further filter the first text information in order to retain key paragraphs that explain or supplement the context information.

[0128] In step S606, text filtering is performed to retain text information related to the question-and-answer system (preset questions), which can be called image-compressed text.

[0129] In step S607, the information database can be vectorized based on contextual information and image compressed text.

[0130] In step S608, in response to the user's question, the user's question is vectorized.

[0131] In step S609, based on the vectorization results of the question and the information database, fragment retrieval is performed to retrieve relevant reference information. These fragments may be text information.

[0132] In step S610, the most relevant fragments to the problem are selected from the recalled fragments.

[0133] In step S611, an answer to the question is generated.

[0134] When a user asks a question, the question-answering system can use "BERT (Browser-Based Similarity Test) - a large model generation method" to find the most relevant fragment to the user's question from the information database, and then generate an answer to the user.

[0135] In summary, the triple filtering method of this disclosure can effectively reduce noise in image information and improve information quality and relevance. Compared with traditional OCR technology, it can extract content that is useful for user understanding more accurately. On the one hand, the increase in effective information improves the accuracy of recall; on the other hand, the reduction of interference and redundant information makes the responses generated by the response system more concise, thereby improving the overall performance of the multimodal information processing system.

[0136] Corresponding to the above method embodiments, this disclosure also provides an image information processing apparatus based on a multimodal knowledge question answering system, which can be used to execute the above method embodiments.

[0137] Figure 7 This is a block diagram of an image information processing apparatus in a multimodal knowledge question-answering system according to an exemplary embodiment of the present disclosure.

[0138] refer to Figure 7 The image information processing device 700 in a multimodal knowledge question answering system may include:

[0139] The image extraction module 71 is configured to obtain a target image from a file to be processed, wherein the file to be processed includes text data and an image, and the target image includes text information;

[0140] The information extraction module 72 is configured to extract text information from the target image as first text information, and to extract context information of the target image in the file to be processed based on the text data;

[0141] The image filtering module 73 is configured to select and retain the target image based on the first text information and the context information.

[0142] The information filtering module 74 is configured to determine, based on the context information, a second text information to be retained from the first text information of the retained target image, wherein the second text information includes paragraphs;

[0143] The association matching module 75 is configured to select third text information corresponding to the multiple preset questions from the second text information based on the matching results of the second text information and multiple preset questions, wherein the third text information includes sentences.

[0144] In one exemplary embodiment of this disclosure, the image extraction module 71 is configured to: determine the category of each image in the image of the file to be processed; and determine the image corresponding to the preset category as the target image.

[0145] In an exemplary embodiment of this disclosure, the image filtering module 73 is configured to: determine the degree of correlation between the first text information and the context information; when the degree of correlation is greater than a first preset value, determine the type of correlation between the first text information and the context information, wherein the type of correlation includes at least explanatory information and supplementary information; and retain the target image when the type of correlation is the explanatory information and / or the supplementary information.

[0146] In an exemplary embodiment of this disclosure, the information filtering module 74 is configured to: segment the first text information to form multiple text units; determine the correlation category of each text unit with the context information; and save the text unit whose correlation category is the explanatory information and / or the supplementary information as the second text information.

[0147] In one exemplary embodiment of this disclosure, the association matching module 75 is configured to: determine the text relevance between the second text information and the preset question; and when the text relevance is greater than a second preset value, determine the second text information as the third text information corresponding to the preset question.

[0148] In one exemplary embodiment of this disclosure, an answer pre-production module is also included, configured to generate candidate answer information for the preset question based on the third text information corresponding to the preset question.

[0149] In one exemplary embodiment of this disclosure, an information collection module is further included, configured to: form candidate information based on the second text information and the context information; obtain question information in response to a user's question; determine relevant information of the question information from at least one of the candidate information; and generate question-and-answer information corresponding to the question information based on the relevant information of the question information.

[0150] Since the functions of the device 700 have been described in detail in their respective method embodiments, they will not be repeated here.

[0151] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0152] In an exemplary embodiment of this disclosure, an electronic device capable of implementing the above-described method is also provided.

[0153] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: entirely hardware implementations, entirely software implementations (including firmware, microcode, etc.), or implementations combining hardware and software aspects, collectively referred to herein as “circuits,” “modules,” or “systems.”

[0154] The following reference Figure 8 To describe an electronic device 800 according to this embodiment of the present invention. Figure 8 The electronic device 800 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0155] like Figure 8 As shown, the electronic device 800 is manifested in the form of a general-purpose computing device. The components of the electronic device 800 may include, but are not limited to: at least one processing unit 810, at least one storage unit 820, and a bus 830 connecting different system components (including storage unit 820 and processing unit 810).

[0156] The storage unit stores program code that can be executed by the processing unit 810, causing the processing unit 810 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 810 can perform the method shown in the embodiments of this disclosure.

[0157] Storage unit 820 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 8201 and / or cache memory 8202, and may further include a read-only memory (ROM) 8203.

[0158] The storage unit 820 may also include a program / utility 8204 having a set (at least one) of program modules 8205, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0159] Bus 830 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0160] Electronic device 800 can also communicate with one or more external devices 900 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 800, and / or with any device that enables electronic device 800 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 850. Furthermore, electronic device 800 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 860. As shown, network adapter 860 communicates with other modules of electronic device 800 via bus 830. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0161] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0162] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible embodiments, various aspects of the invention may also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of the invention described in the "Exemplary Methods" section of this specification.

[0163] The program product for implementing the above-described method according to embodiments of the present invention may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0164] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0165] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0166] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0167] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0168] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0169] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and concept of this disclosure are indicated by the claims.

Claims

1. An image information processing method in a multimodal knowledge question-answering system, characterized in that, include: Obtain the target image from the file to be processed, wherein the file to be processed includes text data and an image, and the target image includes text information; The text information in the target image is extracted as the first text information, and the context information of the target image in the file to be processed is extracted based on the text data; The target image to be retained is selected based on the first text information and the context information; Based on the context information, a second text information to be retained is determined from the first text information of the retained target image, the second text information including paragraphs; Based on the matching results between the second text information and multiple preset questions, third text information corresponding to the multiple preset questions is selected from the second text information, and the third text information includes sentences.

2. The image information processing method in a multimodal knowledge question-answering system as described in claim 1, characterized in that, Obtaining the target image from the file to be processed includes: Determine the category of each image in the image of the file to be processed; The image corresponding to the preset category is determined as the target image.

3. The image information processing method in a multimodal knowledge question-answering system as described in claim 1, characterized in that, The target images selected for retention based on the first text information and the context information include: Determine the degree of relevance between the first text information and the context information; When the relevance is greater than a first preset value, the type of relevance between the first text information and the context information is determined, and the type of relevance includes at least explanatory information and supplementary information. When the relevant category is the explanatory information and / or the supplementary information, the target image is retained.

4. The image information processing method in a multimodal knowledge question-answering system as described in claim 3, characterized in that, The step of determining the second text information to be retained from the first text information of the retained target image based on the context information includes: The first text information is segmented to form multiple text units; Determine the type of association between each text unit and the context information; The text unit whose relevant category is the explanatory information and / or the supplementary information is saved as the second text information.

5. The image information processing method in a multimodal knowledge question-answering system as described in claim 1, characterized in that, Based on the matching results between the second text information and multiple preset questions, the third text information corresponding to the multiple preset questions is selected from the second text information, including: Determine the textual relevance between the second text information and the preset question; When the text relevance is greater than a second preset value, the second text information is determined as the third text information corresponding to the preset question.

6. The image information processing method in a multimodal knowledge question-answering system as described in claim 1, characterized in that, Also includes: Based on the third text information corresponding to the preset question, candidate answer information is generated for the preset question.

7. The image information processing method in a multimodal knowledge question-answering system as described in claim 1, characterized in that, Also includes: Candidate information is formed based on the second text information and the context information; Responding to a user's question, obtaining the question information, and determining relevant information from at least one of the candidate information; Based on the relevant information in the question, generate the corresponding question and answer information.

8. An electronic device, characterized in that, include: Memory; as well as A processor coupled to the memory, the processor being configured to perform the method as described in any one of claims 1-7 based on instructions stored in the memory.

9. A computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the method as described in any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1-7.

Images