A multimodal recognition text-based picture knowledge source warehousing retrieval method

Abstract

The application relates to the technical field of knowledge base management, and provides a picture knowledge source warehousing retrieval method based on multi-modal recognition texturization, which comprises the following steps: when refreshing an index of a local knowledge base, scanning files in a knowledge base folder to recognize picture files; for the recognized picture files, reading a current model configuration, and constructing a restrictive picture recognition prompt, which requires a multi-modal model to extract searchable information in the picture and explicitly mark uncertain content; sending the picture files and the picture recognition prompt to a multi-modal model service to obtain picture recognition text, and when the picture recognition text is not empty, combining a picture source mark with the picture recognition text into picture document text; and fragmenting the picture document text, so that the picture content can be hit through text retrieval by writing the fragmentation result into a local full-text index. The application can effectively control model speculation risks and realize complexity while expanding the coverage range of the knowledge base.

Description

Technical Field

[0001] This invention relates to the field of knowledge base management technology, and in particular to a method for retrieval of image knowledge sources based on multimodal recognition and textualization. Background Technology

[0002] Local knowledge base systems are crucial infrastructure for enterprises and individuals to manage data and provide intelligent question answering. Their core value lies in unifying the indexing of scattered documents, enabling users to quickly retrieve relevant information through natural language queries. With the development of multimodal models and retrieval-enhanced generative technologies, the application scenarios of local knowledge bases are constantly expanding, placing higher demands on the coverage of knowledge sources.

[0003] However, in real-world work scenarios, a large amount of crucial experience and operational information is not stored in plain text format. Screenshots of system errors, configuration interfaces, workflow approval pages, command execution results, tables, instruction diagrams, and various technical architecture diagrams often contain far more information than their filenames suggest. For example, a system error screenshot might include the error code, stack trace, and time of occurrence; a configuration interface screenshot might record the complete parameter setting path. This information is invaluable for troubleshooting and knowledge reuse.

[0004] Existing local knowledge base implementations typically suffer from the following drawbacks in their handling of image files:

[0005] First, the indexing scope is limited to text documents. When scanning the knowledge base folder, the existing system only extracts content and builds an index for text files in formats such as txt, pdf, docx, and md. For image files, it only records the filename and storage path, without parsing the text, tables, error codes, or other information contained within the image. When a user's question involves screenshot content, full-text search cannot find any relevant segments.

[0006] Second, the information conveyed by image filenames is severely insufficient. Image files are usually named with timestamps automatically generated by the system or with short titles by the user, failing to summarize key search elements such as table fields, configuration parameters, command statements, and business module names contained within the image. An effective search entry point cannot be established based solely on the filename.

[0007] Third, information retrieval relies on manual, image-by-image viewing. Because image content is not indexed, users faced with a large volume of images must manually open each file for visual browsing to find the information they need. When the number of images is large or the content is similar, manual screening is extremely inefficient and prone to missing crucial information.

[0008] Fourth, real-time identification during the question-and-answer phase is difficult to implement practically. Some solutions attempt to send retrieved candidate images to a multimodal model in real-time after a user asks a question for identification and response. This approach has significant drawbacks: First, the question-and-answer phase requires determining which images are relevant to the question, which is difficult to achieve accurately without an index; second, image data needs to be sent repeatedly for each question and answer, resulting in high computational costs and response latency; finally, real-time identification lacks an evidence traceability mechanism, making it difficult for users to verify the accuracy of the model's output.

[0009] Fifth, there is a lack of effective constraints on the recognition results of multimodal models. Multimodal models have a natural tendency to make assumptions when recognizing image content, and may supplement false content when information is insufficient. If unconstrained recognition results are directly used as knowledge base content, it may introduce erroneous information and reduce the overall credibility of the knowledge base.

[0010] Chinese patents CN121009961A, CN119272756A, and CN118445395A disclose a method for constructing and retrieving a multimodal knowledge base based on a large language model. CN119272756A discloses a method for managing a multimodal knowledge base. CN118445395A discloses a question-answering method. None of these three methods, in the image content indexing stage, independently segment the image-recognized text and use the same segmenter and unified full-text indexing process as ordinary document text. Their knowledge bases all rely on vector retrieval as the core retrieval mechanism, rather than local full-text indexing. Furthermore, none of these methods establish a constraint mechanism on the multimodal model's recognition results during image content recognition; they do not require explicit annotation of uncertain content or prohibit the model from supplementing information not present in the image, thus failing to effectively control the risk of model conjecture. In the question-answering stage, all methods perform overall recall and answer generation at the paragraph or text block level, failing to treat the image content recognition results as independent fragments for full-text retrieval and display the image source path in the question-answering evidence for user verification of the original image.

[0011] Therefore, how to provide a method to convert image data into searchable text during the local knowledge base index building stage, so that image content can be included in the full-text index along with ordinary documents, while effectively controlling the model recognition risk and improving the local knowledge base's coverage and utilization efficiency of image data knowledge sources, has become an urgent technical problem to be solved. Summary of the Invention

[0012] In view of this, in order to overcome the shortcomings of the prior art, the present invention aims to provide a method for image knowledge source input and retrieval based on multimodal recognition and textualization.

[0013] This invention provides a method for image knowledge source input and retrieval based on multimodal recognition and textualization, the method comprising:

[0014] Step S1: When refreshing the index in the local knowledge base, scan the files in the knowledge base folder and identify image files;

[0015] Step S2: For the identified image file, read the current model configuration and construct a limiting image recognition prompt. This image recognition prompt requires the multimodal model to extract the searchable information in the image and explicitly label the uncertain content.

[0016] Step S3: Send the image file and image recognition prompts to the multimodal model service to obtain the image recognition text. When the image recognition text is not empty, combine the image source mark and the image recognition text into the image document text.

[0017] Step S4: Segment the image document text and write the segmentation results into the local full-text index so that the image content can be retrieved through text retrieval.

[0018] Optionally, in the image knowledge source retrieval method based on multimodal recognition and textualization of the present invention, in step S1, image files are identified by a preset set of image extensions, which includes one or more of png, jpg, jpeg, webp and bmp.

[0019] Optionally, in the image knowledge source retrieval method based on multimodal recognition and textualization of the present invention, in step S1, when the scanned object is not an image file, it is treated as a regular document and a text extraction process is performed. The regular document includes one or more of txt, md, pdf, docx and rtf.

[0020] Optionally, in the image knowledge source retrieval method based on multimodal recognition and textualization of the present invention, if the current model configuration does not exist or is unavailable in step S2, the corresponding image is marked as a recognition failure, the failure information is recorded, and other files are processed.

[0021] Optionally, in the image knowledge source retrieval method based on multimodal recognition and textualization of the present invention, the retrieval information in step S2 includes one or more of the following: text in the image, table fields, error messages, commands, paths, configuration items, and operation steps; the image recognition prompts also require the model to extract key information of the page, window, or business module visible in the screenshot, retain the original proper nouns, and prohibit the addition of information that does not exist in the image.

[0022] Optionally, in the image knowledge source retrieval method based on multimodal recognition textualization of the present invention, before sending the image file to the multimodal model service in step S3, the image file is first read into byte data and its MIME type is determined, and then the image is encoded into a data URL, which together with the image recognition prompt constitutes a multimodal request and then calls the model service interface.

[0023] Optionally, in the image knowledge source retrieval method based on multimodal recognition textualization of the present invention, in step S3, when the image recognition text returned by the multimodal model service is empty or contains only blank characters, the corresponding image is marked as a recognition failure, the failure information is recorded and other files are processed without interrupting the index refresh task.

[0024] Optionally, in the image knowledge source retrieval method based on multimodal recognition and textualization of the present invention, in step S3, the image source tag includes the image file name and the relative path of the image source, which is used to form a traceable binding relationship between the recognition result and the original image.

[0025] Optionally, in the image knowledge source retrieval method based on multimodal recognition and textualization of the present invention, in step S4, when segmenting the image document text, the image document text is first subjected to blank normalization processing, and then segmented according to the preset segment length and overlap length, so that the image document text and ordinary document text adopt the same segmentation rules and index writing process; each segmentation result includes at least a segment identifier, a document identifier, an image file name, an image source path, and a segment text.

[0026] Optionally, in the image knowledge source retrieval method based on multimodal recognition textualization of the present invention, after step S4, when a user asks a question, the local full-text index simultaneously retrieves ordinary document fragments and image recognition fragments. When an image recognition fragment is matched, one or more of the following are displayed in the question-and-answer evidence: image file name, image source path, fragment summary, and matching score, so that the user can verify the original image.

[0027] The image knowledge source retrieval method based on multimodal recognition and textualization of the present invention has the following beneficial technical effects: by converting image files into searchable text with the same fragmentation and indexing structure as ordinary documents after multimodal model recognition during the local knowledge base indexing refresh stage, and binding the recognition results with the image source path to support evidence tracing, the originally unsearchable image data is transformed into knowledge fragments that can be hit by text questions, can be answered, and whose source can be verified. While expanding the coverage of the knowledge base, it effectively controls the risk of model conjecture and the complexity of system implementation. Attached Figure Description

[0028] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1This is a flowchart illustrating the image knowledge source input and retrieval method based on multimodal recognition and textualization according to an exemplary embodiment 1 of the present invention;

[0030] Figure 2 This is a schematic diagram illustrating the process of splitting ordinary documents and image files according to the method of exemplary embodiment 1 of the present invention;

[0031] Figure 3 This is a schematic diagram of the process for image recognition failure degradation handling in the method of exemplary embodiment 1 of the present invention when model configuration is unavailable;

[0032] Figure 4 This is a schematic diagram of the failure handling process when the text in the image recognition is empty, according to the method of exemplary embodiment 1 of the present invention;

[0033] Figure 5 This is a schematic diagram illustrating the process of image segment retrieval and evidence display during the user questioning stage according to the method of Exemplary Embodiment 1 of the present invention. Detailed Implementation

[0034] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0035] It should be noted that, in the absence of conflict, the following embodiments and features can be combined with each other; and, based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0036] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0037] Example 1

[0038] Exemplary Example 1 of the present invention provides a method for image knowledge source input and retrieval based on multimodal recognition and textualization. Figure 1 This is a flowchart illustrating the image knowledge source input and retrieval method based on multimodal recognition and textualization according to Exemplary Embodiment 1 of the present invention, as follows: Figure 1 As shown, in this embodiment, the method of the present invention is implemented in the following manner:

[0039] Step S1: When refreshing the index in the local knowledge base, scan the files in the knowledge base folder and identify image files.

[0040] In this embodiment, image files are identified using a preset set of image file extensions, which includes one or more of png, jpg, jpeg, webp, and bmp. This embodiment accurately distinguishes image files from ordinary documents during the scanning phase, supports batch processing of mainstream image formats, and ensures the accuracy and format compatibility of the image recognition process.

[0041] Figure 2 This is a schematic diagram illustrating the process of splitting ordinary documents and image files according to the method of exemplary embodiment 1 of the present invention, as follows: Figure 2 As shown, in this embodiment, when the scanned object is not an image file, it is treated as a regular document, and a text extraction process is performed. Regular documents include one or more of the following formats: txt, md, pdf, docx, and rtf. This embodiment enables the image recognition process and the regular document processing process to run in parallel without interfering with each other. The image recognition process only applies to image files and does not change the existing indexing and processing path of regular documents, thereby expanding the knowledge base coverage while maintaining efficient processing capabilities for the original document types.

[0042] Step S2: For the identified image file, read the current model configuration and construct a limiting image recognition prompt. This image recognition prompt requires the multimodal model to extract searchable information from the image and explicitly label uncertain content.

[0043] Figure 3 This is a schematic diagram illustrating the process of image recognition failure degradation handling in the method of exemplary embodiment 1 of the present invention when model configuration is unavailable, as shown below. Figure 3 As shown in this embodiment, if the current model configuration does not exist or is unavailable, the corresponding image is marked as a recognition failure, the failure information is recorded, and other files are processed. This ensures that the failure to recognize a single image or multiple images does not affect the normal completion of the entire knowledge base indexing refresh task. Other images and ordinary documents can still be indexed and built, thereby improving the stability and fault tolerance of the indexing process and avoiding the interruption of the overall task due to individual file or configuration problems.

[0044] In this embodiment, the searchable information includes one or more of the following: text in the image, table fields, error messages, commands, paths, configuration items, and operation steps. Image recognition prompts also require the model to extract key information from the visible pages, windows, or business modules in the screenshot, retain the original proper nouns, and prohibit the addition of information not present in the image. This embodiment can guide the multimodal model to output structured text with high retrieval value, making the generated recognized text more closely aligned with the user's search needs. Simultaneously, through uncertainty annotation requirements and information supplementation prohibitions, it can effectively reduce the risk of the model introducing incorrect information or false content through conjecture, thereby improving the accuracy of the recognized text and the overall credibility of the knowledge base.

[0045] Step S3: Send the image file and image recognition prompts to the multimodal model service to obtain the image recognition text. When the image recognition text is not empty, combine the image source mark and the image recognition text into the image document text.

[0046] In this embodiment, before sending the image file to the multimodal model service, the image file is first read as byte data and its MIME type is determined. Then, the image is encoded into a data URL, which, together with the image recognition prompt, forms a multimodal request to call the model service interface. In this embodiment, the image file encoding and multimodal request construction method seamlessly integrates with various multimodal model services compatible with OpenAI interface formats, thereby reducing the integration difficulty with different model services and improving the stability of interface calls.

[0047] Figure 4 This is a schematic diagram of the failure handling process when the text in the image recognition is empty, according to an exemplary embodiment 1 of the present invention. Figure 4 As shown in this embodiment, when the image recognition text returned by the multimodal model service is empty or contains only blank characters, the corresponding image is marked as a recognition failure, the failure information is recorded, and other files continue to be processed without interrupting the index refresh task. This embodiment can eliminate invalid recognition results caused by reasons such as unrecognizable image content, image corruption, or abnormal model service, avoiding the writing of empty or invalid content into the index, which would cause meaningless fragments to occupy storage and retrieval resources, thereby ensuring the quality of index data and maintaining the overall continuity of the indexing task.

[0048] In this embodiment, the image source marker includes the image file name and the relative path of the image source, which is used to form a traceable binding relationship between the recognition result and the original image, so that each piece of recognized text can be traced back to its original image file. This provides a reliable basis for presenting the source of evidence and for users to verify the original image in the subsequent question-and-answer session, thereby enhancing the transparency and credibility of the knowledge base.

[0049] Step S4: Segment the image document text and write the segmentation results into the local full-text index so that the image content can be retrieved through text retrieval.

[0050] In this embodiment, when segmenting image document text, the image document text is first normalized for whitespace, and then segmented according to preset fragment length and overlap length, so that image document text and ordinary document text use the same segmentation rules and index writing process; each segment result includes at least a fragment identifier, a document identifier, an image file name, an image source path, and fragment text. This embodiment enables image-recognized text and ordinary document text to use a completely unified data structure and storage format at the indexing level, eliminating the need to maintain separate indexing logic for image data.

[0051] Figure 5 This is a schematic diagram illustrating the process of image segment retrieval and evidence presentation during the user questioning stage of the method according to Exemplary Embodiment 1 of the present invention. Figure 5 As shown, in this embodiment, after step S4, when a user asks a question, the local full-text index simultaneously searches for ordinary document fragments and image recognition fragments. When an image recognition fragment is matched, one or more of the following are displayed in the question-and-answer evidence: image file name, image source path, fragment summary, and matching score, for the user to verify the original image. This embodiment enables users to directly retrieve detailed content contained in an image using keywords, eliminating the need to manually browse and search through each image file. Furthermore, it allows for quick location and verification of the original image based on the displayed source information, reducing the cost of manual image retrieval and improving knowledge acquisition efficiency and information credibility.

[0052] Example 2

[0053] Exemplary Example 2 of the present invention provides a method for image knowledge source input and retrieval based on multimodal recognition and textualization. This embodiment further describes the method of the present invention in detail with reference to the DDKnowledge local knowledge base system. Specific implementation methods are described below.

[0054] Specifically, in this embodiment, the method of the present invention is implemented in the following manner:

[0055] Step 1: File Type Identification

[0056] The system maintains two sets of file extensions when scanning local folders:

[0057] 1. A collection of common document extensions, which may include txt, md, pdf, docx, rtf;

[0058] 2. A set of image file extensions, which may include png, jpg, jpeg, webp, and bmp.

[0059] The system recursively scans regular files in the knowledge base root directory and filters files belonging to the aforementioned file extension set. This ensures that image files and regular documents are uniformly included in the index refresh process during the scanning phase.

[0060] Step 2: Separate plain documents and image files

[0061] For each scanned file, the system first determines whether it has an image extension. If it is not an image, it proceeds to the plain text extraction process; if it is an image, it proceeds to the image content recognition process. The plain text extraction process can use a document parser to parse formats such as PDF, Word, and RTF, providing a fallback for directly reading text files or Markdown files. The image content recognition process does not directly write the image binary into the full-text index; instead, it first converts it into searchable text using a multimodal model.

[0062] Step 3: Read the image recognition model configuration

[0063] The system reads the current default model configuration. The model configuration may include parameters such as the model interface address, model name, interface key, timeout time, and system prompt. If the user has not yet configured a model, the image recognition module cannot call the multimodal service. In this case, the system throws a recognizable exception, which is caught by the index refresh process. The current image is counted as a failed document, rather than terminating the entire knowledge base refresh. This decouples image recognition capabilities from ordinary document indexing. Even if image recognition is unavailable, the system can still continue processing other ordinary documents or other available sources.

[0064] Step 4: Limited Image Recognition Hints

[0065] When constructing image recognition prompts, the system does not require the model to freely describe the image; instead, the output target is limited to "text suitable for knowledge base retrieval." The prompts require the model to extract the following: 1. Text in the image; 2. Table fields; 3. Error messages; 4. Commands; 5. Paths; 6. Configuration items; 7. Operation steps; 8. Key information about pages, windows, or business modules visible in the screenshot.

[0066] The prompt also requires that original proper nouns be retained, and uncertain content be written as "uncertain," and that no information not present in the image be added. The purpose of this constraint is not to enhance the aesthetic description of the image, but to transform it into searchable, traceable, and minimally speculative knowledge text.

[0067] Step 5: Constructing Multimodal Requests

[0068] The system reads the image as byte data and determines its MIME type based on the file type. If the system cannot detect the MIME type, it can use the file extension as a fallback: image / jpeg, image / webp, image / bmp, or image / png. Subsequently, the system encodes the image into a data URL and combines it with a text prompt to form a multimodal request. The request includes system role prompts, user text prompts, and the image content. The model service returns the image-recognized text. For example, in practical applications, the multimodal request can reuse the OpenAI-compatible chat completions interface, passing the image as the image_url content item.

[0069] Step Six: Verification of Recognition Results

[0070] After the model returns, the system checks if the recognized text is empty. If the return value is empty, blank, or lacks content, the system considers the image recognition to have failed and counts the image as a failed document. If the recognized text is not empty, the system continues to generate the image document text. This check avoids writing empty results into the index, which would result in meaningless fragments.

[0071] Step 7: Image Document Text Generation

[0072] The system uses the image filename as the document title and the image's relative path or source display path as the image source. It then combines the recognized text with the image source into the document text. For example, the document text might include: 1. Image source; 2. Text recognized by the model; 3. Table fields recognized by the model; 4. Error messages, commands, paths, and steps recognized by the model. This structure allows the system to not only return the recognized content when a search is successful, but also display which image that content originated from.

[0073] Step 8: Image, document, and text segmentation

[0074] After the image document text is generated, the system hands it over to the same slicer as a regular document. The slicer can first perform whitespace normalization, and then segment it according to the preset fragment length and overlap length.

[0075] Each image fragment must contain at least: 1. Fragment identifier; 2. Document identifier; 3. Image file name; 4. Image source path; 5. Fragment text. Therefore, image-recognized text and ordinary document text adopt the same structure at the indexing level.

[0076] Step 9: Write the local full-text index

[0077] The system writes the image fragments to the local full-text index. Index fields can include title search fields, source path search fields, and text search fields. After writing, keywords in a user's question can match the image-recognized text, or the image filename or image source path. For example, if a user asks "steps to replace the nginx certificate for a cost control system," and that information comes from an image screenshot, the system can still retrieve the corresponding image fragment.

[0078] Step 10: Failure Degradation and Index Summary

[0079] Image recognition failure does not cause the entire indexing task to fail. The system catches exceptions while processing each file, increments the failure document count for failed images, and continues processing other files. After the index refresh is complete, the system returns a unified summary, including the number of successfully indexed documents, the number of generated fragments, and the number of failed documents. The interface can then use this information to alert the user if any image recognition failed.

[0080] Step 11: Presentation of Evidence Through Questions and Answers

[0081] After a user submits a question, the local retrieval returns the relevant fragment. If the fragment originates from image-recognized text, the system can display the image filename, source path, fragment summary, and relevance score in the evidence panel. The user can then open the original image for verification based on the source path. This approach ensures that image recognition results are not entered into the model in isolation, but rather serve as source-supported evidence in the question-and-answer process.

[0082] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0083] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for image knowledge source input and retrieval based on multimodal recognition and textualization, characterized in that, The method includes: Step S1: When refreshing the index in the local knowledge base, scan the files in the knowledge base folder and identify image files; Step S2: For the identified image file, read the current model configuration and construct a limiting image recognition prompt. This image recognition prompt requires the multimodal model to extract the searchable information in the image and explicitly label the uncertain content. Step S3: Send the image file and image recognition prompts to the multimodal model service to obtain the image recognition text. When the image recognition text is not empty, combine the image source mark and the image recognition text into the image document text. Step S4: Segment the image document text and write the segmentation results into the local full-text index so that the image content can be retrieved through text retrieval.

2. The image knowledge source input and retrieval method based on multimodal recognition and textualization according to claim 1, characterized in that, In step S1, image files are identified using a preset set of image extensions, which includes one or more of png, jpg, jpeg, webp, and bmp.

3. The image knowledge source input and retrieval method based on multimodal recognition and textualization according to claim 1, characterized in that, In step S1, when the scanned object is not an image file, it is treated as a regular document and the text extraction process is performed. Regular documents include one or more of the following: txt, md, pdf, docx, and rtf.

4. The image knowledge source input and retrieval method based on multimodal recognition and textualization according to claim 1, characterized in that, In step S2, if the current model configuration does not exist or is unavailable, the corresponding image is marked as a recognition failure, the failure information is recorded, and other files are processed.

5. The image knowledge source input and retrieval method based on multimodal recognition and textualization according to claim 1, characterized in that, In step S2, the retrievable information includes one or more of the following: text in the image, table fields, error messages, commands, paths, configuration items, and operation steps. The image recognition prompt also requires the model to extract key information of the page, window, or business module visible in the screenshot, retain the original proper nouns, and prohibit the addition of information that does not exist in the image.

6. The image knowledge source input and retrieval method based on multimodal recognition and textualization according to claim 1, characterized in that, In step S3, before sending the image file to the multimodal model service, the image file is first read as byte data and its MIME type is determined. Then, the image is encoded into a data URL, which, together with the image recognition prompt, forms a multimodal request and calls the model service interface.

7. The image knowledge source input and retrieval method based on multimodal recognition and textualization according to claim 1, characterized in that, In step S3, when the image recognition text returned by the multimodal model service is empty or contains only blank characters, the corresponding image is marked as recognition failed, the failure information is recorded, and other files are processed without interrupting the index refresh task.

8. The image knowledge source input and retrieval method based on multimodal recognition and textualization according to claim 1, characterized in that, In step S3, the image source marker includes the image file name and the relative path of the image source, which is used to establish a traceable binding relationship between the recognition result and the original image.

9. The image knowledge source input and retrieval method based on multimodal recognition and textualization according to claim 1, characterized in that, In step S4, when segmenting the image document text, the image document text is first normalized for whitespace, and then segmented according to the preset segment length and overlap length, so that the image document text and ordinary document text use the same segmentation rules and index writing process; each segmentation result includes at least a segment identifier, a document identifier, an image file name, an image source path, and a segment body.

10. The image knowledge source input and retrieval method based on multimodal recognition and textualization according to claim 1, characterized in that, After step S4, when a user asks a question, the local full-text index simultaneously searches for ordinary document fragments and image recognition fragments. When an image recognition fragment is matched, one or more of the following are displayed in the question-and-answer evidence: image file name, image source path, fragment summary, and matching score, so that the user can verify the original image.

Images