Design method of multi-modal knowledge base management system

Abstract

The invention discloses a design method of a multi-modal knowledge base management system, and relates to the technical field of database management systems. Comprising the steps of 1, establishing a knowledge base management system, 2, pushing corresponding modal data into a corresponding analysis module through an identification module according to a heterogeneous data source type, and 3, obtaining corresponding unstructured data from MinIO distributed storage service according to a storage path and semantic retrieval when a large model processing module is called, and obtaining corresponding characters from the knowledge base for common display.

Description

Technical Field

[0001] The invention discloses a design method for a multi-modal knowledge base management system, and relates to the technical field of database management systems. Background Art

[0002] In today's big data era, information data is growing explosively. With the rapid development of information technology, the forms and sources of data are becoming increasingly diverse, including text, images, audio and video. These massive amounts of data contain a large amount of knowledge and information to be discovered. Quickly and effectively searching and analyzing massive amounts of data has become an important requirement for many companies and organizations. Manual analysis methods alone are far from meeting the actual needs of work. Efficient tools must be used to greatly improve work efficiency and reduce work pressure. Existing knowledge base management systems can often only process document-type data and are difficult to adapt to the integration, storage, retrieval and analysis needs of multimodal data. Therefore, there are some limitations:

[0003] 1. Single data format: Usually only supports document-type data, such as TXT, PDF, Word, etc., and lacks support for non-document-type data such as images, audio, and video.

[0004] 2. Lack of multimodal parsing capabilities: Lack of the ability to parse non-text data, such as images, audio, and video, and inability to extract useful information from these data.

[0005] 3. Storage structure limitations: The storage structure is often not suitable for storing multimodal data.

[0006] 4. Lack of intelligent retrieval and visual display: There is a lack of intelligent retrieval function, and the retrieval results can only be displayed in text form, lacking a visual display process. Summary of the invention

[0007] In view of the problems of the prior art, the present invention provides a design method for a multimodal knowledge base management system, which enables the knowledge base management system to integrate, store, retrieve and analyze data from different modalities, including files, texts, images, audio and video, etc. It overcomes the limitations of the prior art and provides a more efficient, comprehensive and intelligent heterogeneous data source parsing and retrieval solution, realizing multi-dimensional interaction and rich expression of information, so as to meet the needs of the knowledge base management system in the information age.

[0008] The specific scheme proposed by the present invention is:

[0009] The present invention provides a design method for a multimodal knowledge base management system, comprising:

[0010] Step 1: Establish a knowledge base management system, which includes a recognition module, an analysis processing module and a large model processing module. The analysis processing module includes a text analysis module, an image analysis module, a video analysis module and an audio analysis module.

[0011] Step 2: Push the corresponding modal data into the corresponding parsing module according to the type of heterogeneous data source through the recognition module, where the text data is parsed through the text parsing module. If there is an image in the text data, the original image is named with a unique identifier UUID, and the original image is uploaded to the MinIO distributed storage service. The storage path of the original image is embedded into the text data according to the custom label format, and the parsed text content and basic information of the file are stored in the vector knowledge base; the image parsing module uses the image text recognition method OCR to convert the text in the image into text, and stores it in the knowledge base in combination with the basic information description of the image, and stores the original image in the MinIO distributed storage service; the video parsing module parses the video data, and after parsing, the text summary of the video data is stored in the knowledge base in combination with the basic information description of the video, and the video data is named with a unique identifier and stored in the MinIO distributed storage service; the audio parsing module uses the speech recognition method to parse the audio data, convert the audio content into text, and store the converted text in the knowledge base;

[0012] Step 3: When the large model processing module is called, it retrieves the corresponding unstructured data from the MinIO distributed storage service according to the storage path based on semantic retrieval, and obtains the corresponding text from the knowledge base for joint display.

[0013] Furthermore, in step 1 of the method for designing a multimodal knowledge base management system, each module in the parsing and processing module is integrated into the knowledge base management system in the form of a plug-in library, and each operates independently to process the corresponding type of data.

[0014] Furthermore, in step 2 of the method for designing a multimodal knowledge base management system, the text data is parsed by a text parsing module, including:

[0015] Determine the text format of text data, including pdf, txt, doc, docx,

[0016] When parsing data in docx text format, deep parsing is performed according to the natural order of the text data.

[0017] After parsing the data in doc text format, the text content and basic information of the file are stored in the vector knowledge base.

[0018] Parse data in PDF text format according to the PDF format layout,

[0019] According to the nature of the txt plain text, the txt plain text content and basic information are stored in the knowledge base.

[0020] Furthermore, in step 2 of the method for designing a multimodal knowledge base management system, the video data is parsed by a video parsing module, including:

[0021] First, extract audio from the read video, and use speech recognition methods to convert the audio content into text. If the video is accompanied by a subtitle file in SRT format, parse the subtitle file to extract text information. Integrate the text obtained by audio transcription and subtitle extraction to extract key information, including the theme, purpose and conclusion, to form a text summary. The text summary is combined with the basic information description of the video and stored in the knowledge base. The video is named with a unique identifier and stored in the MinIO distributed storage service.

[0022] The present invention also provides a multimodal knowledge base management system, comprising a recognition module, a parsing processing module and a large model processing module, wherein the parsing processing module comprises a text parsing module, an image parsing module, a video parsing module and an audio parsing module.

[0023] The recognition module pushes the corresponding modal data into the corresponding parsing module according to the type of heterogeneous data source. The text parsing module parses the text data. If there is an image in the text data, the original image is named with a unique identifier UUID, and the original image is uploaded to the MinIO distributed storage service. The storage path of the original image is embedded into the text data according to the custom label format, and the parsed text content and basic information of the file are stored in the vector knowledge base; the image parsing module uses the image text recognition method OCR to convert the text in the image into text, and stores it in the knowledge base in combination with the basic information description of the image, and stores the original image in the MinIO distributed storage service; the video parsing module parses the video data, and after parsing, stores the text summary of the video data in combination with the basic information description of the video in the knowledge base, and stores the video data in the MinIO distributed storage service with a unique identifier name; the audio parsing module uses the speech recognition method to parse the audio data, converts the audio content into text, and stores the converted text in the knowledge base;

[0024] When called, the large model processing module obtains the corresponding unstructured data from the MinIO distributed storage service according to the storage path based on semantic retrieval, and obtains the corresponding text from the knowledge base for joint display.

[0025] Furthermore, the multimodal knowledge base management system integrates each module in the analysis and processing module into the knowledge base management system in the form of a plug-in library, and each module operates independently to process corresponding types of data.

[0026] Furthermore, the text parsing module of the multimodal knowledge base management system parses the text data, including:

[0027] Determine the text format of text data, including pdf, txt, doc, docx,

[0028] When parsing data in docx text format, deep parsing is performed according to the natural order of the text data.

[0029] After parsing the data in doc text format, the text content and basic information of the file are stored in the vector knowledge base.

[0030] Parse data in PDF text format according to the PDF format layout,

[0031] According to the nature of the txt plain text, the txt plain text content and basic information are stored in the knowledge base.

[0032] Furthermore, the video parsing module of the multimodal knowledge base management system parses the video data, including:

[0033] First, extract audio from the read video, and use speech recognition methods to convert the audio content into text. If the video is accompanied by a subtitle file in SRT format, parse the subtitle file to extract text information. Integrate the text obtained by audio transcription and subtitle extraction to extract key information, including the theme, purpose and conclusion, to form a text summary. The text summary is combined with the basic information description of the video and stored in the knowledge base. The video is named with a unique identifier and stored in the MinIO distributed storage service.

[0034] The benefits of the present invention are:

[0035] Efficiently integrate and process data from different modalities to ensure data diversity and comprehensiveness. The system integrates advanced technologies such as natural language processing, file parsing, OCR technology, audio extraction and video editing, and introduces multimodal parsing modules in the form of plug-in libraries to achieve intelligent content parsing, thereby improving the automation level of data processing and enhancing the scalability and flexibility of the system.

[0036] In the design method, the system has established a unified data management mechanism and equipped with a powerful search function, which can not only perform accurate search, but also deep semantic search. The search results can achieve multi-dimensional interaction and rich expression of information, greatly improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic diagram of the application process of the method of the present invention.

[0038] Figure 2This is a schematic diagram of the intelligent parsing process of docx files. DETAILED DESCRIPTION

[0039] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0040] Example 1

[0041] The present invention provides a design method for a multimodal knowledge base management system, comprising:

[0042] Step 1: Establish a knowledge base management system, which includes a recognition module, a parsing module and a large model processing module. The parsing module includes a text parsing module, an image parsing module, a video parsing module and an audio parsing module.

[0043] Each module in the parsing and processing module can be integrated into the knowledge base management system in the form of a plug-in library, and each module can operate independently to process corresponding types of data. The design greatly enhances the scalability and flexibility of the system.

[0044] Step 2: Push the corresponding modal data into the corresponding parsing module according to the type of heterogeneous data source through the identification module.

[0045] The text data is parsed through the text parsing module. If there are images in the text data, the original image is named with the unique identifier UUID, and the original image is uploaded to the MinIO distributed storage service. The storage path of the original image is embedded in the text data according to the custom label format, and the parsed text content and basic information of the file are stored in the vector knowledge base.

[0046] The method may further include: determining the text format of the text data, the text format including pdf, txt, doc, docx,

[0047] When parsing data in docx text format, deep parsing is performed according to the natural order of the text data.

[0048] After parsing the data in doc text format, the text content and basic information of the file are stored in the vector knowledge base.

[0049] Parse data in PDF text format according to the PDF format layout,

[0050] According to the nature of the txt plain text, the txt plain text content and basic information are stored in the knowledge base.

[0051] The image parsing module uses the image text recognition method OCR to convert the text in the image into text, and stores it in the knowledge base in combination with the basic information description of the image, and stores the original image in the MinIO distributed storage service.

[0052] The video data is parsed through the video parsing module. After parsing, the text summary of the video data is combined with the basic information description of the video and stored in the knowledge base, and the video data is stored in the MinIO distributed storage service with a unique identifier. Specifically, it may include: first extracting audio from the read video, using speech recognition methods to convert the audio content into text, if the video is accompanied by a subtitle file in SRT format, parsing the subtitle file to extract text information, integrating the text obtained by audio transcription and subtitle extraction, extracting key information, including the theme, purpose and conclusion, forming a text summary, storing the text summary in combination with the basic information description of the video in the knowledge base, and storing the video in the MinIO distributed storage service with a unique identifier.

[0053] The audio data is analyzed by the audio analysis module using the speech recognition method, the audio content is converted into text, and the converted text is stored in the knowledge base.

[0054] Step 3: When the large model processing module is called, the corresponding unstructured data is obtained from the MinIO distributed storage service according to the storage path based on semantic retrieval, and the corresponding text is obtained from the knowledge base for joint display. It can also use information such as keywords, content, and questions to perform vectorization processing by calling the large model processing to achieve deep semantic retrieval. When the search results contain unstructured data tags, the corresponding unstructured data and text are obtained from MinIO according to the path and displayed together. The search results not only provide plain text content, but also effectively integrate text with unstructured data such as images, audio, and video, realizing multi-dimensional interaction and rich expression of information, greatly enhancing the user experience.

[0055] Example 2

[0056] The present invention also provides a multimodal knowledge base management system, comprising a recognition module, a parsing processing module and a large model processing module, wherein the parsing processing module comprises a text parsing module, an image parsing module, a video parsing module and an audio parsing module.

[0057] The recognition module pushes the corresponding modal data into the corresponding parsing module according to the type of heterogeneous data source. The text parsing module parses the text data. If there is an image in the text data, the original image is named with a unique identifier UUID, and the original image is uploaded to the MinIO distributed storage service. The storage path of the original image is embedded into the text data according to the custom label format, and the parsed text content and basic information of the file are stored in the vector knowledge base; the image parsing module uses the image text recognition method OCR to convert the text in the image into text, and stores it in the knowledge base in combination with the basic information description of the image, and stores the original image in the MinIO distributed storage service; the video parsing module parses the video data, and after parsing, stores the text summary of the video data in combination with the basic information description of the video in the knowledge base, and stores the video data in the MinIO distributed storage service with a unique identifier name; the audio parsing module uses the speech recognition method to parse the audio data, converts the audio content into text, and stores the converted text in the knowledge base;

[0058] When called, the large model processing module obtains the corresponding unstructured data from the MinIO distributed storage service according to the storage path based on semantic retrieval, and obtains the corresponding text from the knowledge base for joint display.

[0059] As the information interaction and execution process between the modules in the above-mentioned system are based on the same concept as the embodiment of the method of the present invention, the specific contents can be found in the description of the embodiment of the method of the present invention and will not be repeated here.

[0060] Similarly, the system of the present invention efficiently integrates and processes data from different modalities to ensure the diversity and comprehensiveness of the data. The system integrates advanced technologies such as natural language processing, file parsing, OCR technology, audio extraction and video editing, and introduces a multimodal parsing module in the form of a plug-in library to achieve intelligent content parsing, thereby improving the automation level of data processing and enhancing the scalability and flexibility of the system.

[0061] The system has established a unified data management mechanism and is equipped with a powerful search function that can not only perform precise searches, but also deep semantic searches. The search results can achieve multi-dimensional interactions and rich expressions of information, greatly improving the user experience.

[0062] It should be noted that not all steps and modules in the above-mentioned processes and system structures are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The system structure described in the above-mentioned embodiments can be a physical structure or a logical structure, that is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or some components in multiple independent devices may be implemented together.

[0063] The above-described embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or changes made by those skilled in the art based on the present invention are within the protection scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. A design method for a multimodal knowledge base management system, characterized by include: Step 1: Establish a knowledge base management system, which includes a recognition module, an analysis processing module and a large model processing module. The analysis processing module includes a text analysis module, an image analysis module, a video analysis module and an audio analysis module. Step 2: Push the corresponding modal data into the corresponding parsing module according to the type of heterogeneous data source through the recognition module, where the text data is parsed through the text parsing module. If there is an image in the text data, the original image is named with a unique identifier UUID, and the original image is uploaded to the MinIO distributed storage service. The storage path of the original image is embedded into the text data according to the custom label format, and the parsed text content and basic information of the file are stored in the vector knowledge base; the image parsing module uses the image text recognition method OCR to convert the text in the image into text, and stores it in the knowledge base in combination with the basic information description of the image, and stores the original image in the MinIO distributed storage service; the video parsing module parses the video data, and after parsing, the text summary of the video data is stored in the knowledge base in combination with the basic information description of the video, and the video data is named with a unique identifier and stored in the MinIO distributed storage service; the audio parsing module uses the speech recognition method to parse the audio data, convert the audio content into text, and store the converted text in the knowledge base; Step 3: When the large model processing module is called, it retrieves the corresponding unstructured data from the MinIO distributed storage service according to the storage path based on semantic retrieval, and obtains the corresponding text from the knowledge base for joint display.

2. According to the design method of a multimodal knowledge base management system as described in claim 1, it is characterized in that in step 1, each module in the analysis and processing module is integrated into the knowledge base management system in the form of a plug-in library, and each operates independently to process the corresponding type of data.

3. The design method of a multimodal knowledge base management system according to claim 1 is characterized by: In step 2, the text data is parsed by the text parsing module, including: Determine the text format of text data, including pdf, txt, doc, docx, When parsing data in docx text format, deep parsing is performed according to the natural order of the text data. After parsing the data in doc text format, the text content and basic information of the file are stored in the vector knowledge base. Parse data in PDF text format according to the PDF format layout, According to the nature of the txt plain text, the txt plain text content and basic information are stored in the knowledge base.

4. The design method of a multimodal knowledge base management system according to claim 1 is characterized by: In step 2, the video data is parsed by the video parsing module, including: First, extract audio from the read video, and use speech recognition methods to convert the audio content into text. If the video is accompanied by a subtitle file in SRT format, parse the subtitle file to extract text information. Integrate the text obtained by audio transcription and subtitle extraction to extract key information, including the theme, purpose and conclusion, to form a text summary. The text summary is combined with the basic information description of the video and stored in the knowledge base. The video is named with a unique identifier and stored in the MinIO distributed storage service.

5. A multimodal knowledge base management system, characterized by It includes a recognition module, an analysis processing module and a large model processing module. The analysis processing module includes a text analysis module, an image analysis module, a video analysis module and an audio analysis module. The recognition module pushes the corresponding modal data into the corresponding parsing module according to the type of heterogeneous data source. The text parsing module parses the text data. If there is an image in the text data, the original image is named with a unique identifier UUID, and the original image is uploaded to the MinIO distributed storage service. The storage path of the original image is embedded into the text data according to the custom label format, and the parsed text content and basic information of the file are stored in the vector knowledge base; the image parsing module uses the image text recognition method OCR to convert the text in the image into text, and stores it in the knowledge base in combination with the basic information description of the image, and stores the original image in the MinIO distributed storage service; The video parsing module parses the video data, and after parsing, stores the Chinese text summary of the video data combined with the basic information description of the video into the knowledge base, and stores the video data in the MinIO distributed storage service with a unique identifier name; The audio analysis module uses speech recognition methods to analyze audio data, converts the audio content into text, and stores the converted text into the knowledge base; When called, the large model processing module obtains the corresponding unstructured data from the MinIO distributed storage service according to the storage path based on semantic retrieval, and obtains the corresponding text from the knowledge base for joint display.

6. A multimodal knowledge base management system according to claim 5, characterized in that Each module in the parsing and processing module is integrated into the knowledge base management system in the form of a plug-in library, and each operates independently to process the corresponding type of data.

7. A multimodal knowledge base management system according to claim 5, characterized in that The text parsing module parses text data, including: Determine the text format of text data, including pdf, txt, doc, docx, When parsing data in docx text format, deep parsing is performed according to the natural order of the text data. After parsing the data in doc text format, the text content and basic information of the file are stored in the vector knowledge base. Parse data in PDF text format according to the PDF format layout, According to the nature of the txt plain text, the txt plain text content and basic information are stored in the knowledge base.

8. A multimodal knowledge base management system according to claim 5, characterized in that The video parsing module parses the video data, including: First, extract audio from the read video, and use speech recognition methods to convert the audio content into text. If the video is accompanied by a subtitle file in SRT format, parse the subtitle file to extract text information. Integrate the text obtained by audio transcription and subtitle extraction to extract key information, including the theme, purpose and conclusion, to form a text summary. The text summary is combined with the basic information description of the video and stored in the knowledge base. The video is named with a unique identifier and stored in the MinIO distributed storage service.

Images