Multi-modal knowledge matching system

Abstract

The invention belongs to the field of artificial intelligence, and particularly relates to a multi-modal knowledge matching system which comprises a construction module, an image processing module, a matching module and an output module to realize seamless connection of an image and a text knowledge base. According to the method, the user is allowed to use the picture to perform knowledge retrieval, the user experience is improved, the picture is conveniently used to obtain required information, text information can be accurately extracted from the picture and matched with the text knowledge base, the matching accuracy and integrity are improved, the user can directly use the picture to perform knowledge retrieval, and the user experience is improved. And the system automatically identifies characters in the picture and matches the characters with the knowledge base, so that intelligent processing and utilization of image information are realized, and a more intelligent, convenient and efficient information retrieval and question-answering system is realized.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence, and in particular relates to a multimodal knowledge matching system. Background Art

[0002] In today's era of explosive development of AI, natural language processing is increasingly used in the fields of intelligent robots, virtual assistants and knowledge base systems, and plays an extremely important role in them. Traditional knowledge bases, such as customer service robots on current e-commerce platforms, only allow users to interact by entering text and matching it with the knowledge base in the system. Although this can solve most usage scenarios, it still has great limitations in this era of more diversified information, such as automatic interaction through pictures.

[0003] With the continuous development of digital image processing and optical character recognition (OCR) technology, the recognition and understanding of image content has made significant progress. However, matching image content with text knowledge base to achieve intelligent question-answering and other functions has not yet seen an ideal product implementation. Traditional text knowledge base matching systems cannot directly process image inputs. Therefore, when processing queries containing image information, users usually need to convert the text information in the image into text format and then input it into the system for matching. This conversion process not only increases the complexity of the user's operation, but may also lead to information loss and inaccurate matching.

[0004] Therefore, it is necessary to provide a technology that can directly process image input and match it with a text knowledge base, so as to achieve a more intelligent, convenient and efficient information retrieval and question-answering system. In view of the above problems, the present invention proposes an image-assisted knowledge base matching system and method, aiming to overcome the limitations of traditional systems and improve user experience and system performance. Summary of the invention

[0005] The purpose of the present invention is to provide a multimodal knowledge matching system, which can improve the user experience by allowing the user to use pictures for knowledge retrieval, and conveniently use pictures to obtain the required information, which makes knowledge retrieval more intuitive and convenient. By utilizing advanced image processing and optical character recognition technology, it can accurately extract text information from pictures and match it with a text knowledge base, thereby improving the accuracy and completeness of the match, making the knowledge results obtained by the user more accurate. By realizing the seamless integration of image input and text knowledge base, the intelligence of the system is brought to a new level. Users can directly use pictures for knowledge retrieval, and the system will automatically recognize the text in the picture and match it with the knowledge base, thereby realizing the intelligent processing and utilization of image information.

[0006] The technical solution adopted by the present invention is as follows:

[0007] A multimodal knowledge matching system includes a building module, wherein the building module is used to build a knowledge point library;

[0008] An image processing module, the image processing module is used to extract features from an input image and output text content, wherein the features include text features and image features;

[0009] A matching module, which is used to match the text content with the knowledge point library and output relevant knowledge points;

[0010] An output module, the output module is used to return relevant knowledge points to the user interface;

[0011] The image processing module includes an image preprocessing module and an image feature extraction module. The image preprocessing module is used to preprocess and enhance the input image, and the image feature extraction module is used to convert the text information in the processed image into retrievable and analyzable text data.

[0012] In a preferred embodiment, the image preprocessing module includes analyzing the image input by the user and matching the original image format, and extracting edge, texture, shape and color features through features to enhance the contrast and details of the input image and make the image features more obvious. Secondly, it removes noise in the input image to improve image quality and avoid incorrect matching.

[0013] In a preferred embodiment, the processing of the image feature extraction module includes an image recognition OCR module and an Embedding module, wherein the image recognition OCR module extracts text features from the image through image recognition technology, and the Embedding module is used to semantically combine the extracted text features with image features and transform them through the embedding model.

[0014] In a preferred embodiment, the image recognition OCR module includes text positioning, character segmentation, character recognition and post-processing. The text positioning is used to identify the text area in the image, and the EAST algorithm is used to detect and segment the text. The character segmentation divides the located text area into single characters or words in preparation for recognition. The Transformer model is used to recognize the segmented characters. The post-processing corrects the recognition and segmentation results of the characters.

[0015] In a preferred embodiment, the Embedding module is used to convert the text extracted by the image recognition OCR module into a numerical feature vector, which can capture the semantic information and contextual relationship of the text and is used to fuse with the image features to form a multimodal feature representation.

[0016] In a preferred embodiment, the knowledge point library includes knowledge storage, data management and knowledge representation. The knowledge storage serves as the knowledge basis of the system and stores structured or semi-structured knowledge. The structured or semi-structured knowledge can be facts, concepts and entity relationships. The data management is used for indexing, retrieving, updating and managing data. Knowledge can be queried and used. The knowledge representation is used to represent it in a manner suitable for computer processing.

[0017] In a preferred embodiment, the matching module is a bridge connecting the user interface, the numerical feature vector and the knowledge base, so that users can obtain the required information and improve the performance and user experience of the entire system. The matching module includes using an Elasticsearch database as its back-end component. The Elasticsearch database stores and manages a large amount of data in the knowledge base, wherein the Elasticsearch can index structured data and provide efficient query and matching capabilities. The Elasticsearch database is based on the Embedding vector and uses the cosine similarity method to quickly match relevant information in the knowledge base. According to the matching results, the information most relevant to the numerical feature vector is retrieved from the knowledge base, and the relevant information includes entities, concepts, entities with specific attributes, and non-text modal information in the text.

[0018] In a preferred embodiment, the output module is used to provide a user-friendly access point to ensure that users can effectively input queries, receive feedback, and interact with the system intuitively. The output module includes information presentation, operation guidance, and interactive feedback. The information presentation clearly displays the results of the knowledge base matching to the user, the operation guidance guides the user to perform correct operations, and the interactive feedback provides real-time feedback on the results and status of the user's operations, allowing the user to understand the interaction situation.

[0019] In a preferred solution, the output module further includes a feedback mechanism, whereby users can provide feedback on search results through a user interface, and the matching module can use the feedback to optimize the query algorithm.

[0020] In a preferred embodiment, at least one processor;

[0021] and a memory communicatively coupled to the at least one processor;

[0022] Wherein, the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the multimodal knowledge matching system described in any one of claims 1 to 9.

[0023] The technical effects achieved by the present invention are:

[0024] The present invention improves user experience by supporting users to search for information using images, so that users are no longer limited to text input methods, but can intuitively use pictures to find the information they need, which will make knowledge retrieval more intuitive and convenient. By utilizing image processing and optical character recognition technology, text information can be accurately extracted from pictures and matched with a text knowledge base, thereby improving the accuracy and completeness of the match and making the knowledge results obtained by users more precise. At the same time, by achieving seamless integration of image input and text knowledge base, users can directly use pictures for knowledge retrieval, and the system will automatically recognize the text in the picture and match it with the knowledge base, thereby realizing intelligent processing and utilization of image information. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a system module diagram provided by the present invention;

[0026] Figure 2 It is a flow chart of the system execution provided by the present invention. DETAILED DESCRIPTION

[0027] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0028] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0029] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure or characteristic that may be included in at least one implementation of the present invention. The phrase "in a preferred embodiment" that appears in different places in this specification does not refer to the same embodiment, nor is it a separate or selective embodiment that is mutually exclusive with other embodiments.

[0030] See also Figure 1 and Figure 2 As shown, it is a first embodiment of the present invention, which provides a multimodal knowledge matching system, including a building module, the building module is used to build a knowledge point library;

[0031] An image processing module, the image processing module is used to extract features from an input image and output text content, wherein the features include text features and image features;

[0032] Matching module, which is used to match text content with the knowledge point library and output relevant knowledge points;

[0033] Output module, the output module is used to return relevant knowledge points to the user interface.

[0034] The image processing module includes an image preprocessing module and an image feature extraction module. The image preprocessing module is used to preprocess and enhance the input image, and the image feature extraction module is used to convert the text information in the processed image into retrievable and analyzable text data.

[0035] Traditional text knowledge base matching systems cannot directly process image inputs. Therefore, when processing queries containing image information, users usually need to convert the text information in the image into text format and then input it into the system for matching. This conversion process not only increases the complexity of user operations, but may also lead to information loss and inaccurate matching. Therefore, it is particularly important to develop a multimodal knowledge matching system. In this embodiment, the construction module is used to construct a knowledge point library, which can help people find the required information more quickly and avoid information overload and time waste. The image processing module includes an image preprocessing module and an image feature extraction module. The image preprocessing module is used to preprocess and enhance the input image. The image feature extraction module is used to convert the text information in the processed image into retrievable and analyzable text data. The image processing module is responsible for feature extraction of the input image, which includes text features and image features. The matching module is used to match the text content with the knowledge point library and output relevant knowledge points, which greatly shortens the retrieval time and improves the efficiency of knowledge management. The output module is used to return the relevant knowledge points to the user interface. The output module can not only improve the accuracy and efficiency of matching, but also enhance the user's operating experience through intuitive visualization means.

[0036] In a preferred embodiment, the image preprocessing module includes analyzing the image input by the user and matching the original image format, and enhancing the contrast and details of the input image by feature extraction of edge, texture, shape and color features to make the image features more obvious. Secondly, it removes noise from the input image to improve image quality and avoid incorrect matching.

[0037] In this embodiment, the image preprocessing module matches the original image format to reduce the additional format conversion steps, thereby speeding up the entire preprocessing process, improving the efficiency of the system, and avoiding the loss of image quality that may be caused during the format conversion process. Directly using the original image format can avoid this problem. By extracting edge, texture, shape and color features through features, the contrast and details of the input image are enhanced, and features such as edges and textures are strengthened, which helps to highlight important information in the image and reduce the interference of background noise. At the same time, when enhancing image features, the complexity and computational complexity of the algorithm are reduced, the contrast and details of the input image are enhanced, making image segmentation easier and reducing the possibility of segmentation errors. The noise in the input image is removed, and the denoised image has higher feature consistency between different modalities, which is conducive to the fusion and complementarity of multimodal knowledge. The preprocessing module removes random noise in the image through noise reduction technology, improves the visual quality of the image, and makes the subsequent image recognition OCR module extraction more accurate.

[0038] In a preferred embodiment, the processing of the image feature extraction module includes an image recognition OCR module and an Embedding module, wherein the image recognition OCR module extracts text features from the image using image recognition technology, and the Embedding module is used to semantically combine the extracted text features with the image features and transform them through the embedding model.

[0039] In this embodiment, the image recognition OCR module is used to convert the text in the image into a machine-coded text form. The image processed by the image preprocessing module is extracted by the image recognition OCR module. These features include edges, corners, lines, etc. The use of Canny edge detection and other technologies can effectively highlight the text features and lay the foundation for text recognition. The Embedding module converts the recognized text into a vectorized representation with semantic information.

[0040] In a preferred embodiment, the image recognition OCR module includes text positioning, character segmentation, character recognition and post-processing. Text positioning is used to identify the text area in the image. The EAST algorithm is used to detect and segment the text. Character segmentation divides the located text area into single characters or words in preparation for recognition. The Transformer model is used to recognize the segmented characters. The post-processing corrects the recognition and segmentation results of the characters.

[0041] In this embodiment, text positioning is the initial stage of the image recognition OCR module process. The core task of text positioning is to accurately identify the area where the text is located in the image content. This step is very critical to the subsequent character recognition process. Its accuracy will directly determine the overall performance of the image recognition OCR module. In order to achieve efficient and accurate text positioning, the EAST algorithm uses a deep learning model to predict the bounding box of the text area. By leveraging the powerful ability of neural networks to extract features, the EAST algorithm can adapt to texts of different sizes, directions and fonts, thereby significantly improving the speed and accuracy of text detection. When the text area is located, character segmentation can divide the located text area into individual characters or words, and use connected component analysis and other technologies to detect and separate individual characters. Connected components refer to areas composed of connected pixels in an image. Connected component analysis can identify connected pixel areas in an image, thereby accurately locating the position of each character. Through geometric analysis and contour extraction, each character is separated from the text area. The character recognition stage is the core of the image recognition OCR module. Character recognition and the conversion of segmented characters into machine-encoded text. In the field of character recognition, the Transformer model of deep learning has made significant progress. The Transformer model uses the self-attention mechanism to capture long-distance dependencies between characters and improve the accuracy of character recognition. Post-processing is closely linked to character recognition, and the two work together to improve the final text recognition results. Errors may occur in the character recognition stage. Post-processing proofreads and adjusts the format of the recognition results to correct these errors. Post-processing can use contextual information to improve the results of character recognition. For example, if the recognized character sequence is semantically unreasonable, post-processing can infer the correct character based on the context.

[0042] In a preferred embodiment, the Embedding module is used to convert the text extracted by the image recognition OCR module into a numerical feature vector, which can capture the semantic information and contextual relationship of the text and is used to fuse with the image features to form a multimodal feature representation.

[0043] In this implementation, the Embedding module can convert the text extracted by the OCR module into numerical vectors. These numerical feature vectors are constructed in such a way that similar vectors in space represent semantically similar words. At the same time, in the Embedding process, not only is the text simply mapped to the numerical feature vector, but more importantly, the semantic relationship between words is captured. The numerical feature vectors generated by the Embedding module can be combined with image features to form a multimodal feature representation, which facilitates the understanding of the visual content and text information in the image and also enhances the model's ability to understand complex scenes.

[0044] In a preferred embodiment, the knowledge point library includes knowledge storage, data management and knowledge representation. The knowledge storage serves as the knowledge basis of the system and stores structured or semi-structured knowledge. The structured or semi-structured knowledge can be facts, concepts and entity relationships. Data management is used for indexing, retrieving, updating and managing data. Knowledge can be queried and used. The knowledge representation is used to represent it in a manner suitable for computer processing.

[0045] In this implementation, knowledge storage is the basis of the knowledge point library. This knowledge can be structured or semi-structured. Data management plays a core role in the knowledge point library. It is responsible for indexing, searching, updating and managing data. Indexing is to improve search efficiency. By creating indexes for data, the required knowledge can be found quickly. Searching is the process of finding and extracting required data from the knowledge point library. Updating is an important step to ensure that the knowledge in the knowledge point library remains up-to-date and accurate. Management involves aspects such as data maintenance, backup, security and integrity. Knowledge representation is the conversion of knowledge into a format that can be processed by computers. The goal is to enable computers to understand, reason and operate knowledge.

[0046] In a preferred embodiment, the matching module is a bridge connecting the user interface, the numerical feature vector and the knowledge base, so that users can obtain the required information and improve the performance and user experience of the entire system. The matching module uses the Elasticsearch database as its back-end component. The Elasticsearch database stores and manages a large amount of data in the knowledge base, wherein Elasticsearch can index structured data and provide efficient query and matching capabilities. The Elasticsearch database is based on the Embedding vector and uses the cosine similarity method to quickly match relevant information in the knowledge base. According to the matching results, the information most relevant to the numerical feature vector is retrieved from the knowledge base. The relevant information includes entities, concepts, entities with specific attributes, and non-text modal information in the text.

[0047] In this embodiment, the matching module compares the numerical feature vector output from the Embedding module with the entries in the knowledge point library to find the item that best matches the user's query intent. The item to be queried will first be sent to the Embedding module for processing. The Embedding module converts the text information into numerical feature vectors. These numerical feature vectors are presented in a multidimensional space, where each dimension represents a specific semantic attribute of the text, such as lexical meaning, context association, and syntactic structure. After the matching module receives the numerical feature vector, the matching module will start the matching algorithm. The goal of these algorithms is to find the most relevant entry from the knowledge base with the numerical feature vector. The knowledge point library contains a large number of pre-indexed entries, each of which is rich in structured or semi-structured knowledge, such as facts, concepts, and entity relationships. The matching module calculates the similarity between the vector and the knowledge base entry, and uses methods such as cosine similarity to measure the degree of match between the two. At the same time, in order to improve the efficiency and accuracy of matching, the matching module also uses advanced search technologies such as Elasticsearch. Elasticsearch is a distributed, scalable search and data analysis engine that can quickly process large amounts of data and support real-time search and complex query requests. Through Elasticsearch, the matching module can filter out the most relevant information to the user's query intent from the massive knowledge base entries in a short period of time. When the matching module finds the best match, it returns this information to the user interface, and the user can view and evaluate the search results. If the matching results meet the user's needs, the user can further explore the information to obtain the required knowledge and answers. If the matching results are not ideal, the user can optimize the search process by adjusting the query request or providing more contextual information.

[0048] In a preferred embodiment, the output module is used to provide a user-friendly access point to ensure that users can effectively input queries, receive feedback, and interact with the system intuitively. The output module includes information presentation, operation guidance, and interactive feedback. The information presentation clearly displays the results of the knowledge base matching to the user, the operation guidance instructs the user to perform correct operations, and the interactive feedback provides real-time feedback on the results and status of the user's operations, allowing the user to understand the interaction situation.

[0049] In this embodiment, the output module can ensure that users can easily access and use the system. At the same time, when users receive feedback information after the system processes it, whether it is a query result, operation confirmation or an error message, this feedback mechanism enables users to understand the response status of the system in a timely manner. Users can quickly obtain the required information through the output module, reducing the interaction time with the system and improving overall efficiency. Information presentation can help users quickly understand the query results, thereby improving user satisfaction with the system. The output module adopts an adaptable interface design to ensure that information can be clearly displayed on different devices and screen sizes. Secondly, the operation guidance sets prompt information or help icons at key operation points, and users can click to obtain more information. Friendly language and clear instructions are used to avoid technical terms to make the guidance easy to understand. Interactive feedback can provide options to undo or redo operations to reduce the impact of erroneous operations.

[0050] In a preferred embodiment, the output module also includes a feedback mechanism, whereby users can provide feedback on search results through the user interface, and the matching module can use the feedback to optimize the query algorithm.

[0051] In this embodiment, the feedback mechanism in the output module is an important part of the user's interaction with the system, and can provide direct feedback on the search results, such as ratings, comments, or tags. This feedback information is collected and analyzed by the system to identify possible problems or deficiencies in the matching algorithm. By providing feedback in this way, the matching module can refine and optimize the query algorithm to improve the accuracy and relevance of future searches. The feedback mechanism not only enhances the system's learning ability, but also improves user engagement and the overall user experience, making the system more intelligent and personalized.

[0052] The present invention also provides a multimodal knowledge matching system terminal, comprising at least one processor;

[0053] and a memory communicatively coupled to the at least one processor;

[0054] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the above-mentioned multimodal knowledge matching system.

[0055] The above is only a preferred embodiment of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications should also be considered as the protection scope of the present invention. The structures, devices and operating methods not specifically described and explained in the present invention shall be implemented according to the conventional means in the art unless otherwise specified and limited.

Claims

1. Multimodal knowledge matching system, characterized by: include: A construction module, wherein the construction module is used to construct a knowledge point library; An image processing module, the image processing module is used to extract features from an input image and output text content, wherein the features include text features and image features; A matching module, which is used to match the text content with the knowledge point library and output relevant knowledge points; The output module is used to return the relevant knowledge points to the user interface. The image processing module includes an image preprocessing module and an image feature extraction module. The image preprocessing module is used to preprocess and enhance the input image, and the image feature extraction module is used to convert the text information in the processed image into retrievable and analyzable text data.

2. The multimodal knowledge matching system according to claim 1, characterized in that: The image preprocessing module includes analyzing the image input by the user and matching the original image format, and extracting edge, texture, shape and color features through features to enhance the contrast and details of the input image and make the image features more obvious. Secondly, it removes noise in the input image to improve image quality and avoid incorrect matching.

3. The multimodal knowledge matching system according to claim 1, characterized in that: The processing of the image feature extraction module includes an image recognition OCR module and an Embedding module, wherein the image recognition OCR module extracts text features from the image using image recognition technology, and the Embedding module is used to semantically combine the extracted text features with image features and transform them through the embedding model.

4. The multimodal knowledge matching system according to claim 3, characterized in that: The image recognition OCR module includes text positioning, character segmentation, character recognition and post-processing. The text positioning is used to identify the text area in the image, and the EAST algorithm is used to detect and segment the text. The character segmentation divides the located text area into single characters or words in preparation for recognition. The Transformer model is used to recognize the segmented characters. The post-processing corrects the recognition and segmentation results of the characters.

5. The multimodal knowledge matching system according to claim 1, characterized in that: The Embedding module is used to convert the text extracted by the image recognition OCR module into a numerical feature vector. The numerical feature vector can capture the semantic information and contextual relationship of the text, and is used to fuse with the image features to form a multimodal feature representation.

6. The multimodal knowledge matching system according to claim 1, characterized in that: The knowledge point library includes knowledge storage, data management and knowledge representation. The knowledge storage serves as the knowledge basis of the system and stores structured or semi-structured knowledge. The structured or semi-structured knowledge can be facts, concepts and entity relationships. The data management is used for indexing, retrieving, updating and managing data. Knowledge can be queried and used. The knowledge representation is used to represent it in a manner suitable for computer processing.

7. The multimodal knowledge matching system according to claim 1, characterized in that: The matching module is a bridge connecting the user interface, the numerical feature vector and the knowledge base, so that users can obtain the required information and improve the performance and user experience of the entire system. The matching module includes using an Elasticsearch database as its back-end component. The Elasticsearch database stores and manages a large amount of data in the knowledge base, wherein the Elasticsearch can index structured data and provide efficient query and matching capabilities. The Elasticsearch database is based on the Embedding vector and uses the cosine similarity method to quickly match relevant information in the knowledge base. According to the matching results, the information most relevant to the numerical feature vector is retrieved from the knowledge base. The relevant information includes entities, concepts, entities with specific attributes, and non-text modal information in the text.

8. The multimodal knowledge matching system according to claim 1, characterized in that: The output module is used to provide a user-friendly access point to ensure that users can effectively input queries, receive feedback, and interact with the system intuitively. The output module includes information presentation, operation guidance, and interactive feedback. The information presentation clearly displays the results of knowledge base matching to the user, the operation guidance guides the user to perform correct operations, and the interactive feedback provides real-time feedback on the results and status of user operations, allowing the user to understand the interaction situation.

9. The multimodal knowledge matching system according to claim 1, characterized in that: The output module also includes a feedback mechanism, where users can provide feedback on search results through the user interface, and the matching module can use this feedback to optimize the query algorithm.

10. A terminal based on a multimodal knowledge matching system, characterized in that: include: at least one processor; and a memory communicatively coupled to the at least one processor; Wherein, the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the multimodal knowledge matching system described in any one of claims 1 to 9.

Images