Multi-modal evolutionary knowledge injection method and system
By automatically collecting and dynamically updating multimodal evolutionary knowledge in clustering networks, a multimodal evolutionary knowledge data set is constructed, and through knowledge injection tuning and knowledge enhancement methods, the problems of insufficient diversity of evolutionary knowledge, high construction costs and poor generalization effects in the existing technology are solved, and efficient knowledge injection and improvement of multimodal intelligent models are achieved.
Patent Information
- Application Number
- CN202510713707.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-30
- Publication Date
- 2025-07-01
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing technology has shortcomings in terms of the diversity and construction cost of evolutionary knowledge, and the generalization effect of the knowledge injection method is poor, resulting in low accuracy of thing recognition and reasoning accuracy in practical applications.
By automatically and continuously collecting the knowledge of news events and entity updates on the network, a multimodal evolutionary knowledge data set is generated, and clustering operations are used to eliminate abnormal data to construct dynamic clustered multimodal evolution knowledge. Then, through the target evolution knowledge training multimodal intelligent model, knowledge injection and tuning and knowledge enhancement merge and injection collaborative tuning are carried out to improve the effect of knowledge injection.
Automatic collection and evaluation of multimodal evolutionary knowledge is realized, the diversity and comprehensiveness of knowledge injection is improved, and the task performance and reasoning accuracy of multimodal intelligent models in target evolutionary knowledge is enhanced.
Smart Images

Figure CN120235247A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of intelligent agent knowledge injection clustering analysis collaborative optimization in computer systems, and more specifically, to a multi-modal evolutionary knowledge injection method and system. Background Art
[0002] Existing intelligent models obtain reasoning capabilities and rich knowledge through large-scale pre-training on data sources, and capture language patterns and factual information during this process. However, the rapid evolution of global information poses significant challenges to general intelligent models in maintaining knowledge consistency, because the continuous update of knowledge and the emergence of new events and entities hinder their ability to maintain accurate and consistent representations, resulting in outdated and inaccurate knowledge. How intelligent models can flexibly inject and extract evolutionary knowledge is a very important technology, which is of great significance in various fields and tasks and has great potential for technical applications. Currently, the implementation methods for the latest evolutionary knowledge injection mainly stay in single-text modality and language general intelligent models. By constructing a text modality knowledge corpus, the parameters of language general intelligent models are updated in a tuned manner to achieve the goal of adapting to the evolutionary knowledge field. The defects of existing technologies include: insufficient diversity of evolutionary knowledge: existing evolutionary knowledge is limited to a single text modality, and the evaluation diversity is insufficient. There is an urgent need for a multi-modal evolutionary knowledge dataset; high cost of constructing evolutionary knowledge: although the method relying on manual collection can generate high-quality evolutionary knowledge data, the construction cost of this method is high, especially for large-scale evolutionary knowledge data, and it is difficult to continuously iterate and collect evolutionary knowledge; poor effect of existing knowledge injection methods: currently, knowledge injection is mainly in the form of tuning, and the parameters of the model are updated to adapt to evolutionary knowledge, but the generalization effect of this method is not good, and problems such as low accuracy of actual application object recognition and low reasoning accuracy remain to be solved. Therefore, it is necessary to propose a multi-modal evolutionary knowledge injection method and system to at least partially solve the problems existing in the prior art. Summary of the Invention
[0003] A series of simplified concepts are introduced in the Summary of the Invention section, which will be further elaborated in detail in the Detailed Description section. The Summary of the Invention section of the present invention does not mean to attempt to define the key features and essential technical features of the claimed technical solution, nor does it mean to attempt to determine the protection scope of the claimed technical solution.
[0004] To at least partially solve the above problems, the present invention provides a multi-modal evolutionary knowledge injection method, including: S10, automatically and continuously collects network dynamically updated news events and entity knowledge, obtains basic dynamic multimodal evolutionary knowledge, generates evolutionary knowledge text description summary data set, and uses clustering operation to eliminate abnormal image data to obtain dynamic clustering multimodal evolutionary knowledge; S20, constructing a multimodal evolutionary knowledge dataset based on dynamic clustering multimodal evolutionary knowledge, conducting analysis and evaluation of the multimodal evolutionary knowledge dataset, and evaluating the effect of multimodal knowledge injection; S30, train the multimodal intelligent model through target evolution knowledge, verify the improvement of the multimodal intelligent model's task performance and its ability to learn target evolution knowledge, and build a knowledge injection and tuning multimodal intelligent model; S40, merge the data after knowledge enhancement with the data before knowledge enhancement, and then perform knowledge enhancement merging and injection collaborative tuning on the knowledge injection tuning multimodal intelligent model, collaboratively enhance the knowledge injection tuning multimodal intelligent model, and perform network dynamic data knowledge enhancement collaborative precise reasoning.
[0005] Preferably, S10 includes: S101, continuously collect basic dynamic multimodal knowledge text and image data of network dynamically updated news events and thing entity knowledge, and obtain basic dynamic multimodal evolution knowledge; S102, setting a completeness summary object-instruction set, generating an evolutionary knowledge text description summary data set based on basic dynamic multimodal evolutionary knowledge, extracting the corresponding question of each summary, generating the corresponding answer and the corresponding question description object of each summary, and downloading the corresponding question description object image data; S103, using the visual model to extract image features of the image data of the object corresponding to the problem description and clustering, using clustering operation to eliminate abnormal image data, editing multiple examples for multiple fine-grained types ranked top by quantity, and obtaining dynamic clustering multimodal evolution knowledge.
[0006] Preferably, S20 includes: S201, based on dynamic clustering of multimodal evolutionary knowledge, continuously and iteratively collect multimodal evolutionary knowledge and construct a multimodal evolutionary knowledge dataset; S202, based on the multimodal evolutionary knowledge data set, each piece of evolutionary knowledge is divided into injection data and evaluation data; the injection data includes: heuristic questions, knowledge summaries and knowledge corresponding images; the evaluation data includes: evaluation questions, answers, evaluation images; based on the injection data and the evaluation data, the multimodal evolutionary knowledge data set is analyzed and evaluated to evaluate the multimodal knowledge injection effect.
[0007] Preferably, S30 includes: S301, based on the multimodal evolutionary knowledge dataset, training a multimodal intelligent model through target evolutionary knowledge; S302. Verify the improvement degree of the performance of the multi-modal intelligent model and its ability to learn target evolutionary knowledge, perform knowledge injection optimization on the multi-modal intelligent model, and construct a knowledge injection optimized multi-modal intelligent model.
[0008] Preferably, S40 includes: S401. Develop a completeness-enhanced object-instruction set, merge the knowledge-enhanced data with the data before knowledge enhancement, and obtain knowledge-enhanced randomly mixed data; S402. Use the knowledge-enhanced randomly mixed data to perform knowledge enhancement combined injection collaborative optimization on the knowledge injection optimized multi-modal intelligent model, collaboratively enhance the knowledge injection optimized multi-modal intelligent model, and perform network dynamic data knowledge enhancement collaborative precise reasoning.
[0009] The present invention provides a multi-modal evolutionary knowledge injection system, including: A dynamic evolutionary knowledge clustering subsystem, which automatically and continuously collects network dynamic updated news events and thing entity knowledge, obtains basic dynamic multi-modal evolutionary knowledge, generates an evolutionary knowledge text description summary dataset, and uses clustering operations to eliminate abnormal image data to obtain dynamically clustered multi-modal evolutionary knowledge; A multi-modal evolutionary knowledge data evaluation subsystem, which constructs a multi-modal evolutionary knowledge dataset based on the dynamically clustered multi-modal evolutionary knowledge, performs analysis and evaluation of the multi-modal evolutionary knowledge dataset, and evaluates the multi-modal knowledge injection effect; An intelligent model task verification subsystem, which trains a multi-modal intelligent model with target evolutionary knowledge, verifies the improvement degree of the performance of the multi-modal intelligent model and its ability to learn target evolutionary knowledge, and constructs a knowledge injection optimized multi-modal intelligent model; A knowledge enhancement hybrid collaboration subsystem, which merges the knowledge-enhanced data with the data before knowledge enhancement, and then performs knowledge enhancement combined injection collaborative optimization on the knowledge injection optimized multi-modal intelligent model, collaboratively enhances the knowledge injection optimized multi-modal intelligent model, and performs network dynamic data knowledge enhancement collaborative precise reasoning.
[0010] Preferably, the dynamic evolutionary knowledge clustering subsystem includes: A network dynamic knowledge collection subsystem, which continuously collects basic dynamic multi-modal knowledge texts and image data of network dynamic updated news events and thing entity knowledge, and obtains basic dynamic multi-modal evolutionary knowledge; A summary object data download subsystem, which sets a completeness summary object-instruction set, generates an evolutionary knowledge text description summary dataset based on the basic dynamic multi-modal evolutionary knowledge, extracts the corresponding questions for each summary, generates the corresponding answers and corresponding question description objects for each summary, and downloads the image data of the corresponding question description objects; The anomaly elimination clustering editing subsystem uses the visual model to extract the image features of the image data of the object corresponding to the problem description and clusters it, and adopts clustering operation to eliminate abnormal image data. It edits multiple examples for multiple fine-grained types ranked by quantity to obtain dynamic clustering multimodal evolution knowledge.
[0011] Preferably, the multimodal evolutionary knowledge data evaluation subsystem includes: The evolutionary knowledge dataset construction subsystem dynamically clusters multimodal evolutionary knowledge, continuously and iteratively collects multimodal evolutionary knowledge, and constructs a multimodal evolutionary knowledge dataset; The knowledge injection analysis and evaluation subsystem divides each piece of evolutionary knowledge into injection data and evaluation data based on the multimodal evolutionary knowledge dataset; the injection data includes: heuristic questions, knowledge summaries and knowledge corresponding images; the evaluation data includes: evaluation questions, answers, and evaluation images; based on the injection data and evaluation data, the multimodal evolutionary knowledge dataset is analyzed and evaluated to evaluate the effect of multimodal knowledge injection.
[0012] Preferably, the intelligent model task verification subsystem includes: Intelligent model knowledge training subsystem, based on the multimodal evolution knowledge dataset, trains the multimodal intelligent model through target evolution knowledge; The task verification injection tuning subsystem verifies the improvement of the multimodal intelligent model’s task performance and its ability to learn target evolutionary knowledge, performs knowledge injection tuning on the multimodal intelligent model, and constructs a knowledge injection tuned multimodal intelligent model.
[0013] Preferably, the knowledge-enhanced hybrid collaborative subsystem includes: The knowledge-enhanced random mixing subsystem formulates a completeness-enhanced object-instruction set, merges the data after knowledge enhancement with the data before knowledge enhancement, and obtains the knowledge-enhanced random mixing data; Enhance the hybrid collaborative improvement subsystem to enhance random mixed data with knowledge, and then perform knowledge enhancement merge injection collaborative tuning on the knowledge injection tuning multimodal intelligent model to collaboratively enhance the knowledge injection tuning multimodal intelligent model and perform knowledge enhancement collaborative precise reasoning on network dynamic data.
[0014] Compared with the prior art, the present invention has at least the following beneficial effects: A multi-modal evolutionary knowledge injection method and system of the present invention automatically and continuously collects network dynamic updated news events and thing entity knowledge to obtain basic dynamic multi-modal evolutionary knowledge, generates an evolutionary knowledge text description summary data set, and uses clustering operations to eliminate abnormal image data to obtain dynamic clustering multi-modal evolutionary knowledge; constructs a multi-modal evolutionary knowledge data set based on the dynamic clustering multi-modal evolutionary knowledge, conducts analysis and evaluation of the multi-modal evolutionary knowledge data set, and evaluates the multi-modal knowledge injection effect; trains a multi-modal intelligent model with target evolutionary knowledge, verifies the improvement degree of the task performance of the multi-modal intelligent model and the learning ability of the target evolutionary knowledge, and constructs a knowledge injection optimized multi-modal intelligent model; merges the knowledge-enhanced data with the data before knowledge enhancement, and then performs knowledge enhancement merged injection collaborative optimization on the knowledge injection optimized multi-modal intelligent model to collaboratively enhance the knowledge injection optimized multi-modal intelligent model for network dynamic data knowledge enhancement collaborative accurate reasoning; can automatically collect multi-modal evolutionary knowledge, continuously collect text and image data of basic multi-modal knowledge of new events or things constantly emerging in the real world; continuously provides data for the field of knowledge injection; constructs a multi-modal evolutionary knowledge data set, and the multi-modal evolutionary knowledge data set evaluates the multi-modal knowledge injection effect, maintaining the diversity and comprehensiveness of the evaluation task; based on the multi-modal evolutionary knowledge data set, a knowledge injection optimization method for a multi-modal intelligent model is proposed, applying the multi-modal intelligent model to target evolutionary knowledge to improve the task performance of the multi-modal intelligent model on the target evolutionary knowledge; a knowledge enhancement method is proposed, using the knowledge-enhanced evolutionary knowledge in the optimization stage, which can further improve the performance of the multi-modal intelligent model on the target knowledge; the present invention performs knowledge enhancement merged injection collaborative optimization on the knowledge injection optimized multi-modal intelligent model, and the collaborative enhancement of the knowledge injection optimized multi-modal intelligent model includes: based on the multi-modal evolutionary knowledge text data, using a language general intelligent model to rewrite the text data of the multi-modal evolutionary knowledge to obtain paraphrased data, completing the knowledge enhancement operation, constructing a knowledge enhancement data set, using the knowledge enhancement data set to inject and optimize the multi-modal intelligent model, and performing knowledge enhancement optimization of the multi-modal intelligent model; the present invention automatically constructs evolutionary knowledge based on Internet data sources, obtains text and image data of multiple multi-modal evolutionary knowledge from Internet data sources; uses a language general intelligent model to generate a summary of the text data of the multi-modal evolutionary knowledge; uses a language general intelligent model to generate evaluation questions, answers and core description objects according to the summary data; downloads evaluation images from the Internet according to the core description objects; forms a multi-modal evolutionary knowledge data set; can effectively extract the latest basic knowledge from Internet data sources and automatically convert the basic knowledge into data available for general intelligent model optimization, greatly reducing the need for manual collection;For the tuning of knowledge injection into multi-modal intelligent models, it can effectively inject target evolutionary knowledge into multi-modal intelligent models, and propose a knowledge enhancement method in the tuning stage to further improve the learning ability of multi-modal intelligent models for target evolutionary knowledge; the automatic evolutionary knowledge construction method can extract knowledge based on Internet data sources and convert it into diverse and comprehensive multi-modal evolutionary knowledge data; the knowledge injection tuning method for multi-modal intelligent models, combined with the proposed knowledge enhancement method, significantly improves the performance of multi-modal intelligent models on target evolutionary knowledge; the accuracy of news event reasoning and the precision of thing entity recognition are significantly improved; the present invention has important technical significance and remarkable effects.
[0015] A multi-modal evolutionary knowledge injection method and system according to the present invention, other advantages, objectives and features of the present invention will be partially reflected by the following description, and partially will also be understood by those skilled in the art through the research and practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] The drawings are used to provide a further understanding of the present invention, and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation to the present invention. In the drawings: Figure 1 It is a diagram of an embodiment of the basis of the architecture of a multi-modal evolutionary knowledge injection system according to the present invention.
[0017] Figure 2 It is an example diagram of the inference of the multi-modal intelligent model after enhancement of a multi-modal evolutionary knowledge injection system according to the present invention.
[0018] Figure 3 It is an example diagram of a multi-modal evolutionary knowledge injection method according to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0019] The following further describes the present invention in detail with reference to the drawings and embodiments, so that those skilled in the art can implement it according to the description of the specification; as shown in the figure, the present invention provides a multi-modal evolutionary knowledge injection method, including: S10, automatically and continuously collect network dynamic updated news events and thing entity knowledge, obtain basic dynamic multi-modal evolutionary knowledge, generate an evolutionary knowledge text description summary data set, and use clustering operation to eliminate abnormal image data to obtain dynamic clustering multi-modal evolutionary knowledge; S20, according to the dynamic clustering multi-modal evolutionary knowledge, construct a multi-modal evolutionary knowledge data set, conduct analysis and evaluation of the multi-modal evolutionary knowledge data set, and evaluate the effect of multi-modal knowledge injection; S30. Train the multi-modal intelligent model with the target evolutionary knowledge, verify the improvement degree of the task performance of the multi-modal intelligent model and its learning ability of the target evolutionary knowledge, and construct a knowledge-injected and optimized multi-modal intelligent model; S40. Merge the knowledge-enhanced data with the data before knowledge enhancement, and then perform knowledge-enhanced merged injection collaborative optimization on the knowledge-injected and optimized multi-modal intelligent model to collaboratively enhance the knowledge-injected and optimized multi-modal intelligent model for network dynamic data knowledge-enhanced collaborative precise reasoning.
[0020] The principle and effects of the above technical solution are as follows: The present invention provides a multi-modal evolutionary knowledge injection method, including: automatically and continuously collecting network dynamic updated news events and thing entity knowledge, obtaining basic dynamic multi-modal evolutionary knowledge, generating an evolutionary knowledge text description summary data set, and using clustering operations to eliminate abnormal image data to obtain dynamic clustering multi-modal evolutionary knowledge; constructing a multi-modal evolutionary knowledge data set based on the dynamic clustering multi-modal evolutionary knowledge, analyzing and evaluating the multi-modal evolutionary knowledge data set, and evaluating the effect of multi-modal knowledge injection; training a multi-modal intelligent model with target evolutionary knowledge, verifying the improvement degree of the task performance of the multi-modal intelligent model and its learning ability for the target evolutionary knowledge, and constructing a knowledge injection optimized multi-modal intelligent model; merging the knowledge-enhanced data with the data before knowledge enhancement, and then performing knowledge enhancement merging injection collaborative optimization on the knowledge injection optimized multi-modal intelligent model to collaboratively enhance the knowledge injection optimized multi-modal intelligent model and perform network dynamic data knowledge enhancement collaborative precise reasoning; it can automatically collect multi-modal evolutionary knowledge and continuously collect text and image data of basic multi-modal knowledge of new events or things emerging in the real world; continuously provide data for the field of knowledge injection; construct a multi-modal evolutionary knowledge data set, and the multi-modal evolutionary knowledge data set evaluates the effect of multi-modal knowledge injection, maintaining the diversity and comprehensiveness of the evaluation task; based on the multi-modal evolutionary knowledge data set, a knowledge injection optimization method for a multi-modal intelligent model is proposed, applying the multi-modal intelligent model to target evolutionary knowledge to improve the task performance of the multi-modal intelligent model on the target evolutionary knowledge; a knowledge enhancement method is proposed, using the knowledge-enhanced evolutionary knowledge in the optimization stage, which can further improve the performance of the multi-modal intelligent model on the target knowledge; the present invention performs knowledge enhancement merging injection collaborative optimization on the knowledge injection optimized multi-modal intelligent model, and the collaborative enhancement of the knowledge injection optimized multi-modal intelligent model includes: based on the multi-modal evolutionary knowledge text data, using a language general intelligent model to rewrite the text data of the multi-modal evolutionary knowledge to obtain paraphrased data, completing the knowledge enhancement operation, constructing a knowledge enhancement data set, and using the knowledge enhancement data set to inject and optimize the multi-modal intelligent model to perform knowledge enhancement optimization of the multi-modal intelligent model; the present invention automatically constructs evolutionary knowledge based on Internet data sources, obtains text and image data of multiple multi-modal evolutionary knowledge from Internet data sources; uses a language general intelligent model to generate a summary of the text data of the multi-modal evolutionary knowledge; uses a language general intelligent model to generate evaluation questions, answers, and core description objects according to the summary data; downloads evaluation images from the Internet according to the core description objects; forms a multi-modal evolutionary knowledge data set; it can effectively extract the latest basic knowledge from Internet data sources and automatically convert the basic knowledge into data available for optimizing general intelligent models, greatly reducing the need for manual collection;Aiming at the knowledge injection tuning of multimodal intelligent models, the target evolution knowledge can be effectively injected into the multimodal intelligent models, and a knowledge enhancement method in the tuning stage is proposed to further improve the learning ability of the multimodal intelligent models for the target evolution knowledge; the automatic evolution knowledge construction method can extract knowledge based on Internet data sources and convert it into diversified and comprehensive multimodal evolution knowledge data; the knowledge injection tuning method of the multimodal intelligent model, combined with the proposed knowledge enhancement method, significantly improves the performance of the multimodal intelligent model on the target evolution knowledge; the accuracy of news event reasoning and the accuracy of entity recognition are significantly improved; the present invention has important technical significance and significant effects. ;
[0021] In one embodiment, S10 includes: S101, continuously collect basic dynamic multimodal knowledge text and image data of network dynamically updated news events and thing entity knowledge, and obtain basic dynamic multimodal evolution knowledge; S102, setting a completeness summary object-instruction set, generating an evolutionary knowledge text description summary data set based on basic dynamic multimodal evolutionary knowledge, extracting the corresponding question of each summary, generating the corresponding answer and the corresponding question description object of each summary, and downloading the corresponding question description object image data; S103, using the visual model to extract image features of the image data of the object corresponding to the problem description and clustering, using clustering operation to eliminate abnormal image data, editing multiple examples for multiple fine-grained types ranked top by quantity, and obtaining dynamic clustering multimodal evolution knowledge.
[0022] The principle and effect of the above technical solution are as follows: continuously collect basic dynamic multimodal knowledge text and image data of news events and entity knowledge of things that are dynamically updated on the network, and obtain basic dynamic multimodal evolutionary knowledge; set a completeness summary object-instruction set, generate an evolutionary knowledge text description summary data set according to the basic dynamic multimodal evolutionary knowledge, extract the corresponding question of each summary, generate the corresponding answer and the corresponding question description object of each summary, and download the corresponding question description object image data; use a visual model to extract the image features of the corresponding question description object image data and cluster them, use clustering operations to eliminate abnormal image data, edit multiple examples for multiple fine-grained types ranked by quantity, and obtain dynamic clustering multimodal evolutionary knowledge; Continuously collect the basic multimodal knowledge text and image data of dynamically updated news events and thing entity knowledge in the real world, and obtain the basic multimodal evolutionary knowledge, including: using news event knowledge and entity knowledge as the basic multimodal evolutionary knowledge; according to the news knowledge disclosure protocol, extract the calibration sequence of news network resources; and according to the calibration sequence of news network resources, extract the news data within a set time period, and the news data within the set time period includes: news type, news title, news text description, news corresponding image; for entity knowledge, obtain data using the corresponding versions at different time nodes, and by comparing different versions, screen out the newly emerged thing entity data, and the newly emerged thing entity data includes: entity type, entity name, entity text description, entity corresponding image; improve the reliability of the website in collecting and screening basic data; Set a complete summary object - instruction set to generate a summary of the evolutionary knowledge text description, including: for the news data text description or entity data text description or long text content, set the evolutionary knowledge summary rule and the set word count range, call the language model to summarize the news data text description or entity data text description or long text content, and generate a summary content within the set word count range; during the generation of the summary, set a complete summary object - instruction set and provide comprehensive and reliable examples to generate a summary of the evolutionary knowledge text description; Extract the corresponding question for each summary, generate the corresponding answer and the corresponding question description object for each summary, and download the image data of the corresponding question description object, including: for each summary in the evolutionary knowledge text description summary dataset, extract the corresponding question for each summary, generate the corresponding answer for each summary, and generate the corresponding question description object for each summary; according to the corresponding question description object, download the image data of the corresponding question description object; the image data of the corresponding question description object contains abnormal image data, use the visual model to extract the image features, and use the clustering algorithm for clustering to find the abnormal image data and eliminate the abnormal image data; Edit multiple examples for multiple fine - grained types sorted by quantity respectively, including: The first example: The title of the first example: Player A suffered a crushing defeat in the second round of the M Country Open. The description of the first example: World - ranked No. 3 Player A unexpectedly lost to Player B in the second round of the M Country Tennis Open held at Court 1, marking the end of his 15 - game unbeaten record in the tournament season; this 74th - ranked player from Country N won in straight sets, which is his first victory over a top - 10 opponent in the tournament season; the match was held in City X on a certain date in a certain month and year; Player B will face Player C in the third round; The output of the first example: Multimodal question: Who defeated the player in the image in the second round of the M Country Tennis Open? The answer to the first example: Player B The first example entity object: Player A The first example hypernym: Athlete; The second example: The second example title: Team F overcame the boos of the home crowd and narrowly defeated Team G to enter the playoffs; The second example description: Team F overcame early struggles and the boos of the fans, narrowly defeating Team G 105-104 in a semi-final to secure the 7th seed in the Eastern Conference; Player D recovered from knee surgery and played a crucial role in the fourth quarter with the help of Player E's 20-point bench performance; Team F will now face Team H in the playoffs; Meanwhile, Team G faces challenges as Player J may miss due to a knee injury as they prepare for an elimination game against Team K for the last playoff spot; The second example output: The second example multimodal question: After defeating Team G, which team will Team F face in the playoffs in the image? The second example answer: Team H The second example entity object: Team F The second example hypernym: Team; It is able to identify abnormal image data, eliminate the abnormal image data, and significantly improve the quality of image data.
[0023] In one embodiment, S20 includes: S201, continuously iteratively collect multimodal evolutionary knowledge according to dynamic clustering multimodal evolutionary knowledge, and construct a multimodal evolutionary knowledge dataset; S202, divide each piece of evolutionary knowledge into injection data and evaluation data according to the multimodal evolutionary knowledge dataset; the injection data includes: heuristic questions, knowledge summaries, and knowledge-corresponding images; the evaluation data includes: evaluation questions, answers, and evaluation images; according to the injection data and the evaluation data, analyze and evaluate the multimodal evolutionary knowledge dataset to evaluate the multimodal knowledge injection effect.
[0024] The principle and effect of the above technical solution are: continuously iteratively collect multimodal evolutionary knowledge according to dynamic clustering multimodal evolutionary knowledge, and construct a multimodal evolutionary knowledge dataset; divide each piece of evolutionary knowledge into injection data and evaluation data according to the multimodal evolutionary knowledge dataset; the injection data includes: heuristic questions, knowledge summaries, and knowledge-corresponding images; the evaluation data includes: evaluation questions, answers, and evaluation images; according to the injection data and the evaluation data, analyze and evaluate the multimodal evolutionary knowledge dataset to evaluate the multimodal knowledge injection effect; According to the dynamic clustering multi-modal evolutionary knowledge, continuously iterate and collect multi-modal evolutionary knowledge, and construct a multi-modal evolutionary knowledge dataset, including: according to the dynamic clustering multi-modal evolutionary knowledge, collect 9422 multi-modal evolutionary knowledge data, covering 159 fine-grained categories, continuously iterate and collect multi-modal evolutionary knowledge, and construct a multi-modal evolutionary knowledge dataset; significantly improve the diversity and comprehensiveness of the evaluation tasks.
[0025] In one embodiment, S30 includes: S301, based on the multi-modal evolutionary knowledge dataset, train a multi-modal intelligent model through the target evolutionary knowledge; S302, verify the improvement degree of the task performance of the multi-modal intelligent model and its learning ability of the target evolutionary knowledge, perform knowledge injection tuning on the multi-modal intelligent model, and construct a knowledge injection tuned multi-modal intelligent model.
[0026] The principle and effect of the above technical solution are: based on the multi-modal evolutionary knowledge dataset, train a multi-modal intelligent model through the target evolutionary knowledge; verify the improvement degree of the task performance of the multi-modal intelligent model and its learning ability of the target evolutionary knowledge, perform knowledge injection tuning on the multi-modal intelligent model, and construct a knowledge injection tuned multi-modal intelligent model.
[0027] In one embodiment, S40 includes: S401, formulate a completeness-enhanced object-instruction set, merge the knowledge-enhanced data and the data before knowledge enhancement, and obtain knowledge-enhanced random mixed data; S402, use the knowledge-enhanced random mixed data to perform knowledge-enhanced merged injection collaborative tuning on the knowledge injection tuned multi-modal intelligent model, collaboratively enhance the knowledge injection tuned multi-modal intelligent model, and perform network dynamic data knowledge-enhanced collaborative precise reasoning.
[0028] The principle and effect of the above technical solution are: formulate a completeness-enhanced object-instruction set, merge the knowledge-enhanced data and the data before knowledge enhancement, and obtain knowledge-enhanced random mixed data; use the knowledge-enhanced random mixed data to perform knowledge-enhanced merged injection collaborative tuning on the knowledge injection tuned multi-modal intelligent model, collaboratively enhance the knowledge injection tuned multi-modal intelligent model, and perform network dynamic data knowledge-enhanced collaborative precise reasoning; formulating a completeness-enhanced object-instruction set, merging the knowledge-enhanced data and the data before knowledge enhancement, and obtaining knowledge-enhanced random mixed data includes: formulating a completeness-enhanced object-instruction set, using a language intelligent model to rewrite the text data of the evolutionary knowledge to obtain multiple sets of paraphrased data, forming the knowledge-enhanced data; merging the knowledge-enhanced data and the data before knowledge enhancement, and randomly shuffling the order to obtain the knowledge-enhanced random mixed data.
[0029] The present invention provides a multi-modal evolutionary knowledge injection system, including: Dynamic Evolutionary Knowledge Clustering Subsystem, which automatically and continuously collects network dynamic updated news events and thing entity knowledge, obtains basic dynamic multi-modal evolutionary knowledge, generates an evolutionary knowledge text description summary dataset, and uses clustering operations to eliminate abnormal image data to obtain dynamic clustering multi-modal evolutionary knowledge; Multi-modal Evolutionary Knowledge Data Evaluation Subsystem, which constructs a multi-modal evolutionary knowledge dataset based on the dynamic clustering multi-modal evolutionary knowledge, conducts analysis and evaluation of the multi-modal evolutionary knowledge dataset, and evaluates the injection effect of multi-modal knowledge; Intelligent Model Task Verification Subsystem, which trains a multi-modal intelligent model with target evolutionary knowledge, verifies the improvement degree of the multi-modal intelligent model task performance and the learning ability of the target evolutionary knowledge, and constructs a knowledge injection optimized multi-modal intelligent model; Knowledge Enhancement Hybrid Collaboration Subsystem, which merges the data after knowledge enhancement with the data before knowledge enhancement, and then conducts knowledge enhancement merged injection collaborative optimization on the knowledge injection optimized multi-modal intelligent model, collaboratively enhances the knowledge injection optimized multi-modal intelligent model, and conducts network dynamic data knowledge enhancement collaborative precise reasoning.
[0030] The principles and effects of the above technical solutions are as follows: The present invention provides a multi-modal evolutionary knowledge injection system, including: a dynamic evolutionary knowledge clustering subsystem that automatically and continuously collects network dynamic updated news events and thing entity knowledge, obtains basic dynamic multi-modal evolutionary knowledge, generates an evolutionary knowledge text description summary dataset, and uses clustering operations to eliminate abnormal image data to obtain dynamically clustered multi-modal evolutionary knowledge; a multi-modal evolutionary knowledge data evaluation subsystem that constructs a multi-modal evolutionary knowledge dataset based on the dynamically clustered multi-modal evolutionary knowledge, conducts analysis and evaluation of the multi-modal evolutionary knowledge dataset, and evaluates the multi-modal knowledge injection effect; an intelligent model task verification subsystem that trains a multi-modal intelligent model through target evolutionary knowledge, verifies the improvement degree of the multi-modal intelligent model task performance and the learning ability of the target evolutionary knowledge, and constructs a knowledge injection optimized multi-modal intelligent model; a knowledge enhancement hybrid collaboration subsystem that merges the knowledge-enhanced data with the data before knowledge enhancement, and then performs knowledge enhancement merging injection collaborative optimization on the knowledge injection optimized multi-modal intelligent model to collaboratively enhance the knowledge injection optimized multi-modal intelligent model and conduct network dynamic data knowledge enhancement collaborative precise reasoning; it can automatically collect multi-modal evolutionary knowledge, continuously collect text and image data of basic multi-modal knowledge of new events or things constantly emerging in the real world; continuously provide data for the field of knowledge injection; construct a multi-modal evolutionary knowledge dataset, and the multi-modal evolutionary knowledge dataset evaluates the multi-modal knowledge injection effect, maintaining the diversity and comprehensiveness of the evaluation tasks; based on the multi-modal evolutionary knowledge dataset, a knowledge injection optimization method for a multi-modal intelligent model is proposed, applying the multi-modal intelligent model to target evolutionary knowledge to improve the task performance of the multi-modal intelligent model on the target evolutionary knowledge; a knowledge enhancement method is proposed, using the knowledge-enhanced evolutionary knowledge in the optimization stage, which can further improve the performance of the multi-modal intelligent model on the target knowledge; the present invention performs knowledge enhancement merging injection collaborative optimization on the knowledge injection optimized multi-modal intelligent model, and the collaborative enhancement of the knowledge injection optimized multi-modal intelligent model includes: based on the multi-modal evolutionary knowledge text data, using a language general intelligent model to rewrite the text data of the multi-modal evolutionary knowledge to obtain paraphrased data, complete the knowledge enhancement operation, construct a knowledge enhancement dataset, and use the knowledge enhancement dataset to inject and optimize the multi-modal intelligent model to conduct knowledge enhancement optimization of the multi-modal intelligent model; the present invention automatically constructs evolutionary knowledge based on Internet data sources, obtains text and image data of multiple multi-modal evolutionary knowledge from Internet data sources; uses a language general intelligent model to generate a summary of the text data of the multi-modal evolutionary knowledge; uses a language general intelligent model to generate evaluation questions, answers, and core description objects according to the summary data; downloads evaluation images from the Internet according to the core description objects; forms a multi-modal evolutionary knowledge dataset; it can effectively extract the latest basic knowledge from Internet data sources and automatically convert the basic knowledge into data that can be used for optimizing general intelligent models, greatly reducing the need for manual collection.Aiming at the knowledge injection tuning of multimodal intelligent models, the target evolution knowledge can be effectively injected into the multimodal intelligent models, and a knowledge enhancement method in the tuning stage is proposed to further improve the learning ability of the multimodal intelligent models for the target evolution knowledge; the automatic evolution knowledge construction method can extract knowledge based on Internet data sources and convert it into diversified and comprehensive multimodal evolution knowledge data; the knowledge injection tuning method of the multimodal intelligent model, combined with the proposed knowledge enhancement method, significantly improves the performance of the multimodal intelligent model on the target evolution knowledge; the accuracy of news event reasoning and the accuracy of entity recognition are significantly improved; the present invention has important technical significance and significant effects. ;
[0031] In one embodiment, the dynamic evolution knowledge clustering subsystem includes: The network dynamic knowledge collection subsystem continuously collects basic dynamic multimodal knowledge text and image data of network dynamic update news events and thing entity knowledge, and obtains basic dynamic multimodal evolution knowledge; The summary object data download subsystem sets up a complete summary object-instruction set, generates an evolutionary knowledge text description summary data set based on basic dynamic multimodal evolutionary knowledge, extracts the corresponding question of each summary, generates the corresponding answer and the corresponding question description object of each summary, and downloads the corresponding question description object image data; The anomaly elimination clustering editing subsystem uses the visual model to extract the image features of the image data of the object corresponding to the problem description and clusters it, and adopts clustering operation to eliminate abnormal image data. It edits multiple examples for multiple fine-grained types ranked by quantity to obtain dynamic clustering multimodal evolution knowledge.
[0032] The principle and effect of the above technical solution are as follows: a dynamic evolution knowledge clustering subsystem, including: a network dynamic knowledge collection subsystem, which continuously collects basic dynamic multimodal knowledge text and image data of network dynamically updated news events and entity knowledge of things, and obtains basic dynamic multimodal evolution knowledge; a summary object data download subsystem, which sets a complete summary object-instruction set, generates an evolution knowledge text description summary data set according to the basic dynamic multimodal evolution knowledge, extracts the corresponding question of each summary, generates the corresponding answer and the corresponding question description object of each summary, and downloads the corresponding question description object image data; an abnormal elimination clustering editing subsystem, which uses a visual model to extract the image features of the corresponding question description object image data and clusters it, uses clustering operations to eliminate abnormal image data, and edits multiple examples for a variety of fine-grained types ranked in the top order by quantity, and obtains dynamic clustering multimodal evolution knowledge; Continuously collect the basic multimodal knowledge text and image data of dynamically updated news events and thing entity knowledge in the real world, and obtain the basic multimodal evolutionary knowledge, including: using news event knowledge and entity knowledge as the basic multimodal evolutionary knowledge; according to the news knowledge disclosure protocol, extract the calibration sequence of news network resources; and according to the calibration sequence of news network resources, extract the news data within a set time period, where the news data within the set time period includes: news type, news title, news text description, news corresponding image; for entity knowledge, obtain data using the corresponding versions at different time nodes, and by comparing different versions, screen out the newly emerged thing entity data, where the newly emerged thing entity data includes: entity type, entity name, entity text description, entity corresponding image; improve the reliability of the website in collecting and screening basic data; Set a complete summary object - instruction set to generate a summary of the evolutionary knowledge text description, including: for the news data text description or entity data text description or long text content, set the evolutionary knowledge summary rule and the set word count range, call the language model to summarize the news data text description or entity data text description or long text content, and generate the summary content within the set word count range; during the generation of the summary, set the complete summary object - instruction set and provide comprehensive and reliable examples to generate the summary of the evolutionary knowledge text description; Extract the corresponding question for each summary, generate the corresponding answer and the corresponding question description object for each summary, and download the image data of the corresponding question description object, including: for each summary in the evolutionary knowledge text description summary dataset, extract the corresponding question for each summary, generate the corresponding answer for each summary, and generate the corresponding question description object for each summary; according to the corresponding question description object, download the image data of the corresponding question description object; the image data of the corresponding question description object contains abnormal image data, use the visual model to extract the image features, and use the clustering algorithm for clustering to find the abnormal image data and eliminate the abnormal image data; Edit multiple examples for multiple fine - grained types sorted by quantity respectively, including: The first example: The title of the first example: Player A suffered a crushing defeat in the second round of the M Country Open. The description of the first example: World - ranked No. 3 Player A unexpectedly lost to Player B in the second round of the M Country Tennis Open held at Court 1, marking the end of his 15 - match unbeaten record in the tournament season; this 74th - ranked player from Country N won in straight sets, which is his first victory over a top - 10 opponent in the tournament season; the match was held in City X on a certain date in a certain month of a certain year; Player B will face Player C in the third round; The output of the first example: Multimodal question: Who defeated the player in the image in the second round of the M Country Tennis Open? The answer to the first example: Player B The first example entity object: Player A The first example hypernym: Athlete; The second example: The second example title: Team F overcame the boos of the home crowd and narrowly defeated Team G to advance to the playoffs; The second example description: Team F overcame early struggles and the boos of the fans, narrowly defeating Team G 105-104 in a semi-final to secure the 7th seed in the Eastern Conference; Player D recovered from knee surgery and played a crucial role in the fourth quarter with the help of Player E's 20-point bench performance; Team F will now face Team H in the playoffs; Meanwhile, Team G faces challenges as Player J may be absent due to a knee injury as they prepare for an elimination game against Team K for the last playoff spot; The second example output: The second example multimodal question: After defeating Team G, which team will Team F face in the playoffs in the image? The second example answer: Team H The second example entity object: Team F The second example hypernym: Team; It is able to identify abnormal image data, eliminate the abnormal image data, and significantly improve the quality of image data.
[0033] In one embodiment, the multimodal evolutionary knowledge data evaluation subsystem includes: The evolutionary knowledge dataset construction subsystem continuously iteratively collects multimodal evolutionary knowledge according to dynamic clustering multimodal evolutionary knowledge to construct a multimodal evolutionary knowledge dataset; The knowledge injection analysis and evaluation subsystem divides each piece of evolutionary knowledge into injection data and evaluation data according to the multimodal evolutionary knowledge dataset; the injection data includes: heuristic questions, knowledge summaries, and corresponding images of knowledge; the evaluation data includes: evaluation questions, answers, and evaluation images; according to the injection data and evaluation data, it conducts analysis and evaluation of the multimodal evolutionary knowledge dataset to evaluate the multimodal knowledge injection effect.
[0034] The principle and effect of the above technical solution are: The multimodal evolutionary knowledge data evaluation subsystem includes: the evolutionary knowledge dataset construction subsystem continuously iteratively collects multimodal evolutionary knowledge according to dynamic clustering multimodal evolutionary knowledge to construct a multimodal evolutionary knowledge dataset; the knowledge injection analysis and evaluation subsystem divides each piece of evolutionary knowledge into injection data and evaluation data according to the multimodal evolutionary knowledge dataset; the injection data includes: heuristic questions, knowledge summaries, and corresponding images of knowledge; the evaluation data includes: evaluation questions, answers, and evaluation images; according to the injection data and evaluation data, it conducts analysis and evaluation of the multimodal evolutionary knowledge dataset to evaluate the multimodal knowledge injection effect; According to the dynamic clustering multi-modal evolutionary knowledge, continuously iteratively collect multi-modal evolutionary knowledge, and construct a multi-modal evolutionary knowledge data set, including: According to the dynamic clustering multi-modal evolutionary knowledge, collect 9,422 multi-modal evolutionary knowledge data, covering 159 fine-grained categories, continuously iteratively collect multi-modal evolutionary knowledge, and construct a multi-modal evolutionary knowledge data set; significantly improve the diversity and comprehensiveness of the evaluation task.
[0035] In one embodiment, the intelligent model task verification subsystem includes: The intelligent model knowledge training subsystem, based on the multi-modal evolutionary knowledge data set, trains a multi-modal intelligent model through the target evolutionary knowledge; The task verification injection tuning subsystem verifies the improvement degree of the multi-modal intelligent model task performance and the learning ability of the target evolutionary knowledge, performs knowledge injection tuning on the multi-modal intelligent model, and constructs a knowledge injection tuned multi-modal intelligent model.
[0036] The principle and effect of the above technical solution are: The intelligent model task verification subsystem includes: The intelligent model knowledge training subsystem, based on the multi-modal evolutionary knowledge data set, trains a multi-modal intelligent model through the target evolutionary knowledge; The task verification injection tuning subsystem verifies the improvement degree of the multi-modal intelligent model task performance and the learning ability of the target evolutionary knowledge, performs knowledge injection tuning on the multi-modal intelligent model, and constructs a knowledge injection tuned multi-modal intelligent model.
[0037] In one embodiment, the knowledge enhancement hybrid collaboration subsystem includes: The knowledge enhancement random hybrid subsystem formulates a completeness-enhanced object-instruction set, merges the knowledge-enhanced data with the pre-knowledge-enhanced data, and obtains knowledge-enhanced random hybrid data; The enhanced hybrid collaboration improvement subsystem uses the knowledge-enhanced random hybrid data, and then performs knowledge enhancement combined injection collaborative tuning on the knowledge injection tuned multi-modal intelligent model, collaboratively enhances the knowledge injection tuned multi-modal intelligent model, and performs network dynamic data knowledge enhancement collaborative precise reasoning.
[0038] The principle and effect of the above technical solution are as follows: The knowledge-enhanced hybrid collaboration subsystem includes: a knowledge-enhanced random hybrid subsystem that formulates a completeness-enhanced object-instruction set, combines the knowledge-enhanced data with the pre-knowledge-enhanced data to obtain knowledge-enhanced random hybrid data; an enhanced hybrid collaboration improvement subsystem that uses the knowledge-enhanced random hybrid data to perform knowledge-enhanced merging and injection collaborative optimization on the knowledge-injected and optimized multi-modal intelligent model, collaboratively enhances the knowledge-injected and optimized multi-modal intelligent model, and performs network dynamic data knowledge-enhanced collaborative precise reasoning; formulating a completeness-enhanced object-instruction set, combining the knowledge-enhanced data with the pre-knowledge-enhanced data to obtain knowledge-enhanced random hybrid data includes: formulating a completeness-enhanced object-instruction set, using a language intelligent model to rewrite the text data of the evolutionary knowledge to obtain multiple groups of paraphrased data, forming the knowledge-enhanced data; combining the knowledge-enhanced data with the pre-knowledge-enhanced data and randomly shuffling the order to obtain knowledge-enhanced random hybrid data.
[0039] Although the embodiments of the present invention have been disclosed as above, it is not limited to the applications listed in the specification and embodiments. It can be fully applied to various fields suitable for the present invention. For those familiar with the field, additional modifications can be easily made. Therefore, without departing from the general concept defined by the claims and the equivalent scope, the present invention is not limited to the specific details and the illustrated and described examples here.
Claims
1. A multi-modal evolutionary knowledge injection method, characterized in that, include: S10, automatically and continuously collects network dynamically updated news events and entity knowledge, obtains basic dynamic multimodal evolutionary knowledge, generates evolutionary knowledge text description summary data set, and uses clustering operation to eliminate abnormal image data to obtain dynamic clustering multimodal evolutionary knowledge; S20, constructing a multimodal evolutionary knowledge dataset based on dynamic clustering multimodal evolutionary knowledge, conducting analysis and evaluation of the multimodal evolutionary knowledge dataset, and evaluating the effect of multimodal knowledge injection; S30, train the multimodal intelligent model through target evolution knowledge, verify the improvement of the multimodal intelligent model's task performance and its ability to learn target evolution knowledge, and build a knowledge injection and tuning multimodal intelligent model; S40, merge the data after knowledge enhancement with the data before knowledge enhancement, and then perform knowledge enhancement merging and injection collaborative tuning on the knowledge injection tuning multimodal intelligent model, collaboratively enhance the knowledge injection tuning multimodal intelligent model, and perform network dynamic data knowledge enhancement collaborative precise reasoning.
2. The multimodal evolutionary knowledge injection method according to claim 1, wherein S10 includes: S101, continuously collect basic dynamic multimodal knowledge text and image data of network dynamically updated news events and thing entity knowledge, and obtain basic dynamic multimodal evolution knowledge; S102, setting a completeness summary object-instruction set, generating an evolutionary knowledge text description summary data set based on basic dynamic multimodal evolutionary knowledge, extracting the corresponding question of each summary, generating the corresponding answer and the corresponding question description object of each summary, and downloading the corresponding question description object image data; S103, using the visual model to extract image features of the image data of the object corresponding to the problem description and clustering, using clustering operation to eliminate abnormal image data, editing multiple examples for multiple fine-grained types ranked top by quantity, and obtaining dynamic clustering multimodal evolution knowledge.
3. A multimodal evolutionary knowledge injection method according to claim 1, characterized in that The S20 includes: S201, based on dynamic clustering of multimodal evolutionary knowledge, continuously and iteratively collect multimodal evolutionary knowledge and construct a multimodal evolutionary knowledge dataset; S202, based on the multimodal evolutionary knowledge data set, each piece of evolutionary knowledge is divided into injection data and evaluation data; the injection data includes: heuristic questions, knowledge summaries and knowledge corresponding images; the evaluation data includes: evaluation questions, answers, evaluation images; based on the injection data and the evaluation data, the multimodal evolutionary knowledge data set is analyzed and evaluated to evaluate the multimodal knowledge injection effect.
4. A multimodal evolutionary knowledge injection method according to claim 1, wherein S30 includes: S301, based on the multimodal evolutionary knowledge dataset, training a multimodal intelligent model through target evolutionary knowledge; S302, verify the degree of improvement of the multimodal intelligent model's task performance and its ability to learn target evolutionary knowledge, perform knowledge injection tuning on the multimodal intelligent model, and construct a knowledge injection-tuned multimodal intelligent model.
5. A multimodal evolutionary knowledge injection method according to claim 1, characterized in that, S40 includes: S401, formulate a completeness enhancement object-instruction set, merge the data after knowledge enhancement with the data before knowledge enhancement, and obtain knowledge enhancement random mixed data; S402, randomly mix the data with knowledge enhancement, and then perform knowledge enhancement merge injection collaborative tuning on the knowledge injection tuning multimodal intelligent model, collaboratively enhance the knowledge injection tuning multimodal intelligent model, and perform network dynamic data knowledge enhancement collaborative precise reasoning.
6. A multi-modal evolutionary knowledge injection system, characterized in that, include: Dynamic Evolutionary Knowledge Clustering Sub-system, which automatically and continuously collects network dynamic updated news events and thing entity knowledge, obtains basic dynamic multi-modal evolutionary knowledge, generates an evolutionary knowledge text description summary data set, and uses clustering operations to eliminate abnormal image data to obtain dynamic clustering multi-modal evolutionary knowledge; Multi-modal Evolutionary Knowledge Data Evaluation Sub-system, which constructs a multi-modal evolutionary knowledge data set based on the dynamic clustering multi-modal evolutionary knowledge, conducts analysis and evaluation of the multi-modal evolutionary knowledge data set, and evaluates the multi-modal knowledge injection effect; Intelligent Model Task Verification Sub-system, which trains a multi-modal intelligent model through target evolutionary knowledge, verifies the improvement degree of the multi-modal intelligent model task performance and the learning ability of the target evolutionary knowledge, and constructs a knowledge injection optimized multi-modal intelligent model; Knowledge Enhancement Hybrid Collaboration Sub-system, which merges the data after knowledge enhancement with the data before knowledge enhancement, and then conducts knowledge enhancement merged injection collaborative optimization on the knowledge injection optimized multi-modal intelligent model, collaboratively enhances the knowledge injection optimized multi-modal intelligent model, and conducts network dynamic data knowledge enhancement collaborative precise reasoning.
7. A multimodal evolutionary knowledge injection system according to claim 6, characterized in that Dynamic Evolutionary Knowledge Clustering Sub-system, including: Network Dynamic Knowledge Collection Sub-system, which continuously collects the basic dynamic multi-modal knowledge text and image data of network dynamic updated news events and thing entity knowledge, and obtains basic dynamic multi-modal evolutionary knowledge; Summary Object Data Download Sub-system, which sets up a completeness summary object - instruction set, generates an evolutionary knowledge text description summary data set according to the basic dynamic multi-modal evolutionary knowledge, extracts the corresponding questions for each summary, generates the corresponding answers and corresponding question description objects for each summary, and downloads the image data of the corresponding question description objects; Abnormality Elimination Clustering Editing Sub-system, which uses a visual model to extract the image features of the image data of the corresponding question description objects and conducts clustering, uses clustering operations to eliminate abnormal image data, edits multiple examples for multiple fine-grained types sorted by quantity, and obtains dynamic clustering multi-modal evolutionary knowledge.
8. A multimodal evolutionary knowledge injection system according to claim 6, characterized in that Multi-modal Evolutionary Knowledge Data Evaluation Sub-system, including: Evolutionary Knowledge Data Set Construction Sub-system, which continuously iteratively collects multi-modal evolutionary knowledge according to the dynamic clustering multi-modal evolutionary knowledge, and constructs a multi-modal evolutionary knowledge data set; Knowledge Injection Analysis and Evaluation Sub-system, which divides each evolutionary knowledge into injection data and evaluation data according to the multi-modal evolutionary knowledge data set; the injection data includes: heuristic questions, knowledge summaries, and knowledge corresponding images; the evaluation data includes: evaluation questions, answers, and evaluation images; according to the injection data and evaluation data, conducts analysis and evaluation of the multi-modal evolutionary knowledge data set, and evaluates the multi-modal knowledge injection effect.
9. A multimodal evolutionary knowledge injection system according to claim 6, wherein Intelligent Model Task Verification Sub-system, including: Intelligent Model Knowledge Training Sub-system, which trains a multi-modal intelligent model based on the multi-modal evolutionary knowledge data set through target evolutionary knowledge; Task Verification Injection Optimization Sub-system, which verifies the improvement degree of the multi-modal intelligent model task performance and the learning ability of the target evolutionary knowledge, conducts knowledge injection optimization on the multi-modal intelligent model, and constructs a knowledge injection optimized multi-modal intelligent model.
10. A multimodal evolutionary knowledge injection system according to claim 6, characterized in that, Knowledge Enhancement Hybrid Collaboration Sub-system, including: Knowledge-enhanced random hybrid subsystem, formulating a completeness-enhanced object-instruction set, merging the knowledge-enhanced data with the data before knowledge enhancement to obtain knowledge-enhanced random hybrid data; Enhanced hybrid collaborative improvement subsystem, using the knowledge-enhanced random hybrid data, and then performing knowledge-enhanced merging injection collaborative tuning on the knowledge-injected and optimized multi-modal intelligent model, collaboratively enhancing the knowledge-injected and optimized multi-modal intelligent model, and performing network dynamic data knowledge-enhanced collaborative precise reasoning.
Citation Information
Patent Citations
Forensic medicine examination automatic identification system and method based on deep learning
CN110033445A
Exogenous knowledge-based news description generation method and device
CN117648932A
Question processing method and device and electronic equipment
CN119311802A
Cited By
Power edge monitoring method and system based on multi-modal collaboration and environmental knowledge enhancement
CN122313405A
Multimodal synergistic and environment knowledge augmented power edge monitoring method and system
CN122313405B
