Method and system for rapidly interpreting mass image data based on priori knowledge

Abstract

The invention discloses a prior knowledge-based large-batch image data rapid interpretation method and system, and belongs to the field of computer vision and generative artificial intelligence, and the method comprises the steps: S1, constructing a task scene data set, training a CNN-DE algorithm through the data set, and obtaining a model for detecting a target type and a target position; s2, constructing an image-text data set by using the training set in the step S1, and training a fine-tuning LVLM model by using the image-text data set; s3, based on the data set in the step S1, combining the image and using a text to label the image to construct a multi-modal aligned data set, and using the data set to finely adjust CLIP to obtain an auxiliary interpretation model based on content retrieval; and S4, integrating the three models into a system, adjusting input and output of the system, constructing a human-computer interaction interface, and docking a database. The invention provides a multi-model and multi-mode collaborative image automatic target detection scheme based on prior knowledge, and has industrial application value and prospect.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and generative artificial intelligence, and more specifically, to a method and system for rapid interpretation of large batches of image data based on prior knowledge. Background Technology

[0002] With the widespread adoption of smart devices, the hardware costs for capturing, transmitting, and storing image data are decreasing, resulting in massive amounts of image data. Image data is two-dimensional, and compared to one-dimensional text, images can carry a much larger amount of information. Therefore, interpreting image data and identifying target objects within it is more complex and inefficient. Given the contradiction that the efficiency of image interpretation cannot keep up with the speed of image generation, technologies using artificial intelligence algorithms for machine interpretation have emerged. Currently, AI algorithms are applied to target detection tasks, improving human efficiency. However, there is a problem: fast algorithms have low accuracy, while high-accuracy algorithms are slow. Therefore, a better network architecture is needed to balance the trade-off between efficiency and accuracy.

[0003] Artificial intelligence-assisted image interpretation technology has a profound real-world background and broad application prospects, specifically including the following aspects: 1) In the field of screening harmful online information: Using artificial intelligence algorithms to intelligently identify online images containing harmful information can effectively improve the work efficiency of online "pornography screeners" and alleviate their work pressure; 2) Remote Sensing Monitoring: Currently, remote sensing satellites are widely used to monitor land resources and prevent natural disasters. Remote sensing satellites generate terabytes of image data daily, and timely data processing to extract useful information requires the assistance of artificial intelligence. 3) Transportation Sector: Currently, a well-developed highway network has been established, with over 400 million vehicles. Road surveillance cameras capture a large amount of image data daily for evidence collection. Relying entirely on manual interpretation by traffic police would incur enormous labor costs. Using artificial intelligence to assist in interpretation, followed by manual confirmation, would effectively reduce the workload of traffic police. 4) Industrial Robotics: Current industrial robots mainly use path planning robotic arms to complete automated assembly line operations, which are limited in application scenarios and have weak generalization capabilities; computer vision technology, especially artificial intelligence image interpretation technology, can help industrial robots to have the ability to autonomously design operation steps and plan paths.

[0004] Therefore, the present invention aims to propose a technical solution based on artificial intelligence technology to assist humans in quickly and accurately interpreting large batches of image data. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method and system for rapid interpretation of large batches of image data based on prior knowledge. It provides a multi-model, multi-modal collaborative automatic target detection scheme based on prior knowledge, which focuses on practical needs and has industrial application value and prospects.

[0006] The objective of this invention is achieved through the following solution: A fast interpretation method for large-scale image data based on prior knowledge includes: S1. Construct a task scenario dataset and use the dataset to train the CNN-DE algorithm to obtain a model for detecting target type and target location; S2, use the training set from step S1 to construct an image and text dataset, and use the image and text dataset to train and fine-tune the LVLM model so that it can perform image interpretation based on prior knowledge. S3, based on the dataset from step S1, combine images and use text to label the images to construct a multimodal aligned dataset. Use this dataset to fine-tune CLIP to obtain an auxiliary interpretation model that can retrieve content. S4. Using the PyTorch and LangChain framework, integrate the three corresponding models from steps S1, S2, and S3 into a single system, adjust the system's inputs and outputs, build a human-computer interaction interface, and connect to the database.

[0007] Further, in step S1, the construction of the task scene dataset and the training of the CNN-DE algorithm using the dataset to obtain a model for detecting target type and target location include the following sub-steps: S1.1, Construct a typical dataset based on image interpretation task application scenarios, and label the images using annotation tools; S1.2, Use the training set to train the CNN-DE algorithm model, set up the training environment, write the training script and execute it; S1.3, use the validation set to detect the performance metrics of the trained model, and iteratively fine-tune it until a set of parameter weights in which both hyperparameters and performance metrics reach the expected values ​​of the system are trained; S1.4 Save the optimal weights for integration into the system. At the same time, adjust the model output results and standardize them into a prior knowledge format for use as input data for interpreting the model.

[0008] Further, in step S2, the construction of an image-text dataset using the training set from step S1, and the training of a fine-tuned LVLM model using this dataset to enable image interpretation based on prior knowledge, includes the following sub-steps: S2.1, Based on the training set in step S1, construct a text and image dataset for fine-tuning the LVLM model, which is used for model fine-tuning; S2.2, Incremental fine-tuning of the original model based on the LoRA method is used to activate its ability to interpret images; S2.3, merge the fine-tuned weights with the original weights to obtain the LVLM-LoRA model. In addition to the basic capabilities of the original model, this model has learned the images and prior knowledge of the set scene and has the ability to interpret other images in the scene. S2.4. Use the validation set to calculate the interpretation index and compare it with the index of human interpretation. If it does not reach the level of human interpretation, increase the amount of data and refine the labeling of the data to improve the data quality. Repeat the fine-tuning work until the accuracy index reaches or exceeds the level of human interpretation. S2.5 saves the merged weights and integrates them into the system as a judgment module through the LangChain framework.

[0009] Further, in step S3, based on the dataset from step S1, a multimodal aligned dataset is constructed by combining images and labeling them with text. This dataset is then used to fine-tune CLIP to obtain an auxiliary interpretation model capable of content-based retrieval, including the following sub-steps: S3.1, Based on the training set and original data from step S1, construct a feature-aligned multimodal image dataset and label it; S3.2, Use the LoRA method to incrementally fine-tune the CLIP model using a multimodal dataset; S3.3 uses the PyTorch framework to integrate and fine-tune the model, and adjusts the model's input and output to be images. The process of image-to-text conversion and text-to-image matching is encapsulated in the auxiliary interpretation module.

[0010] Further, in step S4, the integration of the three corresponding models from steps S1, S2, and S3 into a single system using the PyTorch and LangChain frameworks, adjustment of the system's inputs and outputs, construction of the human-computer interaction interface, and database integration includes the following sub-steps: S4.1 Receives unlabeled image data input from the user through a human-computer interaction interface; S4.2, Based on the prior knowledge extraction in step S1, the CNN-DE algorithm is called to extract the target knowledge of the image and save it in a normalized format; S4.3 Input the prior knowledge and the original image into the interpretation module at the same time. Check the accuracy of the prior knowledge extracted in S4.2 through the thought chain and prompt words. If all the prior knowledge of the image is correct, store the image and knowledge in the target library for further data mining and statistics. Otherwise, when the prior knowledge and the interpretation result are inconsistent, push the image to the human-computer interaction interface and the auxiliary interpretation module. S4.4 After receiving the original image, the auxiliary interpretation module calls the fine-tuned CLIP model to search for other modal images of the image in the form of content retrieval and push them to the human-computer interaction interface. S4.5, the human-computer interaction interface displays the original image and the results of the auxiliary interpretation module to the user, and provides plotting tools.

[0011] Furthermore, in step S1.3, the performance metrics of the trained model include: precision, recall, average precision, and F1 score.

[0012] Furthermore, in step S2.4, the indicators for judgment include the accuracy indicator.

[0013] Furthermore, in step S4.5, after displaying the original image and the results of the auxiliary interpretation module to the user in the human-computer interaction interface and providing plotting tools, the steps include: after the user completes the manual interpretation, the interpretation results are entered into the database.

[0014] A rapid interpretation system for large-volume image data based on prior knowledge includes a computer device, the computer device including a processor and a memory, the memory storing a computer program, which executes the method described in any of the preceding claims when the computer program is loaded by the processor.

[0015] The beneficial effects of this invention include: Compared with existing artificial intelligence algorithms, the method of this invention has better overall performance, specifically in the following aspects: 1. A fast target detection algorithm based on deep learning is used to extract knowledge information such as the type and location of targets in the image and submit it to a large model for interpretation, which ensures both speed and accuracy. 2. Using a fine-tuned large-scale visual language model as the executor for the judgment task, the fine-tuning process stimulates the professionalism of the open-source model in completing downstream tasks. At the same time, by constraining the model output, the occurrence of model "illusion" problem is effectively reduced. 3. A contrastive language-image model trained with multimodal data can identify other modalities of an image based on content retrieval through feature space alignment, thereby providing auxiliary information for human decision-making; The above methods can be integrated into a single system, enabling manual interpretation of large volumes of image data quickly and accurately. This invention possesses high technological maturity and has broad application prospects in fields such as transportation, remote sensing, and security. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a structural block diagram of the system of the present invention; Figure 2 This is a schematic diagram illustrating the process of training the prior knowledge extraction model of this invention; Figure 3 This is a schematic diagram illustrating the process of training and fine-tuning the large-scale automatic interpretation model of this invention; Figure 4 This is a schematic diagram illustrating the process of training and fine-tuning the auxiliary interpretation model of this invention; Figure 5 This is a system workflow diagram of the present invention. Detailed Implementation

[0018] All features disclosed in all embodiments of this specification, or steps in all methods or processes implied in the disclosure, may be combined and / or extended or replaced in any way, except for mutually exclusive features and / or steps.

[0019] This invention presents a rapid interpretation method and system for large-scale image data based on a deep neural network object detection algorithm (CNN-DE), a large-scale visual language model (LVLM), and a pre-trained contrastive language-image model (CLIP). The method utilizes the CNN-DE algorithm to extract target features from image data in batches to obtain prior knowledge. A fine-tuned LVLM model and prior knowledge are then used for rapid automatic interpretation to improve efficiency. When a small amount of data cannot be automatically interpreted, a fine-tuned CLIP model is used for multimodal assisted manual interpretation to improve the accuracy of image interpretation. Through the organic combination of these three algorithm models, an automatic image object detection system is integrated, overcoming the limitations of single-model application scenarios. Furthermore, compared to purely manual or single-algorithm-assisted systems, this system significantly improves the automation level of large-scale image data interpretation and reduces human labor intensity.

[0020] More specifically, the system of this invention can be integrated into an image interpretation and data mining system. The system block diagram is as follows: Figure 1As shown, the system receives a large amount of unlabeled data input. Through a prior knowledge extraction module, it processes the data to identify the target type and location of the image. Then, the knowledge is input into a large-scale analysis model. If the extracted knowledge is accurate, it is directly entered into the target database. If the analysis result is inconsistent with the prior knowledge, the user is prompted to perform manual analysis. The manually analyzed image will be automatically retrieved from other modalities based on its content by an auxiliary analysis module for the user's reference.

[0021] On the other hand, the present invention provides a method for rapid interpretation of large batches of image data based on prior knowledge, comprising the following steps: Step S1: Construct a dataset of typical task scenarios and use the dataset to train the CNN-DE algorithm to obtain a model capable of detecting target type and target location, which serves as the system's prior knowledge extraction module. (Reference) Figure 2 The specific implementation process of this step is as follows: S1.1, Construct a typical dataset based on image interpretation task application scenarios, such as: in the field of road monitoring, interpreting illegal lane changing over solid lines. Select approximately 2000 images (including video frame extraction images) taken by surveillance cameras, construct a training set and a validation set at a 4:1 ratio, and label the images using annotation tools; S1.2, Use the training set to train the CNN-DE algorithm model, set up the training environment, write and execute the automatic training script; S1.3 Use the validation set to detect the main performance metrics of the trained model, including: precision, recall, average precision, F1 score, etc., and iteratively optimize until a set of parameter weights in which both hyperparameters and performance metrics reach the expected values ​​of the system are trained. S1.4 Save the optimal weights for integration into the system. At the same time, adjust the model output results and standardize them into a prior knowledge format for automatic interpretation of the model's input data.

[0022] Step S2: Construct an image-text dataset using the training set from Step S1, and use this dataset to train and fine-tune the LVLM model so that it can automatically interpret images based on prior knowledge. (Reference) Figure 3 The specific implementation process of this step is as follows: S2.1, Construct a text and image dataset for fine-tuning the LVLM model based on the training set from step one; S2.2, Incremental fine-tuning of the original model based on the LoRA method, mainly to activate its ability to automatically interpret images; S2.3, merge the fine-tuned weights with the original weights to obtain the LVLM-LoRA model. In addition to the basic capabilities of the original model, this model has the ability to automatically interpret other images in the given scene because it has learned a large number of images and prior knowledge in the given scene. S2.4 Use the validation set to calculate the accuracy and other indicators of automatic interpretation, and compare them with the indicators of manual interpretation. If they do not reach the level of manual interpretation, increase the amount of data and refine the labeling of the data to improve the data quality. Repeat the fine-tuning work until the accuracy indicators reach or exceed the level of manual interpretation. S2.5 saves the merged weights and integrates them into the system via the LangChain framework as an automatic judgment module.

[0023] Step S3: Based on the dataset from Step S1, combine images from other modalities and label the images with text to construct a multimodal aligned dataset. Use this dataset to fine-tune CLIP to obtain an auxiliary interpretation model capable of content-based retrieval. (Reference) Figure 4 The specific implementation process of this step is as follows: S3.1, Construct a feature-aligned multimodal image dataset based on the training set from step S1 and the original data from other modalities, and then label it; S3.2, Use the LoRA method to incrementally fine-tune the CLIP model using a multimodal dataset; S3.3 uses the PyTorch framework to integrate and fine-tune the model, adjusting the model's input and output to both images, and encapsulates the image-to-text conversion and text-to-image matching processes in the auxiliary interpretation module.

[0024] Step S4: Integrate the three models into a single system using the PyTorch and LangChain frameworks. Adjust the system's inputs and outputs, build the user interface, and connect to the database. (See attached reference.) Figure 5 The specific implementation process of this step is as follows: S4.1, receives large amounts of unlabeled image data input from users through a human-computer interaction interface; S4.2, through the prior knowledge extraction module, the CNN-DE algorithm is called to automatically extract the target knowledge of the image and save it in a standardized format; S4.3 Input the prior knowledge and the original image into the automatic interpretation module simultaneously. The module automatically checks the accuracy of the prior knowledge extracted in S4.2 through thought chains and prompts. If all the prior knowledge of the image is correct (e.g., there are two instances of illegal lane changes across solid lines in an image, both of which are captured by the prior knowledge and pass the automatic interpretation module's check), then the image and knowledge are stored in the target database for further data mining and statistical analysis. Otherwise, when the prior knowledge and the automatic interpretation results are inconsistent, the image is pushed to the human-computer interaction interface and the auxiliary interpretation module. S4.4 After receiving the original image, the auxiliary interpretation module calls the fine-tuned CLIP model to retrieve other modal images of the image using content retrieval (e.g., if the original image was captured by a visible light camera, retrieve the image from the infrared detector at the same moment) and pushes it to the human-computer interaction interface. In S4.5, the human-computer interaction interface displays both the original image and the results from the auxiliary interpretation module to the user, and provides plotting tools. After completing the manual interpretation, the user can store the interpretation results in the database.

[0025] It should be noted that, within the scope of protection defined in the claims of this invention, the following embodiments can be combined and / or extended or replaced in any logical manner from the above specific embodiments, such as the disclosed technical principles, disclosed technical features or implicitly disclosed technical features.

[0026] Example 1 A fast interpretation method for large-scale image data based on prior knowledge includes: S1. Construct a task scenario dataset and use the dataset to train the CNN-DE algorithm to obtain a model for detecting target type and target location; S2, use the training set from step S1 to construct an image and text dataset, and use the image and text dataset to train and fine-tune the LVLM model so that it can perform image interpretation based on prior knowledge. S3, based on the dataset from step S1, combine images and use text to label the images to construct a multimodal aligned dataset. Use this dataset to fine-tune CLIP to obtain an auxiliary interpretation model that can retrieve content. S4. Using the PyTorch and LangChain framework, integrate the three corresponding models from steps S1, S2, and S3 into a single system, adjust the system's inputs and outputs, build a human-computer interaction interface, and connect to the database.

[0027] Example 2 Based on Example 1, step S1, which involves constructing a task scenario dataset and using the dataset to train the CNN-DE algorithm to obtain a model for detecting target type and target location, includes the following sub-steps: S1.1, Construct a typical dataset based on image interpretation task application scenarios, and label the images using annotation tools; S1.2, Use the training set to train the CNN-DE algorithm model, set up the training environment, write the training script and execute it; S1.3, use the validation set to detect the performance metrics of the trained model, and iteratively fine-tune it until a set of parameter weights in which both hyperparameters and performance metrics reach the expected values ​​of the system are trained; S1.4 Save the optimal weights for integration into the system. At the same time, adjust the model output results and standardize them into a prior knowledge format for use as input data for interpreting the model.

[0028] Example 3 Based on Example 1, in step S2, the construction of an image-text dataset using the training set from step S1, and the training of a fine-tuned LVLM model using this dataset to enable image interpretation based on prior knowledge, includes the following sub-steps: S2.1, Based on the training set in step S1, construct a text and image dataset for fine-tuning the LVLM model, which is used for model fine-tuning; S2.2, Incremental fine-tuning of the original model based on the LoRA method is used to activate its ability to interpret images; S2.3, merge the fine-tuned weights with the original weights to obtain the LVLM-LoRA model. In addition to the basic capabilities of the original model, this model has learned the images and prior knowledge of the set scene and has the ability to interpret other images in the scene. S2.4. Use the validation set to calculate the interpretation index and compare it with the index of human interpretation. If it does not reach the level of human interpretation, increase the amount of data and refine the labeling of the data to improve the data quality. Repeat the fine-tuning work until the accuracy index reaches or exceeds the level of human interpretation. S2.5 saves the merged weights and integrates them into the system as a judgment module through the LangChain framework.

[0029] Example 4 Based on Example 3, in step S3, the dataset from step S1 is combined with images and labeled with text to construct a multimodal aligned dataset. CLIP is then fine-tuned using this dataset to obtain an auxiliary interpretation model capable of content-based retrieval. This includes the following sub-steps: S3.1, Based on the training set and original data from step S1, construct a feature-aligned multimodal image dataset and label it; S3.2, Use the LoRA method to incrementally fine-tune the CLIP model using a multimodal dataset; S3.3 uses the PyTorch framework to integrate and fine-tune the model, and adjusts the model's input and output to be images. The process of image-to-text conversion and text-to-image matching is encapsulated in the auxiliary interpretation module.

[0030] Example 5 Based on Example 4, in step S4, the integration of the three corresponding models from steps S1, S2, and S3 into a single system using the PyTorch and LangChain frameworks, adjustment of the system's inputs and outputs, construction of the human-computer interaction interface, and database integration includes the following sub-steps: S4.1 Receives unlabeled image data input from the user through a human-computer interaction interface; S4.2, Based on the prior knowledge extraction in step S1, the CNN-DE algorithm is called to extract the target knowledge of the image and save it in a normalized format; S4.3 Input the prior knowledge and the original image into the interpretation module at the same time. Check the accuracy of the prior knowledge extracted in S4.2 through the thought chain and prompt words. If all the prior knowledge of the image is correct, store the image and knowledge in the target library for further data mining and statistics. Otherwise, when the prior knowledge and the interpretation result are inconsistent, push the image to the human-computer interaction interface and the auxiliary interpretation module. S4.4 After receiving the original image, the auxiliary interpretation module calls the fine-tuned CLIP model to search for other modal images of the image in the form of content retrieval and push them to the human-computer interaction interface. S4.5, the human-computer interaction interface displays the original image and the results of the auxiliary interpretation module to the user, and provides plotting tools.

[0031] Example 6 Based on Example 2, in step S1.3, the performance metrics of the trained model include: precision, recall, average precision, and F1 score.

[0032] Example 7 Based on Example 3, in step S2.4, the indicators for judgment include the accuracy indicator.

[0033] Example 8 Based on Example 5, in step S4.5, after displaying the original image and the results of the auxiliary interpretation module to the user in the human-computer interaction interface and providing plotting tools, the following steps are included: after the user completes the manual interpretation, the interpretation results are entered into the database.

[0034] Example 9 A rapid interpretation system for large-volume image data based on prior knowledge includes a computer device, which includes a processor and a memory. The memory stores a computer program, and when the computer program is loaded by the processor, it executes the method described in any one of Embodiments 1 to 8.

[0035] The units described in the embodiments of the present invention can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0036] According to one aspect of the present invention, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations described above.

[0037] In another aspect, embodiments of the present invention also provide a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods described in the above embodiments.

Claims

1. A method for rapid interpretation of large batches of image data based on prior knowledge, characterized in that, include: S1. Construct a task scenario dataset and use the dataset to train the CNN-DE algorithm to obtain a model for detecting target type and target location; S2, use the training set from step S1 to construct an image and text dataset, and use the image and text dataset to train and fine-tune the LVLM model so that it can perform image interpretation based on prior knowledge. S3, based on the dataset from step S1, combine images and use text to label the images to construct a multimodal aligned dataset. Use this dataset to fine-tune CLIP to obtain an auxiliary interpretation model that can retrieve content. S4. Using the PyTorch and LangChain framework, integrate the three corresponding models from steps S1, S2, and S3 into a single system, adjust the system's inputs and outputs, build a human-computer interaction interface, and connect to the database.

2. The method for rapid interpretation of large batches of image data based on prior knowledge according to claim 1, characterized in that, In step S1, constructing the task scenario dataset and using the dataset to train the CNN-DE algorithm to obtain a model for detecting target type and target location includes the following sub-steps: S1.1, Construct a typical dataset based on image interpretation task application scenarios, and label the images using annotation tools; S1.2, Use the training set to train the CNN-DE algorithm model, set up the training environment, write the training script and execute it; S1.3, use the validation set to detect the performance metrics of the trained model, and iteratively fine-tune it until a set of parameter weights in which both hyperparameters and performance metrics reach the expected values ​​of the system are trained; S1.4 Save the optimal weights for integration into the system. At the same time, adjust the model output results and standardize them into a prior knowledge format for use as input data for interpreting the model.

3. The method for rapid interpretation of large batches of image data based on prior knowledge according to claim 1, characterized in that, In step S2, the construction of an image-text dataset using the training set from step S1, and the training of a fine-tuned LVLM model using this dataset to enable image interpretation based on prior knowledge, includes the following sub-steps: S2.1, Based on the training set in step S1, construct a text and image dataset for fine-tuning the LVLM model, which is used for model fine-tuning; S2.2, Incremental fine-tuning of the original model based on the LoRA method is used to activate its ability to interpret images; S2.3, merge the fine-tuned weights with the original weights to obtain the LVLM-LoRA model. In addition to the basic capabilities of the original model, this model has learned the images and prior knowledge of the set scene and has the ability to interpret other images in the scene. S2.

4. Use the validation set to calculate the interpretation index and compare it with the index of human interpretation. If it does not reach the level of human interpretation, increase the amount of data and refine the labeling of the data to improve the data quality. Repeat the fine-tuning work until the accuracy index reaches or exceeds the level of human interpretation. S2.5 saves the merged weights and integrates them into the system as a judgment module through the LangChain framework.

4. The method for rapid interpretation of large batches of image data based on prior knowledge according to claim 3, characterized in that, In step S3, based on the dataset from step S1, a multimodal aligned dataset is constructed by combining images and labeling them with text. This dataset is then used to fine-tune CLIP to obtain an auxiliary interpretation model capable of content-based retrieval. This includes the following sub-steps: S3.1, Based on the training set and original data from step S1, construct a feature-aligned multimodal image dataset and label it; S3.2, Use the LoRA method to incrementally fine-tune the CLIP model using a multimodal dataset; S3.3 uses the PyTorch framework to integrate and fine-tune the model, and adjusts the model's input and output to be images. The process of image-to-text conversion and text-to-image matching is encapsulated in the auxiliary interpretation module.

5. The method for rapid interpretation of large batches of image data based on prior knowledge according to claim 4, characterized in that, In step S4, the three models from steps S1, S2, and S3 are integrated into a single system using the PyTorch and LangChain frameworks. This includes adjusting the system's inputs and outputs, constructing a human-computer interface, and connecting to the database. This step includes the following sub-steps: S4.1 Receives unlabeled image data input from the user through a human-computer interaction interface; S4.2, Based on the prior knowledge extraction in step S1, the CNN-DE algorithm is called to extract the target knowledge of the image and save it in a normalized format; S4.3 Input the prior knowledge and the original image into the interpretation module at the same time. Check the accuracy of the prior knowledge extracted in S4.2 through the thought chain and prompt words. If all the prior knowledge of the image is correct, then store the image and knowledge in the target library for further data mining and statistics. Otherwise, when the prior knowledge and the interpretation result are inconsistent, the image is pushed to the human-computer interaction interface and the auxiliary interpretation module; S4.4 After receiving the original image, the auxiliary interpretation module calls the fine-tuned CLIP model to search for other modal images of the image in the form of content retrieval and push them to the human-computer interaction interface. S4.5, the human-computer interaction interface displays the original image and the results of the auxiliary interpretation module to the user, and provides plotting tools.

6. The method for rapid interpretation of large batches of image data based on prior knowledge according to claim 2, characterized in that, In step S1.3, the performance metrics of the trained model include: precision, recall, average precision, and F1 score.

7. The method for rapid interpretation of large batches of image data based on prior knowledge according to claim 3, characterized in that, In step S2.4, the indicators for judgment include the accuracy indicator.

8. The method for rapid interpretation of large batches of image data based on prior knowledge according to claim 5, characterized in that, In step S4.5, after displaying the original image and the results of the auxiliary interpretation module to the user in the human-computer interaction interface and providing plotting tools, the following steps are included: after the user completes the manual interpretation, the interpretation results are entered into the database.

9. A system for rapid interpretation of large-volume image data based on prior knowledge, comprising computer equipment, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program that, when loaded by the processor, executes the method as described in any one of claims 1 to 8.

Images