Multi-modal analysis image identification system and method

Abstract

The invention discloses a multi-modal analysis-based image identification system and method, and the method comprises the steps: constructing a multi-modal feature index model, employing a deep learning-based image segmentation technology, designing a deep learning-based multi-modal matching mechanism, developing a visual analysis module, and displaying the identification process and result in a visual form. The transparency of the system is enhanced; and a man-machine interaction module is designed. According to the image identification system and method based on multi-modal analysis, through combination of deep learning, multi-modal data matching and visualization technologies, the system can automatically segment and match various elements such as topics, seals and picture features in a drawing, and meanwhile, provides visual display of associated information; and in addition, the defects of an existing system in the aspects of diversified data processing, transparency of the identification process and man-machine cooperation are overcome, and scientificity and efficiency of identification work are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of machine vision and artificial intelligence, and in particular to an image identification system and method for multimodal analysis. Background Art

[0002] The identification of traditional Chinese paintings has long relied on the experience of experts and manual judgment by analyzing multiple features such as inscriptions, seals, and brushstrokes, but this process is highly subjective and time-consuming. With the development of computer vision and deep learning technology, computer-assisted identification technology has begun to be applied in this field, which can provide more objective and accurate results.

[0003] For example, in the prior art, a Chinese patent with announcement number CN111539483A discloses a fake image identification system and construction method based on a GAN network, which constructs multiple GAN network models and trains them on their corresponding data sets respectively; uses the trained multiple GAN network models to generate fake images, and selects an equal number of real images from the data set to unify the resolution; performs data enhancement operations on the fake images and the real images respectively to generate a training set; selects a GAN network model, and uses the generated training set to train the discriminator of the GAN network model; generalizes the trained GAN network model to construct a fake image identification system based on the GAN network.

[0004] For example, in the prior art, a Chinese patent with the announcement number CN112418350A discloses a biological image identification system based on a telecentric camera matrix, including: a camera matrix for collecting a plurality of high-precision original images of a local field of view of a tested organism; an image stitching module for registering and stitching a plurality of high-precision original images of a local field of view into a stitched image of a high-precision global field of view; and a genus and species analysis module for analyzing the stitched image and obtaining a conclusion on the prediction of the biological genus and species classification. The present invention also discloses a biological image identification method based on a telecentric camera matrix. Compared with the prior art, the system provided by the present invention can take into account both the accuracy and imaging range of the biological imaging to be tested, greatly reducing the equipment cost and the difficulty of equipment assembly, and facilitating operation.

[0005] However, existing technologies still have problems such as insufficient fusion of multimodal features, poor system interpretability, lack of human-computer interaction mechanism, and single comparison data, which lead to limitations in practical applications. Existing systems usually only analyze single visual features, lack comprehensive analysis of important information such as inscriptions and seals, and their "black box" characteristics make it difficult for experts to understand or trust the identification results given by computers. Summary of the invention

[0006] The object of the present invention is to provide an image identification system and method for multimodal analysis to solve the problem of limitations in practical application mentioned in the above background technology.

[0007] To achieve the above object, the present invention provides the following technical solution: a multimodal analysis image identification system and method, comprising the following contents:

[0008] Construct a multimodal feature index model: Integrate different features of the painting, including inscriptions, seals, picture content and textual research materials;

[0009] Use deep learning-based image segmentation technology to automatically segment the inscriptions, seals, and image content in paintings;

[0010] Design a multimodal matching mechanism based on deep learning to match and determine the similarity of various extracted features;

[0011] Develop a visual analysis module to display the identification process and results in a visual form to enhance the transparency of the system;

[0012] A human-computer interaction module is designed to allow experts to actively participate in feature comparison and verification, thereby improving the scientificity and efficiency of identification work.

[0013] Preferably, the constructing of the multimodal feature index model specifically includes:

[0014] a. Integrate multimodal data including the inscriptions, seals, content of the painting and research materials;

[0015] b. Use deep learning technology to uniformly process and characterize different types of features;

[0016] c. Different types of feature data are integrated through deep learning models to establish a unified multimodal feature representation.

[0017] Preferably, the image segmentation technology specifically includes:

[0018] a. Use a convolutional neural network-based segmentation method to automatically segment the inscriptions, seals, and picture content in the paintings;

[0019] b. Optimize the model training by adjusting the hyperparameters of the segmentation model to improve the segmentation accuracy;

[0020] c. When automatic segmentation cannot achieve the desired effect, a manual assistance function is provided to allow experts to fine-tune and correct the segmentation results.

[0021] Preferably, the multimodal matching mechanism specifically includes:

[0022] a. Design matching algorithms for inscriptions, seals and picture features based on deep learning technology, combined with similarity measurement of multimodal features;

[0023] b. Set the priority and rules for feature matching to ensure the reliability and consistency of matching results;

[0024] c. Supports cross-painting matching, which can perform similarity matching between different paintings to help experts discover possible connections.

[0025] Preferably, the visual analysis module specifically includes:

[0026] a. Display the analysis results of multimodal data through a variety of visualization tools, including charts, heat maps, relationship network diagrams, etc.;

[0027] b. Provide an interactive operation interface, allowing experts to zoom in, zoom out, rotate, etc. on specific areas of the painting;

[0028] c. Dynamically mark inscriptions, seals and other feature areas on the visual interface to help experts quickly locate and analyze.

[0029] Preferably, the human-computer interaction module specifically includes:

[0030] a. Design interactive modes, including manual selection of specific areas of the painting and automatic recommendation of analysis areas by the system;

[0031] b. Provide a quick information retrieval function. The system can quickly retrieve relevant historical documents, painting contents and other related information by connecting with the information database;

[0032] c. The system records the expert's operational feedback and fine-tunes and optimizes the deep learning model based on the expert's judgment.

[0033] A multimodal analysis image identification method, including building a multimodal feature index model, image segmentation technology, multimodal matching mechanism, developing a visual analysis module and designing a human-computer interaction module;

[0034] The construction of the multimodal feature index model comprises the following steps:

[0035] S1. Establish the relevance and integrity of multimodal data and ensure the interrelationship and complementarity between different features;

[0036] S2. Use data fusion mechanism to improve the correlation between data and build a unified feature representation.

[0037] Preferably, the optimization training of the image segmentation technology comprises the following steps:

[0038] S1. Adjust hyperparameters such as learning rate and convolution kernel size to optimize the performance of the segmentation model.

[0039] S2. Use data enhancement technology to improve the generalization ability of the model.

[0040] Preferably, the multimodal matching mechanism comprises the following steps:

[0041] S1. Combining prior knowledge and expert experience, formulate matching confidence threshold;

[0042] S2. Evaluate the credibility of the matching results and provide a confidence index for the matching results.

[0043] Preferably, the developing visualization analysis module comprises the following steps:

[0044] S1. Dynamic update of analysis results to display changes in feature processing in real time;

[0045] S2. Support multi-window or multi-screen display, which is convenient for experts to compare multiple analysis results at the same time;

[0046] Designing a human-computer interaction module includes the following steps:

[0047] S3. Provide expert feedback collection function to record the expert's operations and opinions;

[0048] S4. Based on expert feedback, continuously improve the system’s human-machine collaboration capabilities.

[0049] Compared with the prior art, the beneficial effects of the present invention are as follows: the multimodal analysis image identification system and method adopts a new structural design, and its specific contents are as follows:

[0050] 1. By combining deep learning, multimodal data matching and visualization technology, the system can automatically segment and match various elements in paintings, such as inscriptions, seals, and picture features, while providing a visual display of related information to assist experts in making more comprehensive and objective judgments during the identification process. In addition, it solves the shortcomings of existing systems in terms of diversified data processing, transparency of the identification process, and human-computer collaboration, and significantly improves the scientificity and efficiency of the identification work.

[0051] 2. Improve identification efficiency and accuracy: Through multimodal feature integration and deep learning technology, the system can efficiently extract and analyze key information in paintings, assist experts in comprehensive evaluation, and significantly improve identification efficiency and accuracy;

[0052] Enhance the explainability and transparency of the system: Through the visual analysis module, the system can intuitively display the key steps in the identification process, which enhances the experts' trust in the computer-aided identification system;

[0053] Effective integration of multimodal data: The system can integrate multimodal data such as inscriptions, seals, picture content and research materials, making the identification process more comprehensive and scientific;

[0054] Efficient human-machine collaboration mechanism: The system provides a variety of interactive modes, enabling experts to actively participate in and control the identification process, improving the scientificity and flexibility of the identification work. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a schematic diagram of the picture analysis method for computer-assisted picture identification of the present invention;

[0056] Figure 2 Schematic diagram of two segmentation modes of the SAM model adopted in the present invention;

[0057] Figure 3 It is the preview effect and result of the four traditional segmentation algorithms adopted by the present invention;

[0058] Figure 4 It is a schematic diagram of the heat map mapping effect of semantic segmentation slices;

[0059] Figure 5 is a schematic diagram of the main window of the auxiliary identification tool;

[0060] Figure 6 is a diagram showing a user using the magic wand tool in the image segmentation module;

[0061] Figure 7 This is a schematic diagram of the automatic cutting effect of the painting;

[0062] Figure 8 It is the main program window where text recognition and matching are being performed;

[0063] Fig. 9 On the left is the seal matching module showing the results, and on the right is a detailed preview of the matching results;

[0064] Fig.10 It is the program flow chart of the dynamic refresh function of feature processing results;

[0065] Fig.11 This is the main interface display when the user is performing feature comparison. DETAILED DESCRIPTION

[0066] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] Embodiment 1: This embodiment proposes a traditional painting computer-assisted picture analysis method that combines multiple image methods and deep learning models. Figure 1 The complete picture analysis method of the present invention in computer-assisted picture identification is demonstrated, and (a) to (d) in the figure correspond to four submodules: image segmentation, multimodal identification material matching, visual interpretation, and feature comparison.

[0068] The details are as follows:

[0069] (1) Develop image segmentation module

[0070] (1-1) Data collection and preprocessing

[0071] Use a high-resolution scanner or professional camera to obtain digital images of traditional paintings, ensuring that the images are clear, without reflections and distortion. Convert the images to RGB color space and save them in a lossless format (such as TIFF) to ensure image quality. Perform color correction and geometric correction on the images to eliminate distortion caused by shooting angle or lighting.

[0072] (1-2) Image Cropping and Denoising

[0073] The acquired image is cropped to remove the edges, background or other irrelevant elements, retaining the main painting content. The image is denoised by applying methods such as Gaussian filtering or median filtering to reduce the impact of noise on subsequent processing.

[0074] (1-3) Loading deep learning model

[0075] Segment Anything Model (SAM) is selected as the image segmentation model because of its strong zero-sample generalization ability. In the Python environment, use PyTorch or TensorFlow framework to load the pre-trained SAM model, and set the relevant parameters of the model, such as segmentation threshold, input image size, etc., according to the characteristics of the painting image.

[0076] (1-4) Perform image semantic segmentation

[0077] The preprocessed image is input into the SAM model, and the whole image segmentation is performed to obtain the initial semantic segmentation result (such as Figure 2-a). For areas that require fine segmentation, click on feature points to guide the model to perform fine segmentation (such as Figure 2 -b). Save each segmented area (such as people, animals, landscapes, inscriptions, seals, etc.) as an independent image file, and indicate the category and number when naming.

[0078] (1-5) Manual correction and refinement

[0079] Use traditional segmentation tools such as the Magic Wand tool, Magnetic Lasso tool, and Rectangular Selection Tool to correct the automatic segmentation results. Figure 3 The preview effects and results of four traditional segmentation algorithms are shown. Zoom in to view the edges of each segmented area and manually correct inaccurate parts to ensure the accuracy of the segmentation results. Finally, save the corrected segmented area again to replace the initial segmentation result.

[0080] (2) Establishing a multimodal identification material matching module

[0081] The multimodal semantic identification framework is defined as S as follows:

[0082] S=(T,Y,C,A,H|L)

[0083] Among them, T represents the title, Y represents the seal, C represents the content of the painting image other than the title and seal, A represents the painting, and H represents the textual research. They are connected to each other in the form of an association L.

[0084] (2-1) Feature extraction

[0085] For inscription recognition, we selected the TransOCR model that performs well in handwritten Chinese OCR tasks. We used a specific dataset containing traditional Chinese handwritten data to fine-tune the TransOCR model to improve the recognition accuracy of traditional inscriptions. Then, we input the image of the inscription area into the fine-tuned model to extract the text content of the inscription.

[0086] For seal matching, the SIFT algorithm is used to extract key points and feature vectors from seal slices, and the Euclidean distance between the input seal and the seal features in the database is calculated to find the seal with the highest similarity.

[0087] For content feature extraction, a pre-trained ResNet18 model is used to extract features of images in the content area to obtain high-level image features, including texture, color, shape and other information.

[0088] (2-2) Building a multimodal database

[0089] Collect a large number of inscriptions, seals, content images and related research materials of traditional Chinese paintings. Add detailed metadata tags to each data item, such as author, age, style, source, etc. Store the collected data in a MySQL database and create an index for quick retrieval.

[0090] (2-3) Similarity calculation and material matching

[0091] For inscription matching, the extracted inscription text is preprocessed by converting traditional Chinese to simplified Chinese, removing punctuation, and segmenting words. The similarity with the inscription in the database is calculated using algorithms such as edit distance and Jaccard similarity coefficient. The matching results are sorted according to the similarity score to extract the most likely author and work information. The key information is matched to the textual research database through a string retrieval algorithm based on pattern matching. This part of the work completes the identification framework of any painting A. k Medium Title T i Analysis and title A k 、Painting A k , verify the information H between the relational data construction, there are T i →H and T i →A k .

[0092] For seal matching, the FLANN algorithm is used to quickly match SIFT features. Based on the number and position of matched key points, the similarity score is calculated, and the first few seals with the highest similarity are selected as the matching results.

[0093] For content matching, the cosine similarity between the feature vectors of the content slice and the image in the database is calculated. The results with high similarity are clustered to find potential style or theme associations, and works similar to the input work in composition, brushwork, color, etc. are extracted.

[0094] (2-4) Related research materials

[0095] According to the matched inscriptions, seals, and contents, the corresponding research materials are associated, such as historical documents, expert comments, academic papers, etc. The associated research materials are integrated with the matching results to form a complete identification information set.

[0096] (3) Develop a visual interpretation module

[0097] The amount of information associated with each slice is counted, that is, the number of database items matched. According to the amount of information, color gradient mapping is used, and the area with large amount of information has darker color. Figure 4 The heat map mapping effect of the semantic segmentation slice is demonstrated, and the heat map is superimposed on the original painting image to intuitively show the importance of each area.

[0098] Construct a relationship network diagram, take inscriptions, seals, content slices, research materials, etc. as nodes, and establish connections between nodes based on matching relationships and related information. Use a circular layout to arrange node positions by category and association strength.

[0099] (4) Implementing the feature comparison module

[0100] (4-1) Feature analysis method

[0101] Provides stroke strength erosion analysis, uses the 0penCV library, applies the erosion algorithm to process the stroke area, and analyzes the thickness and strength changes of the stroke. Provides parameters such as kernel size, number of iterations, color transition, etc., which users can adjust as needed.

[0102] Provides ink color multi-threshold screening. First, convert the image into a grayscale image and calculate the grayscale histogram. According to the grayscale mean and variance, set multiple threshold ranges, segment the ink color according to the threshold range, and analyze the changes in ink color density.

[0103] Provide stroke angle fitting, use edge detection algorithm to extract stroke outlines. Perform ellipse or polynomial fitting on the outline, calculate the inclination angle and curvature of the stroke, and extract the geometric features of the stroke for comparison and analysis.

[0104] Provides global color analysis, uses K-means clustering algorithm to cluster global colors, and counts the proportion of main colors. Generates color ratio pie charts to intuitively display the color composition of the work.

[0105] (4-2) Dynamic parameter adjustment and real-time comparison

[0106] In the parameter adjustment interface, controls such as sliders and input boxes are provided for users to adjust the parameters of the analysis algorithm. After the parameters are adjusted, the analysis results are updated in real time, making it convenient for users to observe the effects. In the comparison result display, the feature analysis results of the input work and the matched reference work are displayed side by side, and the differences between the two are highlighted using colors or marks in the comparison chart.

[0107] (5) Experimental results and verification

[0108] (5-1) Model performance evaluation

[0109] In the image segmentation model evaluation, 50 manually annotated traditional Chinese paintings were selected as the test set, and the accuracy, precision, recall, F1-score and IoU of the segmentation results were calculated. The results showed that the SAM model achieved an average IoU of 0.71 on the test set, which was significantly better than the traditional segmentation method.

[0110] In the evaluation of the inscription recognition model, the fine-tuned TransOCR model achieved a recognition accuracy of more than 85% on the test set using images of traditional Chinese handwritten inscriptions in different fonts and styles. An analysis of the samples with recognition errors showed that they were mainly concentrated in extremely sloppy or blurred text.

[0111] (5-2) Actual case verification

[0112] In the seal matching verification, a work to be authenticated was selected, and its seal was unclear. The system successfully matched the seal with the highest similarity, which was highly similar to the seal of a known author (the similarity reached 76.77%). The reliability of the matching result was further verified through the associated textual research materials.

[0113] In the content matching verification, the system retrieves multiple works with similar composition and brushwork based on content features. Authentication experts compare and confirm that these works are indeed from the same school or author.

[0114] (5-3) User feedback and system optimization

[0115] In the feedback from the expert trial, the experts believe that the system interface is intuitive, easy to operate, and has good visualization effects. The feature comparison tool and identification report generation function are very helpful for practical work. According to the system improvement suggestions, it is planned to expand the scale of the database, collect more works and textual research materials, and further optimize the model for specific painting styles to improve matching accuracy.

[0116] Embodiment 2: This embodiment proposes a computer-aided identification system for traditional paintings. FIG. 1 shows the overall framework of the system. Figure 5 The displayed system main window part is mainly divided into the following modules: image operation module, visualization module, feature adjustment module and index information module.

[0117] The image manipulation module is located in the upper left corner of the main window, and includes image selection, image segmentation tools, quick switching between thumbnails of the original and comparison images, and image preview and zoom operation controls. Users can import any painting image to be authenticated, and various processing results will be displayed in real time.

[0118] The visualization module includes the slice horizontal list in the lower left corner of the window and the quick search module in the upper right corner. The slice horizontal list is used to preview the various object slices segmented based on the SAM model. Users can freely select the slices of interest for further identification. The quick search module will display different contents according to the different features selected by the user, including the recognition results of inscriptions and seals and the relationship network diagram constructed by matching objects.

[0119] The index information module is located in the lower right corner of the window. When the user further selects the matched identification material that they are interested in, basic information is displayed. The user can click to view the details and choose whether to add the material to the main window for further comparison with the original.

[0120] The feature adjustment module provides a variety of image feature analysis tools. Users can dynamically adjust parameters to extract and compare features of images to assist in identification work.

[0121] (1) Implementation of the screen segmentation module

[0122] Design the screen segmentation module to make it a relatively independent module, such as Figure 5 This module integrates the basic functions of the system, for example, the user selects the painting to be analyzed and identified from the file system or database and quickly switches between the two thumbnails of the original and the comparison picture to all the contents of other modules.

[0123] A custom multi-purpose image label class (scalableImageLabel) is included to facilitate the visualization, processing and analysis of images in one place. This custom class inherits from PyQt's QLabel class and provides an interactive graphical interface for displaying images and enables users to apply various image processing operations. It includes advanced mouse event handling, tool selection and real-time drawing functions, allowing users to freely scale and move the displayed image by dragging the mouse, sliding the wheel, etc.

[0124] The custom class also integrates several important image segmentation operations, including flood fill (magic wand effect), rectangular cropping, free-form cropping (edge ​​adsorption), seal detection, and multi-threaded calls to the SAM image segmentation model. By seamlessly integrating these functions, the module enables identification experts to efficiently perform image analysis tasks and extract valuable slices from visual data for further analysis and matching.

[0125] Figure 6 Shows a user using the Magic Wand tool in the Image Segmentation module. Figure 7 The results of the model automatic segmentation are shown below the main window. CAPAT selects the slices that the user may be interested in by reading the mask data (Mask) and confidence returned by the model. Expert users can continue the identification work by clicking on the slices, or they can use the above-mentioned image segmentation tools to segment by themselves. Figure 7 Shows a preview of the automatic cutting of the painting.

[0126] (2) Implementation of database matching module

[0127] The database matching module is responsible for identifying the image slices processed by the screen segmentation module through the deep learning model, and matching the identification results in the identification database. It integrates the inscription, seal, and pattern recognition results to establish a matching correlation list, and outputs it to the visual relationship network generation module to dynamically render the relationship diagram.

[0128] The recognition model of TransOCR is called using the multithreaded interface and the operating system pipeline, and the results are fed back to the user interface in real time. An OutputReaderThread class is created to read the output of the child process in another thread. An output_updated signal is defined to emit a signal when the output is updated. In the run() method, the output of the child process is read in a loop, and the output_updated signal is emitted when the running state changes.

[0129] A QLabel is included above the quick search module to display the output results. The update_output slot function is defined to update the text of the QLabel when the output update signal is received. The output reading thread is started through the start_program method and connected to the slot function. With this setting, users can view the deep model processing results in real time in the main interface.

[0130] After obtaining the final recognition result, the main program accesses the MySQL database to perform a joint table query and initializes a new list (QListWidget) to present the matching results containing the text (such as Figure 8 shown).

[0131] The model calling and list display method of the seal matching list are similar to the title. The dynamically loaded UI file defined by Qt Designer is used. In the file, by setting the layout of each list item to a horizontal layout and setting the distribution weight, the program can arrange different controls of the list items according to the window size.

[0132] After the list is generated, users can use the list in two ways: single-click and double-click. When the user clicks on a list item, the "Index Information" module below the list will display detailed research materials and database information based on the selected list item. Users can click to view details and choose whether to use the material as a comparison chart for further processing and comparison. When the user double-clicks on a list item, the interface in the "Quick Search" module will jump to the relationship network, and based on the drawing information of the list item selected by the user, the related identification materials in the database will be organized and visualized as a relationship network diagram. Fig. 9 The seal matching module on the left is displaying the results; the right side is a detailed preview of the matching results

[0133] In order to display different results according to different list items after clicking on the list items, the program stores the parameters in the UserRole of QListWidgetItem. This is a common practice. UserRole is a common role provided by the PyQt library for storing custom data.

[0134] By storing the parameter in the UserRole, you can easily retrieve the parameter value from the QListWidgetItem when needed. Use item.data(QtCore.Qt.UserRole) to get the stored data. Most importantly, the stored parameters are associated with a specific list item, so that the parameters can be passed to the corresponding function when the list item is clicked. By storing the parameters in the list item, the data can be passed along with the list item without having to find the data associated with the list item through other means.

[0135] (3) Implementation of the Visual Relationship Network Generation Module

[0136] The Visual Relationship Network Generation Module establishes a connection with the MySQL database and provides functions for querying and processing data. It allows users to connect to the local MySQL server using specified credentials and open the traditional painting identification database for further operations.

[0137] By using the MySQL Connector / Python library, this module facilitates seamless communication with the MySQL database and is able to execute SQL queries. The module also contains a relationship network graph initialization function, which uses the Graph component in the PyEcharts library to create a graph using nodes, links, and categories to represent data relationships. The generated graph is rendered as an HTML file.

[0138] In the graphical user interface, the module is loaded into a PyQt QWebEngineView control of the main window, which is used to display the rendered graphics in the PyQt application. In the data flow direction, the module executes SQL queries to retrieve related data from the database based on the identification material information manually selected by the user or matched by the deep learning network, and converts the retrieved data into nodes and links for visualization.

[0139] It has an interactive function. When the user hovers the mouse over a node or line, the relevant information of the node or line will be automatically displayed, and all related nodes and lines will be highlighted. At the same time, clicking the legend above can also filter according to the category of the node. In the case of many related nodes, it can help identification experts focus on a certain type of identification material and improve indexing efficiency.

[0140] (4) Implementation of feature extraction and analysis module

[0141] The feature extraction and analysis module is used to perform feature analysis and processing on images. This module provides a series of feature drop-down menus for selecting different feature analysis methods. By clicking these buttons, users can select different functions for image processing.

[0142] In the main window, there are stacked widgets in each area to display the results of different functions.

[0143] The switching between the function button and the stacked widget is realized through the listening function. When the user clicks a function button, the program performs the corresponding operation according to the button identifier. For example, when the user selects the "Strength (stroke)" function, the program will initialize a list of adjustable parameters, and then process the image accordingly according to the selected function and parameters, and display the result in the multi-function image label on the interface.

[0144] The module also provides a list of adjustable parameters for real-time adjustment of parameters. Users can adjust parameters such as erosion core size and erosion times in real time to adjust the results of feature analysis. The result refresh function processes the image according to the selected functions and parameters, and updates the results on the interface. Fig.10 Shows the program flow chart of the dynamic refresh function of feature processing results.

[0145] Fig.11 The main interface is shown when the user is performing feature comparison. The user can adjust the parameters of the processing function through the slider, text edit box and increase and decrease buttons in the parameter list on the right, and preview the results in real time on the left.

[0146] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multimodal analysis image identification system, characterized in that: Includes the following: Construct a multimodal feature index model: Integrate different features of the painting, including inscriptions, seals, picture content and textual research materials; Use deep learning-based image segmentation technology to automatically segment the inscriptions, seals, and image content in paintings; Design a multimodal matching mechanism based on deep learning to match and determine the similarity of various extracted features; Develop a visual analysis module to display the identification process and results in a visual form to enhance the transparency of the system; A human-computer interaction module is designed to allow experts to actively participate in feature comparison and verification, thereby improving the scientificity and efficiency of identification work.

2. The multimodal analysis image identification system according to claim 1, characterized in that: The construction of the multimodal feature index model specifically includes: a. Integrate multimodal data including the inscriptions, seals, content of the painting and research materials; b. Use deep learning technology to uniformly process and characterize different types of features; c. Different types of feature data are integrated through deep learning models to establish a unified multimodal feature representation.

3. The multimodal analysis image identification system according to claim 1, characterized in that: The image segmentation technology specifically includes: a. Use a convolutional neural network-based segmentation method to automatically segment the inscriptions, seals, and picture content in the paintings; b. Optimize the model training by adjusting the hyperparameters of the segmentation model to improve the segmentation accuracy; c. When automatic segmentation cannot achieve the desired effect, a manual assistance function is provided to allow experts to fine-tune and correct the segmentation results.

4. The multimodal analysis image identification system according to claim 1, characterized in that: The multimodal matching mechanism specifically includes: a. Design matching algorithms for inscriptions, seals and picture features based on deep learning technology, combined with similarity measurement of multimodal features; b. Set the priority and rules for feature matching to ensure the reliability and consistency of matching results; c. Supports cross-painting matching, which can perform similarity matching between different paintings to help experts discover possible connections.

5. The multimodal analysis image identification system according to claim 1, characterized in that: The visual analysis module specifically includes: a. Display the analysis results of multimodal data through a variety of visualization tools, including charts, heat maps, relationship network diagrams, etc.; b. Provide an interactive operation interface, allowing experts to zoom in, zoom out, rotate, etc. on specific areas of the painting; c. Dynamically mark inscriptions, seals and other feature areas on the visual interface to help experts quickly locate and analyze.

6. The multimodal analysis image identification system according to claim 1, characterized in that: The human-computer interaction module specifically includes: a. Design interactive modes, including manual selection of specific areas of the painting and automatic recommendation of analysis areas by the system; b. Provide quick information retrieval function. The system can quickly retrieve relevant historical documents, painting contents and other related information by connecting with the information database; c. The system records the expert's operational feedback and fine-tunes and optimizes the deep learning model based on the expert's judgment.

7. A multimodal analysis image identification method, characterized in that: include: Construct multimodal feature index models, image segmentation technology, multimodal matching mechanism, develop visualization analysis modules and design human-computer interaction modules; The construction of the multimodal feature indicator model comprises the following steps: S1. Establish the relevance and integrity of multimodal data and ensure the interrelationship and complementarity between different features; S2. Use data fusion mechanism to improve the correlation between data and build a unified feature representation.

8. The image identification method of multimodal analysis according to claim 7, characterized in that: The optimization training of the image segmentation technology includes the following steps: S1. Adjust hyperparameters such as learning rate and convolution kernel size to optimize the performance of the segmentation model. S2. Use data enhancement technology to improve the generalization ability of the model.

9. The image identification method of multimodal analysis according to claim 7, characterized in that: The multimodal matching mechanism comprises the following steps: S1. Combining prior knowledge and expert experience, formulate matching confidence threshold; S2. Evaluate the credibility of the matching results and provide a confidence index for the matching results.

10. The image identification method of multimodal analysis according to claim 7, characterized in that: The development of the visual analysis module comprises the following steps: S1. Dynamic update of analysis results to display changes in feature processing in real time; S2. Support multi-window or multi-screen display, which is convenient for experts to compare multiple analysis results at the same time; Designing a human-computer interaction module includes the following steps: S3. Provide expert feedback collection function to record the expert's operations and opinions; S4. Based on expert feedback, continuously improve the system’s human-machine collaboration capabilities.

Images