Multi-modal information retrieval method and system based on convolutional neural network and TF-IDF algorithm

Abstract

The application provides a kind of multi-modal information retrieval method and system based on convolutional neural network and TF-IDF algorithm, comprising: step S1: input actual image to obtain label, or directly input label;Step S2: obtain relevant text information by text retrieval method, obtain the keyword with frequency higher than preset standard;Step S3: the classification result and the keyword with frequency higher than preset standard after text retrieval are used as the input of image retrieval, and all relevant image information is obtained in patent library using web crawler;Step S4: output the patent image meeting the preset standard after image retrieval, and the text retrieval structure and image retrieval result are comprehensively output.The application fully explores the characteristics of existing knowledge, and provides a cross-modal knowledge set for the design process to provide analog solution generation.

Description

TECHNICAL FIELD

[0001] The present application relates to the fields of machine learning and industrial design, in particular, to a multi-modal information retrieval method and system based on convolutional neural network and TF-IDF algorithm, and more particularly to a multi-modal information retrieval method and system based on convolutional neural network and TF-IDF algorithm in bionic design. BACKGROUND

[0002] Bionic design is an important design idea in the field of industrial design, and various bionic design products have shown good adaptability and usability in practical applications. In industrial design, bionic design mainly participates in the concept design stage. In this stage, designers need to refer to a large amount of literature or patent materials. Searching for useful information in a large amount of Internet information often takes a lot of time, and the search results directly affect the results of the concept design. In the past, the general method is to extract the text information in the patent library, and use the natural language processing method to obtain relevant information that can provide design inspiration to assist design. This traditional method ignores the picture information in the patent in the process of using the patent library data. In order to realize more comprehensive provision of design inspiration, realize the image information and text information two modalities to assist bionic design, a cross-modal knowledge acquisition method for biological inspiration design is proposed. In view of the problem that the bionic design process relies on single modal knowledge acquisition, a cross-domain and cross-modal knowledge retrieval method is proposed to realize the cross-domain and cross-modal knowledge acquisition in the biological-engineering field to provide a knowledge set.

[0003] In industrial design, bionics knowledge can provide many inspirations for concept design. However, in the process of searching for bionic knowledge, there are many problems such as information retrieval difficulties and single information retrieval mode. Taking the patent library as an example, it is extremely difficult to accurately retrieve the expected text information and image information at the same time in hundreds of millions of patent documents.

[0004] Patent document CN103348348B discloses an information retrieval device and an information retrieval method, which includes: an information retrieval device (1) for retrieving information associated with a word, comprising: an input unit (11) for inputting a word; a mode generation unit for generating a word group mode when a new word is added to the previously input word or when the new word is replaced by the previously input word; an occurrence information derivation unit (14X) for deriving occurrence information corresponding to the probability of occurrence of the generated various word group modes; and a determination unit (15) for determining the word group mode that should be used for new retrieval according to the derived occurrence information. (13) inputs a new word after inputting a predetermined number of words. However, this invention does not realize cross-modal knowledge set for patents. SUMMARY

[0005] In view of the defects in the prior art, the present application aims to provide a multi-modal information retrieval method and system based on a convolutional neural network and a TF-IDF algorithm.

[0006] According to the present application, a multi-modal information retrieval method based on a convolutional neural network and a TF-IDF algorithm is provided, comprising:

[0007] Step S1: input actual images to obtain labels, or directly input labels;

[0008] Step S2: obtain relevant text information by a text retrieval method, and obtain keywords with a frequency higher than a preset standard;

[0009] Step S3: use the classification result and the keywords with a frequency higher than the preset standard after text retrieval as input for image retrieval, and use a web crawler to obtain all relevant image information in a patent library;

[0010] Step S4: output patent images meeting the preset standard after image retrieval, and comprehensively output text retrieval structure and image retrieval results.

[0011] Preferably, in the step S1:

[0012] A classifier pre-trained by a double-layer vgg deep neural network is used for classification;

[0013] Corresponding rank values are obtained for several preset labels respectively;

[0014] The labels with the top two rank values are taken as initial input data for text information processing.

[0015] Preferably, in the step S2:

[0016] A user-defined dictionary is defined to define biological nouns and industrial field nouns, the jieba.load_userdict function is used to load the user-defined dictionary, and a TF-IDF algorithm is used to count word weights in patents;

[0017] The keywords with a frequency higher than the preset standard are obtained as input for image retrieval;

[0018] The text retrieval part comprises the following steps:

[0019] Step a1: using a web crawler to crawl all text results in an authorized patent library according to the input labels, and grouping the text results into long texts;

[0020] Step a2: using a word segmentation technology, using a word segmentation library according to corresponding languages, using a Jieba Chinese word segmentation library for Chinese text patents for word segmentation, and putting preset nouns in the word segmentation results through Jieba by using a user-defined dictionary;

[0021] Step a3: For the word segmentation result, the TF-IDF algorithm is used for word frequency statistics;

[0022] Step a4: The word segmentation result of TF-IDF is used to generate a word cloud diagram through a word cloud diagram generation interface to reflect the designable focus of the query label, and a word cloud diagram of verbs and biological nouns is generated respectively.

[0023] Preferably, in the step S3:

[0024] The overall steps of the image retrieval part are as follows:

[0025] Step b1: Using a web crawler to crawl all image results in the authorized patent library according to the input label;

[0026] Step b2: The image type is identified through the first layer classification network, the image is identified as a chart, an actual image and a patent image, and screening is performed accordingly;

[0027] Step b3: The second layer classification network identifies the image content, judges whether the image content corresponds to the input provided label, and screens accordingly, screens out the results corresponding to the input label, and the last layer of the classification network outputs the probability corresponding to each category, and sets the top three output probability or sets the probability threshold;

[0028] Step b4: Using the style transfer method to provide the user with an end-to-end patent image;

[0029] Step b5: The results of the text retrieval and the image classification are output.

[0030] Preferably, in the step S4:

[0031] The obtained patent information, word frequency statistical information and picture results are output;

[0032] The kinship map between organisms is fused in the network design, the network structure is determined according to the distance of kinship, so that the network utilizes biological knowledge in the process of feedforward, and artificially fuses the explainable learning mode; the style transfer method is used to directly generate an image similar to the patent image through a neural network, and an image style is fused into another image.

[0033] According to the multi-modal information retrieval system based on the convolutional neural network and the TF-IDF algorithm provided by the application, the system comprises:

[0034] Module M1: inputting an actual image to obtain a label, or directly inputting a label;

[0035] Module M2: obtain relevant text information through the method of text retrieval, and obtain keywords with a frequency higher than a preset standard;

[0036] Module M3: take the classification result and the keywords with a frequency higher than a preset standard after text retrieval as input of image retrieval, and use a web crawler to obtain all relevant image information in a patent library;

[0037] Module M4: output patent images meeting a preset standard after image retrieval, and comprehensively output the text retrieval structure and the image retrieval result.

[0038] Preferably, in the module M1:

[0039] a classifier pre-trained through a double-layer vgg deep neural network is used for classification;

[0040] a corresponding rank value is obtained for each preset label;

[0041] the labels with the top two rank values are taken as initial input data for text information processing.

[0042] Preferably, in the module M2:

[0043] a user-defined dictionary is defined, biological nouns and industrial field nouns are defined, the jieba.load_userdict function is used to load the user-defined dictionary, and a TF-IDF algorithm is used to count the word weight in the patent;

[0044] keywords with a frequency higher than a preset standard are obtained as input of image retrieval;

[0045] The steps of the text retrieval part are as follows:

[0046] Step a1: using a web crawler to crawl all text results in the authorized patent library according to the input label, and grouping the text results into long texts;

[0047] Step a2: using a word segmentation technology, using a word segmentation library according to the corresponding language, using a Jieba Chinese word segmentation library for Chinese text in the patent for word segmentation, and putting the preset nouns in the word segmentation result through Jieba according to the user-defined dictionary;

[0048] Step a3: for the word segmentation result, a TF-IDF algorithm is used for word frequency statistics;

[0049] Step a4: generating a word cloud diagram through a word cloud diagram generation interface for the TF-IDF word segmentation result, reflecting the designable key points of the queried label, and generating a word cloud diagram of verbs and biological nouns respectively.

[0050] Preferably, in the module M3:

[0051] The overall steps of the image retrieval part are as follows:

[0052] Step b1: using a web crawler to crawl all image results according to the input label in the authorized patent library;

[0053] Step b2: identifying the image type through a first layer of classification network, identifying the image as a chart, an actual image and a patent image, and screening according to the same;

[0054] Step b3: a second layer of classification network identifies the image content, judges whether the image content corresponds to the input provided label, and screens according to the same, screens out the results corresponding to the input label, the last layer of the classification network outputs the probability corresponding to each category, and sets the top three of the output probability or sets the probability threshold;

[0055] Step b4: using a style transfer method to provide an end-to-end patent image for the user;

[0056] Step b5: comprehensively outputting the results of text retrieval and the results after image classification.

[0057] Preferably, in the module M4:

[0058] The obtained patent information, word frequency statistical information and picture results are outputted;

[0059] The kinship map between organisms is fused in the network design, the network structure is formulated according to the distance of kinship, so that the network utilizes biological knowledge in the process of feedforward, and artificially fuses an interpretable learning mode; a style transfer method is used to directly generate an image similar to a patent image through a neural network, and a kind of image style is fused into another image.

[0060] Compared with the prior art, the present application has the following beneficial effects:

[0061] 1. The present application can quickly retrieve multi-modal fusion useful information to support the field of artificial intelligence in bionic design;

[0062] 2. The present application fully excavates the characteristics of existing knowledge, and provides a cross-modal knowledge set for the design process to provide analog solution generation. BRIEF DESCRIPTION OF DRAWINGS

[0063] Other features, objects and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments, made with reference to the accompanying drawings:

[0064] Fig. 1 The method framework of the present application is shown in the figure;

[0065] Fig. 2The application relates to an image retrieval part applied to surgical anastomat patent image retrieval.

[0066] Fig. 3 A method schematic diagram of the image retrieval part. DETAILED DESCRIPTION

[0067] The application will be described in detail below with specific examples. The following examples will help those skilled in the art to further understand the application, but do not limit the application in any form. It should be pointed out that, for those skilled in the art, without departing from the concept of the application, a number of changes and improvements can be made. These all belong to the protection scope of the application.

[0068] Example 1

[0069] We propose a method that can conveniently retrieve multi-modal information, which contains text retrieval based on TF-IDF algorithm and image retrieval based on convolutional neural network. The method supports inputting original images or keywords, and outputs multi-modal information through text retrieval and image retrieval to stimulate bionic design inspiration.

[0070] According to the multi-modal information retrieval method based on a convolutional neural network and a TF-IDF algorithm, as shown in the figure, Figs. 1-3 the method comprises the following steps.

[0071] Step S1: inputting an actual image to obtain a label, or directly inputting the label;

[0072] Specifically, in the step S1:

[0073] A classifier obtained by pre-training through a double-layer vgg deep neural network is used for classification.

[0074] Corresponding rank values are obtained for several preset labels.

[0075] The labels with the top two rank values are taken as the initial input data for text information processing.

[0076] Step S2: obtaining relevant text information through a text retrieval method, and obtaining keywords with a frequency higher than a preset standard;

[0077] Specifically, in the step S2:

[0078] A user-defined dictionary is defined, biological nouns and industrial field nouns are defined, a jieba.load_userdict function is used to load the user-defined dictionary, and a TF-IDF algorithm is used to count the word weight in the patent.

[0079] The keywords with the frequency higher than the preset standard are obtained as the input for image retrieval.

[0080] The text retrieval part steps as follows:

[0081] Step a1: using a web crawler to crawl all text results in the authorized patent library according to the input label, and composing long text from the text results;

[0082] Step a2: using a word segmentation technology, adopting a word segmentation library according to the corresponding language, and adopting a Jieba Chinese word segmentation library for Chinese text patents for word segmentation, and putting the preset nouns in the word segmentation results through Jieba by a self-defined dictionary;

[0083] Step a3: for the word segmentation results, adopting a TF-IDF algorithm for word frequency statistics;

[0084] Step a4: generating a word cloud map through a word cloud map generation interface for the TF-IDF word segmentation results, reflecting the designable key points of the queried label, and generating a word cloud map of verbs and biological nouns respectively.

[0085] Step S3: taking the classification results and the keywords with a frequency higher than a preset standard after the text retrieval as inputs of image retrieval, and using a web crawler to obtain all relevant image information in the patent library;

[0086] Specifically, in the step S3:

[0087] The overall steps of the image retrieval part are as follows:

[0088] Step b1: using a web crawler to crawl all image results in the authorized patent library according to the input label;

[0089] Step b2: identifying the image type through a first layer of classification network, identifying the image as a chart, an actual image and a patent image, and screening according to the same;

[0090] Step b3: a second layer of classification network identifies the image content, judges whether the image content corresponds to the input provided label, and screens according to the same, screens out the results corresponding to the input label, and the last layer of the classification network outputs the probability corresponding to each category, and sets the top three of the output probability or sets the probability threshold;

[0091] Step b4: using a style transfer method to provide an end-to-end patent image for the user;

[0092] Step b5: comprehensively outputting the results of text retrieval and image classification.

[0093] Step S4: outputting the patent images meeting the preset standard after image retrieval, and comprehensively outputting the text retrieval structure and the image retrieval results.

[0094] Specifically, in the step S4:

[0095] The obtained patent information, word frequency statistical information and picture results are outputted.

[0096] The genetic relationship map between organisms is fused in network design, and the network structure is formulated according to the distance of genetic relationship, so that the network utilizes biological knowledge in the process of feedforward and artificially fuses the explainable learning mode; the method of style transfer is used to directly generate an image similar to the patent image to a preset standard through a neural network, and a kind of image style set is fused into another image.

[0097] Embodiment 2:

[0098] Embodiment 2 is a preferred example of Embodiment 1, and is used to more specifically illustrate the present application.

[0099] The present application also provides a multi-modal information retrieval system based on a convolutional neural network and a TF-IDF algorithm, which can be realized by executing the process steps of the multi-modal information retrieval method based on the convolutional neural network and the TF-IDF algorithm, that is, the multi-modal information retrieval method based on the convolutional neural network and the TF-IDF algorithm can be understood by those skilled in the art as a preferred embodiment of the multi-modal information retrieval system based on the convolutional neural network and the TF-IDF algorithm.

[0100] According to the present application, a multi-modal information retrieval system based on a convolutional neural network and a TF-IDF algorithm is provided, which comprises:

[0101] Module M1: input actual image to obtain label, or directly input label;

[0102] Specifically, in the module M1:

[0103] A classifier pre-trained by a double-layer vgg deep neural network is used for classification;

[0104] For several preset labels, corresponding rank values are obtained respectively;

[0105] The labels with the top two rank values are taken as the initial input data for text information processing.

[0106] Module M2: obtain relevant text information by a text retrieval method, and obtain keywords with a frequency higher than a preset standard;

[0107] Specifically, in the module M2:

[0108] Custom dictionary, define biological terms and industrial terms; use jieba.load_userdict function to load custom dictionary; use TF-IDF algorithm to count word weight in patent;

[0109] Obtain keywords with a frequency higher than a preset standard as input for image retrieval;

[0110] The text retrieval part steps as follows:

[0111] Step a1: use a web crawler to crawl all text results in the authorized patent library according to the input label, and compose the text results into long text;

[0112] Step a2: use word segmentation technology, adopt a word segmentation library according to the corresponding language, and use Jieba Chinese word segmentation library for Chinese text patents for word segmentation; through a custom dictionary, preset nouns in the word segmentation result are proposed by Jieba;

[0113] Step a3: for the word segmentation result, use TF-IDF algorithm to count word frequency;

[0114] Step a4: generate a word cloud map through a word cloud map generation interface for the TF-IDF word segmentation result, reflect the designable focus of the query label, and generate a word cloud map of verbs and biological nouns respectively.

[0115] Module M3: use the classification result and the keywords with a frequency higher than a preset standard after text retrieval as input for image retrieval, and use a web crawler to obtain all relevant image information in the patent library;

[0116] Specifically, in the module M3:

[0117] The overall steps of the image retrieval part are as follows:

[0118] Step b1: use a web crawler to crawl all image results in the authorized patent library according to the input label;

[0119] Step b2: identify the image type through a first layer of classification network, identify the image as a chart, an actual image and a patent image, and filter accordingly;

[0120] Step b3: the second layer of classification network identifies the image content, judges whether the image content corresponds to the input provided label, and filters accordingly, filters out the results corresponding to the input label, the last layer of the classification network outputs the probability corresponding to each category, and sets the top three output probability or sets the probability threshold;

[0121] Step b4: use style transfer method to provide end-to-end patent image for users;

[0122] Step b5: output the results of text retrieval and image classification.

[0123] Module M4: output patent images that meet the preset standard after image retrieval, and output the results of text retrieval and image retrieval.

[0124] Specifically, in the module M4:

[0125] Output the obtained patent information, word frequency statistical information, and picture results;

[0126] Fuse the kinship map between organisms in network design. According to the distance of kinship, design the network structure, so that the network utilizes biological knowledge in the process of feedforward, and artificially fuses the explainable learning mode. Use the style transfer method to directly generate an image that meets the preset standard of similarity with the patent image through the neural network, and fuse a certain set of image style into another image.

[0127] Embodiment 3:

[0128] Embodiment 3 is a preferred example of embodiment 1, which more specifically illustrates the present application.

[0129] An artificial intelligence field method capable of quickly retrieving multi-modal fusion useful information to support bionic design. It includes: proposing an innovative information retrieval model. Adopting an artificial intelligence field method capable of quickly retrieving multi-modal fusion useful information to support bionic design, researching single or multi-modal knowledge acquisition method driven by knowledge collaborative innovation between biological field and engineering field, introducing text knowledge acquisition and acquisition into bionic design process, fully exploring the characteristics of existing knowledge, and providing cross-modal knowledge set for design process and analog solution generation.

[0130] The steps include:

[0131] Step one: input the actual image of a certain organism to obtain its label through a classification network, or directly input its label; use the label as the input of the subsequent part

[0132] Step two: obtain relevant text information through text retrieval method, and obtain high-frequency keywords as image retrieval input;

[0133] Step three: use the classification result and high-frequency keywords after text retrieval as image retrieval input

[0134] Step four: use a web crawler to obtain all relevant image information in the patent library

[0135] Step five: output useful patent images after image retrieval

[0136] Step six: integrate the output text retrieval structure and image retrieval results

[0137] The step one further comprises the following steps:

[0138] (I) using the classifier pre-trained by the double-layer vgg deep neural network for classification

[0139] (II) obtaining the corresponding rank value for each preset label

[0140] The higher the rank value, the more similar the label is to the picture content.

[0141] (III) taking the top two labels of the rank value as the initial input data for text information processing

[0142] The step two further comprises the following steps:

[0143] (I) defining a dictionary, which defines more biological nouns and industry-specific terms

[0144] (II) using the jieba.load_userdict function to load the custom dictionary

[0145] (III) using the TF-IDF algorithm to calculate the word weight in the patent

[0146] (IV) obtaining high-frequency keywords as input for image retrieval

[0147] Using this method, the segmentation results of the segmenter are more in line with expectations.

[0148] The step six further comprises the following steps:

[0149] (I) outputting the patent information, word frequency statistical information and picture results obtained before

[0150] (II) the part that can be expanded (currently not well implemented): integrating the phylogenetic relationship map between organisms into the network design, formulating a special network structure according to the distance of the phylogenetic relationship, so that the network can better utilize biological knowledge in the process of feedforward, that is, artificially integrating interpretable learning mode; using the style transfer method to directly generate an image similar to the patent image through the neural network, and integrating a certain image style into another image.

[0151] Example 4

[0152] Example 4 is a preferred example of example 1, which more specifically illustrates the present application.

[0153] The method integrates text information and image information according to input image content: inputting an actual image of a certain organism to obtain its label through a classification network, or directly inputting the label; using the label as the input of the subsequent part.

[0154] The overall method steps are as follows:

[0155] Step 1: Obtain relevant text information through the method of text retrieval, and the high-frequency keywords obtained can also be used as the input of image retrieval;

[0156] Step 2: The classification result and the high-frequency keywords after text retrieval can be used as the input of image retrieval. First, use a web crawler to obtain all relevant image information in the patent library, and output useful patent images after image retrieval;

[0157] Step 3: Integrate and output the text retrieval structure and the image retrieval result.

[0158] The overall steps of the text retrieval part are as follows:

[0159] Step a1: Use a web crawler to crawl all text results in the authorized patent library or other legal databases according to the input label. Combine them into a long text and wait for processing.

[0160] Step a2: Use word segmentation technology (use appropriate word segmentation library according to the corresponding language, and use Jieba Chinese word segmentation library for Chinese text in this patent), and through a self-defined dictionary, put the verbs, biological nouns, etc. in the word segmentation result which are the focus of this patent through Jieba, and wait for the next step of processing.

[0161] Step a3: For the word segmentation result, use TF-IDF algorithm to count the word frequency. The higher the weight of a word, the more likely it is to provide inspiration for the designer's design.

[0162] Step a4: Generate a word cloud diagram of the TF-IDF word segmentation result by calling a word cloud generation interface, which intuitively reflects the designable key points of the queried label. In addition, generate word cloud diagrams of verbs and biological nouns to focus on the functionality and bionics principles of the design.

[0163] The overall steps of the image retrieval part are as follows:

[0164] Step b1: Use a web crawler to crawl all image results in the authorized patent library or other legal databases according to the input label. At this time, the images contain a large number of results that do not meet the expectations.

[0165] Step b2: Identify the image type through the first layer of classification network, i.e. identify the image as a chart, an actual image, a patent image, etc., and filter accordingly, such as needing a patent image, then filter out all patent images.

[0166] Step b3: The second layer of the classification network identifies the image content and determines whether the image content corresponds to the input provided label, and filters accordingly, filtering out the results corresponding to the input label. It is worth noting that the last layer of the classification network will output the probability corresponding to each category, and the strict classifier will output the category with the highest probability as the result, which is reasonable and correct. However, in the knowledge of bionics, many organisms have a kinship or similar structure, and the corresponding category scores of these organisms may have similar high output probabilities. The text information or image information corresponding to these categories may have a certain stimulating effect on bionic design, and this purpose can be achieved by setting the top three output probability rankings or setting a probability threshold or combining both methods. At the same time, in the case of insufficient data, the network model obtained by training may be under-fitted, and multiple similar probabilities may be generated. The results of this feature are meaningful in bionic design, and can obtain results of other categories but similar structures to stimulate innovation in design.

[0167] Among them: steps b2 and b3 can use existing classifiers based on convolutional neural networks with excellent performance, such as VGG-net. The method has been verified on VGGnet-19, and its classification performance is sufficient to support this method.

[0168] Step b4: Use style transfer method to provide end-to-end patent images for users. The patent images generated in this step are completely novel, but may not be strictly patent images. Only part of the image area has a structure similar to a patent image. If you want to better use style transfer to achieve end-to-end output and stimulate inspiration, a special structure of deep learning network needs to be designed.

[0169] Step b5: Integrate the results of text retrieval and image classification.

[0170] Examples of image retrieval and text retrieval methods are given respectively. Since the method is divided, it is not limited to bionic design, and the following examples will be carried out in the background of industrial design. In the case of sufficient training data, the two parts can and should be used as a whole to play their optimal performance.

[0171] Text retrieval method:

[0172] Step 1: Input the label to the program, and the program starts crawling, obtains long text, performs word segmentation, counts word frequency, and returns a word cloud.

[0173] Image retrieval method:

[0174] Step 1: design the number of labels for the classifier, and collect a large amount of training data for the classifier according to the number of labels, and train with cross-entropy as the loss function.

[0175] The cross-entropy loss function is:

[0176] Step 2: Take the surgical stapler and the mechanical arm as an example. We use the collected images to classify and train on the two classifications. The training epoch number and learning rate and other hyperparameters are determined according to the number of classifications.

[0177] Step 3: Input the search image to be classified obtained by the crawler into the trained neural network to obtain the output result.

[0178] Those skilled in the art know that in addition to implementing the system, device and each module thereof provided by the present application in the form of pure computer readable program code, the same program can be realized by logically programming the method steps in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers. Therefore, the system, device and each module thereof provided by the present application can be considered as a hardware component, and the modules included therein for implementing various programs can also be considered as structures within the hardware component; the modules for implementing various functions can also be considered as both software programs for implementing methods and structures within hardware components.

[0179] The specific embodiments of the present application are described above. It should be understood that the present application is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essential content of the present application. In the case of no conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A multimodal information retrieval method based on convolutional neural networks and the TF-IDF algorithm, characterized in that, include: Step S1: Input the actual image to obtain the label, or input the label directly; Step S2: Obtain relevant text information through text retrieval methods, and obtain keywords with a frequency higher than the preset standard; Step S3: Use the classification results and keywords with frequencies higher than the preset standard after text retrieval as input for image retrieval, and use a web crawler to obtain all relevant image information in the patent database; Step S4: After image retrieval, output patent images that meet preset standards, and combine text retrieval results and image retrieval results; In step S3: The overall steps of image retrieval are as follows: Step b1: Use a web crawler to retrieve all image results from the authorized patent database based on the input tags; Step b2: Identify image types using the first-layer classification network, classifying images as charts, actual images, and patent images, and then filter them accordingly; Step b3: The second-layer classification network identifies the image content, determines whether the image content corresponds to the input label, and filters accordingly, selecting the results that correspond to the input label. The output of the last layer of the classification network corresponds to the probability of each category. The output probabilities are sorted into the top three or a probability threshold is set. Step b4: Use style transfer methods to provide end-to-end patented images to users; Step b5: Combine the results of text retrieval with the results of image classification.

2. The multimodal information retrieval method based on convolutional neural networks and TF-IDF algorithm according to claim 1, characterized in that, In step S1: Classification is performed using a classifier pre-trained with a two-layer VGG deep neural network; Obtain the corresponding rank value for each of the several preset labels; The labels with the top two rank values ​​are taken as the initial input data for text information processing.

3. The multimodal information retrieval method based on convolutional neural networks and TF-IDF algorithm according to claim 1, characterized in that, In step S2: A custom dictionary was created, defining biological and industrial terms; the custom dictionary was loaded using the jieba.load_userdict function; and the TF-IDF algorithm was used to calculate the word weights in the patents. Keywords with a frequency higher than a preset standard are obtained and used as input for image retrieval; The steps involved in text retrieval are as follows: Step a1: Use a web crawler to crawl all text results in the authorized patent database based on the input tags, and combine the text results into a long text; Step a2: Using word segmentation technology, a word segmentation library is adopted according to the corresponding language. For Chinese text, this patent uses the Jieba Chinese word segmentation library for word segmentation. Through a custom dictionary, the preset nouns in the word segmentation results are extracted by Jieba. Step a3: For the word segmentation results, use the TF-IDF algorithm to perform word frequency statistics; Step a4: Generate word cloud diagrams from the TF-IDF segmentation results by calling the word cloud diagram generation interface, reflecting the design focus of the queried tags, and generate word cloud diagrams for verbs and biological nouns respectively.

4. The multimodal information retrieval method based on convolutional neural networks and TF-IDF algorithm according to claim 1, characterized in that, In step S4: Output the obtained patent information, word frequency statistics, and image results; By integrating kinship maps between organisms into network design, the network structure is determined based on the closeness of kinship, enabling the network to utilize biological knowledge during the feedforward process and artificially integrate interpretable learning patterns; using style transfer methods, images that are similar to patent images to a preset standard are directly generated through neural networks, integrating a pre-defined image style into another image.

5. A multimodal information retrieval system based on convolutional neural networks and the TF-IDF algorithm, characterized in that, include: Module M1: Input the actual image to obtain the label, or input the label directly; Module M2: Obtains relevant text information through text retrieval methods, identifying keywords with a frequency higher than the preset standard; Module M3: Uses the classification results and keywords with frequencies higher than the preset standard after text retrieval as input for image retrieval, and uses a web crawler to obtain all relevant image information in the patent database; Module M4: After image retrieval, it outputs patent images that meet preset standards, and combines text retrieval results with image retrieval results; In module M3: The overall steps of image retrieval are as follows: Step b1: Use a web crawler to retrieve all image results from the authorized patent database based on the input tags; Step b2: Identify image types using the first-layer classification network, classifying images as charts, actual images, and patent images, and then filter them accordingly; Step b3: The second-layer classification network identifies the image content, determines whether the image content corresponds to the input label, and filters accordingly, selecting the results that correspond to the input label. The output of the last layer of the classification network corresponds to the probability of each category. The output probabilities are sorted into the top three or a probability threshold is set. Step b4: Use style transfer methods to provide end-to-end patented images to users; Step b5: Combine the results of text retrieval with the results of image classification.

6. The multimodal information retrieval system based on convolutional neural networks and TF-IDF algorithm according to claim 5, characterized in that, In module M1: Classification is performed using a classifier pre-trained with a two-layer VGG deep neural network; Obtain the corresponding rank value for each of the several preset labels; The labels with the top two rank values ​​are taken as the initial input data for text information processing.

7. The multimodal information retrieval system based on convolutional neural networks and TF-IDF algorithm according to claim 5, characterized in that, In module M2: A custom dictionary was created, defining biological and industrial terms; the custom dictionary was loaded using the jieba.load_userdict function; and the TF-IDF algorithm was used to calculate the word weights in the patents. Keywords with a frequency higher than a preset standard are obtained and used as input for image retrieval; The steps involved in text retrieval are as follows: Step a1: Use a web crawler to crawl all text results in the authorized patent database based on the input tags, and combine the text results into a long text; Step a2: Using word segmentation technology, a word segmentation library is adopted according to the corresponding language. For Chinese text, this patent uses the Jieba Chinese word segmentation library for word segmentation. Through a custom dictionary, the preset nouns in the word segmentation results are extracted by Jieba. Step a3: For the word segmentation results, use the TF-IDF algorithm to perform word frequency statistics; Step a4: Generate word cloud diagrams from the TF-IDF segmentation results by calling the word cloud diagram generation interface, reflecting the design focus of the queried tags, and generate word cloud diagrams for verbs and biological nouns respectively.

8. The multimodal information retrieval system based on convolutional neural networks and TF-IDF algorithm according to claim 5, characterized in that, In module M4: Output the obtained patent information, word frequency statistics, and image results; By integrating kinship maps between organisms into network design, the network structure is determined based on the closeness of kinship, enabling the network to utilize biological knowledge during the feedforward process and artificially integrate interpretable learning patterns; using style transfer methods, images that are similar to patent images to a preset standard are directly generated through neural networks, integrating a pre-defined image style into another image.

Images