Archive information retrieval method based on multi-modal model

Abstract

The invention relates to the technical field of electrical digital processing, and discloses a multi-modal model-based archive information retrieval method, which comprises the following steps of: processing a multi-modal archive to obtain a corresponding digital archive; inputting the digital archive into a multi-modal model for identification and classification to obtain corresponding text type description information; based on the descriptive text, constructing an archive database according to a preset conversion rule, and then embedding a feature tag; after training of the neural network model is completed according to the archive database, a retrieval model is obtained; converting the input text type retrieval information into a retrieval picture according to a preset conversion rule; obtaining a retrieval result according to an output result of the retrieval model; according to the method, the retrieval difficulty of files in formats such as pictures, audios and videos can be reduced, file data contents in document formats can be simplified, the retrieval accuracy is continuously improved and ensured, and the retrieval cost and threshold of the multi-modal files are also reduced.

Description

Technical Field

[0001] The present application relates to the technical field of electronic digital processing, and in particular to an archival information retrieval method based on a multimodal model. Background Art

[0002] With the rapid development of information technology, archival information retrieval technology is also constantly improving. Traditional information retrieval is usually based on text data, by inputting textual content such as keywords, phrases or paragraphs, and using natural language processing models for comparison and retrieval. However, this single text retrieval method is incapable of processing archival information containing rich multimedia information.

[0003] Archive information often contains a large amount of non-text data such as pictures, charts, audio, and video, which are crucial for a comprehensive and accurate understanding of the archive content. Traditional text retrieval technology often ignores non-text modal information such as pictures and charts when processing these multimedia data, resulting in incomplete and inaccurate retrieval results.

[0004] A multimodal model is an artificial intelligence model that can process and understand multiple different types of input data (such as text, images, audio, etc.). It can fully understand and process diversified information by learning and integrating the characteristics of different modalities (i.e. data types). These models can be pre-trained on large-scale data to learn the semantic correspondence between different modalities, thereby more accurately understanding and processing multimodal information.

[0005] In this way, this type of multimodal model can effectively guarantee efficiency and accuracy when facing archival retrieval of a certain amount of data. However, when faced with growing multimodal archival data, its retrieval efficiency and accuracy will gradually decline. Moreover, the training threshold of the multimodal model is high, requiring higher computing resources and professional skills, which may be difficult for small and medium-sized enterprises and institutions. Summary of the invention

[0006] In order to lower the threshold for multimodal data retrieval, improve retrieval response efficiency and ensure retrieval accuracy, the present application provides an archival information retrieval method based on a multimodal model.

[0007] The present application provides an archival information retrieval method based on a multimodal model, which adopts the following technical solutions:

[0008] A multimodal model-based archival information retrieval method, comprising:

[0009] After processing the multimodal archives, the corresponding digital archives are obtained;

[0010] Inputting the digital archive into a multimodal model for identification and classification to obtain corresponding textual description information; the description information includes descriptive text reflecting the content of the multimodal archive and corresponding feature tags;

[0011] Based on the descriptive text, a file database is constructed according to preset conversion rules and then corresponding feature tags are embedded;

[0012] After completing the training of the neural network model according to the archive database, a retrieval model is obtained;

[0013] Convert the input text-based search information into a search image according to the preset conversion rules;

[0014] Sending the search image to the search model, and obtaining a search result according to an output result of the search model;

[0015] Among them, the multimodal archives include paper documents, electronic documents, pictures, audio and video formats.

[0016] By adopting the above technical scheme, the present invention can first cut the multimodal archives into data sizes that can be undertaken by the existing multimodal large model after digitizing and denoising the multimodal archives, perform segmented identification and classification, understand and describe the multimodal archives in digital form, and convert them into descriptive texts of a general nature, and then construct an archive database in the form of a picture collection with corresponding labels according to preset conversion rules, so as to achieve the training and use of the neural network model that can be adapted. On the one hand, it can reduce the difficulty of retrieving archives in formats such as pictures, audio, and video, and on the other hand, it can streamline the archive data content in document format. During use, the retrieval model based on the neural network model can be continuously optimized and trained as the total amount of archives increases, thereby continuously improving and ensuring the accuracy of retrieval, and also reducing the retrieval cost and threshold of multimodal archives.

[0017] Optionally, the preset conversion rules include:

[0018] After dividing the text data into sentences, perform vocabulary splitting on the sentences;

[0019] Performing part-of-speech tagging and syntactic analysis on the vocabulary to obtain a semantic two-dimensional pixel code and a logical two-dimensional pixel code corresponding to the vocabulary; the logical two-dimensional pixel code represents a logical relationship between the vocabulary;

[0020] The conversion is completed after the meaning two-dimensional pixel code and the logic two-dimensional pixel code of the vocabulary corresponding to the sentence are loaded into a blank picture in text order.

[0021] Optionally, the method for embedding a feature tag includes:

[0022] Select the area where the two-dimensional pixel code corresponding to the feature label is located;

[0023] Select the designated position of the area as the starting position, take the outline of the area as the route, and establish a unidirectional pixel code line clockwise;

[0024] Wherein, the color sequence of the pixels in the pixel code line uniquely corresponds to the feature label;

[0025] If the feature label includes subdivision categories, the subdivision categories correspond to multiple pixel bars in the pixel code line respectively, and the subdivision category corresponding to the pixel bar closer to the starting position has a higher weight in the sentence.

[0026] Optionally, the semantic two-dimensional pixel code is a rectangular matrix dot code with an X*Y pixel size specification, and the logical two-dimensional pixel code is a rectangular matrix dot code with an A*Y pixel size specification;

[0027] The pixel colors in the word meaning two-dimensional pixel code and the logic two-dimensional pixel code and the arrangement order between different pixels respectively represent the word meaning and the logical relationship between the preceding and following words.

[0028] Optionally, the training process of the neural network model includes:

[0029] Divide the training process into training stages in advance;

[0030] Obtaining a classification accuracy change curve of each feature label in the current training phase;

[0031] For a feature tag, obtain the corresponding current training state evaluation score, and configure the training method for the next training stage according to the current training state evaluation score;

[0032]

[0033] Among them, T j is the jth training stage, Fi(T j ) is the training status evaluation score of the i-th feature label in the corresponding training stage, ΔDi is the improvement of the classification accuracy of the i-th feature label compared with the previous training stage, Dsi(T j ) is the target classification accuracy of the i-th feature label in the j-th training stage, Di(T j ) corresponds to the actual classification accuracy of the i-th feature label in the training stage, Oi(T j+1 ) is the sample size ratio of the samples corresponding to the ith feature label in the j+1th training stage, and Otij is the training time ratio of the samples corresponding to the ith feature label in the jth training stage.

[0034] Optionally, the target classification accuracy Dsi(T j+1 ) is configured as follows:

[0035]

[0036] Among them, mi(T j ) is the number of subdivision categories contained in the i-th feature label in the j-th training stage, XY(T j ) is the pixel capacity of a single rectangular matrix dot code of the sample in the jth training stage.

[0037] Optionally, in the method for obtaining the search result according to the output result of the search model:

[0038] The output result is C feature labels arranged in descending order of classification probability;

[0039] Retrieve digital files containing C feature tags at the same time, and then sort them in descending order according to the number of specified subdivision categories simultaneously contained;

[0040] The specified subdivision category is a subdivision category in the feature label that meets a preset probability ratio.

[0041] Optionally, the method of obtaining a corresponding digital archive after processing the multimodal archive includes:

[0042] Digitalizing the paper documents and converting them into digital form for storage;

[0043] Summarizing and generalizing the electronic document and storing it in digital form;

[0044] The documents in the form of pictures, audio and video are cropped and noise-reduced and then stored in digital form.

[0045] In summary, the present application includes at least one of the following beneficial technical effects:

[0046] The present invention digitizes and performs noise reduction on multimodal archives, and then cuts them into data sizes that can be handled by existing multimodal large models, performs segmented identification and classification, understands and describes the multimodal archives in digital form, and converts them into descriptive texts of a general nature. Then, an archive database in the form of a picture collection with corresponding labels is constructed according to preset conversion rules, so as to achieve the goal of being able to adapt to the training and use of neural network models. On the one hand, it can reduce the difficulty of retrieving archives in formats such as pictures, audio, and video, and on the other hand, it can streamline the archive data content in document format. During use, the retrieval model based on the neural network model can be continuously optimized and trained as the total amount of archives increases, thereby continuously improving and ensuring the accuracy of retrieval, and also reducing the retrieval cost and threshold of multimodal archives. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is an example diagram of the archival information retrieval method based on the multimodal model of the present invention. DETAILED DESCRIPTION

[0048] Embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings.

[0049] In the description of this specification, the description with reference to the terms "certain embodiments", "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0050] The present application embodiment discloses a method for archival information retrieval based on a multimodal model, referring to Figure 1 ,include:

[0051] After processing the multimodal archives, the corresponding digital archives are obtained;

[0052] Inputting the digital archive into a multimodal model for identification and classification to obtain corresponding textual description information; the description information includes descriptive text reflecting the content of the multimodal archive and corresponding feature tags;

[0053] Based on the descriptive text, a file database is constructed according to preset conversion rules and then corresponding feature tags are embedded;

[0054] After completing the training of the neural network model according to the archive database, a retrieval model is obtained;

[0055] Convert the input text-based search information into a search image according to the preset conversion rules;

[0056] Sending the search image to the search model, and obtaining a search result according to an output result of the search model;

[0057] The multimodal archive includes paper documents, electronic documents, pictures, audio, and video. In this embodiment of the present invention, the method for processing the multimodal archive to obtain the corresponding digital archive includes:

[0058] Digitizing the paper documents and converting them into digital form for storage;

[0059] Summarizing and generalizing the electronic document and storing it in digital form;

[0060] The documents in the form of pictures, audio and video are cropped and noise-reduced and then stored in digital form.

[0061] The present invention digitizes and performs noise reduction on multimodal archives, and then cuts them into data sizes that can be handled by existing multimodal large models, performs segmented identification and classification, understands and describes the multimodal archives in digital form, and converts them into descriptive texts of a general nature. Then, an archive database in the form of a picture collection with corresponding labels is constructed according to preset conversion rules, so as to achieve the goal of being able to adapt to the training and use of neural network models. On the one hand, it can reduce the difficulty of retrieving archives in formats such as pictures, audio, and video, and on the other hand, it can streamline the archive data content in document format. During use, the retrieval model based on the neural network model can be continuously optimized and trained as the total amount of archives increases, thereby continuously improving and ensuring the accuracy of retrieval, and also reducing the retrieval cost and threshold of multimodal archives.

[0062] Optionally, the preset conversion rules include:

[0063] After dividing the text data into sentences, perform vocabulary splitting on the sentences;

[0064] Performing part-of-speech tagging and syntactic analysis on the vocabulary to obtain a semantic two-dimensional pixel code and a logical two-dimensional pixel code corresponding to the vocabulary; the logical two-dimensional pixel code represents a logical relationship between the vocabulary;

[0065] The conversion is completed after the meaning two-dimensional pixel code and the logic two-dimensional pixel code of the vocabulary corresponding to the sentence are loaded into a blank picture in text order.

[0066] In this embodiment, after receiving the information, a machine learning algorithm can be used to perform data extraction operations to capture important words in the sentence, and then the important sentences containing the important words are segmented to split the captured sentences into multiple words.

[0067] For example, "The first section of lanes is only for motor vehicles to travel in both directions" will be split into "first section", "of", "lane", "only for", "motor vehicles", "two-way", and "travel". Then, meaningless auxiliary words will be removed. Corresponding to the example above, "of" will be removed, and then the divided words will be tagged with parts of speech. Then, according to the pre-prepared corresponding rules, the split words will be converted into corresponding word meaning two-dimensional pixel codes and logical two-dimensional pixel codes in sequence, and filled in the blank picture; "only for" can be converted into the corresponding logical two-dimensional pixel code to represent the logical relationship between words.

[0068] Optionally, in the method for obtaining the search result according to the output result of the search model:

[0069] The output result is C feature labels arranged in descending order of classification probability;

[0070] Retrieve digital files containing C feature tags at the same time, and then sort them in descending order according to the number of specified subdivision categories simultaneously contained;

[0071] The specified subdivision category is a subdivision category in the feature label that meets a preset probability ratio.

[0072] By adopting the above technical solution, if the search information is A, the output result of the search model is the classification result of A, which is 60% of the a feature label and 40% of the x feature label and the corresponding subcategories. The output result is a digital file containing both the a feature label and the x feature label. If file 1 contains 5 corresponding subcategories and file 2 contains 8 corresponding subcategories, they are sorted and displayed according to the priority of file 2.

[0073] Optionally, the method for embedding a feature tag includes:

[0074] Select the area where the two-dimensional pixel code corresponding to the feature label is located;

[0075] Select the designated position of the area as the starting position, take the outline of the area as the route, and establish a unidirectional pixel code line clockwise;

[0076] Wherein, the color sequence of the pixels in the pixel code line uniquely corresponds to the feature label;

[0077] If the feature label includes subdivision categories, the subdivision categories correspond to multiple pixel bars in the pixel code line respectively, and the subdivision category corresponding to the pixel bar closer to the starting position has a higher weight in the sentence.

[0078] In this embodiment of the present invention, taking "the lane of the first section is only for two-way travel of motor vehicles" as an example, its feature label is "road rules", and its subdivision categories are "motor vehicles" and "two-way travel". In the corresponding area in the picture loaded with the two-dimensional pixel code, the pixel strip related to the "road rules" can be established with the upper left corner of the area as the starting point, such as "color 2011-color 2799-color 2830-color 8166-color 2497-color 0086-color 0000*remaining number of pixels" for representation, and color 0000 can be configured as meaningless filling pixels to completely fill the remaining part of the pixel strip; the pixel strip corresponding to the subdivision category can be established in the pixel strip demarcated area corresponding to the feature label according to the same method, except that the pixel strips corresponding to "motor vehicles" and "two-way travel" are designed to be sorted in front and back. If it is necessary to emphasize motor vehicles, the pixel strips are arranged in the order of "motor vehicles-two-way travel", and if it is necessary to emphasize two-way travel, the pixel strips are arranged in the order of "two-way travel-motor vehicles".

[0079] In this embodiment, the semantic two-dimensional pixel code is a rectangular matrix dot code with an X*Y pixel size specification, and the logical two-dimensional pixel code is a rectangular matrix dot code with an A*Y pixel size specification;

[0080] The pixel colors in the word meaning two-dimensional pixel code and the logic two-dimensional pixel code and the arrangement order between different pixels respectively represent the logical relationship between the word meaning and the previous and following words. X can be 30, Y can be 20, and A can be 2. The configuration size determines the maximum capacity of the meaning expression. The larger the product of X and Y, the more vocabulary meanings can be expressed by this set of word meaning two-dimensional pixel codes, which can also improve the upper limit of recognition accuracy and robustness of the neural network model accordingly. However, it will also reduce the processing efficiency of the neural network model and increase the training time. Therefore, it is necessary to select the configuration according to the specific situation and conduct comprehensive training during the training process.

[0081] Optionally, the training process of the neural network model includes:

[0082] Divide the training process into training stages in advance;

[0083] Obtaining a classification accuracy change curve of each feature label in the current training phase;

[0084] For a feature tag, obtain the corresponding current training state evaluation score, and configure the training method for the next training stage according to the current training state evaluation score;

[0085]

[0086] Among them, T j is the jth training stage, Fi(T j ) is the training status evaluation score of the i-th feature label in the corresponding training stage, ΔDi is the improvement of the classification accuracy of the i-th feature label compared with the previous training stage, Dsi(T j ) is the target classification accuracy of the i-th feature label in the j-th training stage, Di(T j ) corresponds to the actual classification accuracy of the i-th feature label in the training stage, Oi(T j+1 ) is the sample size ratio of the samples corresponding to the i-th feature label in the j+1-th training stage, and Otij is the training time ratio of the samples corresponding to the i-th feature label in the j-th training stage. In this way, the sample size corresponding to different feature labels can be adjusted at different training stages to ensure global training efficiency and accuracy.

[0087] Optionally, the target classification accuracy Dsi(T j+1 ) is configured as follows:

[0088]

[0089] Among them, mi(T j ) is the number of subdivision categories contained in the i-th feature label in the j-th training stage, XY(T j ) is the pixel capacity of a single rectangular matrix dot code of the sample in the jth training stage.

[0090] By adopting the above technical solution, after the training of a certain size of two-dimensional pixel code is completed, when entering the next training stage, it is necessary to select a unified image sample with a new size of two-dimensional pixel code for continued training. At this time, it is necessary to calculate the target classification accuracy Dsi(T j+1 ) are readjusted to ensure global training efficiency and accuracy.

[0091] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for archival information retrieval based on a multimodal model, characterized in that: include: After processing the multimodal archives, the corresponding digital archives are obtained; Inputting the digital archive into a multimodal model for identification and classification to obtain corresponding textual description information; the description information includes descriptive text reflecting the content of the multimodal archive and corresponding feature tags; Based on the descriptive text, a file database is constructed according to preset conversion rules and then corresponding feature tags are embedded; After completing the training of the neural network model according to the archive database, a retrieval model is obtained; Convert the input text-based search information into a search image according to the preset conversion rules; Sending the search image to the search model, and obtaining a search result according to an output result of the search model; Among them, the multimodal archives include paper documents, electronic documents, pictures, audio and video formats.

2. The archival information retrieval method based on a multimodal model according to claim 1, characterized in that: The preset conversion rules include: After dividing the text data into sentences, perform vocabulary splitting on the sentences; Performing part-of-speech tagging and syntactic analysis on the vocabulary to obtain a semantic two-dimensional pixel code and a logical two-dimensional pixel code corresponding to the vocabulary; the logical two-dimensional pixel code represents a logical relationship between the vocabulary; The conversion is completed after the meaning two-dimensional pixel code and the logic two-dimensional pixel code of the vocabulary corresponding to the sentence are loaded into a blank picture in text order.

3. The archival information retrieval method based on a multimodal model according to claim 2 is characterized in that: The method for embedding feature tags comprises: Select the area where the two-dimensional pixel code corresponding to the feature label is located; Select the designated position of the area as the starting position, take the outline of the area as the route, and establish a unidirectional pixel code line clockwise; Wherein, the color sequence of the pixels in the pixel code line uniquely corresponds to the feature label; If the feature label includes subdivision categories, the subdivision categories correspond to multiple pixel bars in the pixel code line respectively, and the subdivision category corresponding to the pixel bar closer to the starting position has a higher weight in the sentence.

4. The archival information retrieval method based on a multimodal model according to claim 2 is characterized in that: The semantic two-dimensional pixel code is a rectangular matrix dot code with an X*Y pixel size specification, and the logical two-dimensional pixel code is a rectangular matrix dot code with an A*Y pixel size specification; The pixel colors in the word meaning two-dimensional pixel code and the logic two-dimensional pixel code and the arrangement order between different pixels respectively represent the word meaning and the logical relationship between the preceding and following words.

5. The archival information retrieval method based on a multimodal model according to claim 4 is characterized in that: The training process of the neural network model includes: Divide the training process into training stages in advance; Obtaining a classification accuracy change curve of each feature label in the current training phase; For a feature tag, obtain the corresponding current training state evaluation score, and configure the training method for the next training stage according to the current training state evaluation score; Among them, T j is the jth training stage, Fi(T j ) is the training status evaluation score of the i-th feature label in the corresponding training stage, ΔDi is the improvement of the classification accuracy of the i-th feature label compared with the previous training stage, Dsi(T j ) is the target classification accuracy of the i-th feature label in the j-th training stage, Di(T j ) corresponds to the actual classification accuracy of the i-th feature label in the training stage, Oi(T j+1 ) is the sample size ratio of the samples corresponding to the ith feature label in the j+1th training stage, and Otij is the training time ratio of the samples corresponding to the ith feature label in the jth training stage.

6. The archival information retrieval method based on a multimodal model according to claim 5 is characterized in that: The target classification accuracy Dsi(T j+1 ) is configured as follows: Among them, mi(T j ) is the number of subdivision categories contained in the i-th feature label in the j-th training stage, XY(T j ) is the pixel capacity of a single rectangular matrix dot code of the sample in the jth training stage.

7. The archival information retrieval method based on a multimodal model according to claim 3 is characterized in that: In the method for obtaining the search result according to the output result of the search model: The output result is C feature labels arranged in descending order of classification probability; Retrieve digital files containing C feature tags at the same time, and then sort them in descending order according to the number of specified subdivision categories simultaneously contained; The specified subdivision category is a subdivision category in the feature label that meets a preset probability ratio.

8. The archival information retrieval method based on a multimodal model according to claim 1 is characterized in that: The method for obtaining a corresponding digital archive after processing a multimodal archive comprises: Digitizing the paper documents and converting them into digital form for storage; Summarizing and generalizing the electronic document and storing it in digital form; The documents in the form of pictures, audio and video are cropped and noise-reduced and then stored in digital form.

Images