An archive information retrieval method based on a multi-modal model

Abstract

The application relates to the technical field of electric digital processing, and discloses a file information retrieval method based on a multi-modal model, which comprises the following steps: obtaining corresponding digital files after processing multi-modal files; inputting the digital files into a multi-modal model for identification and classification to obtain corresponding text type description information; embedding feature labels corresponding to a file database according to preset conversion rules based on the descriptive text; obtaining a retrieval model after training a neural network model based on the file database; converting input text type retrieval information into retrieval pictures according to preset conversion rules; obtaining a retrieval result according to an output result of the retrieval model; the retrieval difficulty of file archives in formats such as pictures, audio and video can be reduced, the file data content in the document format can be simplified, the accuracy of retrieval can be continuously improved and ensured, and the retrieval cost and threshold of multi-modal archives are reduced.

Description

Technical Field

[0001] This application relates to the technical field of electronic digital processing, and in particular to a method for retrieving archival information based on a multimodal model. Background Technology

[0002] With the rapid development of information technology, archival information retrieval technology is also constantly improving. Traditional information retrieval is usually based on text data, which involves inputting keywords, phrases, or paragraphs and then using natural language processing models for comparison and retrieval. However, this single text retrieval method is inadequate when processing archival information that contains rich multimedia information.

[0003] Archival information often contains a large amount of non-textual data such as images, charts, audio, and video, which is crucial for a comprehensive and accurate understanding of the archival content. Traditional text retrieval techniques often overlook non-textual modalities such as images and charts when processing this multimedia data, leading to incomplete and inaccurate search results.

[0004] Multimodal models are artificial intelligence models capable of processing and understanding various types of input data (such as text, images, and audio). By learning and integrating the features of different modalities (i.e., data types), they achieve a comprehensive understanding and processing of diverse information. These models can be pre-trained on large-scale data to learn the semantic correspondences between different modalities, thereby more accurately understanding and processing multimodal information.

[0005] Thus, such multimodal models can effectively guarantee efficiency and accuracy when retrieval a certain amount of data in archives. However, when faced with the ever-increasing amount of multimodal archive data, their retrieval efficiency and accuracy will gradually decline. Moreover, the training threshold for multimodal models is relatively high, requiring significant computing resources and professional skills, which may pose certain difficulties for small and medium-sized enterprises and institutions. Summary of the Invention

[0006] To lower the barrier to multimodal data retrieval, improve retrieval efficiency, and ensure retrieval accuracy, this application provides a method for retrieving archival information based on a multimodal model.

[0007] This application provides a method for retrieving archival information based on a multimodal model, employing the following technical solution:

[0008] A method for retrieving archival information based on a multimodal model, comprising:

[0009] The corresponding digital files are obtained after processing the multimodal files;

[0010] The digital archives are input into a multimodal model for recognition and classification to obtain corresponding textual description information; the description information includes descriptive text reflecting the content of the multimodal archives and corresponding feature tags;

[0011] Based on the descriptive text, an archive database is constructed according to preset conversion rules, and then corresponding feature tags are embedded.

[0012] After training the neural network model based on the aforementioned archive database, a retrieval model is obtained;

[0013] The input text-based search information is converted into search images according to the preset conversion rules;

[0014] The search image is sent to the search model, and the search results are obtained based on the output of the search model.

[0015] The multimodal archives include paper documents, electronic documents, images, audio, and video formats.

[0016] By adopting the above technical solution, this invention, after digitizing and denoising multimodal archives, can first cut them into data sizes that conform to the capacity of existing large multimodal models, perform segmented recognition and classification, understand and describe the digital multimodal archives, and convert them into descriptive text of a general nature. Then, according to preset conversion rules, it constructs an archive database in the form of image sets with corresponding tags, in order to be suitable for training and use of neural network models. On the one hand, it can reduce the difficulty of searching for archives in formats such as images, audio, and video; on the other hand, it can simplify the content of archive data in document formats. During use, as the total number of archives increases, the retrieval model based on the neural network model can be continuously optimized and trained, thereby continuously improving and ensuring the accuracy of retrieval, and reducing the retrieval cost and threshold of multimodal archives.

[0017] Optionally, the preset conversion rules include:

[0018] After segmenting the text data into sentences, the sentences are then broken down into words.

[0019] Part-of-speech tagging and syntactic analysis are performed on the vocabulary to obtain the semantic two-dimensional pixel code and logical two-dimensional pixel code corresponding to the vocabulary; the logical two-dimensional pixel code represents the logical relationship between the vocabulary.

[0020] The conversion is completed by loading the semantic two-dimensional pixel codes and logical two-dimensional pixel codes of the words corresponding to the statement into a blank image in the order of the text.

[0021] Optionally, the method for embedding feature labels includes:

[0022] Select the region containing the two-dimensional pixel code corresponding to the feature label;

[0023] Select a specified position in the region as the starting position, and use the outline of the region as the route to establish a unidirectional pixel code line in a clockwise direction;

[0024] The color order of the pixels in the pixel code line uniquely corresponds to the feature label;

[0025] If the feature label includes subcategories, the subcategories correspond to multiple pixel segments in the pixel code line, and the subcategories corresponding to the pixel segments closer to the starting position have a higher weight in the statement.

[0026] Optionally, the semantic two-dimensional pixel code is a rectangular matrix dot code with an X*Y pixel size, and the logical two-dimensional pixel code is a rectangular matrix dot code with an A*Y pixel size.

[0027] The pixel colors and the arrangement order of different pixels in the semantic two-dimensional pixel code and the logical two-dimensional pixel code respectively represent the semantic meaning and the logical relationship between the preceding and following words.

[0028] Optionally, the training process of the neural network model includes:

[0029] The training process is divided into training phases in advance;

[0030] Obtain the classification accuracy change curves of each feature label during the current training phase;

[0031] For a given feature label, obtain the corresponding current training state evaluation score, and configure the training method for the next training stage based on the current training state evaluation score.

[0032]

[0033] Among them, T j For the j-th training phase, Fi(T) j ) represents the training state evaluation score of the i-th feature label in the corresponding training phase, ΔDi represents the improvement in classification accuracy of the i-th feature label compared to the previous training phase, and Dsi(T) represents the improvement in classification accuracy of the i-th feature label compared to the previous training phase. j Let Di(T) be the target classification accuracy of the i-th feature label in the j-th training phase. j ) corresponds to the actual classification accuracy of the i-th feature label during the training phase, Oi(T) j+1 ) represents the proportion of the sample size corresponding to the i-th feature label in the (j+1)-th training stage, and Otij represents the proportion of the training time for the sample corresponding to the i-th feature label in the j-th training stage.

[0034] Optionally, the target classification accuracy Dsi(T) of the i-th feature label in the (j+1)-th training phase j+1 The configuration method for ) is as follows:

[0035]

[0036] Among them, mi(T) j XY(T) represents the number of subcategories contained in the i-th feature label during the j-th training phase. j ) represents the pixel capacity of a single rectangular matrix dot code of a sample in the j-th training phase.

[0037] Optionally, in the method for obtaining search results based on the output of the search model:

[0038] The output consists of C feature labels arranged in descending order of classification probability;

[0039] Retrieve digital files that simultaneously contain C feature labels, and then sort them in descending order according to the number of files that simultaneously contain the specified subcategories;

[0040] The specified subcategories are the subcategories in the feature labels that meet the preset probability percentage.

[0041] Optionally, the method for obtaining the corresponding digital file after processing the multimodal file includes:

[0042] The paper documents are digitized and stored in digital form.

[0043] The electronic documents are summarized and stored in digital form;

[0044] The documents in the form of images, audio, and video are cropped and denoised, and then stored in digital form.

[0045] In summary, this application includes at least one of the following beneficial technical effects:

[0046] This invention digitizes and denoises multimodal archives, then segments them to a size suitable for existing large multimodal models. It then classifies and identifies these segments, interpreting and describing the digital multimodal archives into general descriptive text. Following preset conversion rules, it constructs an archive database of image sets with corresponding tags, enabling training and use of neural network models. This reduces the difficulty of searching for image, audio, and video archives, while simplifying document-format archive data. As the total number of archives increases, the neural network-based retrieval model can be continuously optimized and trained, continuously improving and ensuring retrieval accuracy, and lowering the cost and barrier to multimodal archive retrieval. Attached Figure Description

[0047] Figure 1 This is an example diagram of the archival information retrieval method based on a multimodal model according to the present invention. Detailed Implementation

[0048] The embodiments of this application are described in detail below, and examples of the embodiments are shown in the accompanying drawings.

[0049] In the description of this specification, the references to "certain embodiments," "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples" refer to specific features, structures, materials, or characteristics described in connection with the described embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0050] This application discloses a method for retrieving archival information based on a multimodal model, referring to... Figure 1 ,include:

[0051] The corresponding digital files are obtained after processing the multimodal files;

[0052] The digital archives are input into a multimodal model for recognition and classification to obtain corresponding textual description information; the description information includes descriptive text reflecting the content of the multimodal archives and corresponding feature tags;

[0053] Based on the descriptive text, an archive database is constructed according to preset conversion rules, and then corresponding feature tags are embedded.

[0054] After training the neural network model based on the aforementioned archive database, a retrieval model is obtained;

[0055] The input text-based search information is converted into search images according to the preset conversion rules;

[0056] The search image is sent to the search model, and the search results are obtained based on the output of the search model.

[0057] The multimodal archives include paper documents, electronic documents, images, audio, and video formats. In this embodiment of the invention, the method for processing the multimodal archives to obtain the corresponding digital archives includes:

[0058] The paper documents are digitized and stored in digital form.

[0059] The electronic documents are summarized and stored in digital form;

[0060] The documents in the form of images, audio, and video are cropped and denoised, and then stored in digital form.

[0061] This invention digitizes and denoises multimodal archives, then segments them to a size suitable for existing large multimodal models. It then classifies and identifies these segments, interpreting and describing the digital multimodal archives into general descriptive text. Following preset conversion rules, it constructs an archive database of image sets with corresponding tags, enabling training and use of neural network models. This reduces the difficulty of searching for image, audio, and video archives, while simplifying document-format archive data. As the total number of archives increases, the neural network-based retrieval model can be continuously optimized and trained, continuously improving and ensuring retrieval accuracy, and lowering the cost and barrier to multimodal archive retrieval.

[0062] Optionally, the preset conversion rules include:

[0063] After segmenting the text data into sentences, the sentences are then broken down into words.

[0064] Part-of-speech tagging and syntactic analysis are performed on the vocabulary to obtain the semantic two-dimensional pixel code and logical two-dimensional pixel code corresponding to the vocabulary; the logical two-dimensional pixel code represents the logical relationship between the vocabulary.

[0065] The conversion is completed by loading the semantic two-dimensional pixel codes and logical two-dimensional pixel codes of the words corresponding to the statement into a blank image in the order of the text.

[0066] In this embodiment, after receiving the information, a machine learning algorithm can be used to extract data, capture important words in the sentence, and then segment the important sentences containing the important words into multiple words.

[0067] For example, the phrase "The first lane is for two-way motor vehicle traffic only" will be broken down into "first lane", "of", "lane", "only", "motor vehicle", "two-way", and "travel". Then, meaningless auxiliary words will be removed; in the example above, "of" will be removed. Next, the segmented words will be tagged with their parts of speech. Then, according to pre-prepared corresponding rules, the segmented words will be converted into corresponding semantic two-dimensional pixel codes and logical two-dimensional pixel codes in sequence and filled into a blank image. "Only" can be converted into the corresponding logical two-dimensional pixel code to represent the logical relationship between the words.

[0068] Optionally, in the method for obtaining search results based on the output of the search model:

[0069] The output consists of C feature labels arranged in descending order of classification probability;

[0070] Retrieve digital files that simultaneously contain C feature labels, and then sort them in descending order according to the number of files that simultaneously contain the specified subcategories;

[0071] The specified subcategories are the subcategories in the feature labels that meet the preset probability percentage.

[0072] By adopting the above technical solution, if the search information is A, the output result of the retrieval model is the classification result of A, which is 60% of the a feature label and 40% of the x feature label and the corresponding subcategories. The output result is a digital file that contains both the a feature label and the x feature label. If file 1 contains 5 corresponding subcategories and file 2 contains 8 corresponding subcategories, then the files are sorted and displayed according to the priority of file 2.

[0073] Optionally, the method for embedding feature labels includes:

[0074] Select the region containing the two-dimensional pixel code corresponding to the feature label;

[0075] Select a specified position in the region as the starting position, and use the outline of the region as the route to establish a unidirectional pixel code line in a clockwise direction;

[0076] The color order of the pixels in the pixel code line uniquely corresponds to the feature label;

[0077] If the feature label includes subcategories, the subcategories correspond to multiple pixel segments in the pixel code line, and the subcategories corresponding to the pixel segments closer to the starting position have a higher weight in the statement.

[0078] In this embodiment of the invention, taking "the first lane is for two-way traffic of motor vehicles only" as an example, its feature label is "road rules", and its subcategories are "motor vehicles" and "two-way traffic". In the corresponding area of ​​the image with loaded two-dimensional pixel code, pixel bars related to "road rules" can be established starting from the upper left corner of the area, such as "color 2011-color 2799-color 2830-color 8166-color 2497-color 0086-color 0000*number of remaining pixels". Color 0000 can be configured as meaningless fill pixels to completely fill the remaining part of the pixel bar. The pixel bars corresponding to the subcategories can be established in the same way within the area defined by the pixel bars corresponding to the feature label. However, the pixel bars corresponding to "motor vehicles" and "two-way traffic" have a sequential design. If it is necessary to emphasize motor vehicles, the pixel bars are arranged in the order of "motor vehicles-two-way traffic". If it is necessary to emphasize two-way traffic, the pixel bars are arranged in the order of "two-way traffic-motor vehicles".

[0079] In this embodiment, the semantic two-dimensional pixel code is a rectangular matrix dot code with an X*Y pixel size, and the logical two-dimensional pixel code is a rectangular matrix dot code with an A*Y pixel size.

[0080] The pixel colors and the arrangement order of different pixels in the semantic two-dimensional pixel code and the logical two-dimensional pixel code represent the meaning of the word and the logical relationship between the preceding and following words, respectively. X can be 30, Y can be 20, and A can be 2. The configuration size determines the maximum capacity of meaning expression. The larger the product of X and Y, the more word meanings this set of semantic two-dimensional pixel codes can express, which can also improve the upper limit of the recognition accuracy and robustness of the neural network model. However, it will also reduce the processing efficiency of the neural network model and increase the training time. Therefore, the configuration needs to be selected according to the specific situation, and comprehensive training should be carried out during the training process.

[0081] Optionally, the training process of the neural network model includes:

[0082] The training process is divided into training phases in advance;

[0083] Obtain the classification accuracy change curves of each feature label during the current training phase;

[0084] For a given feature label, obtain the corresponding current training state evaluation score, and configure the training method for the next training stage based on the current training state evaluation score.

[0085]

[0086] Among them, T j For the j-th training phase, Fi(T) j ) represents the training state evaluation score of the i-th feature label in the corresponding training phase, ΔDi represents the improvement in classification accuracy of the i-th feature label compared to the previous training phase, and Dsi(T) represents the improvement in classification accuracy of the i-th feature label compared to the previous training phase. j Let Di(T) be the target classification accuracy of the i-th feature label in the j-th training phase. j ) corresponds to the actual classification accuracy of the i-th feature label during the training phase, Oi(T) j+1 Let Otij be the percentage of the sample size corresponding to the i-th feature label in the (j+1)-th training stage, and Otij be the percentage of the training time taken for the sample corresponding to the i-th feature label in the j-th training stage. In this way, the sample size corresponding to different feature labels can be adjusted at different training stages, ensuring overall training efficiency and accuracy.

[0087] Optionally, the target classification accuracy Dsi(T) of the i-th feature label in the (j+1)-th training phase j+1 The configuration method for ) is as follows:

[0088]

[0089] Among them, mi(T) j XY(T) represents the number of subcategories contained in the i-th feature label during the j-th training phase. j ) represents the pixel capacity of a single rectangular matrix dot code of a sample in the j-th training phase.

[0090] By adopting the above technical solution, after training on a two-dimensional pixel code of a certain size is completed, when entering the next training stage, it is necessary to select image samples with a uniform two-dimensional pixel code of a new size for continued training. At this time, it is necessary to adjust the target classification accuracy Dsi(T). j+1 The system will be readjusted to ensure overall training efficiency and accuracy.

[0091] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A method for retrieving archival information based on a multimodal model, characterized in that, include: The corresponding digital files are obtained after processing the multimodal files; The digital archives are input into a multimodal model for recognition and classification to obtain corresponding textual description information; the description information includes descriptive text reflecting the content of the multimodal archives and corresponding feature tags; Based on the descriptive text, an archive database is constructed according to preset conversion rules, and then corresponding feature tags are embedded. After training the neural network model based on the aforementioned archive database, a retrieval model is obtained; The input text-based search information is converted into search images according to the preset conversion rules; The search image is sent to the search model, and the search results are obtained based on the output of the search model. The multimodal archives include paper documents, electronic documents, images, audio, and video formats; The preset conversion rules include: After segmenting the text data into sentences, the sentences are then broken down into words. Part-of-speech tagging and syntactic analysis are performed on the vocabulary to obtain the semantic two-dimensional pixel code and logical two-dimensional pixel code corresponding to the vocabulary; the logical two-dimensional pixel code represents the logical relationship between the vocabulary. The conversion is completed by loading the semantic two-dimensional pixel codes and logical two-dimensional pixel codes of the words corresponding to the statement into a blank image in the order of the text. The method for embedding corresponding feature labels includes: Select the region containing the two-dimensional pixel code corresponding to the feature label; Select a specified position in the region as the starting position, and use the outline of the region as the route to establish a unidirectional pixel code line in a clockwise direction; The color order of the pixels in the pixel code line uniquely corresponds to the feature label; If the feature label includes subcategories, the subcategories correspond to multiple pixel segments in the pixel code line, and the subcategories corresponding to the pixel segments closer to the starting position have a higher weight in the statement.

2. The archival information retrieval method based on a multimodal model according to claim 1, characterized in that, The semantic two-dimensional pixel code is a rectangular matrix dot code with an X*Y pixel size, and the logical two-dimensional pixel code is a rectangular matrix dot code with an A*Y pixel size. The pixel colors and the arrangement order of different pixels in the semantic two-dimensional pixel code and the logical two-dimensional pixel code respectively represent the semantic meaning and the logical relationship between the preceding and following words.

3. The archival information retrieval method based on a multimodal model according to claim 2, characterized in that, The training process for the neural network model includes: The training process is divided into training phases in advance; Obtain the classification accuracy change curves of each feature label during the current training phase; For a given feature label, obtain the corresponding current training state evaluation score, and configure the training method for the next training stage based on the current training state evaluation score. ； ； in, For the first Each training phase For the corresponding training phase Training state evaluation score for each feature label, For the first The improvement in classification accuracy corresponding to each feature label compared to the previous training stage. For the first The training phase The target classification accuracy of each feature label The corresponding training phase The actual classification accuracy of each feature label For the first The sample corresponding to the feature label is in the _th ... The percentage of sample size in each training phase. For the first The sample corresponding to the feature label is in the _th ... The percentage of training time spent in each training phase.

4. The archival information retrieval method based on a multimodal model according to claim 3, characterized in that, No. The target classification accuracy of the i-th feature label under each training phase The configuration method is as follows: ； Among them, is the first The training phase The number of subcategories contained in each feature label For the first The pixel capacity of a single rectangular matrix dot code of a sample in each training phase.

5. The archival information retrieval method based on a multimodal model according to claim 1, characterized in that, In the method for obtaining search results based on the output of the search model: The output consists of C feature labels arranged in descending order of classification probability; Retrieve digital files that simultaneously contain C feature labels, and then sort them in descending order according to the number of files that simultaneously contain the specified subcategories; The specified subcategories are the subcategories in the feature labels that meet the preset probability percentage.

6. The archival information retrieval method based on a multimodal model according to claim 1, characterized in that, The method for obtaining corresponding digital files after processing multimodal files includes: The paper documents are digitized and stored in digital form. The electronic documents are summarized and stored in digital form; The documents in the form of images, audio, and video are cropped and denoised, and then stored in digital form.

Images