An archive cross-modal retrieval method and device based on bidirectional semantic texturization

Abstract

The application provides an archive cross-modal retrieval method and device based on bidirectional semantic texturization. The method comprises the following steps: preprocessing a multi-modal original archive on a resource side to obtain a standardized archive resource, completing semantic texturization processing according to modal differences, generating structured enhanced semantic text conforming to archive business specifications, initializing index parameters based on an inverted index, precomputing parameters required for BM25 sorting, and generating a retrieval index that can be used for relevance sorting. The retrieval end receives a multi-modal query request, converts the query request into a standardized text retrieval instruction through modal recognition and semantic text conversion, and generates the standardized text retrieval instruction through deep semantic analysis and optimization. The relevance score is calculated by matching the index, the candidate archive resources are threshold filtered, and the retrieval result is output. The application can map heterogeneous archive data and multi-modal queries to an archive standardized semantic text space, eliminate semantic heterogeneity barriers, improve the recall rate and precision rate of retrieval, and realize intelligent cross-modal retrieval of deep archive semantics.

Description

Technical Field

[0001] This application relates to the field of retrieval technology, and in particular to a method and apparatus for cross-modal retrieval of archives based on bidirectional semantic textualization. Background Technology

[0002] With the deepening of digital archives informatization, the carrier form of archival resources is gradually shifting from traditional single-text paper archives to multimodal digital archives that coexist with text, images, audio, and video. Massive, heterogeneous, and multi-sourced multimodal archival data has become the main data carrier for archival management. In the context of smart archives construction, users' retrieval methods are becoming increasingly diversified, no longer limited to traditional text retrieval. There are frequent non-text retrieval needs such as images, voice, and short videos. This places higher demands on the cross-modal adaptation capabilities, semantic understanding capabilities, and unified retrieval capabilities of archival retrieval systems. Currently, various archival retrieval systems have basically achieved the storage and basic retrieval functions of digital archives, and can meet simple text keyword matching retrieval needs, and are widely used in routine archive access and document retrieval scenarios.

[0003] Most existing archival retrieval technologies employ single-modal keyword matching or low-level visual and audio feature comparison, exhibiting significant technical limitations and failing to meet the intelligent retrieval needs of today's multimodal archives. Traditional retrieval methods can only achieve precise matching of literal keywords, failing to understand the deep semantic information of archival data. This makes them highly susceptible to missed or false detections due to differences in expression, inconsistent terminology, and semantic ambiguity, resulting in low recall and precision rates. Furthermore, the feature spaces of different archival modalities are independent, and the modal heterogeneity barrier makes cross-modal correlation retrieval difficult, unable to support multimodal queries and cross-modal resource linkage output scenarios. In addition, existing technologies typically rely on complex multimodal vector space alignment and end-to-end deep learning training, lacking semantic enhancement and correction mechanisms tailored to archival industry standards. The semantic descriptions generated by general models lack professionalism and business relevance, requiring large-scale labeled data and high computing power, making engineering implementation difficult and deployment and maintenance costs high. This makes them unsuitable for the constraints of sparse labeled data and strong business standards in the archival field, further hindering the efficiency of intelligent retrieval and in-depth mining and utilization of multimodal archival resources. Summary of the Invention

[0004] In view of this, the embodiments of this application provide a method and apparatus for cross-modal archival retrieval based on bidirectional semantic textualization. By constructing a bidirectional semantic textualization channel between the resource processing side and the retrieval side, heterogeneous multimodal archival resources and user multimodal query requests are uniformly mapped to a standardized semantic textual space in the archival field, eliminating the semantic heterogeneity barrier of multimodal data. Relying on the BM25 algorithm, deep semantic cross-modal intelligent retrieval is achieved, adapting to diverse archival retrieval scenarios and effectively improving the accuracy of multimodal archival retrieval and the efficiency of resource utilization.

[0005] The technical solution of this application embodiment is implemented as follows: In a first aspect, embodiments of this application provide a cross-modal retrieval method for archives based on bidirectional semantic textualization, comprising the following steps: The received multimodal raw archival data is preprocessed to obtain standardized archival resources; The standardized archival resources are processed into differentiated semantic texts based on modality type to generate structured enhanced semantic texts that conform to archival business specifications. The structured, enhanced semantic text is stored, and a multimodal archive retrieval index is constructed based on the inverted index; It receives multimodal query requests from users, performs modality recognition and semantic text conversion on the query modality, and generates standardized text retrieval instructions through deep semantic parsing and optimization processing. The standardized text retrieval command is matched with the multimodal archive retrieval index, a relevance score is calculated, candidate archive resources that meet the preset similarity threshold are selected, and multimodal retrieval results are generated.

[0006] Secondly, embodiments of this application also provide a cross-modal archive retrieval device based on bidirectional semantic textualization, the device comprising: The preprocessing module is used to preprocess the received multimodal raw archival data to obtain standardized archival resources; The semantic text conversion module is used to perform differentiated semantic textification on the standardized archival resources according to modality type, and generate structured enhanced semantic text that conforms to archival business specifications; An index building module is used to store the structured enhanced semantic text and build a multimodal archive retrieval index based on the inverted index; The query modality recognition and conversion module is used to receive multimodal query requests from users and perform modality recognition and semantic text conversion on the query modality. The query semantic parsing and optimization module is used to generate standardized text retrieval instructions through deep semantic parsing and optimization processing; The retrieval matching and result output module is used to match the standardized text retrieval command with the multimodal archive retrieval index, calculate the relevance score, filter candidate archive resources that meet the preset similarity threshold, and generate multimodal retrieval results.

[0007] Thirdly, embodiments of this application also provide an electronic device, including: a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the cross-modal file retrieval method based on bidirectional semantic textualization as described in any of the first aspects.

[0008] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the cross-modal file retrieval method based on bidirectional semantic textualization as described in any of the first aspects.

[0009] The embodiments of this application have the following beneficial effects: By constructing a complete closed-loop retrieval chain—including differentiated semantic textualization on the resource processing side, pre-calculation of BM25 ranking parameters based on an inverted index structure, unified multimodal transformation and deep semantic parsing on the retrieval side, quantified relevance matching, and threshold filtering output—this approach unifies the previously heterogeneous and semantically fragmented multimodal archival resources (text, images, audio, and video) and diverse user retrieval requests into a standardized semantic text space within the archival domain. This completely avoids the modal barriers and semantic gaps caused by traditional archival retrieval's reliance on shallow keyword matching and low-level feature comparison. The process involves first standardizing archival resources, then using differentiated semantic textualization and domain knowledge enhancement to achieve unified semantic representation, pre-calculating BM25 ranking parameters based on an inverted index structure, performing deep semantic-level parsing queries, and quantifying... The progressive technical architecture of matching and filtering realizes a paradigm upgrade in archival retrieval from manual coarse-grained cataloging and literal precise matching to fully automated machine fine-grained semantic understanding and cross-modal association retrieval. It effectively supports various modal archives and various modal query inputs, and stably outputs multimodal retrieval results with reliable semantic association and comprehensive coverage. While significantly improving the recall and precision of multimodal archival retrieval, it does not rely on large-scale labeled data and complex multimodal alignment training models. It can achieve cross-modal unified retrieval by relying on differentiated prompting engineering technology and mature BM25 text retrieval engine, simplifying the engineering implementation difficulty of cross-modal intelligent retrieval, reducing the deployment and maintenance costs of multimodal unified archival retrieval, and significantly improving the intelligent retrieval capability and comprehensive utilization efficiency of digital archival multimodal resources. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a flowchart illustrating steps S101-S105 provided in the embodiments of this application; Figure 2 This is a flowchart illustrating steps S201-S203 provided in the embodiments of this application; Figure 3 This is a flowchart illustrating steps S301-S304 provided in the embodiments of this application; Figure 4 This is a flowchart illustrating steps S401-S402 provided in the embodiments of this application; Figure 5 This is a flowchart illustrating steps S501-S504 provided in the embodiments of this application; Figure 6 This is a flowchart of the overall multimodal raw archive data preprocessing provided in the embodiments of this application; Figure 7 This is a flowchart of the multimodal archive resource side differentiated semantic unified representation processing provided in the embodiments of this application; Figure 8 This is a flowchart illustrating the semantic description generation process of photo files and video keyframe images after keyframe extraction, as provided in the embodiments of this application. Figure 9 This is a flowchart of the user retrieval side multimodal unified conversion and deep semantic parsing provided in the embodiments of this application; Figure 10 This is a schematic diagram of the structure of the cross-modal file retrieval device based on bidirectional semantic textualization provided in the embodiments of this application; Figure 11 This is a schematic diagram of the composition structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0013] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0014] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0015] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0016] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0017] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application and is not intended to limit this application.

[0018] See Figure 1 , Figure 1This is a flowchart illustrating steps S101-S105 of the cross-modal archive retrieval method based on bidirectional semantic textualization provided in this application embodiment. Figure 1 Steps S101-S105 are explained below.

[0019] In step S101, the received multimodal raw archival data is preprocessed to obtain standardized archival resources; In step S102, the standardized archival resources are processed into differentiated semantic text based on modality type to generate structured enhanced semantic text that conforms to archival business specifications. In step S103, the structured enhanced semantic text is stored, and a multimodal archive retrieval index is constructed based on the inverted index; In step S104, a user's multimodal query request is received, the query modality is identified and semantic text is converted, and a standardized text retrieval instruction is generated through deep semantic parsing and optimization. In step S105, the standardized text retrieval instruction is matched with the multimodal archive retrieval index, a relevance score is calculated, candidate archive resources that meet the preset similarity threshold are selected, and multimodal retrieval results are generated.

[0020] The principle of this embodiment is as follows: by constructing a two-way semantic textualization channel between the resource processing side and the retrieval side, heterogeneous multimodal archive resources and user multimodal queries are uniformly transformed into domain-standardized enhanced semantic text, and the cross-modal semantic matching problem is transformed into a mature text semantic retrieval problem based on the BM25 algorithm, thereby achieving a deep semantic understanding upgrade of archive retrieval from the architectural level.

[0021] In practice, the first step is to preprocess the multimodal original archival data. Through data cleaning, format standardization, and normalization, problems such as messy formats, incomplete data, and interference from invalid noise data in the original archival data are eliminated, outputting standardized archival resources that can be directly used for algorithm analysis, providing a high-quality data foundation for subsequent semantic textualization processing. Secondly, considering the differences in the underlying characteristics of different modal archival data, differentiated technologies are used to achieve semantic textualization conversion. Relying on the metadata standard mapping, professional terminology correction, and compliance verification capabilities of the domain knowledge-enhanced large language model, the limitations of traditional manual coarse-grained annotation are overcome, generating structured enhanced semantic text that conforms to archival business standards, is rich in deep semantics, and has a unified structure, completing the semantic unified representation of all modal archival resources.

[0022] The generated structured enhanced semantic text is then persistently stored to build a full-text database. Based on the inverted index structure, BM25 sorting parameters are pre-calculated to construct a dedicated multimodal archival retrieval index. This transforms the semantic text into a fast-searchable and quantifiable matching index structure, supporting efficient retrieval operations. In the query processing stage, the system is compatible with common multimodal query inputs in archival scenarios, such as text, audio, images, and video. Through a full-process processing of modality recognition, cross-modal text conversion, deep semantic parsing, and optimization, it eliminates modal differences, semantic ambiguities, and non-standard expressions in user queries, outputting unified, standardized, and accurately matching text retrieval commands. Finally, based on the constructed retrieval index, relevance quantification is performed. Using a preset similarity threshold as an objective screening standard, qualified candidate resources are accurately selected, outputting fully modal related retrieval results, achieving true cross-modal integrated semantic retrieval.

[0023] In some embodiments, the differential semantic textification processing of the standardized archival resources based on modality type includes: For video archives, keyframe extraction is performed first to extract keyframe images from the video. Then, an initial semantic text description is generated by combining a multimodal big model with structured prompt word templates. Finally, a domain knowledge-enhanced big language model is used to complete metadata specification mapping, professional terminology correction and compliance verification to generate a structured enhanced semantic text description. For image-based archives, which refer to native photo archives, an initial semantic text description is first generated by combining a multimodal large model with structured prompt word templates. Then, a domain-knowledge-enhanced large language model is used to complete metadata specification mapping, professional terminology correction, and compliance verification to generate a structured enhanced semantic text description. For audio archives, an initial structured transcribed text with timestamps is generated through speech recognition and audio understanding models. Then, a large language model with archive domain knowledge enhancement is used to complete metadata specification mapping, professional terminology correction and compliance verification, generating structured enhanced semantic text. For the digitization of paper archives, layout analysis and OCR recognition technologies are used to convert them into structured text, generating structured enhanced semantic text; For text-based archives, their standardized text is directly used as structured, semantically enhanced text.

[0024] For visual archives such as images and videos, which lack natural text information but contain rich visual semantic information such as scenes, entities, spatial relationships, and image details, this embodiment first samples the continuous video stream using a keyframe extraction algorithm, removing redundant frames and retaining keyframes that cover the core business information of the video, transforming the dynamic video data into a static image sequence, reducing the difficulty of subsequent model analysis. Then, based on a multimodal large model combined with structured prompt word templates, it accurately identifies titles, seals, handwritten signatures, text regions, entity objects, and spatial semantic relationships in the image, generating a structured semantic text description containing complete visual information and spatial relationships. Finally, based on a large language model enhanced with domain knowledge embedded with archival industry standards (DA / T 62—2017, DA / T 63—2017, DA / T 78—2019, etc.) and laws and regulations (the "Archives Law of the People's Republic of China," the "Implementation Regulations of the Archives Law of the People's Republic of China," etc.), it completes metadata specification mapping, professional terminology correction, and content compliance verification, correcting the professional biases of the general model description, and outputting structured enhanced semantic text that conforms to archival business specifications.

[0025] For audio archives, this type of data uses time-series sound signals as its carrier, containing speech content and background sound field information. This embodiment combines an automatic speech recognition model and an audio understanding model (such as Qwen-Audio) to not only accurately transcribe the speech text content, but also identify non-linguistic semantic information such as ambient sounds and background sound effects in the audio, and retain timestamp information throughout the process, generating a complete initial structured audio transcription text with time-series attributes. Subsequently, the initial transcribed text is input into a large language model enhanced with archival domain knowledge, which performs archival metadata standard mapping, professional terminology correction, and content compliance verification, converting colloquial expressions into archival standard terminology, and conducting compliance review of the audio content according to archival standards and specifications, ultimately generating a structured enhanced semantic text of the audio archive that conforms to archival business specifications.

[0026] For digitized paper archives, this type of data records the complete layout and text content of the paper archives, and inherently contains rich structured text information. This embodiment uses layout analysis technology (such as RapidLayout) to identify the document's layout structure, column distribution, and text areas, and then uses OCR recognition technology to accurately extract the text content, ultimately converting it into directly indexable structured text and generating structured enhanced semantic text.

[0027] For raw text archives, which possess inherent textual attributes and require no cross-modal conversion, this embodiment directly uses their normalized text as structured enhanced semantic text, uniformly incorporating it into the multimodal archive retrieval index. This completely eliminates differences in multimodal data representation formats, achieving uniformity in the semantic representation of all modal archives.

[0028] In some embodiments, see Figure 2 , Figure 2 This is a flowchart illustrating steps S201-S203 provided in the embodiments of this application. The construction of a multimodal archive retrieval index based on the inverted index can be achieved through steps S201-S203, which will be explained in conjunction with each step.

[0029] In step S201, text length normalization is performed on all structured enhanced semantic text to ensure that the text data after different modal transformations maintain consistency in document length distribution and information density. In step S202, the structured enhanced semantic text after all text lengths are standardized is segmented and stop words are removed to construct a retrieval dictionary and an inverted list; In step S203, the number of terms in each document, the global average number of terms in each document, the document frequency of each term, and the total number of documents are counted. The inverse document frequency of each term is pre-calculated. The index parameters are initialized based on the inverted index structure. The parameters required for BM25 sorting are pre-calculated. A retrieval index that can be used for relevance sorting is generated.

[0030] Here, text preprocessing is first performed, and text length standardization is applied to text data from different modalities. This ensures that the refined semantic descriptions generated from image and video modalities are consistent with those from audio transcription, OCR extraction, and native text in terms of document length distribution and information density, guaranteeing the fairness of BM25 relevance calculations within a unified index space. Based on this, precise word segmentation, stop word removal, and text normalization are performed on all structured enhanced semantic texts that have undergone text length standardization. Function words, auxiliary words, and general redundant words without retrieval significance are filtered out, and core terms with retrieval value are selected. A globally unique retrieval dictionary is constructed based on these effective terms. Simultaneously, a dedicated inverted list is created for each independent term in the dictionary, accurately recording all document IDs containing the term, the term's frequency (TF) in the corresponding document, and the term's position information. This achieves a reverse mapping from terms to documents, supporting the subsequent rapid retrieval of candidate document sets.

[0031] Based on this, the global index parameter statistics and initialization are completed. The core statistical parameters include the number of terms in a single document |D|, the sum of the number of terms in all global documents, and the total number of documents, and are calculated using the following formula: ; The global document term count (avgdl) is calculated. Simultaneously, all terms are traversed, and the inverse document frequency formula is used: ; Calculate the global IDF value for each term, where N represents the total number of global documents, and n(qi) represents the number of documents containing the target term. Finally, integrate the search dictionary, inverted list, global avgdl, IDF values ​​of each term, and document term count parameters to complete index initialization and generate a standardized multimodal archive retrieval index that can be directly used for BM25 relevance calculation.

[0032] In some embodiments, the modality recognition and semantic text conversion of the query modality includes: Identify the user's query input modality. If the input modality is a text query, use it directly as the query text. If the input modality is non-textual audio, image, or video queries, they are converted into corresponding modality query texts using differentiated modality conversion methods based on the query modality type: For audio queries, after being converted into text by a speech recognition and audio understanding model, they are processed by a large language model enhanced with archival domain knowledge into structured enhanced semantic text that conforms to archival domain business specifications; for image queries, they are converted into structured enhanced semantic text descriptions that conform to archival domain business specifications through a multimodal large model and a large language model enhanced with archival domain knowledge; for video queries, after keyframe images are generated by extracting keyframes from the video, they are converted into structured enhanced semantic text descriptions that conform to archival domain business specifications through a multimodal large model and a large language model enhanced with archival domain knowledge.

[0033] Here, this embodiment first performs modality recognition on the original query request initiated by the user, distinguishing between text modality and non-text modality queries, and executing differentiated branch processing logic. For text queries directly entered by the user, no cross-modality conversion is required, and the query can directly enter the subsequent deep semantic parsing and optimization process, simplifying the retrieval process and improving response efficiency.

[0034] For three non-text query modalities—audio, image, and video—corresponding cross-modal conversion methods are matched. Audio queries utilize automatic speech recognition technology combined with an audio understanding model to convert speech signals into text, preserving temporal semantic information and generating initial structured audio-transcribed text. This text is then processed by a large language model enhanced with archival domain knowledge to perform archival domain terminology correction and compliance enhancement, transforming it into standardized query text conforming to archival domain standards. Image queries directly use a multimodal large model to perform visual content parsing, generating text descriptions containing entities, scenes, and business information. These descriptions are then processed by a large language model enhanced with archival domain knowledge to perform archival domain metadata standard mapping, terminology correction, and compliance enhancement, transforming it into standardized query text conforming to archival domain business standards. Video queries first undergo keyframe decomposition, converting dynamic video streams into static keyframe sequences. Visual parsing then generates corresponding text descriptions containing entities, scenes, and business information. Finally, a large language model enhanced with archival domain knowledge performs archival domain metadata standard mapping, terminology correction, and compliance verification, transforming it into standardized query text conforming to archival domain business standards. After undergoing the aforementioned differentiated modal transformation, all non-text queries are ultimately converted into standardized query text that conforms to the business specifications of the archival field, achieving unified textification of any query modality and providing a unified input source for subsequent full-domain semantic parsing and instruction optimization.

[0035] In some embodiments, see Figure 3 , Figure 3 This is a flowchart illustrating steps S301-S304 provided in the embodiments of this application. The generation of standardized text retrieval instructions through deep semantic parsing and optimization processing can be achieved through steps S301-S304, which will be explained in conjunction with each step.

[0036] In step S301, it is determined whether the query text meets the preset semantic refinement triggering conditions; when it is determined that the semantic refinement triggering conditions are met, the query text is semantically refined to generate a simplified query text; when it is determined that the semantic refinement triggering conditions are not met, the query text is directly used as the simplified query text. In step S302, based on a domain knowledge-enhanced large language model, intent recognition and semantic disambiguation are performed on the simplified query text; In step S303, the query text after intent recognition and semantic disambiguation is expanded with domain synonyms and reconstructed to complete the domain adaptation optimization of query semantics; In step S304, a standardized text retrieval instruction is generated based on the optimized query statement.

[0037] The first step involves core execution condition judgment and semantic refinement branch processing. The semantic refinement trigger condition is based on preset text length thresholds and / or semantic density thresholds: when the text length of the query text exceeds the preset length threshold and / or the semantic density is lower than the preset density threshold, the semantic refinement trigger condition is met, and the query text is semantically refined. This process removes colloquial and redundant expressions, background sound field descriptions, and duplicate confirmation information from long text queries, compressing the text length and increasing the semantic density to generate concise query text. This avoids the term dilution effect and computational waste caused by direct matching of long queries. When the query text is a short text keyword query, and its text length does not exceed the preset length threshold and the semantic density meets the requirements, the semantic refinement trigger condition is not met, and the query text is directly used as concise query text for subsequent processing. This avoids unnecessary semantic loss and ensures the retrieval accuracy and response efficiency of short queries.

[0038] The second step involves intent recognition and semantic disambiguation. Using a large language model enhanced with archival domain knowledge, combined with archival professional context, business scenarios, and terminology, the simplified query text is deeply analyzed to accurately identify the true search intent behind the user's surface query text. At the same time, it resolves the polysemy, ambiguity, and vague expressions inherent in natural language, eliminates invalid semantic interference, locks in the core search needs, and makes up for the shortcomings of traditional search that only matches literal text and cannot understand semantic connotations.

[0039] The third step involves domain semantic expansion and sentence reconstruction. Based on an archival terminology database, industry standards, and business logic, the query text after intent recognition and semantic disambiguation undergoes domain synonym expansion and sentence reconstruction. Domain synonym expansion includes expanding relevant professional terms to address issues such as incomplete user query expressions, non-standard terminology, and semantic gaps. Sentence reconstruction involves structurally reconstructing the original query statement, optimizing statement logic, standardizing expression formats, and adapting general natural language queries to archival domain-specific retrieval language, thus achieving deep optimization of query semantics.

[0040] The fourth step generates standardized text retrieval instructions. Based on the optimized, expanded, and reconstructed query statements, standardized text retrieval instructions with accurate semantics, clear logic, and uniform format are output. These instructions are fully aligned with the structured enhanced semantic text semantic system generated by the resource processing side and can be directly used for semantic matching calculations of the retrieval index, providing core query input support for accurate retrieval.

[0041] In some embodiments, see Figure 4 , Figure 4 This is a flowchart illustrating steps S401-S402 provided in the embodiments of this application. The calculation of the correlation score can be achieved through steps S401-S402, which will be explained in conjunction with each step.

[0042] In step S401, based on the term occurrence frequency, single document term count, global average document term count, and term inverse document frequency parameters stored in the retrieval index, a set of candidate documents containing the query term is recalled through the inverted index. In step S402, the overall relevance score between the query text and each candidate document is calculated by summing each word, and the degree of relevance is represented by the score value.

[0043] This embodiment uses pre-statistically collected and stored core index parameters, including the frequency of each term in the target document f(qi,D), the number of terms in a single candidate document |D|, the global average number of terms in a document avgdl, and the inverse document frequency (IDF) of each term Iqi. This eliminates the need for real-time repetitive statistical calculations, significantly improving retrieval response efficiency. During the retrieval process, the system first uses an inverted index to quickly filter out the set of all candidate documents containing the query term, narrowing the matching calculation range and avoiding the computational waste caused by traversing all documents.

[0044] Based on the standard BM25 score calculation formula: ; The relevance calculation is performed word-by-word. For each core term in the query text, its matching contribution value with candidate documents is calculated, and these values ​​are accumulated to obtain the overall document relevance score. Here, k1 and b are preset adjustment coefficients of the BM25 algorithm, used to balance the term frequency weighting and the document length normalization effect. Finally, the quantitative score accurately represents the lexical relevance between the query text and the archival documents, providing an objective quantitative basis for subsequent result filtering and ranking.

[0045] In some embodiments, see Figure 5 , Figure 5 This is a flowchart illustrating steps S501-S504 provided in the embodiments of this application. The step of filtering candidate archive resources that meet the preset similarity threshold and outputting multimodal retrieval results can be achieved through steps S501-S504, which will be explained in conjunction with each step.

[0046] In step S501, a preset similarity threshold is used to filter out low-matching candidate documents with relevance scores below the threshold, thereby obtaining a set of qualified candidate documents. In step S502, the candidate documents that meet the criteria are sorted in descending order of relevance score, and the preset Top-K compliant results are selected. In step S503, the sorted structured enhanced semantic text is back-mapped to the corresponding original multimodal archive resource; In step S504, the original archive data and standardized semantic description information are output together.

[0047] This embodiment pre-sets a similarity score threshold, traverses the recalled candidate document set, and automatically filters out low-match and weakly related documents with relevance scores below the threshold. This avoids invalid search results with low relevance and low matching from the root, improves search accuracy, solves the problems of false detection and mis-detection in traditional search, and obtains a high-matching candidate document set.

[0048] Then, the results are sorted and optimized. All qualified candidate documents are sorted in descending order based on the relevance score calculated by the BM25 algorithm. The top-ranked high-quality search results are selected according to the preset number of Top-K documents. The archive resources with the highest matching degree and the closest relevance are output first to meet the search needs of users who want to obtain core target resources first.

[0049] Finally, the system completes the reverse mapping from structured semantic text to the original archives and integrates the results. The sorted, enhanced semantic text descriptions are reverse-associated and mapped to their corresponding original image, video, audio, and text archive resources, breaking down modal isolation barriers. The system ultimately provides users with simultaneous feedback of the original multimodal archive data and standardized domain semantic description information, achieving a one-stop retrieval effect of "any modal input, full modal associative output," fully restoring the multi-dimensional information value of archive resources and meeting users' in-depth and comprehensive needs for archive utilization.

[0050] The embodiments of this application will now be described in full and in detail in conjunction with practical applications.

[0051] Please see Figure 6 , Figure 6 This is a flowchart of the preprocessing of multimodal original archival data provided in the embodiments of this application. The specific implementation process is as follows: The system first receives five types of multimodal original archival data in batches: original electronic archives, digital copies of paper archives, photo archives, audio archives, and video archives. It then performs preprocessing operations such as data cleaning, format normalization, and noise removal to obtain standardized archival resources.

[0052] Please see Figure 7 , Figure 7 This is a flowchart of the multimodal archive resource-side differentiated semantic unified representation processing provided in the embodiments of this application, such as... Figure 7As shown, taking an archival dataset as the processing object, this paper performs multimodal refined semantic transformation on five types of archival data within the dataset: original electronic archives, digitized copies of paper archives, photo archives, audio archives, and video archives. The specific implementation process is as follows: For video archives, a keyframe extraction algorithm is used to filter video keyframes covering meeting scenes, attendees, displayed documents, and screen content. A multimodal big data model combined with prompt word templates is used to identify core visual information in the video, such as meeting banners, attendees, document annotations, seal markings, layout, and semantic scenes, generating initial visual semantic text. This initial text is then corrected by a domain-knowledge-enhanced big data language model to correct official document terminology, supplement archival metadata, and verify content compliance, generating standardized enhanced text descriptions. For audio archives, speech recognition and audio understanding models (such as Qwen-Audio) are used to complete speech transcription and background sound field recognition, preserving accurate timestamp information. This is then integrated to generate initial structured audio transcription text, which is then further processed by a domain-knowledge-enhanced big data language model. The model completes metadata specification mapping, terminology correction, and compliance verification, generating structured enhanced semantic text for audio archives. For digitized copies of paper archives, layout analysis technology is used to identify document headers, footers, seals, text areas, and column distribution. OCR recognition technology is used to accurately extract text content and convert it into structured enhanced semantic text. For native text-based electronic archives, their standardized text is directly used as structured enhanced semantic text. For photo archives, a multimodal large model, combined with prompt word templates, identifies core visual elements in the image such as seals, handwritten comments, layout, document titles, and people and scenes to generate initial semantic text. This initial semantic text is then completed by a domain-knowledge-enhanced large language model to complete metadata specification mapping, terminology correction, and compliance verification, generating structured enhanced semantic text descriptions. All modal archives ultimately output structured enhanced semantic text descriptions with consistent specifications and in line with archival business standards, completely eliminating the problem of heterogeneous underlying features of multimodal data and providing homogeneous semantic data support for the construction of a unified index.

[0053] Please see Figure 8 , Figure 8 This is a flowchart illustrating the semantic description generation process of photo files and video keyframe images after keyframe extraction, as provided in the embodiments of this application. Figure 8 As shown, the specific implementation process is as follows: For photo archives and video archive frames extracted from keyframes, the system first inputs the image data into a multimodal large model. Relying on differentiated prompt word templates for guidance and control, it accurately identifies core visual elements in the image, such as seals, handwritten instructions, document titles, page layouts, people and scenes, and spatial semantic relationships, and generates an initial semantic text description. Subsequently, relying on a domain knowledge-enhanced large language model, the initial semantic text description is subjected to archive metadata standard mapping, official document terminology correction, and content compliance verification to generate a structured enhanced semantic text description in a unified and standardized format, thus completing the semantic description generation of image archives.

[0054] Please see Figure 9 , Figure 9 This is a flowchart of the user retrieval side multimodal unified conversion and deep semantic parsing provided in the embodiments of this application, such as... Figure 9 As shown, the specific implementation process is as follows: The system first receives user query input and determines the query modality through input parsing. If it is a text query, it is directly used as the query text. If it is a non-text query, it enters the multimodal input understanding stage: For audio queries, audio preprocessing is performed to complete the speech-to-text conversion and generate audio semantic text; for video queries, video preprocessing is performed to extract keyframes and convert them into image data; for image queries, image data is directly obtained and then input into the multimodal big model to generate semantic text descriptions; subsequently, the query text, audio semantic text, and image semantic text descriptions are uniformly converged into the query semantic parsing and optimization module to perform intent recognition and semantic disambiguation, and generate / refine domain-adapted standardized text retrieval instructions; finally, through query text processing, candidate file resource acquisition, and result feedback, the system completes the end-to-end modal unified conversion and retrieval response on the user's retrieval side.

[0055] In summary, the embodiments of this application have the following beneficial effects: By implementing differentiated modality-adaptive semantic enhancement processing on text, image, video, and audio archives, and relying on traditional machine learning algorithms and multimodal large models to collaboratively complete the structured translation of visual and auditory semantics, and leveraging domain-knowledge-enhanced large language models to achieve professional standardization correction of archives, a structured enhanced semantic text system that is domain-adaptive, format-unified, and semantically dense has been constructed. This completely breaks down the semantic heterogeneity barriers between multimodal archives. Combined with a standardized inverted index and quantitative relevance matching mechanism built on the BM25 algorithm, unified semantic parsing, intent disambiguation, and standardized reconstruction of search commands are achieved for users' queries of any modality, enabling precise mining of the deep semantics of multimodal archives. By linking information, the precision and recall of archival retrieval are effectively improved. Furthermore, this application's embodiments do not rely on large-scale labeled data and complex multimodal alignment training models. Instead, they leverage differentiated prompting engineering techniques and a mature text retrieval engine to achieve unified cross-modal retrieval, significantly reducing system deployment, iteration, and maintenance costs. It possesses excellent robustness, scalability, and system compatibility, adapting to massive, heterogeneous, and low-quality existing archives as well as diverse new archive retrieval scenarios. It truly achieves a one-stop intelligent archival retrieval effect of "any modal input, full-modal semantic linkage output," significantly improving the efficiency of in-depth utilization and intelligent management of multimodal archival resources.

[0056] Based on the same inventive concept, this application also provides a cross-modal retrieval device for archives based on bidirectional semantic text, which corresponds to the cross-modal retrieval method for archives based on bidirectional semantic text in the first embodiment. Since the principle of the device in this application is similar to the cross-modal retrieval method for archives based on bidirectional semantic text, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0057] like Figure 10 As shown, Figure 10 This is a schematic diagram of the structure of a cross-modal file retrieval device 1000 based on bidirectional semantic textualization provided in this application embodiment. The cross-modal file retrieval device 1000 based on bidirectional semantic textualization includes: Preprocessing module 1001 is used to preprocess the received multimodal raw archival data to obtain standardized archival resources; The semantic text conversion module 1002 is used to perform differentiated semantic textification processing on the standardized archival resources according to modality type, and generate structured enhanced semantic text that conforms to archival business specifications; The index building module 1003 is used to store the structured enhanced semantic text and build a multimodal archive retrieval index based on the inverted index; The query modality recognition and conversion module 1004 is used to receive multimodal query requests from users and perform modality recognition and semantic text conversion on the query modality. The query semantic parsing and optimization module 1005 is used to generate standardized text retrieval instructions through deep semantic parsing and optimization processing; The retrieval matching and result output module 1006 is used to match the standardized text retrieval instruction with the multimodal archive retrieval index, calculate the relevance score, filter candidate archive resources that meet the preset similarity threshold, and generate multimodal retrieval results.

[0058] Those skilled in the art should understand that Figure 10 The functions of each unit in the bidirectional semantic textualization-based cross-modal archive retrieval device 1000 shown can be understood by referring to the relevant description of the bidirectional semantic textualization-based cross-modal archive retrieval method described above. Figure 10 The functions of each unit in the bidirectional semantic textual archive cross-modal retrieval device 1000 shown can be implemented by a program running on a processor or by specific logic circuits.

[0059] In one possible implementation, the semantic text conversion module 1002 is specifically used for: For video archives, keyframe extraction is performed first to extract keyframe images from the video. Then, an initial semantic text description is generated by combining a multimodal big model with structured prompt word templates. Finally, a domain knowledge-enhanced big language model is used to complete metadata specification mapping, professional terminology correction and compliance verification to generate a structured enhanced semantic text description. For image-based archives, which refer to native photo archives, an initial semantic text description is first generated by combining a multimodal large model with structured prompt word templates. Then, a domain-knowledge-enhanced large language model is used to complete metadata specification mapping, professional terminology correction, and compliance verification to generate a structured enhanced semantic text description. For audio archives, an initial structured transcribed text with timestamps is generated through speech recognition and audio understanding models. Then, a large language model with archive domain knowledge enhancement is used to complete metadata specification mapping, professional terminology correction and compliance verification, generating structured enhanced semantic text. For the digitization of paper archives, layout analysis and OCR recognition technologies are used to convert them into structured text, generating structured enhanced semantic text; For text-based archives, their standardized text is directly used as structured, semantically enhanced text.

[0060] In one possible implementation, the index building module 1003 is specifically used for: Text length normalization is performed on all structured enhanced semantic text to ensure that the text data after different modal transformations maintains consistency in document length distribution and information density; All structured enhanced semantic texts after standardization of text length are segmented and stop words are removed to construct a retrieval dictionary and inverted list; The system calculates the number of terms in each document, the global average number of terms in each document, the document frequency of each term, and the total number of documents. It also pre-calculates the inverse document frequency of each term, initializes the index parameters based on the inverted index structure, pre-calculates the parameters required for BM25 sorting, and generates a retrieval index that can be used for relevance sorting.

[0061] In one possible implementation, the query modality recognition and conversion module 1004 is specifically used for: Identify the user's query input modality. If the input modality is a text query, use it directly as the query text. If the input modality is non-textual audio, image, or video queries, they are converted into corresponding modality query texts using differentiated modality conversion methods based on the query modality type: For audio queries, after being converted into text by a speech recognition and audio understanding model, they are processed by a large language model enhanced with archival domain knowledge into structured enhanced semantic text that conforms to archival domain business specifications; for image queries, they are converted into structured enhanced semantic text descriptions that conform to archival domain business specifications through a multimodal large model and a large language model enhanced with archival domain knowledge; for video queries, after keyframe images are generated by extracting keyframes from the video, they are converted into structured enhanced semantic text descriptions that conform to archival domain business specifications through a multimodal large model and a large language model enhanced with archival domain knowledge.

[0062] In one possible implementation, the query semantic parsing and optimization module 1005 is specifically used for: Determine whether the query text meets the preset semantic refinement triggering conditions; when it is determined that the semantic refinement triggering conditions are met, perform semantic refinement processing on the query text to generate simplified query text; when it is determined that the semantic refinement triggering conditions are not met, directly use the query text as simplified query text. Based on a domain knowledge-enhanced large language model, intent recognition and semantic disambiguation are performed on the simplified query text. The simplified query text after intent recognition and semantic disambiguation is expanded with domain synonyms and reconstructed to complete the domain adaptation optimization of query semantics; Generate standardized text retrieval instructions based on the optimized query statement.

[0063] In one possible implementation, the retrieval matching and result output module 1006 is specifically used for: Based on the term occurrence frequency, single-document term count, global average document term count, and term inverse document frequency parameters stored in the retrieval index, a set of candidate documents containing the query term is retrieved through the inverted index. The overall relevance score between the query text and each candidate document is calculated by summing each word and item, and the degree of relevance is represented by the score value.

[0064] In one possible implementation, the retrieval matching and result output module 1006 is specifically used for: A preset similarity threshold is used to filter out low-match candidate documents with relevance scores below the threshold, resulting in a set of qualified candidate documents. The candidate documents that meet the criteria are sorted in descending order of relevance score, and the preset Top-K compliant results are selected. The sorted structured enhanced semantic text is back-mapped to the corresponding original multimodal archive resource; The system outputs original archive data and standardized semantic description information in a linked manner.

[0065] The aforementioned cross-modal archival retrieval device based on bidirectional semantic textualization constructs a bidirectional semantic textualization channel between the semantic text conversion module at the resource processing end and the modality recognition and semantic parsing module at the retrieval end. This unifies the mapping of heterogeneous multimodal archival resources and user multimodal query requests to a standardized semantic text space in the archival domain. Combined with a standardized inverted index and a quantitative relevance matching mechanism built based on the BM25 algorithm, it achieves accurate matching and full-modal linkage output for cross-modal archival retrieval, effectively improving the precision and recall rates of archival retrieval. Furthermore, this embodiment does not rely on large-scale labeled data and complex multimodal alignment training models. It leverages prompting engineering technology and a mature text retrieval engine to achieve unified cross-modal retrieval, significantly reducing system deployment, iteration, and maintenance costs. It possesses good robustness, scalability, and system compatibility, adapting to massive, heterogeneous, and low-quality existing archives and diverse new archive retrieval scenarios. It truly achieves a one-stop intelligent archival retrieval effect of "any modal input, full-modal semantic linkage output," significantly improving the efficiency of deep utilization of multimodal archival resources and the level of intelligent management.

[0066] like Figure 11 As shown, Figure 11 This is a schematic diagram of the composition structure of the electronic device 1100 provided in the embodiments of this application. The electronic device 1100 includes: The device 1100 includes a processor 1101, a storage medium 1102, and a bus 1103. The storage medium 1102 stores machine-readable instructions executable by the processor 1101. When the electronic device 1100 is running, the processor 1101 communicates with the storage medium 1102 via the bus 1103. The processor 1101 executes the machine-readable instructions to perform the steps of the cross-modal file retrieval method based on bidirectional semantic textualization described in this application embodiment.

[0067] In practical applications, the various components in the electronic device 1100 are coupled together via bus 1103. It is understood that bus 1103 is used to achieve communication between these components. In addition to a data bus, bus 1103 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 11 The general labeled all buses as Bus 1103.

[0068] The above electronic device performs differentiated modality-adaptive semantic enhancement processing on text, image, video, and audio archives, and relies on the collaboration of traditional machine learning algorithms and a multimodal large model to complete structured translation of visual and auditory semantics; it relies on a domain knowledge-enhanced large language model to achieve professional standard correction of archives, and constructs a structured enhanced semantic text system with domain adaptation, unified format and dense semantics, which completely breaks through the semantic heterogeneity barrier between multimodal archives. Combined with the standardized inverted index and quantitative relevance matching mechanism built based on the BM25 algorithm, it realizes unified semantic parsing, intent disambiguation and standardized reconstruction of retrieval instructions for user queries of any modality, can accurately mine deep semantic association information of multimodal archives, and effectively improve the precision ratio and recall ratio of archive retrieval. By executing the aforementioned archive cross-modal retrieval method based on bidirectional semantic textualization, the above electronic device can realize bidirectional semantic textualization mapping between the resource end and the retrieval end, cross-modal unified semantic retrieval and full-modal linkage output. Meanwhile, the embodiments of the present application do not need to rely on large-scale labeled data and complex multimodal alignment training models, and can realize the implementation of cross-modal unified retrieval relying on prompt engineering technology and a mature text retrieval engine, which greatly reduces the costs of system deployment, iteration and operation and maintenance, has good robustness, scalability and system compatibility, can adapt to the retrieval scenarios of massive, heterogeneous, low-quality existing archives and diverse newly added archives, truly achieves the one-stop intelligent archive retrieval effect of "any modality input, full-modal semantic linkage output", and significantly improves the deep utilization efficiency and intelligent management level of multimodal archive resources.

[0069] The embodiments of the present application further provide a computer-readable storage medium, where the storage medium stores executable instructions, and when the executable instructions are executed by at least one processor 1101, the archive cross-modal retrieval method based on bidirectional semantic textualization described in the embodiments of the present application is implemented.

[0070] In some embodiments, the storage medium may be Ferroelectric Random Access Memory (FRAM), Read Only Memory (ROM), Programmable Read Only Memory), Erasable Programmable Read Only Memory), Electrically Erasable Programmable Read Only Memory), Flash Memory, magnetic surface memory, optical disc, or Compact Disc Read-Only (CD ROM, Compact Disc Read It can be a memory such as a memory only; or it can be a device that includes one or any combination of the above-mentioned memories.

[0071] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0072] As an example, executable instructions may, but do not necessarily, correspond to files in the file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0073] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0074] The aforementioned computer-readable storage medium stores executable instructions, enabling the processor to implement the aforementioned cross-modal archival retrieval method based on bidirectional semantic textualization during execution. Specifically, it performs differentiated modality-adaptive semantic enhancement processing on text, image, video, and audio archives; relies on traditional machine learning algorithms and multimodal large models to collaboratively complete the structured translation of visual and auditory semantics; and utilizes a domain-knowledge-enhanced large language model to achieve professional standardization correction of archives, constructing a domain-adaptive, format-unified, and semantically dense structured enhanced semantic text system. This completely breaks down the semantic heterogeneity barriers between multimodal archives. Combined with a standardized inverted index and quantitative relevance matching mechanism built based on the classic information retrieval algorithm BM25, it realizes unified querying of any modality by the user. Semantic parsing, intent disambiguation, and standardized reconstruction of retrieval commands can accurately mine deep semantic associations in multimodal archives, effectively improving the precision and recall of archive retrieval. Furthermore, this application's embodiments do not rely on large-scale labeled data and complex multimodal alignment training models; instead, they leverage prompting engineering technology and mature text retrieval engines to achieve unified cross-modal retrieval, significantly reducing system deployment, iteration, and maintenance costs. It possesses excellent robustness, scalability, and system compatibility, adapting to massive, heterogeneous, and low-quality existing archives as well as diverse new archive retrieval scenarios. It truly achieves a one-stop intelligent archive retrieval effect of "any modal input, full-modal semantic linkage output," significantly improving the efficiency of in-depth utilization and intelligent management of multimodal archive resources.

[0075] In the several embodiments provided in this application, it should be understood that the disclosed methods and electronic devices can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another system, or some features may be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0076] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0077] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0078] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a platform server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0079] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A cross-modal retrieval method for archives based on bidirectional semantic textualization, characterized in that, Includes the following steps: The received multimodal raw archival data is preprocessed to obtain standardized archival resources; The standardized archival resources are processed into differentiated semantic texts based on modality type to generate structured enhanced semantic texts that conform to archival business specifications. The structured, enhanced semantic text is stored, and a multimodal archive retrieval index is constructed based on the inverted index; It receives multimodal query requests from users, performs modality recognition and semantic text conversion on the query modality, and generates standardized text retrieval instructions through deep semantic parsing and optimization processing. The standardized text retrieval command is matched with the multimodal archive retrieval index, a relevance score is calculated, candidate archive resources that meet the preset similarity threshold are selected, and multimodal retrieval results are generated.

2. The method according to claim 1, characterized in that, The process of performing differentiated semantic textification of the standardized archival resources based on modality type includes: For video archives, keyframe extraction is performed first to extract keyframe images from the video. Then, an initial semantic text description is generated by combining a multimodal big model with structured prompt word templates. Finally, a domain knowledge-enhanced big language model is used to complete metadata specification mapping, professional terminology correction and compliance verification to generate a structured enhanced semantic text description. For image-based archives, which refer to native photo archives, an initial semantic text description is first generated by combining a multimodal large model with structured prompt word templates. Then, a domain-knowledge-enhanced large language model is used to complete metadata specification mapping, professional terminology correction, and compliance verification to generate a structured enhanced semantic text description. For audio archives, an initial structured transcribed text with timestamps is generated through speech recognition and audio understanding models. Then, a large language model with archive domain knowledge enhancement is used to complete metadata specification mapping, professional terminology correction and compliance verification, generating structured enhanced semantic text. For the digitization of paper archives, layout analysis and OCR recognition technologies are used to convert them into structured text, generating structured enhanced semantic text; For text-based archives, their standardized text is directly used as structured, semantically enhanced text.

3. The method according to claim 1, characterized in that, The multimodal archive retrieval index built based on the inverted index includes: Text length normalization is performed on all structured enhanced semantic text to ensure that the text data after different modal transformations maintains consistency in document length distribution and information density; All structured enhanced semantic texts after standardization of text length are segmented and stop words are removed to construct a retrieval dictionary and inverted list; The system calculates the number of terms in each document, the global average number of terms in each document, the document frequency of each term, and the total number of documents. It also pre-calculates the inverse document frequency of each term, initializes the index parameters based on the inverted index structure, pre-calculates the parameters required for BM25 sorting, and generates a retrieval index that can be used for relevance sorting.

4. The method according to claim 1, characterized in that, The modality recognition and semantic text conversion of the query modality includes: Identify the user's query input modality. If the input modality is a text query, use it directly as the query text. If the input modality is non-textual audio, image, or video queries, they are converted into corresponding modality query texts using differentiated modality conversion methods based on the query modality type: For audio queries, after being converted into text by a speech recognition and audio understanding model, they are processed by a large language model enhanced with archival domain knowledge into structured enhanced semantic text that conforms to archival domain business specifications; for image queries, they are converted into structured enhanced semantic text descriptions that conform to archival domain business specifications through a multimodal large model and a large language model enhanced with archival domain knowledge; for video queries, after keyframe images are generated by extracting keyframes from the video, they are converted into structured enhanced semantic text descriptions that conform to archival domain business specifications through a multimodal large model and a large language model enhanced with archival domain knowledge.

5. The method according to claim 1, characterized in that, The process of generating standardized text retrieval instructions through deep semantic parsing and optimization includes: Determine whether the query text meets the preset semantic refinement triggering conditions; when it is determined that the semantic refinement triggering conditions are met, perform semantic refinement processing on the query text to generate simplified query text; when it is determined that the semantic refinement triggering conditions are not met, directly use the query text as simplified query text. Based on a domain-knowledge-enhanced large language model, intent recognition and semantic disambiguation are performed on the concise query text; After intent recognition and semantic disambiguation, the query text is expanded with domain synonyms and reconstructed to achieve domain adaptation optimization of query semantics; Generate standardized text retrieval instructions based on the optimized query statement.

6. The method according to claim 3, characterized in that, The parameters required for pre-calculating BM25 sorting include: Based on the term occurrence frequency, single-document term count, global average document term count, and term inverse document frequency parameters stored in the retrieval index, a set of candidate documents containing the query term is retrieved through the inverted index. The overall relevance score between the query text and each candidate document is calculated by summing each word and item, and the degree of relevance is represented by the score value.

7. The method according to claim 1, characterized in that, The process of filtering candidate archive resources that meet a preset similarity threshold and generating multimodal search results includes: A preset similarity threshold is used to filter out low-match candidate documents with relevance scores below the threshold, resulting in a set of qualified candidate documents. The candidate documents that meet the criteria are sorted in descending order of relevance score, and the preset Top-K compliant results are selected. The sorted structured enhanced semantic text is back-mapped to the corresponding original multimodal archive resource; The system outputs original archive data and standardized semantic description information in a linked manner.

8. A cross-modal retrieval device for archives based on bidirectional semantic textualization, characterized in that, The device includes: The preprocessing module is used to preprocess the received multimodal raw archival data to obtain standardized archival resources; The semantic text conversion module is used to perform differentiated semantic textification on the standardized archival resources according to modality type, and generate structured enhanced semantic text that conforms to archival business specifications; An index building module is used to store the structured enhanced semantic text and build a multimodal archive retrieval index based on the inverted index; The query modality recognition and conversion module is used to receive multimodal query requests from users and perform modality recognition and semantic text conversion on the query modality. The query semantic parsing and optimization module is used to generate standardized text retrieval instructions through deep semantic parsing and optimization processing; The retrieval matching and result output module is used to match the standardized text retrieval command with the multimodal archive retrieval index, calculate the relevance score, filter candidate archive resources that meet the preset similarity threshold, and generate multimodal retrieval results.

9. An electronic device, characterized in that, include: The device includes a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the cross-modal file retrieval method based on bidirectional semantic textualization as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the cross-modal retrieval method for archives based on bidirectional semantic textualization as described in any one of claims 1 to 7.

Images