A Multimodal Document Storage and Retrieval Method and System Based on Two-Layer Evidence Graph and Joint Optimization
By employing a two-layer evidence graph and joint optimization method, the problem of inaccurate resource calculation in document retrieval was solved, the correspondence between document source relationships and output objects was determined, resource utilization was optimized, and the original text structure was preserved.
Patent Information
- Application Number
- CN202611641642.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-22
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies cannot effectively handle situations in document retrieval where images on the same page may be shared by multiple cells, and image files in different reports may be exactly the same, leading to inaccurate resource calculations and loss of the original text structure in the output results.
A method based on two-layer evidence graphs and joint optimization is adopted to construct the structural and query relationships of documents respectively. By using the source structure graph and query evidence graph, the correspondence between content sources and selection variables is determined, and resource calculation is optimized through an integer optimization model.
It achieves a definite correspondence between document source relationships and shared output objects, reduces model size, preserves the structural relationships of the original text, optimizes resource computation, and avoids duplicate transmission and loss of original text information.
Smart Images

Figure CN123115497A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer information storage and retrieval technology, specifically to a multimodal document storage and retrieval method and system based on a two-layer evidence graph and joint optimization. Background Technology
[0002] Inspection reports, technical manuals, and other documents often scatter a fact across text, tables, and images. For example, numerical cells need to be combined with column headings and unit explanations, and allowable deviations need to correspond to the measurements in the same row; retrieval systems must not only find relevant content but also retain its position and structural relationship within the original text.
[0003] However, when output resources are limited, the number of fragments does not correspond to the actual overhead; images on the same page may be shared by multiple cells, and image files in different reports may be identical; accumulating fragment overhead line by line may overcount shared images, while directly merging identical content may lose their respective sources; source records, record connections, and header fields also consume output space. Existing vector retrieval, graph relationship retrieval, and budget constraint selection can organize candidates from different perspectives and solve the above problems, but the following issues still need to be addressed in specific data structures and execution processes:
[0004] 1) Content files can be shared, but the source location and structural scope need to be saved separately, and the selection calculation must reflect both identities simultaneously;
[0005] 2) Requirements such as the same row or the same table often only constrain specific needs: for example, the width and allowable deviation need to come from the same row, while specifying the attached figure and the conclusion paragraph undertakes other requirements. If all selected content is uniformly applied to the same row conditions, it will change the original meaning of the query.
[0006] 3) An entry may enable multiple source records and multiple actual output objects: Some of these records are always enabled together, and setting selection flags for each record separately will increase the model size; these calculations can only be merged if the enabling conditions remain equivalent and the records are expanded to remain complete.
[0007] Therefore, how to provide a multimodal document storage and retrieval method, system, computer device, and readable storage medium based on a two-layer evidence graph and joint optimization that enables a definite correspondence between document source relationships, shared output objects, and selection variables, and allows resource computation to directly correspond to the final written record, is the problem that this invention urgently needs to solve. Summary of the Invention
[0008] The purpose of this invention is to propose a multimodal document storage and retrieval method and system based on a two-layer evidence graph and joint optimization, which enables a definite correspondence between document source relationships, shared output objects and selection variables, and allows resource computation to directly correspond to the final written record.
[0009] To achieve the above objectives, the technical solution adopted by the present invention is as follows:
[0010] This invention constructs separate graphs for the original structural relationships in the document and the output relationships of the current query: the source structure graph is responsible for explaining where the content comes from and what context is needed; the query evidence graph is responsible for explaining which source records and actual data will be used when an entry is selected, so that the candidate support set in the query evidence graph is merged and the selection mark is selected, and the list of records within the group is retained, so that the model size reduction and the original source retention are achieved simultaneously.
[0011] In a first aspect, the present invention provides a multimodal document storage and retrieval method based on a two-layer evidence graph and joint optimization, comprising the following steps:
[0012] Step 1: Parse the multimodal document, store the content assets, the source appearance of the content in the document, and the source bindings with scope, role, and domain respectively, and save the association between the content assets, the source appearance, and the source bindings;
[0013] Step 2: Using the source bindings stored in Step 1 as nodes, establish a source structure graph consisting of inclusion relationships and necessary dependencies. Establish a multimodal index and range index for encoding input to all source bindings. Map the source structure graph, indexes and document versions to the same release version.
[0014] Step 3: Receive the query and resource limit, fix the release version registered in Step 2, obtain the requirement items, mandatory items, and source pairing conditions between requirement items, locate the source domain through the index in the release version, enumerate candidate items in the domain and determine the requirement matching relationship between the requirement and the candidate, calculate the closure of each candidate along the mandatory dependency relationship in the source structure graph, and determine the allowed requirement pairing according to the complete source key; for each requirement and candidate combination that has satisfied the requirement matching relationship, calculate the non-negative integer requirement matching score according to the similarity of the corresponding modality, and set a positive integer requirement weight for each requirement item;
[0015] Step 4: Based on the candidate entries, closures, and demand matching relationships obtained in Step 3, pre-generate records of text, images, sources, and the relationships between text, images, and sources according to a fixed output configuration, and record their actual byte lengths. Record the coordinate mapping from the source region to the output image. For loads with the same type, size, and bytes, share a single load record. Establish a query evidence graph that binds candidate entries to closure sources and sources to actual loads. For each demand item and candidate entry combination that satisfies the demand matching relationship, pre-generate the corresponding demand allocation record and record its complete byte length. Whether the demand allocation record is written into the search results is determined by the solution results in Step 7.
[0016] Step 5: Based on the query evidence graph established in Step 4, determine which candidate entries reference each source binding and payload. Group source bindings and payloads with the same candidate support set into a group, and save the binding set, payload set and corresponding pre-generated record list within the group respectively. Based on the actual byte length and payload size recorded in Step 4, calculate the byte quantity, number of images and number of pixels of each group, and save the three statistical values as the group resource quantity of the corresponding record group respectively.
[0017] Step 6: Based on the demand matching relationship in Step 3, establish allocation variables for the matched demands and candidates, and establish selection variables for the candidates and the record groups formed in Step 5 respectively; establish output resource measurement relationship based on the pre-generated record length in Step 4 and the group resource quantity counted and saved in Step 5, and write demand coverage, mandatory retention conditions, source matching conditions triggered by demand allocation, and the upper limit of resources received in Step 3 into the same zero-one integer optimization model; calculate the sum of weighted matching scores of actual demand allocation using the demand matching score calculated in Step 3, the set demand weight, and the allocation variables, add one to the upper limit of total bytes as the score weight coefficient, multiply the coefficient by the sum of the aforementioned weighted matching scores, subtract the total output bytes, and use the resulting value as the objective function to be maximized, where the demand matching score is a non-negative integer, the demand weight is a positive integer, and the upper limit of total bytes is a non-negative integer;
[0018] Step 7: For query inputs with non-empty feasible regions under joint constraints, solve the zero-one integer optimization model established in Step 6 to maximize the objective function and obtain the joint results of demand allocation, item selection, and record group activation.
[0019] Step 8: Based on the combined results obtained in Step 7, retrieve the selected entry record, the corresponding associated record to the entry, and the adopted demand allocation record from the pre-generated records in Step 4. Expand the records of the enabled group according to the group member record list saved in Step 5, and write the search results according to the stable identifier and fixed frame format, so that the actual output resource quantity is equal to the corresponding resource measurement value in Step 6.
[0020] Furthermore, source binding is a record with source location, content scope, and structural role: for example, two cells on the same page each have their own source binding, but can share the same page image; images with the same content in two documents also retain their source separately, and bytes are reused only in the content asset or output payload layer.
[0021] Furthermore, the candidate support set is used to indicate when a record enters the results: it consists of all candidate entry identifiers that enable the record; records with identical sets share a single selection flag, and the group cost is obtained by adding the actual lengths of the member records; the record itself, the source field, and the reference relationships between records are all stored in the group member list.
[0022] Furthermore, the requirement matching condition is written on the requirement allocation relationship. The model constraint assumes that the items of the specified requirement come from the same document, table or row, while allowing other requirements to use their own appropriate content. This condition is expressed separately from the necessary dependencies such as table headers and units, and they all participate in the same solution.
[0023] Furthermore, the output format is determined before solving: each record uses a stable identifier, the content bytes are generated in advance, the record length field and the header count field adopt a fixed width, and after selecting and connecting these records according to the solution results, the total number of bytes is the same as the calculated value in the model.
[0024] Furthermore, in step 1, content assets are searched using content summaries and reuse relationships are determined based on content byte consistency; the source appearance is determined by document identifier, document version, and original location, and the source binding is jointly determined by the source appearance, content scope, structural role, and scope; content with the same bytes corresponds to the same content asset, and source bindings with different original locations or different scopes are retained respectively.
[0025] The inclusion relationship in step 2 is used to determine the member scope of chapters, pages, tables, and cells; the necessary dependency relationship is used to connect the headers, units, figure captions, and referenced objects necessary to explain the target content, and records the document version, table, row and column, or coordinate range to which the relationship applies. The association key of the coded input is determined by the actual input summary, model version, and preprocessing configuration. The same coded key is associated with all its source bindings. The published version records the document version mapping, source structure diagram identifier, and index identifier. Vector hits, member enumeration, dependency expansion, and payload preparation for a single query all read the data corresponding to the published version.
[0026] Furthermore, in step 3, the source bindings and explicitly specified required entries matched by the index are used as seeds. A limited source domain is determined based on the page, table, and chapter to which the seed belongs. Member enumeration is performed on the source domain, and members that meet all the requirements of at least one requirement are merged with the seed, and duplicates are removed using the source binding identifier. When matching requirements with candidates, the target object, structural role, content type, numerical conditions, unit conditions, and the conditions specified in the source constraints are checked item by item. A matching relationship is established only if all conditions are met, and unspecified conditions are not included in the constraints. For requirement items specified as same-source pairings in the query, the allowed pairing relationships are recorded based on the complete source key of the candidate target binding.
[0027] Furthermore, the pre-generated records in step 4 include entry records, entry-to-binding association records, source binding records, binding-to-payload association records, payload records, and demand allocation records. Using the top-left corner of the original page as the origin, the page is scaled proportionally to the specified canvas with a centered margin. The source region in the output image is calculated using the same scaling ratio and offset. The scaling ratio, horizontal and vertical offsets, and output region are written into the association records bound to the image payload. The source binding retains the original source coordinates. Each record and mapping parameter is fixed before solving. The record type field, length field, and header record count field have fixed widths, and each field value is within its corresponding representation range. The actual record length includes all bytes of the frame field, identifier, and content. The closure of the candidate entry consists of its target source binding and the source bindings reachable along the necessary dependencies. The number of images and the number of pixels in the images are obtained by summing the corresponding cumulative values of the image records in the enabled group.
[0028] Furthermore, the candidate support set for source bindings consists of all candidate entries in the closure containing the binding. In step 5, the candidate support set for payloads is obtained by taking the union of the candidate support sets of the source bindings that reference the payload. The complete equality of the candidate support sets is used as the grouping condition, and the source field, content bytes, and record type retain their original values. Each group stores the source binding set and payload set within the group. The record list of the group consists of the source records of each binding within the group, the records associated with the payloads, and the payload records of each payload within the group. The actual byte length in the list is accumulated, and the number of images and the number of pixels are accumulated according to the image payloads within the group. Each source binding and payload is assigned to a group, and the group activation variable is equal to the logical union of the selection variables of each entry in the support set of the group. All records in the group are activated together.
[0029] Furthermore, in step 6, each requirement item is assigned to a matching and selected candidate item, and the selected item must be reserved; the same-source pairing in the pairing uses a complete source key that expands step by step from the document identifier and its document version, the same table condition further includes the table identifier, and the same row condition further includes the uniform row identifier within the table; pairing is allowed only when the required fields are complete and the complete source keys are equal; the remaining candidate pairings are subject to the constraint that the sum of the two requirement assignment variables is no greater than one, so that the source pairing condition only applies to the combination of items that actually undertake the requirement;
[0030] In step 6, the length of the entry record and all its associated closure records are included in the entry cost, the cumulative length of the group member records is included in the group cost, and the length of the demand allocation record is included in the allocation cost. The total number of bytes is obtained by adding the fixed header length, the cost of the selected entry, the cost of the enabled group, and the cost of the actual demand allocation. The demand matching score is a non-negative integer, the demand weight is a positive integer, and the upper limit of the total number of bytes is an integer. The score weight coefficient of the objective function is the value obtained by adding one to the upper limit of the total number of bytes. This coefficient is multiplied by the sum of the weighted matching scores of the actual demand allocation, and then the total number of bytes output is subtracted. The maximum value of the obtained objective function is then calculated to obtain the highest weighted matching score first, and then select the combination with the smaller total number of bytes output from the feasible combinations with the same score.
[0031] Furthermore, in step 8, the enabled source bindings and payload records are restored according to the group member record list to stably identify the connection entries, sources, and payloads; the records are sorted by type and identifier, and the same payload record is written once, while each source binding and its respective image position mapping are retained; the reading end of the retrieval result locates the image content based on the output area in the associated record and locates the original text based on the original coordinates in the source binding; a fixed-length header is written at the beginning of the byte stream of the retrieval result, and then the pre-generated record bytes are connected; the group division only merges the selection markers, and different records within the group exist separately in the output.
[0032] Secondly, the present invention provides a multimodal document storage and retrieval system based on a two-layer evidence graph and joint optimization, for executing a multimodal document storage and retrieval method based on a two-layer evidence graph and joint optimization, comprising:
[0033] The module includes a storage index module, a candidate processing module, a record preparation module, a joint solution module, and a result output module.
[0034] Furthermore, a multimodal document storage and retrieval system based on a two-layer evidence graph and joint optimization includes:
[0035] The storage index module is used to manage content assets, source occurrence, source binding, source structure graph, and multimodal indexes;
[0036] The candidate processing module is used to locate the source domain, enumerate members, expand necessary dependencies and register requirement matching conditions, calculate requirement matching scores and set requirement weights.
[0037] The record preparation module is used to generate fixed-byte records, build query evidence graphs, group candidates by support set, and count the group resource quantity of each group.
[0038] The joint solution module is used to establish and solve the integer optimization model based on the group resource quantity and demand conditions;
[0039] The results output module is used to expand the list of group members according to the same joint result and write the search results.
[0040] Thirdly, a computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement a multimodal document storage and retrieval method based on a two-layer evidence graph and joint optimization.
[0041] Fourthly, a computer-readable storage medium having a computer program stored thereon, wherein when executed by a processor, the computer program implements a multimodal document storage and retrieval method based on a two-layer evidence graph and joint optimization.
[0042] Compared with the prior art, the beneficial effects of the present invention are:
[0043] The method provided by this invention separates content assets, source occurrence, and source binding, while still allowing references in shared files to be located separately. It relies on entries entering the selection range to process cells together with their headers and units. Demand pairing constraints limit the combination of entries that actually fulfill the demand, avoiding cross-line concatenation of values and descriptions.
[0044] The method provided by this invention, when merging records based on the same candidate support set, requires only one enable tag for each group. Different sources within the group can still be fully expanded. Let the sum of the source binding number and the payload number be the original resource tag number. The resource tag number after grouping is equal to the number of different support sets and is not greater than the original value. This merging maintains the feasibility of entry selection, target value, and output resource quantity.
[0045] The pre-generated records and fixed framing provided by this invention enable the byte overhead to be precisely added on a record-by-record basis; the number of images and the number of pixels are accumulated separately according to different payloads, and the output directly uses the same selection result and group member list, which can reduce the repeated transmission of shared images while retaining independent original text positioning information. Attached Figure Description
[0046] Figure 1 This is a flowchart illustrating the overall process of a multimodal document storage and retrieval method based on a two-layer evidence graph and joint optimization according to the present invention.
[0047] Figure 2 This is a flowchart illustrating the document storage, source relationships, and retrieval index establishment process of this invention.
[0048] Figure 3 This is a flowchart illustrating the processing of candidate content, necessary context, and shared output data for this invention.
[0049] Figure 4 This is a flowchart illustrating how the present invention supports set grouping, joint solution, and result output. Detailed Implementation
[0050] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The duplicate content determination in the process is used to determine byte reuse relationships; source location, demand matching, and support set grouping always belong to the same processing flow. The document data used below has a determined structural location and dependencies, and the query candidates and resource parameters satisfy the condition that the feasible domain of the joint constraint is not empty.
[0051] Example 1: A multimodal document storage and retrieval method based on two-layer evidence graphs and joint optimization:
[0052] like Figure 1 As shown, the storage phase processes the original document, the query phase reads a fixed published version, and the system outputs the search entries, source location, actual bytes of text and images, and records connecting these objects at the end of the query. The output can be used by the document viewer to locate the source, or it can be passed to the subsequent content processing program.
[0053] Step 1: Parse the document and record the content and source respectively:
[0054] Read the main text paragraphs, tables, cells, page images, captions, and explicit references from the document to obtain the original object paths and page coordinates. For tables, record the row numbers, column numbers, merge ranges, and the table identifiers they belong to. Standardize the page coordinates to a coordinate system with the top-left corner as the origin, the horizontal axis to the right, and the vertical axis downwards, based on the original page size. Record the original page width and height and the rectangle of the source area, and retain the correspondence with the original position. The text position is represented by the original object path and character range, with the original page width and height being positive, and the source area located within the page range.
[0055] according to Figure 2 The process shown establishes three types of records: content assets store the bytes, content summary, and type of the original or media; the summary is used for searching; byte consistency is used to confirm reuse; and source occurrence stores the document identifier, document version, and original location, pointing to the content asset. The source is bound to a source occurrence to further specify the content scope, role, and domain.
[0056] For example, when a chart appears in two reports, the content assets can be shared, but the two sources are saved separately. The width cell and the deviation cell in a table page are also bound separately. The source binding identifier is determined by the document identifier, document version, original location, content scope, structural role and scope. The identifier is encoded according to field type, field length and field content to avoid different field combinations forming the same identifier.
[0057] The structural roles include body text, data cells, column headers, unit declarations, images, and captions. The scope of unit declarations is determined according to the hierarchy of cells, columns, and tables, taking the innermost explicit declaration that covers the target data. Table headers are associated according to the column span, header hierarchy, and merging range in the same table. Tables that continue across pages use the same table identifier and row and column mapping determined when the database is created.
[0058] Step 2: Save the source structure diagram and create an index:
[0059] The source structure graph is built using source bindings, with edges representing the affiliation of chapters, pages, tables, and structural members. Required dependency edges point from the content to be explained to its necessary context. Each dependency stores the complete document source key, table or chapter identifier, scope of application, and relational basis. For example, a data cell points to its column header and unit declaration, and a cross-page entry points to its contiguous section.
[0060] Text vector index, page visual index, local image and text index, and range index are established. The association key of the coded input consists of the actual input byte summary, model version, and preprocessing configuration. All source bindings are stored under one association key. If the content is the same, only the duplicate storage of the coded input is reduced. All sources can still be expanded when querying. The range index retrieves structural members by document, version, page, table, and coordinate range.
[0061] Register the document version mapping, source structure diagram, and index identifier in the release list V. e Each release generation e corresponds to a set of completed storage records. Queries start with e fixed, and subsequently read the graph, index, and documents corresponding to that generation. Document updates form new release generations, and the version read by existing queries remains fixed.
[0062] Step 3: Find candidate content and required context:
[0063] The query is broken down into a set of requirement items T, with each requirement item being handled by a candidate item. Information that requires multiple items to provide is broken down into multiple requirement items. Each requirement item records the target object, structural role, content type, numerical conditions, unit conditions, and source restrictions specified in the query. Unspecified fields are marked as unrestricted. Explicitly specified attached figures or paragraphs are registered as mandatory items. For requirements such as "the value and its deviation come from the same row", the requirement items that need to be paired and the pairing level are recorded.
[0064] Fixed output configuration, including text encoding, page rendering size, image encoding, and recording frame format, with a maximum total received byte size of B. bytes Maximum number of images B img and the maximum number of total pixels in the image B pix All three upper limits are non-negative integers. The byte budget only constrains the application layer result byte stream defined by this method, without considering network protocol encapsulation or converting it to the word segmentation number of the external model. The pixel count is the sum of the products of the width and height of each output image. The image width and height are calculated according to the actual integer dimensions.
[0065] In the fixed publishing generation, text, visual, and graphic indexes are retrieved separately, and a preset number of hit results are read. The hit code key is expanded into source binding, and the seed is obtained after document and version restrictions. Explicitly required entries are directly added to the seed, and a limited source domain is determined based on the page, table, and chapter where the seed is located.
[0066] like Figure 3 As shown, a range query and edge traversal are performed for each source domain, listing the structural members within the domain. Members that meet at least one requirement are merged with the seed and deduplicated using the source binding identifier to obtain a candidate item set R. This enumeration is completed at this stage of each query, allowing finer cell and page-level hits to enter the selection range together.
[0067] Each requirement t is examined for all specified conditions regarding the target object, structural role, content type, value, unit, and source. Only requirements that satisfy all conditions and candidate pairs are included in the matching set A. The target object is determined by the candidate target binding and its necessary context. For example, the width field and the allowable deviation field are distinguished based on the associated table header. The structural role and content type are taken from the target binding and are not replaced by the roles or types of other bindings in the closure. If the fields required by the specified conditions are missing or cannot be determined, the candidate will not match. No additional restrictions are imposed for unspecified conditions. The numerical comparison is first unified according to the registered unit relationship. The semantic similarity is only used for the scoring after matching. The closure of the candidate r is denoted as Cl(r), which includes its target source binding and all bindings reachable along the necessary dependency edges.
[0068] Dependency traversal records visited nodes by binding identifier. Visited bindings are not expanded repeatedly. Bindings in each candidate closure are merged into set B. Even if the table header, unit, and caption do not directly undertake query requirements, they can still enter the query through the closure.
[0069] Register the required pairs of demands as a set Q. Each pair of demands is registered only once. The source key for the same document version consists of the document identifier and the document version. The source key for the same table is the table identifier after the above two items. The source key for the same row is the unified row identifier within the table. When continuing a table across pages, the same table identifier and unified row mapping confirmed when the database was created are used. Local row numbers within a page are not used as a separate pairing criterion. The published data is used to retrieve fixed data and does not replace the document version.
[0070] For a combination of candidate r undertaking demand t and candidate s undertaking demand u, read the complete source keys of both at the specified level according to the demand. If all required fields exist and are equal item by item, then pair the constant K. turs Take 1; take 0 for other combinations. Unregistered fields are not considered to be from the same source based on the same null value or default line number. The same version number, table number or line number in different documents do not constitute the same source. Pairing is based only on the complete hierarchical key of the original document, not on the shared load identifier or output image coordinates, and only applies to the actual allocation of the pair to the requirements.
[0071] After the matching set A and the demand pairing relationships are determined, the demand matching score s is calculated for each demand in A and the candidate matching pair. tr The similarity of the corresponding modes is normalized to 0 to 1 according to a predetermined rule, then multiplied by 1000 and rounded to the nearest integer, resulting in an integer between 0 and 1000. A positive integer demand weight w is assigned to each demand item in the demand item set T. t The demand matching score and demand weight are passed as output parameters from step 3 to step 6 and remain unchanged during this solution process.
[0072] Step 4: Prepare output records in advance:
[0073] The source binding set B is obtained by merging the source bindings contained in the closures of each candidate entry after deduplication according to the binding identifier, where b represents any source binding in set B. According to the fixed output configuration, actual output data is generated for the content involved in the source bindings in set B. The main text and independent text descriptions use defined text bytes. The page visual evidence is drawn onto a specified canvas with positive integer width and height, and saved according to a defined image encoding. The image type, size, and bytes together determine its output identity.
[0074] When drawing the page, first calculate the ratio of the specified canvas width to the original page width and the ratio of the specified canvas height to the original page height, and take the smaller value as the scaling ratio. Multiply the original page width and height by this ratio respectively, use half of the remaining canvas width as the horizontal offset and half of the remaining height as the vertical offset, draw proportionally at this position and center with white space. The scaling ratio and offset use a defined numerical code, and the page drawing and position mapping use the same set of parameters.
[0075] For each source region, the x-coordinate of its original rectangle is multiplied by the scaling factor and then the horizontal offset is added; the y-coordinate is multiplied by the scaling factor and then the vertical offset is added. The left and top boundaries of the resulting rectangle are rounded down, and the right and bottom boundaries are rounded up, and the rectangle is confined within the canvas boundary to obtain an integer pixel rectangle covering the region. The original source coordinates are stored in the binding record, and the scaling factor, horizontal and vertical offsets, and output rectangle are stored in the associated record bound to the image payload, all of which are fixed before solving.
[0076] The original image, which is saved separately, uses its own width and height and the region coordinates with the top left corner as the origin. It is processed according to the same proportional scaling and centering rules. The original text location is still saved in the source binding. Even if different sources share the same image payload, their respective mappings are registered separately, and the region of one place does not cover other sources.
[0077] Output objects with the same type, size, and complete payload bytes share a single payload identifier. All output payloads, after deduplication by payload identifier, are grouped into a set P, where p represents any payload in set P, and P(b) represents the set of payloads referenced by source binding b. Source bindings from different origins are retained separately; payload reuse does not merge source bindings.
[0078] Establish the evidence graph for this query, with binding edges from candidate r to Cl(r) and load edges from candidate b to P(b). Based on the connection relationship between candidates, sources, and actual data, determine which records must be written when each candidate enters the result.
[0079] Six types of records are pre-generated: Entry records store target bindings and stable entry identifiers; Entry-to-binding association records store closure connections; Source binding records store document, version, native location, original scope, role, and domain; Binding-to-load association records store load references and, when referencing an image, store the scaling ratio, horizontal and vertical offsets, and output area of the binding; Load records store the actual text or image; Requirement assignment records store the connections between requirement items and assigned entries, with nodes and records distinguished by type and stability identifiers.
[0080] In this embodiment, frames are divided sequentially according to "type, identifier length, identifier bytes, content length, and content bytes". The type is 1 byte, the identifier length is a 2-byte unsigned integer, and the content length is a 4-byte unsigned integer. The multi-byte integers are encoded in big-endian. The number of identifier bytes does not exceed 65535, and the number of content bytes does not exceed 4294967295. The payload record also includes a 1-byte media type and two 4-byte unsigned size fields. The image size is an integer from 1 to 4294967295, and the text size is 0. The content, mapping parameters, and stability identifier of each record are determined before solving.
[0081] The header of the retrieval result consists of a 4-byte format identifier, a 2-byte unsigned format version, an 8-byte unsigned publication code, a 32-byte query summary, and six 4-byte unsigned record count fields, with a length of 70 bytes, denoted as H. The format version does not exceed 65535, the publication code does not exceed 18446744073709551615, and the number of pre-generated records for each type does not exceed 4294967295. The output is a subset of these records, so each count is within the range of 0 to 4294967295. The solution only changes the value of the count field, and the header length is fixed.
[0082] For each candidate entry r in the candidate entry set R, let c be the sum of the full byte lengths of its entry record and the associated records to all closure members. r For each source binding b in the source binding set B, let c be the sum of the full byte lengths of its source binding record and its associated record to the payload. b This includes the bytes occupied by the image location mapping; for each load p in the load set P, the complete byte length of its load record is denoted as c. p Demand allocation records are pre-generated only for matching pairs of demand item t and candidate item r in matching set A, and the full byte length of the record is denoted as c. tr The above lengths include all bytes of the framing field, identifier, and content, and are defined as non-negative integers.
[0083] Step 5: Merge records with the same enable conditions:
[0084] The candidate support set is calculated from the query evidence graph. The support set of binding b consists of all candidates whose closure contains the binding. The support set of payload p is obtained by merging the support sets of each binding that references the payload. The same candidate is only kept once in the set. Both B and P are expanded from the current candidate, so each support set is not empty.
[0085] The support sets are sorted by candidate identifiers and compared item by item. Bindings and payloads that are completely equal in the support sets are grouped into a record group. The group index set is denoted as G. A summary is calculated and can be used to find the same set. The equality of the groups is still determined by the equality of the sorted identifier list. A binding or payload is only assigned to one group.
[0086] For group g, store the group binding set B respectively. g The load set P within the group g and the common support set S g B g With P g At least one of them is not empty. The binding sets of different groups do not overlap and are merged into B; the load sets of different groups do not overlap and are merged into P. The group record list is D. g List B gEach bound source record and all records bound to the payload association, as well as P g Load records for each load; each entry saves the record type, stability flag, and pre-generated bytes, as shown in Listing D. g It is the actual record list, B g and P g It is a collection of two types of nodes, and the three types are stored separately.
[0087] For example, if a unit text and its source binding are both referenced only by the same set of numerical entries, they may both appear in the results or neither may appear in the results under any candidate selection. They can share an enable flag, but the unit text and the source record are still written separately. Figure 4 The "group by support set" in the text corresponds to this step.
[0088] Group cost d g Equals List D g The sum of the lengths of all records in the memory, which is B g Internal binding costs c b With P g Cost of each load within the container c p The sum of the number of images in the group n g Only count P g Image payload within, group pixel count v g The sum of the width-to-height ratios of these images is used. Each payload is assigned to only one group, so its bytes, number of images, and pixels are counted only once. The group cost, number of images in the group, and number of pixels in the group correspond to the number of bytes, the number of images, and the number of pixels, respectively, which together constitute the group resource quantity of the group. The three statistical values are stored separately in bytes, frames, and pixels, and are used for the measurement and constraint of the corresponding resources after being passed to step 6.
[0089] The associated record bound to the load belongs to the group to which the binding belongs. Even if it points to a load in another group, it is only measured once within the binding group. The item and the associated record of the item to the binding are included in the item cost, and the demand allocation record is included in the allocation cost. The recording scope of the three types of costs does not overlap.
[0090] Step 6: Establish a joint constraint model:
[0091] Establish zero-one variables a only for the requirement and candidate pairs in the matching set A tr This indicates that demand t is undertaken by candidate r; the remaining combinations are logically zero and no variables are created, while a zero-one variable x is created for each candidate r. r For each group g, establish a zero-to-one variable u g , respectively, indicate that the item has been selected and the record group has been enabled; the m flag must be retained. r Set the value to 1, and the rest to 0. All matching, pairing, closure, grouping, and cost parameters are fixed during the solution process.
[0092] For demand t, R t T represents the set of candidates that match it; for candidate r, T r Let T represent the set of requirements that match it. Each requirement t belonging to T satisfies the summation equation of equation (1), and each requirement belonging to A and its candidate pair satisfies the inequality therein, such that each requirement is undertaken by exactly one matched and selected candidate:
[0093]
[0094] Each candidate r belonging to R satisfies equation (2), and must be retained in the result. The remaining candidates can only be selected if they fulfill at least one requirement; T r When the summation is empty, the corresponding summation is set to 0:
[0095]
[0096] When each group g belongs to G, its enabled state is to support set S. g The logical union of candidate selection states, and the lower bound of equation (3) for each r belonging to S g If the candidate is valid, the upper bound holds true for the entire set of support:
[0097]
[0098] For each pair of requirements t and u belonging to Q, check one by one whether r belongs to R. t s belongs to R u Candidate combinations, pairing constant K turs Combinations with a value of 0 satisfy:
[0099]
[0100] In equation (4), t and u represent two demand items that need to be paired, and r and s represent candidate items that undertake demand t and demand u, respectively; Q is the set of demand pairs that need to be paired, and R t and R u Let a be the candidate set that matches demand t and demand u, respectively. tr and a us All are zero- or one-demand allocation variables: a tr Setting it to 1 indicates that demand t is assigned to candidate r, a us Setting it to 1 indicates that the requirement u is assigned to the candidate s, and setting it to 0 indicates that no corresponding assignment is made.
[0101] K turs Assign corresponding source pairing constants to the above two items. When two candidates satisfy this requirement—having the specified source level, all required fields, and the same complete source key—K... turs Take 1; otherwise, take 0. K tursWhen the value is 0, equation (4) limits a. tr With a us Both cannot be 1, meaning that these two allocations that do not meet the source pairing conditions are prohibited from being valid at the same time.
[0102] Equation (4) restricts the two requirement allocation relationships. For example, when "width" and "deviation" are required to be on the same row, only the complete row source key that bears these two requirements is compared; the appearance diagram that is specified to be retained can be located on other pages of the same report, and the document and version constraints of a single requirement itself have been checked during the creation of the matching set A.
[0103] The total output byte size L is calculated based on the layer to which the record belongs. The summation of the three terms in equation (5) traverses all candidate R, all record groups G, and the matching set A, respectively. Equation (6) simultaneously restricts the byte size, the number of images, and the number of pixels:
[0104]
[0105]
[0106] In equation (5), L is the total number of bytes in the byte stream of the current retrieval result, H is the fixed header length of the byte stream, and H is 70 bytes in this embodiment; R is the set of candidate entries, G is the set of indexes of record groups, and A is the set of matching pairs between requirements and candidates; t, r and g are the indices of the requirement item, candidate entry and record group respectively, and the summation of each item traverses the sets listed in the equation.
[0107] c r The sum of the byte lengths of the candidate r's entry records and all its entries associated with the closure member records; d g This refers to the sum of the byte lengths of all records in the member list of record group g, including records bound to each source within the group and their associated records bound to the payload, as well as the payload records of each payload within the group; c tr Assign the full byte length of the record corresponding to demand t and candidate r. The above length includes the framing field, identifier, and content of each record. r u is the selection variable for candidate r. g A is the enable variable for record group g. tr The allocation variables are for demand t to candidate r. All three values are either 0 or 1. When the value is 1, the cost of the corresponding record is included in L. When the value is 0, it is not included.
[0108] In equation (6), B bytes This represents the maximum allowed total output bytes for this query, in bytes; B img The maximum number of allowed output images, in units of frames; B pix n represents the maximum allowed total number of pixels in the output image, expressed in pixels. g v represents the number of different image payloads within group g.g The sum of the products of the width and height of each of these image payloads; for the record group without image payloads, its n g and v g All are 0, g, G and u g The meaning is the same as in equation (5). Each image load is assigned to only one group, so the sum of the two groups is the actual number of output images and the total number of pixels, respectively.
[0109] Using the demand matching score calculated in step 3 and the set demand weights, with equation (7) as the objective, together with equations (1) to (6) and the zero-to-one value conditions, an integer optimization model is formed:
[0110]
[0111] In equation (7), max represents maximizing the objective function within the feasible region defined by equations (1) to (6) and the condition of taking zero and one values; A is the set of matching pairs of demand and candidate, t and r are the demand index and candidate index respectively, and w t Let s be the positive integer weights of demand t. tr The matching score of candidate r to requirement t is an integer from 0 to 1000 according to the aforementioned rules; s tr In this context, 's' represents the score, which differs from the meaning of 's' used as a candidate index in equation (4). tr A value of 1 indicates that the matching pair is actually used, and therefore the summation term is the sum of the weighted matching scores assigned to the actual needs.
[0112] L is the total number of output bytes determined by equation (5), B bytes Let B be the upper limit of the total number of bytes in equation (6), M be the score weight coefficient in the objective function, and let B be the upper limit of the total number of bytes in equation (6). bytes Add 1, and jointly determine the demand allocation variable a during the solution process. tr Candidate selection variable x r and group enable variable u g Matching score, demand weight, recording cost, and resource limit are all kept to predetermined parameters.
[0113] The target coefficient M is an integer B. bytes Add 1, the absolute value of the difference in bytes between any two feasible results does not exceed B. bytes Since the weighted score is an integer, the change in the target brought about by a unit score is greater than the difference in the number of bytes. Equation (7) first selects the highest weighted matching score, and then selects the smaller number of bytes among the combinations with the same score.
[0114] Step 7: Solve and restore the selection results for each layer:
[0115] For this query with a non-empty feasible region, the optimal solution of the above model is obtained using a zero-one integer programming solver, yielding the joint values of demand allocation, candidate selection, and group activation. Expanding u... g List of records D for each group with a value of 1 g Obtain the enabled source record, payload record, and associated records.
[0116] Grouping equivalence can be directly described by the support set. After fixing any candidate selection set, a binding record and its associated load record, or a load record, are enabled if and only if at least one selected candidate appears in its support set. Members of the same support set have the same value. Equation (3) gives the logical union. The grouping cost is the sum of the lengths of each member record. Therefore, when using the same matching set, pairing conditions and resource limits, the feasible candidates, resource quantity and target value are the same before and after grouping.
[0117] Let N be the number of candidates, and A be the number of requirements and candidate pairs in the matching set A. n The number of source bindings is B. n The number of loads is P n The number of different support sets is G. n When setting the enable variable separately for each binding and payload, there are a total of N+A. n +B n +P n There are zero or one variables; after grouping by support set, the result is N+A. n +G n One, and G n Not greater than B n With P n The sum is the variable count during the model construction phase, which has not yet undergone solver preprocessing. Grouping reduces resource-enabled variables, and integer solving is still a combinatorial optimization on a finite number of candidates.
[0118] Step 8: Directly write out the search results:
[0119] Retrieve the selected candidate entry record and its closure association record, then select by a. tr Obtain the demand allocation record, retrieve the source, payload and its associated record according to the member list of the enabled group, sort all records by type and stability identifier, first write the 70-byte header determined in step 4 at the beginning of the retrieval result byte stream, then concatenate each prepared byte sequence, write each payload identifier once, and write each different source binding separately.
[0120] Each group of record lists does not overlap with each other. The entry record and its associated record are written only with the corresponding candidate. The demand allocation record is written only with the corresponding allocation. Therefore, the final byte length is equal to L in equation (5), and the number of images and the total number of pixels are equal to the sum of the corresponding two terms in equation (6).
[0121] The result reading end is a result processing program that receives and parses the byte stream output from step 8. The reading end parses the packet header and records according to a fixed frame format, finds the entries from the demand allocation records, and then obtains the text, images, and context based on the association between the entries and bindings, and between the bindings and the payload. The image content is located according to the output area in the associated record. The original text is located according to the document identifier, document version, original path, and original coordinates in the source binding. The same images from different sources can share the actual bytes, and the original text position and image mapping of each occurrence can still be found separately.
[0122] Specific calculation examples are recorded;
[0123] The following example of a cross-page inspection table illustrates the above processing. The query requires a width value, its allowable deviation, and a conclusion. The width and deviation must originate from the same row, and a specified appearance image must be retained. The target objects for the width and deviation requirements are the width field and the allowable deviation field, respectively. Both are data cells in structure and numeric in content type, with millimeters as the unit and no separate numerical range specified. The target object for the conclusion requirement is the conclusion paragraph, with the body text as its structure and text type. The first row in the table has a width of 12.0 mm and a deviation of ±0.1 mm; the second row has a width of 12.1 mm and a deviation of ±0.2 mm.
[0124] The candidates are first row width, first row deviation, second row width, second row deviation, specified appearance drawing, and conclusion paragraph. The two rows are located on page one and page two, respectively. The width and deviation are each bound to the image on the page where they are located. The two width entries share the width header binding, the two deviation entries share the deviation header binding, the four numerical entries share the table-level unit binding, the appearance drawing and the figure caption belong to the same closure, and the conclusion paragraph has an independent text load.
[0125] This example has 10 source bindings and 8 payloads. The payloads include two pages of images, an appearance diagram, a conclusion text, a unit text, two table header texts, and figure caption texts. The table headers, units, and figure captions are located through their own source records. Each page image is 640×480 pixels, and the appearance diagram is 320×240 pixels.
[0126] The matching scores of the four numerical candidates are 950, 850, 900 and 980 respectively, the conclusion score is 700, the demand weight is 1 for each, the width demand matches two width entries, the deviation demand matches two deviation entries, and the conclusion demand matches the conclusion paragraph. There are a total of 5 demand-candidate matching pairs. The two lines belong to the 3rd edition of the document "Inspection Report 1" and the table "Dimension Inspection Table". The uniform row identifiers are 1 and 2 respectively. The width and deviation are matched with a complete source key consisting of the document identifier, document version, table identifier and uniform row identifier.
[0127] After grouping by support set, the 18 binding and payload resource tags are grouped into 11 groups. For example, unit binding and unit text have the same four-entry support set; appearance drawing binding, caption binding, image payload and caption text are all supported only by specified appearance drawing entries, and the four share an enable tag, while each independent record is still stored within the group.
[0128] The image limit is set to 2, the total pixel limit is 400,000, and the byte limit is an integer not less than the sum of the lengths of all pre-generated records and the header length in this example. Therefore, the byte limit in this example does not exclude candidate combinations. The model selects the second row width, the second row deviation, the specified appearance image, and the conclusion section. The weighted score is 2580. The width and deviation share the same image on page 2. In addition to the appearance image, there are a total of 2 images and 384,000 pixels. The score of the same row combination in the first row is 2500. The combination of the first row width and the second row deviation does not meet the condition that the source keys of the complete row are the same, and it also requires more than 3 images above the limit.
[0129] The result includes 4 entry records, 9 entry-to-binding association records, 8 source binding records, 8 binding-to-load association records, 7 load records and 3 demand allocation records. Each of the two values has 3 closure associations of target, header and unit. The appearance diagram has 2 targets and captions, and the conclusion section has 1, for a total of 9. After being merged according to the binding identifier, there are 8 source bindings. The 7 loads are the page 2 image, appearance diagram and the text of conclusion, unit, two headers and captions. The actual lengths of these six types of selected records are added together, and then 70 bytes of header are added to get the total number of bytes in equation (5).
[0130] In this example, there are 6 candidate selection variables and 5 matching assignment variables. When variables are enabled for each binding and load setting, there are a total of 29 zero and one variables in the model construction stage; after grouping, there are 22. The above count includes variables that have been fixed by the mandatory retention condition and have not yet been preprocessed by the solver. The 8 source bindings and 7 loads in the output still exist as originally recorded. The reduction of variables does not omit the source.
[0131] Example 2: A Multimodal Document Storage and Retrieval System Based on Two-Layer Evidence Graphs and Joint Optimization
[0132] The storage index module executes steps 1 and 2 to save the content bytes, original text position, structural relationships, index, and publication list;
[0133] The candidate processing module executes step 3, outputting the requirement matching relationship, requirement matching score, requirement weight, requirement pairing constant, must-retain flag, and the required dependency closure of each candidate;
[0134] The record preparation module executes steps 4 and 5, and outputs pre-generated records, query evidence graphs, candidate support sets, grouping lists, and corresponding group resource quantities.
[0135] The joint solution module executes steps 6 and 7, and provides the same set of selection results for the requirements, candidates, and record groups;
[0136] The results output module executes step 8, thereby reading existing records and generating search results. All modules transmit the same release code and a fixed output configuration identifier.
[0137] Example 3: Computer Equipment
[0138] The computer device includes a processor, memory, and input / output interfaces. The memory stores the data structure, index, record bytes, and computer program of this method. The processor executes the program to complete document parsing, source registration, candidate retrieval, support set grouping, joint solution, and result output. The input / output interfaces receive documents, queries, and resource parameters and return multimodal retrieval results.
[0139] Example 4: Computer-readable storage medium
[0140] A computer program is stored in a computer-readable storage medium, which, when executed by a processor, implements the method described in Example 1.
[0141] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any modifications or equivalent changes made based on the technical essence of the present invention shall still fall within the scope of protection claimed by the present invention.
Claims
1. A multimodal document storage and retrieval method based on two-layer evidence graphs and joint optimization, characterized in that, Includes the following steps: Step 1: Parse the multimodal document, store the content assets, the source appearance of the content in the document, and the source bindings with scope, role, and domain respectively, and save the association between the content assets, the source appearance, and the source bindings; Step 2: Using the source bindings stored in Step 1 as nodes, establish a source structure graph consisting of inclusion relationships and necessary dependencies. Establish a multimodal index and range index for encoding input to all source bindings. Map the source structure graph, indexes and document versions to the same release version. Step 3: Receive the query and resource limit, fix the release version registered in Step 2, obtain the requirement items, mandatory items, and source pairing conditions between requirement items, locate the source domain through the index in the release version, enumerate candidate items in the domain and determine the requirement matching relationship between the requirement and the candidate, calculate the closure of each candidate along the mandatory dependency relationship in the source structure graph, and determine the allowed requirement pairing according to the complete source key; for each requirement and candidate combination that has satisfied the requirement matching relationship, calculate the non-negative integer requirement matching score according to the similarity of the corresponding modality, and set a positive integer requirement weight for each requirement item; Step 4: Based on the candidate entries, closures, and demand matching relationships obtained in Step 3, pre-generate records of text, images, sources, and the relationships between text, images, and sources according to a fixed output configuration, and record their actual byte lengths. Record the coordinate mapping from the source region to the output image. For loads with the same type, size, and bytes, share a single load record. Establish a query evidence graph that binds candidate entries to closure sources and sources to actual loads. For each demand item and candidate entry combination that satisfies the demand matching relationship, pre-generate the corresponding demand allocation record and record its complete byte length. Whether the demand allocation record is written into the search results is determined by the solution results in Step 7. Step 5: Based on the query evidence graph established in Step 4, determine which candidate entries reference each source binding and payload. Group source bindings and payloads with the same candidate support set into a group, and save the binding set, payload set, and corresponding pre-generated record list within each group. Based on the actual byte length and payload size recorded in Step 4, calculate the byte volume, number of images, and number of pixels for each group. Save the three statistical values of byte volume, number of images, and number of pixels as the group resource volume of the corresponding record group. Step 6: Based on the demand matching relationship in Step 3, establish allocation variables for the matched demands and candidates, and establish selection variables for the candidates and the record groups formed in Step 5 respectively; establish output resource measurement relationship based on the pre-generated record length in Step 4 and the group resource quantity counted and saved in Step 5, and write the demand coverage, mandatory retention conditions, source matching conditions triggered by demand allocation, and the upper limit of resources received in Step 3 into the same zero-one integer optimization model. The sum of the weighted matching scores of the actual demand allocation is calculated using the demand matching score calculated in step 3, the set demand weight, and the allocation variables. The value obtained by adding one to the upper limit of the total byte size is used as the score weight coefficient. This coefficient is multiplied by the sum of the aforementioned weighted matching scores, and then the total output byte size is subtracted. The resulting value is used as the objective function to be maximized, where the demand matching score is a non-negative integer, the demand weight is a positive integer, and the upper limit of the total byte size is a non-negative integer. Step 7: For query inputs with non-empty feasible regions under joint constraints, solve the zero-one integer optimization model established in Step 6 to maximize the objective function and obtain the joint results of demand allocation, item selection, and record group activation. Step 8: Based on the combined results obtained in Step 7, retrieve the selected entry record, the corresponding associated record to the entry, and the adopted demand allocation record from the pre-generated records in Step 4. Expand the records of the enabled group according to the group member record list saved in Step 5, and write the search results according to the stable identifier and fixed frame format, so that the actual output resource quantity is equal to the corresponding resource measurement value in Step 6.
2. The multimodal document storage and retrieval method based on two-layer evidence graphs and joint optimization according to claim 1, characterized in that, In step 1, content assets are searched using content summaries and reuse relationships are determined based on the consistency of content bytes. The source of occurrence is determined by document identifier, document version, and original location, while the source binding is determined by the source of occurrence, content scope, structural role, and scope. Content with the same bytes corresponds to the same content asset, and source bindings with different original locations or different scopes are retained respectively. The inclusion relationship in step 2 is used to determine the member scope of chapters, pages, tables, and cells; the required dependency relationship is used to connect the headers, units, figure captions, and referenced objects necessary to interpret the target content, and records the document version, table, row and column, or coordinate range to which the relationship applies. The association key of the coded input is determined by the actual input summary, model version, and preprocessing configuration. The same coded key is associated with all its source bindings; the release version records the document version mapping, source structure diagram identifier, and index identifier. Vector hits, member enumeration, dependency expansion, and payload preparation for a single query all read the data corresponding to the release version.
3. The multimodal document storage and retrieval method based on two-layer evidence graphs and joint optimization according to claim 1, characterized in that, In step 3, the source bindings and explicitly specified required entries matched by the index are used as seeds. A limited source domain is determined based on the page, table, and chapter to which the seed belongs. Member enumeration is performed on the source domain, and members that meet all the requirements of at least one requirement are merged with the seed, and duplicates are removed using the source binding identifier. When matching requirements with candidates, the target object, structural role, content type, numerical conditions, unit conditions, and the conditions specified in the source constraints are checked item by item. A matching relationship is established only if all conditions are met, and unspecified conditions are not included in the constraints. For requirement items specified as same-source pairings in the query, the allowed pairing relationships are recorded based on the complete source key of the candidate target binding.
4. The multimodal document storage and retrieval method based on two-layer evidence graphs and joint optimization according to claim 1, characterized in that, Step 4 pre-generates records of text, images, sources, and the associations between text, images, and sources, including entry records, entry-to-binding association records, source binding records, binding-to-payload association records, payload records, and demand allocation records. Using the top-left corner of the original page as the origin, the page is scaled proportionally to the specified canvas with a centered margin. The source region in the output image is calculated using the same scaling ratio and offset. The scaling ratio, horizontal and vertical offsets, and output region are written into the association records bound to the image payload. The source binding retains the original source coordinates. Each record and mapping parameter is fixed before solving. The record type field, length field, and header record count field have fixed widths, and each field value is within its corresponding representation range. The actual record length includes all bytes of the frame field, identifier, and content. The closure of the candidate entry consists of its target source binding and source bindings reachable along necessary dependencies. The number of images and the number of pixels in each image are obtained by summing the corresponding cumulative values of the image records in the enabled group.
5. The multimodal document storage and retrieval method based on two-layer evidence graphs and joint optimization according to claim 1, characterized in that, The candidate support set for source binding consists of all candidate entries whose bindings are contained in the closure. In step 5, the candidate support set for the payload is obtained by taking the union of the candidate support sets of the source bindings that reference the payload. The complete equality of the candidate support sets is used as the grouping condition, and the source field, content byte, and record type retain their original values. Each group stores the source binding set and payload set within the group. The record list of a group consists of the source records bound to each group, the records associated with the loads bound to the loads, and the load records of each load within the group. The actual byte length in the list is accumulated, and the number of images and the number of pixels are accumulated according to the image loads within the group. Each source binding and load is assigned to a group. The group enable variable is equal to the logical union of the selection variables of each entry in the group support set. All records in the group are enabled together.
6. The multimodal document storage and retrieval method based on two-layer evidence graphs and joint optimization according to claim 1, characterized in that, Each requirement item in step 6 is assigned to a matching and selected candidate item, and the selected item must be retained; the same-source pairing in the pairing uses the complete source key that expands step by step from the document identifier and its document version, the same table condition further includes the table identifier, and the same row condition further includes the uniform row identifier within the table; pairing is allowed only when the required fields are complete and the complete source keys are equal; the remaining candidate pairings are subject to the constraint that the sum of the two requirement assignment variables is no greater than one, so that the source pairing condition only applies to the combination of items that actually undertake the requirement; In step 6, the length of the entry record and all its associated closure records are included in the entry cost, the cumulative length of the group member records is included in the group cost, and the length of the demand allocation record is included in the allocation cost. The total number of bytes is obtained by adding the fixed header length, the cost of the selected entry, the cost of the enabled group, and the cost of the actual demand allocation. The demand matching score uses a non-negative integer, the demand weight uses a positive integer, and the upper limit of the total number of bytes is an integer. The score weight coefficient of the objective function is the value obtained by adding one to the upper limit of the total number of bytes. This coefficient is multiplied by the sum of the weighted matching scores of the actual demand allocation, and then the total number of bytes output is subtracted. The maximum value of the obtained objective function is then calculated to obtain the highest weighted matching score first, and then select the combination with the smaller total number of bytes output from the feasible combinations with the same score.
7. The multimodal document storage and retrieval method based on two-layer evidence graphs and joint optimization according to claim 1, characterized in that, Step 8 restores the enabled source bindings and payload records according to the group member record list to stably identify connection entries, sources, and payloads; sorts by record type and identifier, writes the same payload record once, and retains each source binding and its respective image position mapping; the reading end of the retrieval result locates the image content based on the output area in the associated record and locates the original text based on the original coordinates in the source binding; writes a fixed-length header at the beginning of the byte stream of the retrieval result, and then connects the pre-generated record bytes; the group division only merges the selection markers, and different records within the group exist separately in the output.
8. A multimodal document storage and retrieval system based on a two-layer evidence graph and joint optimization, characterized in that, For performing the method according to any one of claims 1 to 7, comprising: The storage index module is used to manage content assets, source occurrence, source binding, source structure graph, and multimodal indexes; The candidate processing module is used to locate the source domain, enumerate members, expand necessary dependencies and register requirement matching conditions, calculate requirement matching scores and set requirement weights. The record preparation module is used to generate fixed-byte records, build query evidence graphs, group candidates by support set, and count the group resource quantity of each group. The joint solution module is used to establish and solve the integer optimization model based on the group resource quantity and demand conditions; The results output module is used to expand the list of group members according to the same joint result and write the search results.
