A multi-modal large model inference acceleration method based on cross-modal feature fusion

Abstract

This invention discloses a method for accelerating multimodal large-scale model inference based on cross-modal feature fusion, specifically relating to the field of multimodal intelligent inference technology. It addresses the problem of balancing inference efficiency and the preservation of key cross-modal evidence in existing multimodal large-scale model inference acceleration processes. By uniformly encoding the input image and text content, candidate evidence units are generated according to spatial attribution, layout attribution, and semantic fragment attribution. Evidence retention levels are assigned based on intramodal stability, cross-modal responsibility, and inference post-validation. Furthermore, the evidence participation process is collaboratively controlled through the main inference channel, candidate evidence channel, and directed backflow. This achieves the effect of prioritizing the preservation of key cross-modal evidence, avoiding premature loss of local key evidence, and balancing inference efficiency and conclusion accuracy during inference acceleration.

Description

Technical Field

[0001] This invention relates to the field of multimodal intelligent reasoning technology, specifically to a method for accelerating multimodal large model reasoning based on cross-modal feature fusion. Background Technology

[0002] With the increasing application of multimodal large-scale models in scenarios such as industrial quality inspection, equipment inspection, invoice verification, and image-text question answering, these models need to simultaneously receive image and text content and complete target localization, field recognition, relationship judgment, and conclusion generation during a unified inference process. Existing multimodal inference schemes typically encode images and text separately first, and then obtain the result through cross-modal alignment and joint inference. To reduce inference latency and resource consumption, existing technologies are gradually adopting methods such as visual feature compression, irrelevant fragment pruning, cross-modal interaction simplification, and cache reuse to accelerate the inference process. These solutions can play a certain role in general image-text understanding tasks, and are particularly suitable for scenarios with prominent targets, simple layouts, and clear text directions.

[0003] However, in real-world business scenarios, image content often contains multiple pieces of information simultaneously, including nameplate parameters, local labels, layout blocks, explanatory text, and background structure. Text content also frequently carries complex constraints such as objects, attributes, locations, and relationships. What truly influences the conclusion often isn't the main area of ​​the entire image, but rather a small amount of fine-grained evidence distributed in local locations, such as a specific field row, a local label, an adjacent relationship, or specific content within a layout block. Existing acceleration methods mostly rely on overall compression, uniform filtering, or fixed rule retention. While these can reduce the number of features involved in reasoning, they typically lack detailed control over which local evidence should be prioritized in the current reasoning and which should be deferred but retained for later use. This often involves considering image spatial location, layout attribution, and text semantics. Therefore, in scenarios with mixed image and text input, existing solutions tend to compress crucial local evidence during acceleration, leading to partial conclusions in the early stages of reasoning. However, when encountering tasks involving field comparison, local verification, or relationship closure, the stability of the conclusions is insufficient.

[0004] Based on this, in the process of accelerating multimodal large model reasoning, how can existing technologies reduce the reasoning burden while avoiding the premature discarding of local cross-modal evidence that is critically related to the current problem, thus affecting the integrity and stability of subsequent conclusions, when faced with mixed text and image inputs that include local text, local targets, and layout structure? Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a method for accelerating multimodal large model inference based on cross-modal feature fusion, in order to solve the problems mentioned in the background.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for accelerating multimodal large model inference based on cross-modal feature fusion, comprising: S1. Encode the input image content and input text content, generate candidate evidence units according to spatial attribution, layout attribution, and semantic fragment attribution, and establish a corresponding index from text fragments to candidate evidence units; S2. Calculate the intramodal stability, cross-modal responsiveness, and inference validity for each candidate evidence unit, and assign an evidence retention level based on the joint judgment result; S3. Send the main participation level evidence units into the main reasoning channel, send the candidate retention level evidence units into the candidate evidence channel, and record the position mapping, summary representation, and backflow index of the candidate evidence units; S4. Perform cross-modal fusion in the main reasoning channel according to the segment and output the intermediate conclusion status. The intermediate conclusion status includes the answer candidate distribution, slot filling status, used evidence mapping set, and conflict status. S5. Determine the reasoning gap based on the intermediate conclusion status, extract the candidate evidence unit corresponding to the reasoning gap from the candidate evidence channel and return it to the main reasoning channel to participate in subsequent cross-modal fusion. S6. Reorganize the reasoning cache based on the evidence units that finally participate in the fusion, perform decoding output according to the reorganized fusion result, and generate reasoning evidence records.

[0007] Furthermore, S1 includes: After encoding the input image content and input text content, spatial assignment is determined based on the boundary distance and overlap ratio of adjacent image feature segments; Page layout is determined based on line spacing, block spacing, border continuity, and background dividers. The semantic segment attribution is determined based on the positional relationship between text segments and the collocation relationship of business terms; Based on spatial attribution and page layout as boundary constraints, candidate evidence units are generated by combining the attention area corresponding to the semantic fragment attribution, and a corresponding index from text fragment to candidate evidence unit is established.

[0008] Furthermore, S2 includes: Intramodal stability is determined by comparing the adjacent differences, boundary continuity, texture repetition, and brightness fluctuation amplitude of image feature segments within candidate evidence units. Cross-modal responsivity is determined by comparing object words, attribute words, and location words in a text segment with the page layout, spatial location, and adjacent text content of candidate evidence units; Post-inference validity is determined by comparing the influence trends of candidate evidence units on candidate answer ranking, key slot filling, and conflict reduction.

[0009] Furthermore, S2 also includes: The joint judgment results are formed in the order of intramodal stability, cross-modal responsiveness, and inference after-validity. Candidate evidence units with intramodal stability below the baseline value are excluded first. The remaining candidate evidence units are then ranked according to cross-modal response and the order is adjusted in conjunction with inference post-validation. Subsequently, evidence retention levels were assigned to the primary participation level, the alternate retention level, the delayed awakening level, and the release level, respectively.

[0010] Furthermore, S3 includes: When a candidate-reserved evidence unit is sent into the candidate evidence channel, a location map, a summary representation, and a return index are generated simultaneously. The location mapping consists of the coordinates of the top left corner, bottom right corner, width, height, page layout, and adjacent unit relationships of the candidate reserved evidence unit; The abstract indicates that it is formed by merging fragments within the candidate-reserved evidence unit in spatial order; The backflow index consists of session identifier, candidate evidence channel number, candidate evidence unit number, associated text fragment number, location mapping address, and summary representation address.

[0011] Furthermore, S4 includes: Intermediate conclusion states are formed at the end of each segment; The distribution of candidate answers is determined based on the number of times a candidate answer is supported by object words, attribute words, position words, and relation words, the length of consecutive support, and the mutual exclusion of each other. The slot filling status is determined based on the unique corresponding evidence of the business element position, the vacancy status, and the multi-value competition status. The conflict state is determined based on the conflict of business element values, the conflict of location area attributes, and the reverse change of conclusions in adjacent layers.

[0012] Furthermore, S5 includes: The reasoning gap is determined based on the intermediate conclusion state; Inference gaps include slot vacancies, value range conflicts, uncovered regions, and unclosed relationships. Slot vacancy is determined based on slot filling status. Value range conflict is determined based on conflict status. The area not covered was determined based on the evidence mapping set and the location words of the text fragments; The relationship is not closed, which is determined by the distribution of candidate answers and the relation words in the text fragment.

[0013] Furthermore, S5 also includes: Candidate evidence units are extracted based on the backflow index, location mapping, and summary representation; First, retrieve candidate evidence units with the same sequence number as the associated text fragments based on the backflow index; Then, based on the location mapping, candidate evidence units that do not meet the requirements of target area, page ownership, and adjacency relationship are filtered out; Subsequently, candidate evidence units are identified based on the summary and written back to the main reasoning channel in the order of reflow.

[0014] Furthermore, S6 includes: After deduplication based on session identifier, candidate evidence unit sequence number, and reflow order, the inference cache is rearranged based on text fragment sequence number, evidence citation order, and location mapping proximity relationship. The fusion result is determined based on the state of the last intermediate conclusion and the rearranged inference cache; The inference evidence record includes session identifier, candidate evidence unit number, location mapping, corresponding text segment number, layer number, reflow order, and conclusion segment number.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. By uniformly encoding the input image content and input text content, and generating candidate evidence units according to spatial attribution, layout attribution, and semantic fragment attribution, and then assigning evidence retention levels based on intramodal stability, cross-modal responsiveness, and post-inference validity, the system further sends the main participation level evidence units into the main inference channel and the candidate retention level evidence units into the candidate evidence channel. After determining the inference gap based on the intermediate conclusion status, a directional backflow is executed. This achieves the effect of prioritizing the retention of cross-modal evidence that is critically related to the current problem during the process of accelerating multimodal inference, avoiding premature discarding of local evidence that determines the conclusion, and balancing inference efficiency and inference accuracy.

[0016] 2. By performing cross-modal fusion in the main inference channel in stages, an intermediate conclusion state is formed, which includes the distribution of answer candidates, slot filling status, used evidence mapping set, and conflict status. Based on the evidence units that finally participate in the fusion, the inference cache is reorganized, the decoding output is executed, and the inference evidence record is generated. This achieves the effects of improving the controllability of the inference process, enhancing the traceability of the conclusion generation process, reducing the invalid cache occupation, and improving the stability of actual deployment. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the overall process of a method to accelerate inference of a multimodal large model based on cross-modal feature fusion; Figure 2 Diagram illustrating the determination of evidence preservation levels and dual-channel diversion; Figure 3 This is a schematic diagram of the reasoning gap determination and the return of candidate evidence based on the intermediate conclusion state. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Example: Combined with Appendix Figure 1-3 This embodiment provides a method for accelerating multimodal large model inference based on cross-modal feature fusion, including: S1. Encode the input image content and input text content, generate candidate evidence units according to spatial attribution, layout attribution, and semantic fragment attribution, and establish a corresponding index from text fragments to candidate evidence units. Specifically, this is implemented as follows: First, the encoding execution program deployed on the multimodal inference server reads the input image content and input text content under the same session identifier after a single inference session is established. The input image content is taken from the pixel matrix submitted by the industrial vision camera, document scanning device, terminal screenshot acquisition module, or video frame extraction module within the current inference trigger period, with the unit being pixels. The submission rhythm can be set to single frame submission or submission at fixed frame intervals, and the timestamp precision can be set to milliseconds. The input text content is taken from the character sequence submitted by the operator question interface, work order description area, alarm description area, or business instruction area within the same trigger period. The encoding format is locked to a unified character set and includes a session timestamp. Encoding refers to converting the input image content into image feature fragments arranged by position and the input text content into text feature fragments arranged by word order. This conversion is completed within a preset observation window after the current session begins. The observation window can be set to 200ms to 1500ms. If the upper limit is exceeded, a timeout code is recorded and the current round of unit building action is terminated.

[0020] Spatial assignment is determined within the image coordinate system. Each image feature segment retains its top-left corner coordinates, width, and height, with the coordinate unit being pixels. Adjacent segments are assigned to the same spatial assignment by comparing boundary distances and overlap ratios. Page layout assignment is performed for on-site scenarios containing text blocks, such as invoice review, equipment nameplate recognition, and inspection report verification. In the image, stable page layout blocks such as the title area, body text area, parameter area, and signature area are first determined based on line spacing, block spacing, border continuity, and background separators. Then, image feature segments are assigned to the corresponding page layout according to their respective blocks. The line spacing and block spacing are obtained by comparing the center distance of adjacent text boxes segment by segment within the current image. Semantic fragment attribution is performed on the input text content. First, the text is segmented according to punctuation, line breaks, field separators and fixed business word boundaries. Then, semantic fragment attribution is formed within the current text sequence based on the positional relationship between the fragments and the collocation relationship of business words. For example, in the production line quality inspection question and answer scenario, "read the rated current of the nameplate and determine whether it is consistent with the work order" is split into object fragments, attribute fragments and judgment fragments. Each fragment can be directly located within the character offset range.

[0021] Candidate evidence units consist of at least one image feature fragment. They are generated based on spatial attribution, with layout attribution as boundary constraint, and then aggregated by combining the semantic fragment attribution with the corresponding region of interest. Once the aggregation rule version is locked, it will not be changed during the session. For example, if the model line, voltage line, and current line are arranged continuously in the same nameplate area, and the block spacing does not exceed the preset threshold and the background is continuous, a candidate evidence unit is generated. The threshold can be set to 0.5 times to 2 times the average height of adjacent text, with a default value of 1. This threshold is determined by the lowest point of erroneous block rate in the historical labeled samples.

[0022] The corresponding index is established between the text fragment and the candidate evidence unit. The index item includes at least the session identifier, text fragment number, candidate evidence unit number, establishment time, rule version, and confidence flag. The confidence flag is obtained by comparing the object words, attribute words, and location words in the text fragment with the page block, spatial location, and adjacent text content of the candidate evidence unit. For example, when the text fragment contains "rated current of the nameplate in the upper left corner", the candidate evidence unit located in the upper left area, falling into the nameplate page, and whose adjacent text contains the words "rated current" is preferentially selected and written into the corresponding index.

[0023] To ensure on-site feasibility, image timestamps and text timestamps are first aligned according to a unified timekeeping benchmark, with an allowable deviation of 20ms to 100ms. If the deviation exceeds this range, it will be recorded in the rearrangement log. For images with a lot of noise, only isolated bright spots and scanning burrs are removed without changing the main outline of the text strokes. When a frame is missing, it is only allowed to be filled in once from the previous frame. If two consecutive frames are missing, the generation of new candidate evidence units will be stopped and the missing frame code will be returned. Image content that arrives repeatedly within the same session is deduplicated using the session identifier, image summary, and timestamp triplet. Duplicate requests will directly return the existing index results to achieve idempotency.

[0024] Upon completion, a candidate evidence unit list and a corresponding index list are generated. The structured records with field names are written to the session-level high-speed storage area and simultaneously written to the evidence trace area. The next step reads the data in the order of session identifiers, and cross-session reuse is not allowed. Communication is achieved through local process calls or LAN message calls. The minimum message set includes the session identifier, image batch number, text batch number, rule version, and timeout. The minimum return set includes the number of candidate evidence units, the number of corresponding indexes, and the status code. The status codes include at least 0000 success, 1001 timestamp mismatch, 1002 image missing, 1003 text missing, 1004 rule version inconsistency, and 1005 observation window timeout.

[0025] The number of concurrent sessions can be set to an upper limit based on the server's video memory capacity. The number of retries per session can be set to 2. If this limit is exceeded, the failure record will be retained and the session will exit. The security boundary is limited to the desensitized business images and business text. Personal sensitive information blocks are masked before entering the encoding process and do not participate in the generation of candidate evidence units.

[0026] During on-site verification, the coverage rate of candidate evidence units and the corresponding index hit rate are checked with no less than 300 consecutive real samples. If the coverage rate is lower than 95% or the hit rate is lower than 93%, the previous rule version is rolled back.

[0027] In a preferred embodiment, for the power distribution cabinet nameplate verification scenario, the input image content comes from a single frame captured by a 5-megapixel industrial camera, the input text content comes from work order verification statements, the observation window is set to 600ms, the timestamp allowable deviation is set to 50ms, the block spacing threshold is taken as 1 times the average text height, 8 to 15 candidate evidence units are generated per session, the corresponding index establishment time is controlled within 180ms, the candidate evidence unit coverage rate is 97.3% in 300 samples, and the corresponding index hit rate is 95.8%; alternatively, in the mobile terminal inspection scenario, the input image content can also be taken from keyframes extracted from the video stream at fixed intervals, as long as candidate evidence units are still generated according to spatial attribution, layout attribution, and semantic fragment attribution and the corresponding index from text fragments to candidate evidence units is established, it falls under the same technical concept.

[0028] S2. Calculate the intramodal stability, cross-modal responsiveness, and inference validity for each candidate evidence unit, and assign an evidence retention level based on the joint judgment result. The specific implementation is as follows: After the candidate evidence unit list corresponding to the current session is formed, the candidate evidence units, text fragments and corresponding indexes are read one by one in the order of session identifier. Under the same rule version, the continuous determination of intramodal stability, cross-modal responsiveness and inference post-validation is completed. This action is executed immediately after the candidate evidence unit generation stage. On-site, it can be performed on the side of multimodal inference devices such as production line quality inspection question and answer server, equipment inspection auxiliary terminal, and invoice review workstation. The scoring execution program calls the candidate evidence unit list, corresponding index list, image feature fragment record and text fragment record in the session-level high-speed storage area. The time period of action is limited to before a single inference session enters the main inference channel. The purpose is to distinguish which candidate evidence units should participate in subsequent inference in the current scenario without changing the boundaries of the original candidate evidence units. It is suitable for scenarios with local key evidence, such as image and text co-understanding, layout verification, nameplate recognition question and answer, and inspection image question and answer. It is not suitable for scenarios with only single text content and no image feature fragments.

[0029] Intramodal stability is obtained within the current image feature segment signal. Its physical meaning is whether the same candidate evidence unit has continuous, complete, and low-disturbance usable information within this modality. Within the preset observation window after the current session is established, it is obtained by comparing the adjacent differences, boundary continuity, texture repetition, and brightness fluctuation amplitude of each image feature segment within the candidate evidence unit. The unit adopts a dimensionless score, and the value range can be set from 0 to 100. For example, in the scenario of checking the nameplate of the power distribution cabinet, if the text boundary is continuous, the gray scale change of adjacent segments is stable, and there is no large area of ​​occlusion within the same candidate evidence unit, the intramodal stability is improved. If there are reflective bands, stains, or scanning shadows, the intramodal stability is reduced. This quantity is formed by comparing the internal segments one by one within a single frame image. In the video key frame scenario, the consistency comparison of segments at the same position in adjacent frames is also added.

[0030] Cross-modal responsivity is obtained under the joint constraints of text fragment records and corresponding index records. Its physical meaning is the strength of support of the current candidate evidence unit for the object, attribute, position, and relationship expressed by the current text fragment. Within the same observation window, it is obtained by comparing the object words, attribute words, and position words in the text fragment with the page affiliation, spatial position, adjacent text content, and historical hit records of the candidate evidence unit. The unit adopts a dimensionless score, and the value range can be set from 0 to 100. For example, if the text fragment contains "read the rated current of the upper left nameplate", then the cross-modal responsivity of the candidate evidence unit located in the upper left area, belonging to the nameplate page affiliation, and containing the rated current field in adjacent text is increased, while the cross-modal responsivity of the candidate evidence unit located in the lower description area is decreased.

[0031] Post-inference validity is estimated before the current candidate evidence unit enters the main reasoning channel. Its physical meaning is whether the candidate evidence unit is likely to change the conclusion of the current question in subsequent reasoning. Within the set of candidate evidence units corresponding to the current text segment, it is obtained by comparing the trend of the influence of the candidate evidence unit on the ranking of candidate answers, key slot filling, and conflict reduction after it is included. The unit adopts a dimensionless score, and the value range can be set from 0 to 100. This quantity is not given out of thin air, but is first registered in the historical sample playback record locked by the rule version, and then mapped and determined in the current session according to the hit frequency and conclusion change of similar text segments, similar page ownership, similar spatial area. For example, in the equipment inspection question and answer scenario, although the small area of ​​the label block located near the alarm light is small, it often has a conclusion-reversing effect on questions such as "What is the fault code?", so the post-inference validity remains high.

[0032] To ensure full transparency, timestamp alignment, empty segment removal, isolated noise suppression, and missing field completion boundary determination are completed before scoring. The allowable deviation of timestamps follows the session timing benchmark from the previous step, and those exceeding the allowable deviation will not participate in this round of scoring. Missing segments within candidate evidence units are only allowed to be interpolated once by adjacent segments. If two consecutive missing segments are found, the reason for downgrading will be marked in the record.

[0033] The joint judgment result is obtained by comparing intramodal stability, cross-modal responsiveness, and post-inference validity in a fixed order. First, candidate evidence units with intramodal stability below the baseline value are excluded. Then, the remaining candidate evidence units are sorted by cross-modal responsiveness. Finally, the order is adjusted based on the post-inference validity. Based on the joint judgment result, evidence retention levels are assigned. Evidence retention levels include at least the active participation level, the alternate retention level, the delayed awakening level, and the release level. The active participation level corresponds to candidate evidence units that directly enter the main reasoning channel. The alternate retention level corresponds to candidate evidence units that retain the index and summary records first. The delayed awakening level corresponds to candidate evidence units that are only brought in when a reasoning gap occurs later. The release level corresponds to candidate evidence units that no longer participate in this round of conversation.

[0034] The observation window can be set from 150ms to 800ms, with the default value determined by the upper limit of on-site hardware latency and the sample stability rate. The number of times a single session is scored can be set to 1. If the list of upstream candidate evidence units is updated, one re-evaluation is allowed. The same candidate evidence unit that arrives repeatedly is deduplicated using the session identifier, unit number, and rule version as idempotent keys.

[0035] Upon completion, an evidence retention level list is generated. The list fields include at least the session identifier, candidate evidence unit number, intramodal stability, cross-modal responsiveness, inference validity, evidence retention level, rule version, and generation time. The list is written to the session-level high-speed storage area using structured records and simultaneously written to the evidence trace area for subsequent stages to access in sequence according to the session identifier. Communication can be achieved through local process calls or LAN message calls. The minimum request fields include the session identifier, rule version, and timeout. The minimum return fields include the total number of candidate evidence units, the number of each evidence retention level, and the status code. The status codes include at least 0000 success, 2001 candidate evidence unit missing, 2002 corresponding index missing, 2003 rule version inconsistency, 2004 scoring timeout, and 2005 duplicate request.

[0036] Concurrency is controlled by the upper limit of video memory usage, and the upper limit of single session latency can be set by the deployment side. When the upper limit is exceeded, the retention level of the previous round of valid evidence is directly retained and recorded in the timeout log; text blocks involving personal information are masked before scoring and do not participate in cross-modal response comparison.

[0037] During on-site testing, no fewer than 300 real-world working condition samples are used to verify the accuracy of evidence retention level and the first-round hit rate of subsequent main reasoning channels. If the accuracy is lower than the preset threshold, the previous rule version will be rolled back.

[0038] In a preferred embodiment, for the question-and-answer scenario of the power distribution cabinet nameplate, the single-frame image resolution is 2592×1944, the observation window is set to 400ms, the intramodal stability baseline value is set to 35, and after the evidence retention level is divided, each session averages 4 primary participation level, 3 candidate retention level, 2 delayed wake-up level, and 1 release level. The evidence retention level accuracy rate reaches 95.1% in 300 samples, and the first-round hit rate of the subsequent main reasoning channel reaches 93.7%. Alternatively, in the invoice review scenario, the candidate evidence unit can also be changed from the text block level to the field row level. As long as the intramodal stability, cross-modal responsiveness, and reasoning post-validation are still calculated for each candidate evidence unit and the evidence retention level is assigned based on the joint judgment result, it belongs to the same technical concept.

[0039] S3. Send the primary participation level evidence units into the primary reasoning channel, and send the candidate retention level evidence units into the candidate evidence channel, and record the position mapping, summary representation, and backflow index of the candidate evidence units. The specific implementation is as follows: First, the evidence retention level list is read according to the session identifier. Candidate evidence units with a retention level of primary participation are written into the main inference channel, and candidate evidence units with a retention level of alternate retention are written into the alternate evidence channel. At the same time as writing into the alternate evidence channel, the location mapping, summary representation, and backflow index are generated and saved. This action is executed immediately after the evidence retention level is formed. On-site, it can be performed on the session scheduling side of the production line quality inspection question and answer server, equipment inspection edge computing node, and invoice review workstation. The channel scheduling program calls the candidate evidence unit list, corresponding index list, and evidence retention level list in the session-level high-speed storage area to complete the task. The effective period is limited to before the main inference channel starts cross-modal fusion. The purpose is to separate and store the candidate evidence units that need to participate in the inference immediately in the current session from the candidate evidence units that do not participate immediately but need to retain the ability to be retrieved later. It is suitable for scenarios where there is both key local evidence and backup local evidence, such as mixed text and image understanding, partial field verification, and inspection image question and answer.

[0040] The main inference channel refers to the high-priority evidence carrying area that directly participates in subsequent cross-modal fusion within the current session. Within the preset transport window after the current session is established, the main participation level evidence units are written one by one in the order of evidence retention level and candidate evidence unit number. The transport window can be set from 50ms to 300ms. The alternate evidence channel refers to the retention area within the current session that does not enter subsequent cross-modal fusion for the time being, but only saves basic information that can be called back. Within the same transport window, the alternate retention level evidence units are written one by one.

[0041] The location mapping is obtained from the image feature segment record corresponding to the candidate-level evidence unit. Its physical meaning is the spatial and page layout of the candidate-level evidence unit in the original image content. It is formed by reading the upper left corner coordinates, lower right corner coordinates, width, height, page layout, and adjacent unit relationship covered by the candidate-level evidence unit in the image coordinate system of the current session. The coordinate unit is pixels, and the page layout uses the locking result of the previous step. For example, in the power distribution cabinet nameplate Q&A scenario, when the current field block located in the upper right corner of the nameplate area is written to the candidate evidence channel, its rectangular range in the whole image, its nameplate page layout, and its adjacent direction with the model field block are recorded. It is then directly located based on this during subsequent reflow.

[0042] The abstract indicates that it is obtained from the original image feature fragment set of the candidate-level evidence unit. Its physical meaning is that it is a compressed representation that can represent the main content of the candidate-level evidence unit and occupy less storage. It is obtained by merging the fragments inside the candidate-level evidence unit in spatial order, removing duplicate fragments, retaining boundary fragments and high-response fragments within the current transfer window. The unit adopts fixed-length vector length record and byte length record. For example, in the scenario of checking the field of the invoice, the continuous repeated background fragments in the same field row are no longer fully retained. Only the representative fragments of the front end, middle and rear end of the field and their aggregation results are retained, so that the core information of the field can be recovered during subsequent reflow.

[0043] The backflow index is obtained under the joint constraints of the corresponding index list and the evidence retention level list. Its physical meaning is that when a reasoning gap occurs in the subsequent main reasoning channel, the retrieval path for the candidate retention level evidence unit is re-entered. It is formed in the current session by binding the session identifier, candidate evidence channel number, candidate evidence unit number, associated text fragment number, location mapping address, summary representation address, rule version, and generation time. For example, when the text fragment corresponds to "whether the rated current is consistent", the backflow index synchronously writes the text fragment number and the corresponding candidate retention level evidence unit number. Subsequently, the original candidate evidence unit can be located by simply pressing this index.

[0044] To ensure on-site implementation, before writing, the consistency of session identifiers, rule versions, and the continuity of candidate evidence unit numbers are verified. For duplicate arrivals of the same candidate evidence unit, the session identifier, candidate evidence unit number, and rule version are used as idempotent keys for deduplication. For candidate retention-level evidence units with missing location coordinates, one attempt is allowed to complete them. If they are still missing, they are not written into the candidate evidence channel and are recorded in the trace area. When writing main participation-level evidence units into the main reasoning channel, a first-in-first-out order is adopted, and cross-session queueing is not allowed.

[0045] Upon completion, a main inference channel loading list and a candidate evidence channel retention list are generated. The former includes at least the session identifier, main inference channel sequence number, candidate evidence unit sequence number, write time, and rule version. The latter includes at least the session identifier, candidate evidence channel sequence number, candidate evidence unit sequence number, location mapping, digest representation storage address, reflow index, write time, and rule version. Both are written to the session-level high-speed storage area using structured records and simultaneously written to the evidence trace area for the next stage to read in the order of session identifiers. Communication can be achieved through local process calls or LAN message calls. The minimum request fields include the session identifier, rule version, and timeout time. The minimum return fields include the number of writes in the main inference channel, the number of writes in the candidate evidence channel, and the status code. The status codes include at least 0000 success, 3001 missing evidence retention level list, 3002 missing candidate evidence unit, 3003 inconsistent rule version, 3004 missing location mapping, and 3005 write timeout.

[0046] The number of concurrent sessions can be set according to the upper limit of video memory and memory. The number of retries per session can be set to 2. After exceeding this limit, the previous valid retention list is retained and the current round of transfer is stopped. Candidate retention level evidence units involving sensitive personal information continue to be masked before forming a digest representation and plaintext fields are not written into the digest representation.

[0047] During on-site testing, the first-round hit rate of the main inference channel, the locatability rate of the backup evidence channel return, and the write latency qualification rate can be checked on no less than 300 real samples. If any indicator is lower than the preset threshold, the previous rule version will be rolled back.

[0048] In a preferred embodiment, for the power distribution cabinet nameplate question-and-answer scenario, the average number of primary participation level evidence units per session is 4, and the number of candidate retention level evidence units is 3. The transfer window is set to 120ms. The position mapping adopts the upper left corner coordinate plus width and height recording method. The summary representation length is controlled to be 35% to 50% of the original fragment set storage volume. In 300 samples, the first-round hit rate of the main inference channel reaches 93.4%, the return flow location rate of the candidate evidence channel reaches 98.1%, and the write latency qualification rate reaches 96.7%. Alternatively, in the mobile terminal inspection scenario, the main inference channel and the candidate evidence channel can also be established in different memory areas of the same device. As long as the primary participation level evidence units are still sent to the main inference channel and the candidate retention level evidence units are sent to the candidate evidence channel, and the position mapping, summary representation, and return flow index of the candidate evidence units are recorded, they belong to the same technical concept.

[0049] S4. Perform cross-modal fusion by segment within the main reasoning channel, and output intermediate conclusion states. The intermediate conclusion states include answer candidate distribution, slot filling status, used evidence mapping set, and conflict status. The specific implementation is as follows: First, the main participant-level evidence units are sequentially read from the main inference channel according to the session identifier. Then, the evidence alignment and semantic convergence are advanced layer by layer in the main inference channel in combination with the corresponding text fragments. An intermediate conclusion state is formed after each preset layer is passed. This action is executed immediately after the main inference channel is completed. On-site, it can be executed on the inference execution side of the production line quality inspection question and answer server, equipment inspection inference node, and invoice review workstation. The cross-modal fusion execution program calls the main participant-level evidence units in the main inference channel, the text fragment records in the session-level high-speed storage area, the corresponding index list, and the location mapping records to complete the process. The time period of action is limited to before the candidate evidence channel has returned. The purpose is to use the determined main participant-level evidence units to form the first round of inference skeleton of the current session. It is suitable for mixed text and image inference scenarios where there is a clear question and some key evidence has entered the main inference channel.

[0050] A segment refers to a set of fixed-order inference layers that are executed continuously within the main inference channel. Its physical meaning is the smallest execution unit of a local fusion judgment. The segment boundary is determined by reading the version-locked inference configuration within the current session. Typical values ​​can be set to 2 to 6 consecutive layers. The default value is determined by the latency limit of the deployment device and the stability rate of intermediate conclusions. Once the segment boundary is determined in this session, it will not be modified in the middle of the session.

[0051] Cross-modal fusion is performed segment by segment between the image feature fragment sequence and the text fragment sequence of the main participant level evidence unit. First, the text fragment is initially aligned with the main participant level evidence unit according to the corresponding index. Then, within the current segment, the object words, attribute words, position words, relation words and the layout, spatial position, adjacent text content and contextual semantic fragments of the main participant level evidence unit are compared in turn. This ensures that mutually supportive content is kept related, and content that cannot support each other is given a lower priority. For example, in the question-and-answer scenario of the power distribution cabinet nameplate, when the text fragment contains "whether the rated current is consistent with the work order", the main participant level evidence unit located in the nameplate area and whose adjacent text contains the current field is first aligned with the "rated current" fragment, and then continues to form a comparison relationship with the "whether consistent" fragment. If the main participant level evidence unit can only support "rated current" but not "work order value", then this segment only forms a partial conclusion and does not make a final determination.

[0052] To ensure feasibility on-site, before proceeding to this step, session identifier verification, rule version verification, and text fragment sequence order verification are performed. Duplicate primary participant-level evidence units in the main inference channel are deduplicated by session identifier, candidate evidence unit number, and rule version. For primary participant-level evidence units with high local noise, only isolated abnormal fragments are suppressed, without altering their position mapping or original boundaries. Missing fragments are only allowed to be supplemented once by adjacent valid fragments; if two consecutive missing fragments occur, the primary participant-level evidence unit is retained, but a credibility decrease flag is recorded in the intermediate conclusion state. The intermediate conclusion state is formed once at the end of each segment, and its fields and definitions remain consistent throughout the session.

[0053] The candidate answer distribution is obtained from the set of candidate answers after the current segment ends. Its physical meaning is the strength of support for the current question among multiple candidate answers. It is obtained by statistically analyzing the number of times each candidate answer is supported by object words, attribute words, position words, and relation words, the length of continuous support, and the mutual exclusion of each candidate answer on the correlation signal between the current text segment and the main participant level evidence unit. The unit adopts dimensionless scoring, and the value range can be set from 0 to 100. For example, in the nameplate verification scenario, "consistent" and "inconsistent" are two candidate answers. If the main participant level evidence unit supports the same nameplate current value and work order current value, the candidate distribution of the answer "consistent" will increase.

[0054] The slot filling status is obtained from the business element position corresponding to the current text fragment. Its physical meaning is whether the objects, attributes, values, positions, and relationships required to complete the current problem have been filled item by item by the main participant level evidence unit. In the text fragment association record after the end of the current level segment, it is obtained by checking whether each business element position has a unique corresponding evidence, whether there is a gap, and whether there is multi-value competition. For example, "equipment name", "rated current", and "location area" can all be used as business element positions. In the scenario of invoice field verification, when the field value has been found and corresponds to only 1 main participant level evidence unit, it is recorded as filled; when it has not been found, it is recorded as gap; when 2 competing values ​​are found, it is recorded as pending.

[0055] The evidence mapping set is obtained on the main participating level evidence unit that actually participates in the conclusion formation in the current segment. Its physical meaning is the range of evidence that has been cited by the current conclusion. It is formed in the current segment by recording the candidate evidence unit number, position mapping, corresponding text segment number, and calling order that are actually called by the text segment. Subsequently, it is possible to directly trace which main participating level evidence unit the conclusion comes from, where it is located in the image, and which text segment it supports.

[0056] The conflict state is jointly obtained from the distribution of answer candidates, slot filling status, and used evidence mapping set after the current segment ends. Its physical meaning is whether there are mutually exclusive evidence references, contradictory slot filling values, and inconsistent candidate answer trends under the same question. It is obtained by comparing whether different values ​​appear in the same business element position, whether different attributes are assigned to the same location area, and whether the conclusions of adjacent segments change in opposite directions within the current session. For example, if two different current values ​​are identified in the same nameplate area at the same time, or if the previous segment tends to be "consistent" while the next segment tends to be "inconsistent", then the conflict state is set to have a conflict.

[0057] The observation window can be set from 300ms to 1500ms. The trigger condition is that the main inference channel loads the list and the text fragment is recorded completely. The stopping condition is that the preset number of segments is reached or the distribution of answer candidates meets the termination threshold. The termination threshold can be set to the difference between the score of the first candidate answer and the score of the second candidate answer reach the preset difference, and there are no gaps in the slot filling state and the conflict state is eliminated. The default value is determined by the highest stable hit rate of the first round of historical samples.

[0058] Upon completion, an intermediate conclusion status record is generated, which includes at least the session identifier, segment number, answer candidate distribution, slot filling status, used evidence mapping set, conflict status, rule version, and generation time. This record is written to the session-level high-speed storage area and simultaneously to the evidence tracking area, allowing the next stage to read it sequentially by session identifier and segment number. Communication can be achieved through local process calls or LAN message calls. The minimum request fields include the session identifier, rule version, segment limit, and timeout. The minimum return fields include the segment number, number of intermediate conclusion states, and status code. Status codes must include at least: 0000 Success, 4001 Missing main inference channel loading list, 4002 Missing text fragment, 4003 Inconsistent rule version, 4004 Segment execution timeout, and 4005 Intermediate conclusion status writing failure.

[0059] The number of concurrent sessions is limited by both video memory usage and response time limit. Each session is allowed to retry once per segment. Before retrying, the temporary records of this segment must be cleared to prevent duplicate accounting. If the retry still fails, the valid intermediate conclusion status of the previous segment is retained and timeout is recorded.

[0060] Text blocks containing sensitive personal information will remain masked in this step, with only their slot placeholders retained, and will not be included in the answer candidate distribution comparison in plaintext.

[0061] During on-site inspection, the stability rate of the first-round conclusions within the segment, the slot filling completeness rate, the traceability rate of used evidence, and the accuracy rate of conflict identification can be checked on no less than 300 real samples. If any indicator is lower than the preset threshold, the previous rule version will be rolled back.

[0062] In a preferred embodiment, for the question-and-answer scenario of the power distribution cabinet nameplate, the segment is set to form an intermediate conclusion state every 3 layers, with a maximum of 4 segments executed per session. The observation window is set to 900ms, and the termination threshold is set to the first candidate answer score being 15 points higher than the second candidate answer score and all key slots being filled. In 300 samples, the first-round conclusion stability rate reaches 92.8%, the slot filling completeness rate reaches 95.4%, the traceability rate of used evidence reaches 99.1%, and the conflict identification accuracy rate reaches 94.6%. Alternatively, in the mobile terminal inspection question-and-answer scenario, the segment boundary can be changed from a fixed number of layers to a fixed calculation time window. As long as cross-modal fusion is still performed by segment within the main inference channel, and an intermediate conclusion state containing the answer candidate distribution, slot filling state, used evidence mapping set, and conflict state is formed, it belongs to the same technical concept.

[0063] S5. Based on the intermediate conclusion status, determine the reasoning gap, extract the candidate evidence unit corresponding to the reasoning gap from the candidate evidence channel, and return it to the main reasoning channel to participate in subsequent cross-modal fusion. The specific implementation is as follows: After the intermediate conclusion state of the current layer is generated, the answer candidate distribution, slot filling status, used evidence mapping set, and conflict status are read according to the session identifier. These are then compared item by item under the same rule version to determine if there are any inference gaps requiring new evidence to continue the current session. Next, candidate evidence units corresponding to these inference gaps are extracted from the candidate evidence channel and written back to the main inference channel in the order of backflow for subsequent cross-modal fusion. This action is executed immediately after the intermediate conclusion state is formed, and can be implemented on-site at the production line quality inspection question-and-answer server, equipment inspection inference node, and invoice review workstation. On the other hand, the reflow execution program calls the intermediate conclusion status record, candidate evidence channel retention list, reflow index, location mapping, and summary representation in the session-level high-speed storage area. The effective period is limited to the period when the current session has not reached the termination threshold and the main inference channel can still accept new evidence. The purpose is to only add candidate evidence units that can make up for the deficiencies of the current conclusion, and to avoid adding all candidate evidence units to the main inference channel at once, which would cause invalid calculations. It is suitable for scenarios such as mixed text and image understanding, partial field verification, equipment nameplate Q&A, and inspection defect Q&A, where the first round of inference may only form partial conclusions.

[0064] Reasoning gaps are identified in the current intermediate conclusion state. Physically, they represent gaps in evidence, elements, or contradictions that prevent the current conclusion from being closed. These gaps are identified in the intermediate conclusion state record after the current segment ends by comparing the difference between the first and second candidate in the answer candidate distribution, checking for gaps or multi-value competition in the slot filling state, verifying whether the used evidence mapping set covers the key area pointed to by the current text fragment, and comparing whether mutually exclusive terms still exist in conflict states. For example, in a power distribution cabinet nameplate question-and-answer scenario, if the text fragment requires determining whether the rated current matches the work order, and the current used evidence mapping set only covers the nameplate current field but not the work order value area, then this session has a numerical comparison-type reasoning gap. Similarly, in a bill review scenario, if two competing recognition results appear in the same amount field, then this session has a conflict resolution-type reasoning gap.

[0065] To ensure full disclosure, inference gaps are not marked with abstract symbols, but are recorded as four categories based on their ability to directly drive backflow actions: slot vacancy, value domain conflict, region uncovered, and relationship unclosed. Slot vacancy is obtained by checking whether the target business element position is empty in the slot filling status signal. Value domain conflict is obtained by comparing whether the same business element position corresponds to multiple different values ​​in the conflict status signal. Region uncovered is obtained by checking whether there is no evidence to cite the key position in the joint signal of the used evidence mapping set and the text fragment position words. Relationship unclosed is obtained by checking whether there is still no definite support for the preceding and following relationship, the superior and inferior relationship, and the identity relationship in the joint signal of the answer candidate distribution and the text fragment relationship words.

[0066] The candidate evidence units corresponding to the inference gaps in the candidate evidence channel are not manually designated, but are screened item by item based on the backflow index, position mapping, and summary representation: First, candidate evidence units with the same sequence number of related text fragments are retrieved in the backflow index according to the type of inference gap. Then, candidate evidence units that are not in the target area, do not meet the page assignment, or do not have an adjacency relationship are filtered out according to the position mapping. Finally, the summary representation is checked to see if it contains the main content that can fill the current inference gap. For example, when the inference gap is "the upper left nameplate area is not covered", candidate evidence units that are located in the upper left area, belong to the nameplate area, and have been bound to the "rated current" text fragment in the backflow index are extracted first. When the inference gap is "two values ​​conflict in the same slot", candidate evidence units that retain the field header and field footer in the summary representation and can help determine which value belongs to the target field are extracted first.

[0067] When candidate evidence units are fed back to the main inference channel, the consistency of the session identifier, rule version, candidate evidence channel number, and candidate evidence unit number is first verified. Then, they are written to the end of the main inference channel in the order of feeding back. The feeding back order is determined according to the inference gap type, the corresponding text fragment number, and the proximity of the candidate evidence unit to the target area. The proximity is obtained by comparing the distance between the center coordinates of the candidate evidence unit and the center coordinates of the area pointed to by the target location word on the location mapping record, in pixels.

[0068] To prevent duplicate backflow, the same candidate evidence unit is allowed to be backflowed only once within the same session, using the session identifier, candidate evidence unit number, rule version, and inference gap number as idempotent keys. If the same type of inference gap still exists after the intermediate conclusion status is updated, candidate evidence units that have not yet been backflowed will be extracted first.

[0069] The observation window can be set from 80ms to 400ms. The trigger condition is that the intermediate conclusion state has not reached the termination threshold and there is a reasoning gap. The stopping condition is that the candidate evidence unit corresponding to the current reasoning gap has been returned, there is no matching candidate evidence unit in the candidate evidence channel, or the number of returned in this round reaches the preset limit. The upper limit of the number of returned samples can be set from 1 to 4 per round. The default value is determined by the latency limit of the deployment device and the secondary fusion benefit of historical samples.

[0070] Before writing back, only necessary expansion of the summary representation is allowed, without changing the original boundaries of the candidate evidence unit; if the location mapping is missing, it is allowed to check once by the backflow index, and if it is still missing, the backflow of the candidate evidence unit is stopped and recorded in the trace area; if the summary representation cannot cover the content required for the current reasoning gap, the candidate evidence unit is retained in the candidate evidence channel but marked as not applicable in this round.

[0071] Upon completion, inference gap records and backflow records are generated. The former includes at least the session identifier, layer sequence number, inference gap sequence number, inference gap type, associated text fragment sequence number, generation time, and rule version. The latter includes at least the session identifier, layer sequence number, candidate evidence channel sequence number, candidate evidence unit sequence number, backflow order, write-back time, and rule version. Both are written to the session-level high-speed storage area using structured records and simultaneously written to the evidence trace area for subsequent cross-modal fusion to read sequentially according to the session identifier and backflow order. Communication can be achieved through local process calls or LAN message calls. The minimum request fields include the session identifier, layer sequence number, rule version, and timeout time. The minimum return fields include the number of inference gaps, the number of backflows, and the status code. The status codes include at least 0000 success, 5001 missing intermediate conclusion status, 5002 missing candidate evidence channel retention list, 5003 missing backflow index, 5004 missing location mapping, and 5005 backflow timeout.

[0072] The number of concurrent sessions is limited by video memory, main memory, and response time limit. Each segment of a single session is allowed to retry the backflow once. Before retrying, the unfinished write-back records of the current round must be cleared to prevent duplicate entry into the channel. If the retry still fails, the current intermediate conclusion status is retained and the backflow of the current segment is marked as incomplete.

[0073] Candidate evidence units involving sensitive personal information remain masked before being returned to the system; only placeholder fragments related to the current inference gap are restored, and plaintext fields are not.

[0074] During on-site testing, the accuracy of inference gap identification, the correctness of candidate evidence unit matching, the conclusion improvement rate after re-entry, and the repeated re-entry suppression rate can be checked on no less than 300 real samples. If any indicator is lower than the preset threshold, the previous rule version will be rolled back.

[0075] In the preferred embodiment, for the question-and-answer scenario of the power distribution cabinet nameplate, the observation window is set to 180ms, the maximum number of backflows per round is set to 2, and an average of 1.6 inference gaps are identified per session in 300 samples. The matching accuracy of the candidate evidence unit reaches 96.2%, the average difference between the first and second positions of the answer candidate distribution after backflow increases by 18 points, the conclusion improvement rate reaches 91.5%, and the repeated backflow suppression rate reaches 99.3%. Alternatively, in the question-and-answer scenario of mobile terminal inspection, the candidate evidence channel can be searched in sections according to the type of inference gap before backflow is performed. As long as the inference gap is still determined based on the intermediate conclusion state, and the candidate evidence unit corresponding to the inference gap is extracted from the candidate evidence channel and backflowed to the main inference channel to participate in subsequent cross-modal fusion, it belongs to the same technical concept.

[0076] S6. Reorganize the inference cache based on the evidence units that ultimately participate in the fusion, perform decoding output according to the reorganized fusion result, and generate inference evidence records. The specific implementation is as follows: First, after the current session reaches the termination threshold, the evidence units, corresponding text fragments, intermediate conclusion states, used evidence mapping sets, and backflow records that ultimately participated in the fusion within the main inference channel are summarized according to the session identifier. Under the same rule version, the order of evidence, fragments, and index relationships in the inference cache are rearranged. Then, the decoding output is completed based on the reorganized fusion result, and the inference evidence record is simultaneously solidified. This action is executed immediately after the subsequent cross-modal fusion is completed. On-site, it can be located on the result generation side of the production line quality inspection question and answer server, equipment inspection inference node, and invoice review workstation. The cache reorganization execution program calls the session-level high-speed... Once the main inference channel in the storage area has completed loading the list, reflow record, intermediate conclusion status record, and location mapping record, its effect is limited to the period after the current session has stopped reflowing candidate evidence units and the answer candidate distribution, slot filling status, and conflict status meet the termination conditions. The purpose is to compress and rearrange the evidence units that actually participate in the formation of the conclusion in this round according to their final usage relationship, so that the decoding stage only revolves around the valid evidence and leaves a complete trace of the source of the conclusion. It is suitable for scenarios that require giving textual conclusions and tracing the source of evidence, such as mixed text and image understanding, nameplate Q&A, invoice field verification, and inspection defect Q&A.

[0077] The inference cache refers to the session-level storage area that retains the evidence sequence, text sequence, corresponding index, segment state, and backflow relationship during the inference process of the current session. It is obtained before the end of the current session by reading the actual reference records of each segment, removing temporary fragments that are not referenced by the final conclusion, and merging duplicate fragments of the same candidate evidence unit. The evidence units that finally participate in the fusion are jointly determined on the used evidence mapping set and backflow records. On the correlation signal after the last intermediate conclusion state is generated, it is distinguished by checking which candidate evidence units are actually referenced by the first conclusion of the answer candidate distribution and which candidate evidence units only participate in the intermediate comparison but do not enter the final conclusion.

[0078] During reassembly, deduplication is first performed by session identifier, candidate evidence unit sequence number, and return order. Then, the evidence units are rearranged by text segment sequence number, evidence citation order, and location mapping proximity relationship, so that the evidence units corresponding to the same text segment are arranged continuously in the cache, so that unused evidence does not need to be traversed again during subsequent decoding. For example, in the power distribution cabinet nameplate question and answer scenario, if the final conclusion simultaneously references the nameplate current field block, work order value block, and location verification block, these three evidence units that ultimately participate in the fusion are arranged continuously into the inference cache in the order of "problem object - comparison value - location confirmation", and the background block that is not used by the final conclusion is no longer retained.

[0079] The fusion result is determined under the joint constraints of the last intermediate conclusion state and the reorganized inference cache. Its physical meaning is the evidence synthesis state that can directly generate text conclusions. It is obtained at the end of the current session by comparing whether the first and second terms of the answer candidate distribution are stable, whether the slot filling state is complete, and whether the conflict state is eliminated. The decoding output generates the conclusion text word by word based on the fusion result. When generating, the object, attribute, value, and relationship are referenced in the order of the evidence units that finally participated in the fusion. If a slot comes from a backflow candidate evidence unit, the reference mark of the candidate evidence unit is attached when the corresponding text fragment is generated.

[0080] The reasoning evidence record is written at the same time as the conclusion text is generated. Its physical meaning is which evidence units ultimately participated in the fusion to support this conclusion, where each evidence unit is located, when it participated, and whether it was formed through backflow. In the reference record after the current session ends, it is formed item by item by summarizing the session identifier, candidate evidence unit number, location mapping, corresponding text segment number, layer number, backflow order, and conclusion segment number, which can directly support result traceability and verification.

[0081] To ensure on-site feasibility, before reassembly, the consistency of session identifiers, rule versions, and the continuity of layer order are verified. For repeated reassembly requests of the same type, the existing result is directly returned using the session identifier, rule version, and termination time as idempotent keys. For evidence units that are missing location mappings and are ultimately involved in fusion, one supplementary check is allowed according to the backflow record. If they are still missing, the missing location is marked in the reasoning evidence record, but the generation of conclusions is not blocked. For sessions where the conflict state has not been completely eliminated, no definitive conclusion is generated, but a conclusion pending review is generated, and the reference chain of all evidence units that are ultimately involved in fusion is retained.

[0082] The observation window can be set from 100ms to 500ms. The trigger condition is that the current session reaches the termination threshold, and the stop condition is that the decoding output is completed and the inference evidence record is written. The number of retries per session can be set to 1. After exceeding this, the previous valid fusion result is retained and recorded in the trace area.

[0083] Upon completion, a conclusion text record and a reasoning evidence record are generated. The former includes at least a session identifier, conclusion text, generation time, and rule version. The latter includes at least a session identifier, candidate evidence unit number, location mapping, corresponding text segment number, layer segment number, return order, conclusion segment number, generation time, and rule version. The structured records are written to the session-level high-speed storage area and simultaneously to the evidence trace area for the result display interface, review terminal, or subsequent auditing program to call according to the session identifier order. Communication can be achieved through local process calls or LAN message calls. The minimum request fields include session identifier, rule version, and timeout time. The minimum return fields include conclusion status, conclusion length, number of evidence records, and status code. The status codes include at least 0000 success, 6001 missing evidence unit for final fusion, 6002 missing intermediate conclusion status, 6003 rule version inconsistency, 6004 cache reorganization timeout, and 6005 reasoning evidence record writing failure.

[0084] Text blocks containing sensitive personal information remain obscured in the conclusion text and the record of reasoning evidence, with only placeholders and location mappings retained.

[0085] During on-site testing, decoding consistency rate, conclusion traceability rate, latency reduction rate after cache reorganization, and duplicate request reuse rate can be checked on no less than 300 real samples. If any indicator is lower than the preset threshold, the previous rule version will be rolled back.

[0086] In a preferred embodiment, for the power distribution cabinet nameplate question-and-answer scenario, the observation window is set to 220ms, the average number of evidence units participating in the fusion in a single session is 5, the number of fragments retained after cache reorganization is reduced by 41% compared to before reorganization, the decoding consistency rate reaches 96.4% in 300 samples, the traceability rate of the conclusion reaches 99.2%, and the average decoding latency decreases by 23.7% after reorganization. Alternatively, in the mobile terminal inspection question-and-answer scenario, the inference cache can also be divided into a video memory area and a memory area and rearranged separately. As long as the inference cache is still reorganized according to the evidence units that ultimately participate in the fusion, the decoding output is performed according to the reorganized fusion result, and inference evidence records are generated, it belongs to the same technical concept.

[0087] In the operating scenario shown in this embodiment: taking the automatic verification of nameplate parameters on the power distribution cabinet factory verification station as an example, after the operator initiates the inference request of "reading the rated current of the upper left nameplate and judging whether it is consistent with the work order" on the verification terminal, the multimodal inference server receives the single-frame cabinet door image captured by the industrial vision camera and the work order verification statement issued by the manufacturing execution system under the same session identifier. The image content includes the cabinet nameplate area, circuit identification area, warning label area and background connector area, and the text content includes the object word "nameplate", the attribute word "rated current", the position word "upper left" and the judgment word "consistent".

[0088] The encoding execution program then converts the image content into image feature fragments arranged by position, and the text content into text feature fragments arranged by word order. Within the image coordinate system, spatial assignment is formed based on the boundary distance and overlap ratio of adjacent fragments, layout assignment is formed based on line spacing, block spacing, border continuity, and background separators, and semantic fragment assignment is formed based on the positional relationship of the text fragments. As a result, candidate evidence units containing model lines, voltage lines, and current lines are generated in the upper left nameplate area, candidate evidence units for work order parameters are generated in the work order value display area, and irrelevant candidate evidence units are generated in the warning label area. At the same time, a corresponding index from text fragments to candidate evidence units is established.

[0089] The scoring procedure then calculates the intramodal stability, cross-modal response, and post-inference validity for each candidate evidence unit. Among them, the nameplate current field block is given a high joint judgment result and marked as the main participation level because of its continuous boundaries, small brightness fluctuations, adjacent text containing "rated current", and significant influence on the conclusion reversal in historical samples. The work order value block is given the candidate retention level because of its slightly off-center position in the current image and the fact that it may not directly participate in the first round but may be used for comparison later. The warning label block is marked as the release level because of its weak supporting relationship with the current text fragment.

[0090] The channel scheduling program then sends the main participation level evidence units into the main inference channel and the candidate retention level evidence units into the candidate evidence channel. It also records the location mapping, summary representation, and backflow index for the candidate retention level evidence units. The location mapping stores the rectangular range and layout of the work order value block in the image. The summary representation retains the segment represented by the front, middle, and rear of the field. The backflow index is bound to the text segment number corresponding to "whether the rated current is consistent".

[0091] The cross-modal fusion execution program then performs cross-modal fusion by segment within the main inference channel. In the first round of segments, the nameplate current field block and the "rated current" text fragment form a stable alignment. The answer candidate distribution has not yet created a sufficient difference between "consistent" and "inconsistent". In the slot filling state, "nameplate current value" is filled while "work order current value" is empty. The evidence mapping set used only covers the nameplate area. Although there is no numerical contradiction in the conflict state, the relationship is not closed. Therefore, the intermediate conclusion state indicates that there are inference gaps in the current session, such as slot vacancies and uncovered areas.

[0092] The backflow execution program retrieves the candidate evidence unit corresponding to the inference gap in the candidate evidence channel, locks the work order value block according to the backflow index, and confirms that it belongs to the work order parameter display area according to the position mapping. The summary indicates that it contains key value segments corresponding to the current field. Then, the candidate evidence unit is written back to the main inference channel in the backflow order.

[0093] In subsequent stages, the main inference channel simultaneously holds the nameplate current field block and the work order value block. The cross-modal fusion execution program continues to compare object words, attribute words, and relation words, and finds that the two values ​​are consistent. In the answer candidate distribution, "consistent" is significantly higher than "inconsistent". In the slot filling state, both "nameplate current value" and "work order current value" have been uniquely filled. The nameplate area and work order parameter area have been completely covered by the evidence mapping set. The conflict state is eliminated, and the session reaches the termination threshold.

[0094] The cache reorganization execution program then summarizes the evidence units, intermediate conclusion states, used evidence mapping sets, and reflow records that ultimately participate in the fusion. It rearranges the inference cache in the order of "problem object - nameplate value - work order value - location confirmation", retaining only the valid evidence fragments that support the final conclusion. It executes the decoding output according to the reorganized fusion result, generating the conclusion text "the rated current of the upper left nameplate is consistent with the work order record", and simultaneously generates an inference evidence record. The inference evidence record states that this conclusion is supported by the nameplate current field block and the work order value block, and indicates the candidate evidence unit number, location mapping, corresponding text fragment number, layer number, and reflow order of the two.

[0095] If the on-site review interface needs to be reviewed, the system can directly retrieve the conclusion text and reasoning evidence records by clicking the session icon, and locate the specific image positions in the upper left nameplate area and work order parameter area for the operator to complete the final review.

[0096] If the same image and the same work order statement are submitted repeatedly, the system will directly reuse the existing session results according to the established idempotency key, without repeating the entire process. This allows for a traceable, reproducible, and implementable multimodal large model inference acceleration process in a real production line review scenario.

[0097] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A multi-modal large model inference acceleration method based on cross-modal feature fusion, characterized in that, include: S1. Encode the input image content and input text content, generate candidate evidence units according to spatial attribution, layout attribution, and semantic fragment attribution, and establish a corresponding index from text fragments to candidate evidence units; S2. Calculate the intramodal stability, cross-modal responsiveness, and inference validity for each candidate evidence unit, and assign an evidence retention level based on the joint judgment result; S3. Send the main participation level evidence units into the main reasoning channel, send the candidate retention level evidence units into the candidate evidence channel, and record the position mapping, summary representation, and backflow index of the candidate evidence units; S4. Perform cross-modal fusion in the main reasoning channel according to the segment and output the intermediate conclusion status. The intermediate conclusion status includes the answer candidate distribution, slot filling status, used evidence mapping set, and conflict status. S5. Determine the reasoning gap based on the intermediate conclusion status, extract the candidate evidence unit corresponding to the reasoning gap from the candidate evidence channel and return it to the main reasoning channel to participate in subsequent cross-modal fusion. S6. Reorganize the reasoning cache based on the evidence units that finally participate in the fusion, perform decoding output according to the reorganized fusion result, and generate reasoning evidence records.

2. The multi-modal large model inference acceleration method based on cross-modal feature fusion according to claim 1, characterized in that, S1 includes: After encoding the input image content and input text content, spatial assignment is determined based on the boundary distance and overlap ratio of adjacent image feature segments; Page layout is determined based on line spacing, block spacing, border continuity, and background dividers. The semantic segment attribution is determined based on the positional relationship between text segments and the collocation relationship of business terms; Based on spatial attribution and page layout as boundary constraints, candidate evidence units are generated by combining the attention area corresponding to the semantic fragment attribution, and a corresponding index from text fragment to candidate evidence unit is established.

3. The multi-modal large model inference acceleration method based on cross-modal feature fusion according to claim 1, characterized in that S2 include: Intramodal stability is determined by comparing the adjacent differences, boundary continuity, texture repetition, and brightness fluctuation amplitude of image feature segments within candidate evidence units. Cross-modal responsivity is determined by comparing object words, attribute words, and location words in a text segment with the page layout, spatial location, and adjacent text content of candidate evidence units; Post-inference validity is determined by comparing the influence trends of candidate evidence units on candidate answer ranking, key slot filling, and conflict reduction.

4. The method for accelerating multimodal large model inference based on cross-modal feature fusion according to claim 3, characterized in that, S2 also includes: The joint judgment results are formed in the order of intramodal stability, cross-modal responsiveness, and inference after-validity. Candidate evidence units with intramodal stability below the baseline value are excluded first. The remaining candidate evidence units are then ranked according to cross-modal response and the order is adjusted in conjunction with inference post-validation. Subsequently, evidence retention levels were assigned to the primary participation level, the alternate retention level, the delayed awakening level, and the release level, respectively.

5. The method for accelerating multimodal large model inference based on cross-modal feature fusion according to claim 1, characterized in that, S3 include: When a candidate-reserved evidence unit is sent into the candidate evidence channel, a location map, a summary representation, and a return index are generated simultaneously. The location mapping consists of the coordinates of the top left corner, bottom right corner, width, height, page layout, and adjacent unit relationships of the candidate reserved evidence unit; The abstract indicates that it is formed by merging fragments within the candidate-reserved evidence unit in spatial order; The backflow index consists of session identifier, candidate evidence channel number, candidate evidence unit number, associated text fragment number, location mapping address, and summary representation address.

6. The method for accelerating multimodal large model inference based on cross-modal feature fusion according to claim 1, characterized in that, S4 includes: Intermediate conclusion states are formed at the end of each segment; The distribution of candidate answers is determined based on the number of times a candidate answer is supported by object words, attribute words, position words, and relation words, the length of consecutive support, and the mutual exclusion of each other. The slot filling status is determined based on the unique corresponding evidence of the business element position, the vacancy status, and the multi-value competition status. The conflict state is determined based on the conflict of business element values, the conflict of location area attributes, and the reverse change of conclusions in adjacent layers.

7. The method for accelerating multimodal large model inference based on cross-modal feature fusion according to claim 1, characterized in that, S5 include: The reasoning gap is determined based on the intermediate conclusion state; Inference gaps include slot vacancies, value range conflicts, uncovered regions, and unclosed relationships. Slot vacancy is determined based on slot filling status. Value range conflict is determined based on conflict status. The area not covered was determined based on the evidence mapping set and the location words of the text fragments; The relationship is not closed, which is determined by the distribution of candidate answers and the relation words in the text fragment.

8. The method for accelerating multimodal large model inference based on cross-modal feature fusion according to claim 7, characterized in that, S5 also includes: Candidate evidence units are extracted based on the backflow index, location mapping, and summary representation; First, retrieve candidate evidence units with the same sequence number as the associated text fragments based on the backflow index; Then, based on the location mapping, candidate evidence units that do not meet the requirements of target area, page ownership, and adjacency relationship are filtered out; Subsequently, candidate evidence units are identified based on the summary and written back to the main reasoning channel in the order of reflow.

9. The method for accelerating multimodal large model inference based on cross-modal feature fusion according to claim 1, characterized in that, S6 include: After deduplication based on session identifier, candidate evidence unit sequence number, and reflow order, the inference cache is rearranged based on text fragment sequence number, evidence citation order, and location mapping proximity relationship. The fusion result is determined based on the state of the last intermediate conclusion and the rearranged inference cache; The inference evidence record includes session identifier, candidate evidence unit number, location mapping, corresponding text segment number, layer number, reflow order, and conclusion segment number.

Images