Text and image bidirectional alignment method and system based on multi-hop parallel reasoning

Abstract

The invention discloses a text and image bidirectional alignment method and system based on multi-hop parallel reasoning, and the method comprises the steps: analyzing and marking a text, and obtaining a multi-granularity text feature group; image generation is processed, grids and target output are combined into a multi-scale visual feature group, an alignment index table is established, a sub-target chain is constructed based on visual features, the position and the dependency relationship are recorded, and explicit constraints are injected into five elements; recalling the candidate area, generating an anchor point through verification and fusion, writing the anchor point into a cross-modal index, locally completing three-layer decoupling scoring based on anchor point geometry and the cross-modal index, and outputting a three-state evidence in combination with a threshold value; the method comprises the steps of generating an evidence set through geometric and semantic consistency calibration, generating cross-hop evidence according to a rule, weighting and pooling the evidence set, generating a shared vector and traceable meta-information, inputting a task head output result and generating an evidence list and an auditing path, and according to the method, local alignment, constraint verification and evidence accumulation are completed hop by hop. And high-precision, traceable and interpretable cross-modal correspondence is realized.

Description

Technical Field

[0001] This invention relates to the field of modal alignment, and more particularly to a method and system for bidirectional alignment of text and images based on multi-hop parallel reasoning. Background Technology

[0002] With the continuous increase in the scale and complexity of multimedia data applications, the cross-modal alignment between image content and natural language description is inevitably moving from global similarity to fine-grained correspondence. Traditional methods often extract features separately using pre-trained visual / language encoders, and then perform one-time matching at a high level through self-attention or global pooling. This type of unidirectional, global alignment often fails to accurately capture local relationships such as word-region and phrase-object in multi-entity, complex action, or detail-intensive scenes, resulting in inaccurate alignment and difficulty in interpretation.

[0003] To improve alignment accuracy, existing research has attempted to introduce multi-head attention, cross-modal interaction, or joint training objectives to achieve finer fusion at the feature level. However, such improvements often remain within a single global alignment paradigm, lacking explicit binding to the textual hierarchical structure and visual multi-scale structure, making it difficult to perform traceable verification of fields such as color, shape, orientation, and interaction relationships. At the same time, semantic interference and constraint conflicts are prone to occur in multi-entity relationship chains, which is not conducive to maintaining stable performance and auditability on large-scale, open-domain data. The patent with publication number CN120146051A provides a method and system for multimodal entity and relation extraction based on cross-modal alignment and fusion, including: processing and encoding input text and images to obtain multiple image and text features; using the semantic representation of the image as an anchor, aligning fine-grained and coarse-grained text features with pixel-level image representations respectively, mapping image and text features to the same semantic space; performing multi-granularity feature fusion through text-guided dynamic gating aggregation, visual prefix cross-modal fusion, and cross-modal image-text matching, increasing feature complementarity while modeling the association between noun phrases in the text and image objects, obtaining multi-granularity multimodal feature representations; fusing multi-granularity multimodal features through entity-guided attention gating, aggregating visual information related to text entities, obtaining the final multimodal fused representation; and performing task prediction for multimodal named entity recognition and multimodal relation extraction based on the multimodal fused representation. By employing cross-modal alignment and fusion methods in multimodal named entity recognition and multimodal relation extraction, this method utilizes pixel-level and semantic representations of images for feature alignment and fusion, solving the problem of difficult image information integration, improving task performance, and providing more comprehensive contextual information. It focuses on learning-based alignment, pays more attention to feature diversity, solves the modal noise problem, and directly outputs multimodal named entity recognition and relation extraction results.

[0004] To address the aforementioned issues, there is an urgent need for a transparent, sub-target-driven interactive framework that can preserve the hierarchical semantics and role dependencies of words, phrases, and sentences on the text side, integrate multi-scale features of grids and object detection on the visual side, and use explicit field mapping as a bridge to complete local alignment, constraint verification, and evidence accumulation hop by hop, thereby achieving high-precision, traceable, and interpretable cross-modal correspondence. Summary of the Invention

[0005] One objective of this invention is to propose a text-image bidirectional alignment method based on multi-hop parallel reasoning, which can preserve the hierarchical semantics and role dependencies of words, phrases, and sentences on the text side, and integrate multi-scale features of grid and object detection on the visual side, thereby achieving high-precision, traceable, and interpretable cross-modal correspondence and effectively improving the accuracy of cross-modal semantic alignment.

[0006] To achieve the above objectives, in a first aspect, the present invention provides a method for bidirectional alignment of text and images based on multi-hop parallel inference, comprising the following steps: S1. Parse the text for words / phrases / sentences and label the subject, attributes, actions, recipients, and scenes to obtain multi-granularity text feature sets. The image generation grid and target outputs are merged into a multi-scale visual feature set. And create an alignment index table from text fields to visual fields; S2. Construct a sub-target chain based on the two multi-scale features of the grid and the target, record the position and dependency of the sub-targets in the sub-target chain, and inject explicit constraints into the subject, attributes, actions, receptors and scenes; S3. Use the alignment index table to recall candidate regions of semantic nearest neighbors. Based on the sub-target dependencies and explicit constraints, perform field-level attribute verification and orientation prior fusion on the candidate regions, generate anchor points and write them to the cross-modal index. S4. Based on the generated anchor point geometry and the updated cross-modal index, perform three-layer decoupled scoring at the local anchor point, and output three-state evidence by judging through the role sensitivity threshold. S5. Using three-state evidence combined with sub-target chains and explicit constraint geometry, geometric and semantic consistency calibration is performed in the neighborhood of the anchor point according to the dependency relationship to obtain the calibrated evidence set; the merging rule of necessary condition priority and sufficient condition supplementary evidence is adopted to generate calibrated jump evidence. S6. Weight the calibrated evidence set according to the priority and dependency of the sub-target chain, and generate a shared vector and traceable meta-information; S7. Input the shared vector and traceable metadata into the task header, output the task results, and simultaneously generate an evidence list and auditable path based on the traceable metadata.

[0007] Furthermore, S1 includes the following steps: S11. Parse the input text by word, phrase, and sentence, and build a vector sequence. With location index Includes text type markers, color tags, action location entities. ; Segmenting sentence dependencies into phrase fragments And assign the character's main attributes, actions, and the scene. The feature sequence from word to phrase to sentence is concatenated into a multi-granularity text feature set. Explicitly reserved To support subsequent sub-target retrieval; S12. Decompose the image into two branches: grid and target. Generate a set of regions for the grid. Extracting regional features , and a color histogram are attached. Outline description Normalized coordinates Generate an object set for candidate targets. Extract semantic features , and attached The two outputs are merged into a multi-scale visual feature set. And record the source marker and spatial index; S13. Establish a mapping between text types and visual fields and solidify it into an alignment index table. Record a quadruple for each text item. According to Select field key (color →) Action → rel, Location → Entity → ), and provide the source grid target. Align the index table Binding to multi-granularity text feature groups With multi-scale visual feature groups In the description.

[0008] Furthermore, S2 includes the following steps: S21. Group multi-granularity text features The words and phrases in the text are traversed sequentially, and entity attributes, actions, and scenarios are marked by type. Categorize and index by position Bind to the original text position for each phrase / segment Determining semantic roles based on sentence dependency relations Within the same sentence range, records them as a sequence according to their order of appearance. Set the cross-sentence boundary as a non-mergeable constraint; S22, Semantic sequence Sentences are assembled sequentially into sub-target chains. It adopts a fixed order of subject, attribute, action, recipient, and scene, and records the minimum necessary field for each sub-target: text location. Literal content Character Type and the dependency pointers to the previous sub-target. The missing subject's actions and the receptor are marked as a state to be completed; S23. Inject each sub-objective in the sub-objective chain into an executable set of explicit constraints. Fill in the subject and receptor category key class Same as the same key Fill the attribute with key color Or shape key To fill in the interactive prompts for actions Fill the scene with directional key bindings Or background key scene Match each key with the text position Bind to field-level key-value descriptions to form a ternary index of key, value, and position; S24, Sub-target chain Organize into a structured sequence and assign an execution sequence number. Target category hints Attribute filtering conditions with position reference Assembled into a control signal set ; Explicitly define cross-subtarget dependencies as edges This record is used to trigger subsequent association verification and local search scope limitation. As a direct input to the candidate search and filtering strategy, visual regions that are irrelevant to the current sub-target are eliminated.

[0009] Furthermore, S3 includes the following steps: S31. Transfer the text fragment of the current sub-target. With multi-scale visual feature groups Pair all regions one by one to generate candidate pair sequences. ,in, For each candidate pair, the main semantic comparison is performed using the cosine similarity between the word vector and the region semantic vector, covering both the grid and the target region, to obtain semantic segmentation. Righteousness Along with text position index With regional spatial index These are all included in the candidate list to cover potential alignment regions using a semantic nearest neighbor approach, providing a searchable base set for the overlay of attributes and orientation constraints. S32. Calculate attribute matching for each candidate in the candidate list. Color terms are mapped to color fields, shape terms to outline fields, and category hints to subcategory fields. Field-level consistency checks are performed, and entries lacking required fields or exhibiting obvious conflicts are directly removed, retaining only those that pass the checks. and genus Write the entry identifier as a parallel record, and specify the text position. Maintain binding with field keys; S33, Calculate the prior quantiles of the retained entries. For absolute directional terms, compare with normalized coordinate fields; for relative directional terms, check the relative layout of paired regions; and define the meaning. , belonging to ,Bit The comprehensive score is obtained by weighting and merging the results using preset weights. Perform a full sort on all entries, and mark the samples that rank high in the overall sort but have conflicting attributes or positions with conflict flags. S34. Fix the first few combinations in the upper-threshold interval of the comprehensive ranking result as pairs to generate a triplet semantic attribute summary. And write it into the cross-modal index table. Assign a unique identifier to each anchor point. Preserve references to source branches and field keys.

[0010] Furthermore, S4 includes the following steps: S41. The cross-modal index table for the current jump... Perform batch processing, reading the text position of anchor pairs based on unique identifiers. With regional index Extract text fragments for each anchor pair. Multi-granularity representation of regions and the description of the jump target. Sequentially concatenate the data to form a ternary input. ; S42. Read the explicit constraints of the jump target from the key-value table as a control vector. Mapping to ternary inputs by field alignment; assigning separate fields to color, shape, category, orientation, and interactive prompts. Independent slots, and with , Establish alignment pointers for the corresponding fields, and... By adding an independent channel to the ternary input, we obtain ; S43, will Perform interpretable calculations and limit the calculations to a local range of anchor points, performing layer-by-layer calculations to obtain the sub-item scores and total scores of the three-layer output; S44. Generate master-slave thresholds by assigning the output sub-scores to the roles in a sensitive manner, and complete the traceability determination and database entry in a segmented process. Write the three-state results and field-level reasons into the evidence entries. S45. Summarize the valid evidence for this jump in parallel within the batch, remove duplicates by anchor point unique identifier, and maintain the reference relationship with the index table. Add jump number and timestamp to each piece of evidence, output as a structured list to the shared buffer, and retain the dependency tags and restricted retention status in the list to the entries. Furthermore, Performing interpretable calculations, and limiting them to pairwise layer-by-layer calculations within a local area of ​​the anchor point, includes: The first layer only performs channel-by-channel multiplication and addition on the semantic core, and The fusion yields the main semantic response, answering whether it is the stated subject or object, without introducing attributes or location; The second layer will control vectors The attribute slots are aligned with the corresponding fields of the region. Gated multiplication is used to apply gain to consistent attributes and suppress conflicting attributes, maintaining decoupling from the first layer and only answering whether the attributes are consistent. The third layer will The orientation and relationship slots are mapped to the geometry and interaction channels. The absolute orientation and relative relationship are verified only in the local neighborhood of the current anchor point. Neighborhood queries are triggered to confirm the mutual pointing of the action and the receiver, and to answer whether the position and interaction are valid. The three-layer output is generated in a unified manner with the sub-scores and the total score, and a single auditable fusion is adopted: first, the semantic backbone score, attribute consistency score and directional relationship score are generated, and then fused into the final compatibility within the same link. Each component is established with a field-level reference to its source slot, and the keys and indexes used for alignment are retained. Fusion formula:

[0011] in: For final compatibility; For text semantic vectors; For region semantic vectors; This is channel-by-channel multiplication; , , Three-layer fusion weights; , These are the control sub-vectors for attribute and orientation relationship, respectively; , For the region's attribute fields and geometry / relationship fields; , This is a function for verifying the gating function and its neighborhood relationship.

[0012] Furthermore, the output sub-scores are used to generate master-slave thresholds based on role sensitivity, and a segmented process is used to complete traceability determination and data entry, including: The first paragraph maps the sub-target roles to the main dimension: the subject corresponds to the semantic core, the attributes correspond to the consistent attributes, and the actions and the recipients correspond to the directional relationship; for the non-main dimensions, a threshold is set and bound to the sub-target number, forming a configuration table of threshold groups and corresponding entries; The second section performs a judgment on each anchor point in the order of primary and secondary dimensions: if the primary dimension does not meet the threshold, it is directly rejected and the rejection field is recorded; if the primary dimension passes, the secondary dimensions are checked in turn. If the primary dimension is slightly insufficient, it is marked as restricted and retained with the trigger condition. If there is a clear conflict, it is immediately removed and the conflict source key is recorded. The third stage will save the text position, region index, sub-score, threshold configuration, judgment status, and triggering conditions by writing the evidence set of the current jump with the restricted entries, so that subsequent consistency calibration can directly read and close the evidence chain according to the conditions:

[0013] in: For the sub-target role; , , These are respectively: semantic core score, attribute consistency score, and location relationship score; The sensitivity coefficient for character integration; The main dimension is divided according to the role; Divide into any dimension; , Master / slave thresholds are bound to the character; This serves as a reference value for the role-sensitive threshold used for recording. Output is a three-state determination; The above formula presents the rules in the order of primary dimension priority and secondary dimension verification, and provides explicit condition entry for restricted retention. The three-state output results and field-level reasons are written into the evidence entries: the primary field key and corresponding score for the rejected sample record that did not reach the threshold; the secondary field key and triggering condition for the insufficient restricted retention records; and all field keys and sub-items of the passed sample records. The evidence set is output as a structured list, and it is referenced one by one with the anchor unique identifier, text position and area index.

[0014] Furthermore, S5 includes the following steps: S51. Parse the dependencies between adjacent sub-targets in the sub-target chain into directed relationships: category, master, owner, active, passive, master, scene. The anchor point is uniquely identified by pairing the previous hop evidence set with the next hop candidate set as the two ends. With text position Establish connection keys, generate a candidate connection table, where each connection is limited to the local neighborhood of the corresponding anchor point for calibration, and sort the connection table by relation category; S52. Extract the geometric fields of paired regions from the candidate join table. Relationship hints from control vector The system reads data from the middle, compares the absolute orientation relationship with the normalized coordinate range, verifies the center distance between the two boxes, the occlusion ratio and the orientation indicator for the relative relationship, triggers neighborhood restrictions for verbs involving interaction, searches and verifies proximity or opposite signs only within the local range of the subject and the receiver, marks connections that do not meet the above geometric constraints as geometrically inconsistent, and retains the conflict source key. S53. Read the categories and attribute constraints determined in the previous hop from the evidence entries as an inheritance set, and then apply this set to the fields of the next hop candidate. The fields are checked one by one according to color, shape, material, category and quantity. Fields that must be inherited are set as strong consistency and are removed if there is a conflict. Descendable descriptions are set as subordinate attributes. If there is a deficiency, it is marked as needing to be supplemented and the triggering conditions and missing fields are registered. The consistency check results are written back to the join table and coexist with the geometric inconsistency mark to ensure that the subsequent rule merging has a complete basis. S54. Assign the join table to the table by relation type: primary, secondary, active, passive, primary. Grouping, constructing a rule-priority merger, solidifying the judgment logic based on the order of necessary prerequisites to sufficient supplementation, explicitly exposing the restricted and retained trigger conditions and insufficient fields, and obtaining a three-state judgment result; S55. Write the retained and restricted retention entries from the three-state determination results into the calibrated evidence set, and retain the anchor point identifiers. Text position Regional Index 1. Relationship category and judgment status; 2. For restricted retention, attach trigger conditions and fields to be verified, clarify the direction to be verified in the next hop, record the reason key for the removal of the removed entries, and merge the calibrated evidence set with the original evidence set to generate a cross-hop evidence set.

[0015] Furthermore, the join tables are categorized by relation type: primary, secondary, active, passive, and primary. Grouping, building rules-first merger includes The geometric validation result and semantic attribute validation result of each connection are used as two inputs. First, a necessary condition judgment is performed: geometric necessary condition is used for subject → action and action → recipient, attribute necessary condition is used for subject → attribute, and category consistency necessary condition is used for subject → scene. Only entries that pass the necessary condition are then evaluated for sufficient condition, ensuring that the prerequisite for the relationship is not affected by attribute details.

[0016] The sufficient conditions are set as a set of attributes, and the auxiliary descriptions of color, shape, material, quantity and orientation are verified in turn. Entries without conflicts but with deficiencies are retained with restrictions, and the fields to be verified and the triggering conditions are registered in the connecting entries, pointing to subsequent jumps for lateral verification by the relevant sub-targets; entries with strong conflicts (such as color conflicts or opposite relative orientations) are directly marked as conflicted and removed, and do not enter the verification process. The merged output is written to the calibration decision, using a three-state structure with a source key (geometric or attribute), allowing the audit to point back to the triggering rule and corresponding field. The rule-based merge formula is as follows:

[0017] in: For three-state determination; It is a relation type; This is the vector of geometric verification results; This is a vector of attribute validation results; This is a necessary condition function that switches according to the relation type (geometry / attribute / category consistency); For a sufficient set of condition fields (color, shape, material, quantity, and orientation auxiliary); A set of strongly conflicting fields; For the first Consistency indicators for fields; Based on the above formula, the judgment logic is solidified in the order of necessary prerequisites to sufficient supplementation, and the limited and retained trigger conditions and insufficient fields are explicitly exposed to facilitate subsequent jump-oriented supplementary verification and backtracking.

[0018] Furthermore, S6 includes the following steps: S61. Aggregate the calibrated and retained jump evidence into a set according to the sub-target number. And read the role type of each sub-target. With priority sequence Map the role sensitivity weight table to sub-target weights It is used to control the contribution ratio of subject, attribute, action, receptor, and scene in the final representation, and to transform the textual features of evidence entries. With visual features Uniquely identified by anchor point Pairing into units; S62. Construct a weighted concatenation vector sum for each piece of evidence. Among them, To ensure compatibility, multiple pieces of evidence under the same sub-target are obtained through weighted pooling. The weight is determined by the overall and To jointly determine and unify the anchor points and avoid strong semantics or high priority dominance by one side, the anchor points will be identified during the pooling process. Text position Regional Index The sub-score pointers are written into the metadata index; S63. Pooling results of each sub-target Consolidate and lightweightly merge sub-target chains in order to generate a shared vector. As a unified representation across modalities; parallel output structured subchannels Record the list of valid evidence and anchor point identifiers for each sub-target. Sub-objective weights Overall compatibility With the triggering conditions, and They are bound to the same record for direct reading by retrieval, location, or question-and-answer task headers.

[0019] Furthermore, S7 includes the following steps: S71. Input the shared vector into the corresponding task header, select the reading rule according to the task type, and generate the target image identifier, target region identifier, and answer identifier. Then, combine the target image identifier, target region identifier, and answer identifier output from the task header with the sub-target number and anchor point identifier. Establish a reference; S72. Transcribe the valid evidence for each sub-target into a ternary record with the sub-target number. ,in For anchor point identification, For text location indexing, append records with compatibility scores. The system determines the state and triggering conditions, forming a structured evidence list. This list is then organized into a visual path according to the sub-target chain sequence and relational direction, sequentially presenting the alignment relationships of the subject, attributes, actions, and recipients or scenes. This is synchronized with the output of target image identifiers, target region identifiers, and answer identifiers. S73. Bind the target image identifier, target area identifier, answer identifier, and evidence list to the same auditable entry, and provide sub-target number and area index. Text position The system offers filtering and redirection capabilities, explicitly prompting for evidence marked with restricted retention, allowing users to view the fields to be verified and triggering conditions on the interface, and identifying them by anchor points. Returning to the original candidate, the current session is exported as a structured report, including the main result, evidence path, ternary record, and key judgment criteria.

[0020] Secondly, the present invention provides a text and image bidirectional alignment system based on multi-hop parallel reasoning, including a preprocessing module, a target chain construction module, an anchor point generation module, a three-state evidence acquisition module, a calibration evidence generation module, a traceability information generation module, and a result generation module; The preprocessing module is used to parse the text into words / phrases / sentences and annotate the subject, attributes, actions, recipients, and scenes to obtain multi-granular text feature sets. The image generation grid and target outputs are merged into a multi-scale visual feature set. And create an alignment index table from text fields to visual fields; The target chain construction module constructs sub-target chains based on two multi-scale features: mesh and target. It records the position and dependency of sub-targets in the sub-target chains and injects explicit constraints into the subject, attributes, actions, receptors, and scene. The anchor point generation module uses the alignment index table to recall candidate regions of semantic nearest neighbors. Based on the dependencies of sub-targets and explicit constraints, it performs field-level attribute verification and orientation prior fusion on the candidate regions, generates anchor points, and writes them into the cross-modal index. The three-state evidence acquisition module performs three-layer decoupled scoring at the local anchor point based on the generated anchor point geometry and the updated cross-modal index, and outputs three-state evidence after judging by the role sensitivity threshold. The calibration evidence generation module uses three-state evidence combined with sub-target chains and explicit constraint geometry to perform geometric and semantic consistency calibration in the neighborhood of the anchor point according to the dependency relationship, and obtains the calibrated evidence set; it adopts the merging rule of necessary condition priority and sufficient condition supplementary evidence to generate calibrated jump evidence; The traceability information generation module is used to aggregate evidence by weighted pooling of the calibrated evidence set according to the priority and dependency of the sub-target chain, and generate shared vectors and traceable meta-information by weighted pooling. The results generation module is used to input shared vectors and traceable metadata into the task header, output task results, and simultaneously generate an evidence list and auditable path based on traceable metadata.

[0021] Compared with the prior art, the present invention has at least the following beneficial effects: This invention replaces one-time global alignment with sub-target chains, explicit constraints, and local anchor point calculation. By mapping elements including color, shape, orientation, and interaction to text keys and visual color histograms, contours, and geometric / relational fields, and progressively generating and calibrating anchor points across multiple hops, it achieves fine-grained word-region and phrase-object correspondence. This mechanism proposed in this application can stably distinguish subjects / attributes / relationships even in multi-entity, complex action scenarios, significantly reducing mismatches and providing field-level traceability.

[0022] This invention employs a three-layer decoupled scoring and role sensitivity threshold within the anchor point neighborhood: the first layer only verifies the semantic backbone, the second layer uses gating to gain / suppress attribute consistency, and the third layer performs absolute / relative orientation and interaction verification; the sub-items of each layer and the source key are fused and recorded by a single link. This design strengthens the correspondence between key regions and keywords, avoids non-primary factors interfering with subject identification, and significantly improves cross-modal matching accuracy and the ability to identify subtle differences and local relationships. This invention, through cross-jump consistency calibration and chain-like evidence accumulation, solidifies the prioritization of necessary conditions and the supplementary evidence of sufficient conditions into rules. It adaptively selects geometric or attribute-based prerequisites according to relation type and generates restricted retention and supplementary fields for insufficient samples. Finally, it aggregates these into a shared vector using role-sensitive weights and synchronously outputs an auditable secondary channel. Based on the above process, it reduces reliance on prior data and re-annotation, decreases parameter tuning costs, and ensures flexibility and robustness on heterogeneous and diverse text and image data. Attached Figure Description

[0023] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention, but do not constitute a limitation thereof; in the drawings: Figure 1 This is a flowchart of a text-image bidirectional alignment method based on multi-hop parallel inference proposed in this invention; Figure 2 This is a flowchart of the sub-target chain and control key generation process for a text and image bidirectional alignment method based on multi-hop parallel reasoning proposed in this invention. Figure 3 This is a flowchart of a local three-layer interpretable verification process for a text-image bidirectional alignment method based on multi-hop parallel reasoning proposed in this invention. Figure 4 This is a flowchart of the cross-hop consistency calibration process for a text and image bidirectional alignment method based on multi-hop parallel inference proposed in this invention. Figure 5 The image shown is an example of a text-image bidirectional alignment method based on multi-hop parallel reasoning proposed in this invention. Detailed Implementation

[0024] S1. Perform word / phrase / sentence parsing on the text and label the subject, attributes, actions, receptors, and scenes; generate two multi-scale features for the image: grid and target, and establish an alignment index table from text fields to visual fields; S2. Construct sub-target chains by sentence, record text positions and dependency pointers, and inject explicit constraints such as category, color, shape, orientation, and interaction into the subject, attributes, actions, receptors, and scenes; S3. Based on semantic nearest neighbor, recall candidate regions, perform field-level attribute verification and orientation prior fusion, generate anchor points and write them to the cross-modal index; S4. Perform three-layer decoupling scoring (semantic backbone, attribute consistency, and orientation relationship) at the anchor point and determine the role sensitivity threshold, and output three-state evidence; S5. Based on the sub-target dependency, perform geometric and semantic consistency calibration in the local neighborhood, and use the merging rule of prioritizing necessary conditions and supplementing with sufficient conditions to generate calibrated evidence. S6. Aggregate evidence by sub-target and weight pool it to obtain shared vectors and traceable meta-information; S7. Input the shared vector into the task header to output the retrieval / location / question answering results, and simultaneously generate an evidence list and audit path.

[0025] The present invention will now be described in further detail with reference to the accompanying drawings, which are simplified schematic diagrams illustrating only the basic structure of the invention and therefore only show the components relevant to the invention. refer to Figure 1 A text-image bidirectional alignment method based on multi-hop parallel inference includes the following steps: S1. Parse the text into words / phrases / sentences and label the subject, attributes, actions, receptors, and scenes; generate two multi-scale features for the image: mesh and target, and establish an alignment index table from text fields to visual fields; Step S1 includes the following sub-steps: S11. Parse the input text by word, phrase, and sentence, and build a vector sequence. With location index Includes text type markers, color tags, action location entities. .

[0026] Segmenting sentence dependencies into phrase fragments And assign the character's main attributes, actions, and the scene. .

[0027] The feature sequence from word to phrase to sentence is concatenated into a text feature group. Explicitly reserved This is to support subsequent sub-target retrieval.

[0028] S12. Decompose the image into two branches: the grid and the target. Generate a set of regions for the grid. Extracting regional features , and a color histogram are attached. Outline description Normalized coordinates Generate an object set for candidate targets. Extract semantic features , and attached The two outputs are merged into a multi-scale visual feature set. It also records the source marker and spatial index for subsequent selective matching by sub-target type.

[0029] S13. Establish a mapping between text types and visual fields and solidify it into an alignment index table. Record a quadruple for each text item. According to Select field key (color →) Action → rel, Location → Entity → ), and provide the source grid target. Align the index table. Binding to text feature groups With multi-scale visual feature groups The description allows subsequent sub-targets to be directly located to the corresponding visual dimension and complete field-level constraint binding.

[0030] S2. Construct sub-target chains by sentence It records text position and dependency pointers, and injects explicit constraints, including category, color, shape, orientation, and interaction, into the subject, attribute, action, receptor, and scene; Step S2 includes the following sub-steps: S21. Group multi-granularity text features The words and phrases in the text are traversed sequentially, and entity attributes, actions, and scenarios are marked by type. Categorize and index by position Bind to the original text position. For each phrase / segment Determining semantic roles based on dependency relationships , each phrase fragment Within the same sentence, records are categorized into semantic sequences based on their order of appearance. Setting cross-sentence boundaries as non-mergeable constraints avoids cross-sentence confusion and provides stable input and traceable position references for subsequent fixed logic to generate sub-targets.

[0031] S22, Semantic sequence Sentences are assembled sequentially into sub-target chains. It adopts a fixed order of subject, attribute, action, recipient, and scene. For each sub-target, the minimum necessary field is recorded: text location. Literal content Character Type and the dependency pointer to the previous sub-target (e.g., attribute-dependent subjects, action-dependent subjects, or receptors). The missing subject's action and receptor are marked as pending completion, serving only as placeholders without expansion, ensuring semantic continuity within the chain and clear dependency direction, thus establishing an execution order for hop-by-hop alignment and evidence transfer.

[0032] S23, Sub-target chain Inject an executable set of explicit constraints into each sub-objective. Fill in the category key classes for the subject and receptor. Same as the same key Fill the attribute with key color Or shape key To fill in the interactive prompts for actions Fill the scene with directional key bindings Or background key scene Match each key with its text position. Binding is done as a field-level key-value description, forming a ternary index of key, value, and position, which allows for direct filtering, constraint binding, and selection by field on the visual side, avoiding ambiguity caused by global fuzzy matching.

[0033] S24, Sub-target chain Organize into a structured sequence and assign an execution sequence number. Target category hints Attribute filtering conditions with position reference Assembled into a control signal set .

[0034] Depend across sub-targets as explicit edges This record is used to trigger subsequent association verification and local search scope limitation. As a direct input to the candidate search and filtering strategy, visual regions unrelated to the current sub-target are eliminated, ensuring that complex descriptions have a clear and interpretable alignment path in multi-entity and action scenarios.

[0035] S3. Based on semantic nearest neighbor, recall candidate regions, perform field-level attribute verification and orientation prior fusion, generate anchor points and write them to the cross-modal index; Step S3 includes the following sub-steps: S31. Transfer the text fragment of the current sub-target. With multi-scale visual feature groups Pair all regions one by one to generate candidate pair sequences. ,in The regions covering both the grid and the target are defined. For each candidate pair, a main semantic comparison is performed using the cosine similarity between the word vectors and the region semantic vectors to obtain semantic segmentation. Semantic separation Along with text position index With regional spatial index They are also written into the candidate list to cover potential alignment areas in a semantic nearest neighbor manner, providing a searchable basic set for the superposition of attributes and orientation constraints.

[0036] S32. Calculate attribute matching for each candidate in the candidate list. Color terms are mapped to color fields, shape terms to outline fields, and category hints to subcategory fields, with field-level consistency checks performed. Entries lacking required fields or exhibiting obvious conflicts are directly removed, retaining only those that pass the verification, and semantically delimited. Matching attributes Write the entry identifier as a parallel record. (Text position) Bind to the field key to ensure that subsequent traceable filtering and hierarchical sorting can be performed to gradually narrow down the search scope.

[0037] S33, Calculate the prior quantiles of the retained entries. For absolute directional terms, the coordinate fields are compared to normalized coordinate fields; for relative directional terms, the relative layout of paired regions is checked. Semantic segmentation is performed. Attribute matching and classification Prior quantiles of position The comprehensive score is obtained by weighting and merging the results using preset weights. A full sort is performed on all entries. Samples that rank highly in the overall sort but have conflicting attribute or positional markers are marked with conflict flags to prevent them from being selected as valid anchors in the future, ensuring that the dual constraints of semantics and space are implemented in the candidate stage.

[0038] S34. Fix the first few combinations in the upper-threshold interval of the comprehensive ranking result as pairs to generate a triplet semantic attribute summary. And write it into the cross-modal index table. Assign a unique identifier to each anchor point. It retains references to source branches and field keys to ensure that cross-jump retrieval and evidence backtracking can directly locate text fragments and image regions. The upper threshold interval is the range of scores that are higher than the preset qualified score line (threshold) in the sorted results of the "comprehensive score after the fusion of semantic score, attribute score, and orientation score". Candidate regions whose comprehensive scores fall within this interval can serve as anchor points.

[0039] S4. Perform three-layer decoupling scoring (semantic backbone, attribute consistency, and orientation relationship) at the anchor point and perform role sensitivity threshold judgment to output three-state evidence; Step S4 includes the following sub-steps: S41. The cross-modal index table for the current jump... Perform batch processing, reading the text position of anchor pairs based on unique identifiers. With regional index Extract text fragment representations for each anchor point pair. Multi-granularity representation of regions and the description of the jump target. Sequentially concatenate the data to form a ternary input. The number of jumps refers to the independent processing round of a single sub-target in the sub-target chain. For example, processing "body (cup)" is "jump 1", processing "attribute (red)" is "jump 2", and processing "action (place)" is "jump 3". Each jump corresponds to the complete alignment process of a sub-target (from anchor point reading, scoring to threshold judgment).

[0040] S42. Read the explicit constraints of the jump target from the key-value table as a control vector. Mapping to ternary inputs by field alignment Each of the following uses a control vector: color, shape, category, orientation, and interactive cues. Independent slots, and with , Establish alignment pointers for the corresponding fields. Set the control vector... Add a separate channel to the ternary input This allows the explicit constraints of the jump target to participate in the calculation in an auditable manner, supporting differential verification by semantic dimension and avoiding alignment ambiguity caused by implicit fusion.

[0041] S43, Input the triplet Interpretable calculations are performed, but are limited to pairwise layer-by-layer calculations within the local area of ​​the anchor point.

[0042] The first layer only performs channel-by-channel multiplication and addition on the semantic core, and The fusion yields the main semantic response, answering whether it is the subject or object, without introducing attributes and location, thus avoiding interference from non-main factors in subject identification.

[0043] The second layer will control vectors The attribute slots are aligned with the corresponding fields of the region. Gated multiplication is used to apply gain to consistent attributes and suppress conflicting attributes, maintaining decoupling from the first layer and only answering whether the attributes are consistent.

[0044] The third layer will control vectors The orientation and relationship slots are mapped to the geometry and interaction channels. The absolute orientation and relative relationship are verified only in the local neighborhood of the current anchor point. Neighborhood queries are triggered to confirm the mutual pointing of the action and the receiver, and to answer whether the position and interaction are valid.

[0045] The three-layer output is generated uniformly with sub-scores and a total score, employing a single-time auditable fusion method: first, semantic backbone score, attribute consistency score, and orientation relationship score are generated, and then fused within the same link to obtain the final compatibility score. Each component is linked to its source slot at the field level, retaining the keys and indexes used for alignment. This allows subsequent threshold determination and jump calibration to directly read and replay the decisions made in this step, fully recording the field sources, fusion parameters, and boundary conditions of the three-layer scoring in this step. This ensures that subsequent processes (such as threshold determination and calibration) can trace "how the score was obtained" and reproduce the decision-making process, transforming the alignment logic from a "black box" to "transparent and verifiable," providing a basis for result auditing and error tracing. Furthermore, to avoid interpretive ambiguity caused by global attention, all calculations are restricted to anchor pairs and their necessary neighborhoods, without looking back at irrelevant areas or introducing historical weights into the current jump decision.

[0046] Optional fusion formula (used to generate final compatibility in one step):

[0047] in: For final compatibility; For text semantic vectors; For region semantic vectors; This is channel-by-channel multiplication; , , Three-layer fusion weights; , These are the control sub-vectors for attribute and orientation relationship, respectively; , For the region's attribute fields and geometry / relationship fields; , This is a function for verifying the gating function and its neighborhood relationship.

[0048] The above formula expresses the technical route of this step in the order of trunk, attribute, and relationship, and ensures interpretable alignment in multi-entity and complex action scenarios through field-level gating and local verification.

[0049] S44. Generate master-slave thresholds from the sub-scores output by S43 in a role-sensitive manner, and complete the traceability determination and storage in a segmented process.

[0050] The first section maps sub-target roles to the main dimension: the subject corresponds to the semantic core, the attributes correspond to the same attributes, and the actions and recipients correspond to the directional relationship; for non-main dimensions, a threshold is set and bound to the sub-target number, forming a configuration table of threshold groups and corresponding entries.

[0051] The second step performs a primary-secondary judgment on each anchor point: if the primary dimension does not meet the threshold, it is directly rejected and the rejection field is recorded; if the primary dimension passes, the secondary dimensions are checked in turn. If the primary dimension is slightly insufficient, it is marked as restricted and retained with the trigger condition. If a clear conflict occurs, it is immediately removed and the conflict source key is recorded.

[0052] The third stage will save the text position, region index, sub-score, threshold configuration, judgment status, and triggering conditions by writing the evidence set of the current jump with the restricted entries, so that subsequent consistency calibration can directly read and close the evidence chain according to the conditions:

[0053] in: For the sub-target role; , , These are respectively: semantic core score, attribute consistency score, and location relationship score; The sensitivity coefficient for character integration; The main dimension is divided according to the role; Divide into any dimension; , Master / slave thresholds are bound to the character; This serves as a reference value for the role-sensitive threshold used for recording. This is the output result for the three-state determination.

[0054] The above formula presents the rules in the order of primary dimension priority and secondary dimension verification, and provides explicit condition entry points for restricted retention.

[0055] Write the three-state determination output along with the field-level reasons into the evidence entry: the main field key and corresponding score for rejected sample records that did not reach the threshold; the secondary field key and triggering conditions for insufficient limited retention records (such as the need to confirm the relative orientation with the recipient); and all field keys and sub-items for passed sample records.

[0056] The evidence entries are output as a structured list, with each entry referenced one-to-one with the anchor point unique identifier, text location, and region index, for the next step of consistency calibration to perform geometric and semantic transfer verification based on the connection relationship.

[0057] S45. Summarize the valid evidence for this hop in parallel within the batch, deduplicate it by the anchor point's unique identifier, and maintain the reference relationship with the index table. Attach a hop number and timestamp to each piece of evidence, and output a structured list to the shared buffer. Save the dependency tags and restricted retention states from the list to the entries so that subsequent cross-hop consistency calibration can be conditionally triggered for reading, ensuring that the results of fine-grained hop-by-hop alignment can be directly used for chain evidence accumulation and traceable display.

[0058] S5. Based on the sub-target dependency, perform geometric and semantic consistency calibration in the local neighborhood, and use the merging rule of prioritizing necessary conditions and supplementing with sufficient conditions to generate calibrated evidence. Step S5 includes the following sub-steps: S51. Parse the dependencies between adjacent sub-targets in the sub-target chain into directed relationships: category, master, owner, active, passive, master, scene. The evidence set of the previous hop and the candidate set of the next hop are used as the two ends for pairing. The anchor point is uniquely identified. With text position Establish connection keys and generate a candidate connection table, where each connection is limited to the local neighborhood of its corresponding anchor point for calibration to avoid introducing irrelevant regions. Sort the connection table by relation type to provide clear object pairs and scopes for subsequent geometric and semantic verification.

[0059] S52. Extract the geometric fields of paired regions from the candidate join table. Relationship hints from control vector The system reads data from the data and compares the absolute orientation relationships with normalized coordinate ranges. For relative relationships, it verifies the center distance between the two bounding boxes, the occlusion ratio, and the orientation indicator. Verbs involving interaction trigger neighborhood constraints, searching and verifying proximity or opposing signs only within the local area of ​​the subject and receiver. Connections that do not meet the above geometric constraints are marked as geometrically inconsistent, and the conflict source key is retained for subsequent merging and determination.

[0060] S53. Read the categories and attribute constraints determined in the previous hop from the evidence entries as an inheritance set, and then apply this set to the fields of the next hop candidate. Perform a field-by-field check based on color, shape, material, category, and quantity. Fields that must be inherited are set to strong consistency, and any conflicts are eliminated. Descendants that can be weakened are set as subordinate attributes, and any deficiencies are marked as requiring supplementary verification, with the triggering conditions and missing fields recorded. The consistency check results are written back to the join table and coexist with the geometric inconsistency markers to ensure that subsequent rule merging has a complete basis.

[0061] S54. Assign the join table to the table by relation type: primary, secondary, active, passive, primary. Grouping and building a rule-priority merger.

[0062] The geometric validation result and semantic attribute validation result of each connection are used as two inputs. First, a necessary condition judgment is performed: geometric necessary condition is used for subject → action and action → recipient, attribute necessary condition is used for subject → attribute, and category consistency necessary condition is used for subject → scene. Only entries that pass the necessary condition are then evaluated for sufficient condition, ensuring that the prerequisite for the relationship is not affected by attribute details.

[0063] The sufficient conditions are set as a set of attributes, and the auxiliary descriptions of color, shape, material, quantity, and orientation are verified in sequence. Entries without conflicts but with deficiencies are retained with restrictions, and the fields to be verified and the triggering conditions are registered in the connecting entries, pointing to subsequent jumps where the relevant sub-targets provide supplementary verification. Entries with strong conflicts (such as color conflicts or opposite relative orientations) are directly marked as conflicting and removed, and do not enter the supplementary verification process.

[0064] The merged output is written to the calibration decision, using a three-state structure with a source key (geometric or attribute), allowing the audit to point back to the triggering rule and corresponding field. The rule-based merge formula is as follows:

[0065] in: For three-state determination; It is a relation type; This is the vector of geometric verification results; This is a vector of attribute validation results; This is a necessary condition function that switches according to the relation type (geometry / attribute / category consistency); For a sufficient set of condition fields (color, shape, material, quantity, and orientation auxiliary); A set of strongly conflicting fields; For the first Consistency indicators for fields.

[0066] Based on the above formula, the judgment logic is solidified in the order of necessary prerequisites to sufficient supplementation, and the limited and retained trigger conditions and insufficient fields are explicitly exposed to facilitate subsequent jump-oriented supplementary verification and backtracking.

[0067] S55. Write the retained and restricted retention entries into the calibrated evidence set, and retain the anchor point identifier. Text position Regional Index The relationship category and judgment status are defined; for restricted entries, additional trigger conditions and fields to be verified are retained, clarifying the direction to be verified in the next hop. The removal reason key is recorded for entries to be removed to prevent duplicate calibration. The calibrated evidence set is merged with the original evidence set to generate a cross-hop evidence set, which is used in the next stage for feature fusion and traceable output based on anchor points and relationship references, maintaining the continuous accumulation of chain evidence.

[0068] S6. Aggregate evidence by sub-target and weight pool it to obtain shared vectors and traceable meta-information; Step S6 includes the following sub-steps: S61. Aggregate the calibrated and retained jump evidence into a set according to the sub-target number. And read the role type of each sub-target. With priority sequence Map the role sensitivity weight table to sub-target weights. This is used to control the contribution ratio of the subject, attribute, action, recipient, and scene in the final representation. It incorporates the textual features of evidence entries. With visual features Uniquely identified by anchor point Pairing is used as a unit to ensure that the same evidence participates in subsequent calculations only as a single aggregation unit, avoiding duplication and semantic confusion.

[0069] S62. Construct a weighted concatenation vector sum for each piece of evidence. Among them, For compatibility, multiple pieces of evidence under the same sub-target are obtained through weighted pooling. The weight is determined by the overall and The anchor points are jointly determined and normalized to avoid strong semantics or high priority dominance by one side. Anchor point identification is performed during the pooling process. Text position Regional Index The sub-score pointers are written together into the metadata index, supporting [the following]: It directly traces back to specific words, specific regions, and corresponding judgment criteria.

[0070] S63. Pooling results of each sub-target Consolidate and lightweightly merge sub-target chains in order to generate a shared vector. As a unified representation across modalities; parallel output structured subchannels Record the list of valid evidence and anchor point identifiers for each sub-target. Sub-objective weights Overall compatibility With triggering conditions. and Binding them to the same record allows for direct reading from retrieval, location, or question-and-answer task headers, enabling synchronous delivery and traceable retrieval of overall similar expressions and fine-grained alignment information.

[0071] S7. Input the shared vector into the task header to output the retrieval / location / question answering results, and simultaneously generate an evidence list and audit path.

[0072] Step S7 includes the following sub-steps: S71, Shared Vectors Input the data into the corresponding task header, select the reading rule according to the task type, and generate the main result. For the retrieval task, ... The target image is ranked based on similarity with candidate representations in the database and an identifier is output; for localization tasks, Region-related components and candidate region index Align and output target region identifiers; for question-and-answer tasks, Compare the results with the answer candidate list and output the answer identifier. Include the task header output with the sub-target number and anchor point identifier. Establish references that can point back to the set of sub-goals that participated in the contribution.

[0073] S72. Transcribe the valid evidence for each sub-target into a ternary record with the sub-target number. ,in For anchor point identification, Indexed by text location. Append records with compatibility summary. The system determines the state and triggering conditions to form a structured evidence list. This list is then organized into a visual path according to the sub-target chain order and relational direction, sequentially presenting the alignment relationships of the subject, attributes, actions, recipients, or scenes. This is output synchronously with the main result, allowing users to view the anchoring method and cross-verification process hop by hop.

[0074] S73. Bind the main result and the evidence list to the same auditable entry, providing indexes by sub-target number and region. Text position The system offers filtering and redirection capabilities. It provides explicit prompts for evidence marked with restricted retention, allows users to view the fields to be verified and triggering conditions on the interface, and identifies them by anchor points. Return to the original candidate. Export the current session as a structured report, including the main result, evidence path, ternary record, and key judgment criteria, supporting review, retention, and reuse in retrieval, location, and question-answering scenarios.

[0075] Example 1: A content platform deployed this invention in a product trial and evaluation scenario to solve the problem of inaccurate and uninterpretable alignment of text and images in descriptions involving multiple entities and complex actions. Traditional solutions use one-time global attention matching, which often confuses subject-attribute-action-receptor. For example, if a blue glass contains a gold spoon, the spoon handle pointing towards the plate is prone to mismatch or makes it impossible to trace the cause.

[0076] The system of this invention first parses the evaluation text into words / phrases / sentences according to S1, annotating the subject, attributes, actions, recipients, and scenes, and constructing a word-phrase-sentence hierarchical index. On the image side, it generates two multi-scale features in parallel: a grid and the target, attaching color histograms, contours, and normalized coordinates, and solidifying alignment index tables for color → color field, shape → contour, orientation → geometry, and action → relationship. Then, according to S2, it assembles the subject-attribute-action-recipient-scene sub-target chain, injecting explicit key-value constraints and dependency pointers such as category, color, shape, orientation, and interaction to form a control signal set.

[0077] In stage S3, the system recalls candidate regions for each sub-target based on semantic nearest neighbors, performs field-level attribute verification and orientation prior fusion, filters out supra-threshold anchor points and writes them into the cross-modal index (recording semantic backbone score, attribute consistency score, orientation relationship score and source branch). In stage S4, three-layer decoupled scoring is performed only locally at the anchor points: first, the main semantics are identified, then color / shape consistency is verified by gating, and finally, absolute / relative orientation and interaction relationship are verified in the neighborhood; combined with the role sensitivity threshold, three-state evidence of pass / restricted retention / rejection is output, retaining field-level reasons and triggering conditions.

[0078] To ensure consistency across entities and action chains, S5 performs consistency calibration in the anchor point neighborhood based on sub-target dependencies: geometric necessity conditions are applied for agent → action and action → recipient, and attribute necessity conditions are applied for agent → attribute. Sufficient conditions (color, shape, material, quantity, and orientation auxiliary) are then verified after passing these conditions. Insufficient samples are recorded as fields to be supplemented and retained with restrictions; conflicting samples are recorded with conflicting keys and removed. S6 weights and pools the retained evidence according to sub-targets to obtain shared vectors and traceable metadata (anchor ID, text location, region index, and sub-pointers). S7 reads the shared vectors from the retrieval / location / question-answering task header, outputs the main result, and simultaneously generates an evidence list and a visualization path, supporting hop-by-hop playback and field-level auditing.

[0079] On three test sets of tableware plating and matching tools and accessories installation, compared with the global alignment baseline, the present invention improved word-region alignment mAP by 11.8%, relationship recognition F1 by 14.2%, and mismatch rate of samples with multiple receptors by 37.5%. The audit interface can directly locate the source of conflict between color key, shape key, and orientation key, significantly shortening the quality inspection backtracking time and improving online search click satisfaction.

[0080] Specific examples: 1. Input: 1) Input text: A red cup sits on a white table, with a silver spoon stuck inside, the spoon's handle pointing towards the plate. 2) Input as follows Figure 5 Image shown: The image contains four visual targets: Target 1: Red cylindrical cup (coordinates: [0.3, 0.2, 0.4, 0.5], color histogram) Outline description = "cylindrical") Objective 2: White rectangular table (coordinates: [0.1, 0.6, 0.8, 0.8], color histogram) Outline description = "rectangle") Target 3: Silver long-handled spoon (coordinates: [0.32, 0.15, 0.38, 0.4], color histogram) Outline description = "long-handled spoon shape") Target 4: White circular plate (coordinates: [0.5, 0.2, 0.6, 0.4], color histogram) Outline description = "circle") 2. Process S1: Text parsing, visual feature extraction, and index building S11, Text Parsing Word segmentation and vector sequence: The text is split into a word-level sequence (red / cup / placed / on / white / table / , / cup / with / silver / spoon / inserted / , / spoon / handle / facing / plate), and each word corresponds to a vector (e.g., red: ,cup: ), location index (red), (cup),…, (plate).

[0081] Type marker ( ):red( =color), cup ( =Entity), placed in ( =Action), white ( =color), table ( =physical), inserted ( =Action), Silver ( =color), spoon ( =Entity), Orientation ( =Direction), plate ( =Entity).

[0082] Phrase segmentation and semantic roles ( ):phrase ="Red Cup" =Main body) = "Place it on the white table" = Action + Receptor, where the action is "placed" and the receptor is "white table". =“A silver spoon is stuck in the cup” =Subject "cup" + Action "inserted" + Recipient "silver spoon") ="Spoon handle facing the plate" =Subject "spoon handle" + Action "direction" + Recipient "plate") ="white table" =Scene).

[0083] Text feature groups : Connecting words-phrases-sentence features, preserved Information, such as Include( ), ( )wait.

[0084] S12, Visual Feature Extraction Grid feature: Divide the image into a 10×10 grid (K=100) and extract each grid region. Features Such as a grid containing a cup , Color histogram ,contour ="Contains cylindrical targets", normalized coordinates .

[0085] Target features: Extract features from four targets. ( , , , ), additional ( = ), ( = ), ( ).

[0086] Visual feature group Merge mesh and target features, and label the source. (like of , of ).

[0087] S13, Alignment Index Table A according to Mapping field key ,like: red( → (Color histogram) ; cup( → (Semantic features) ; Place in ( → (relation), ; Orientation ( → (coordinate), .

[0088] Example of index table A:

[0089] S2: Sub-target chain construction and constraint injection S21, Semantic Sequence Classification Categorization sequence based on sentence order : = (Subject: Cup, Attribute: Red, Action: Placed, Recipient: White Table, Scene: Table) =(Subject: cup, Action: inserted, Recipient: silver spoon) =(Subject: Spoon handle, Action: Orientation, Recipient: Plate) S22, Sub-target Chain

[0090] Assembled in the order of "Subject → Attribute → Action → Recipient → Scene", dependent on pointers. Example:

[0091] S23, Explicit Constraint Injection Inject constraint sets into sub-objectives ,like: Sub-objective 1 (main object "cup"): ="container", =“The only cup”; Sub-target 2 (attribute "red"): =“RGB(255,0,0)”; Sub-target 3 (action "place"): =“The subject is above the receptor, without obstruction”; Sub-target 8 (Action "Orientation"): =“Subject coordinates point to receptor coordinates”.

[0092] S24, Control Signal Set

[0093] Generate execution signals, such as =3 (action "place") Include: = ( =3, =“Placement action” =“The subject is a cup, and the recipient is a table” = Dependency edge .

[0094] S3: Candidate Recall and Anchor Point Generation Sub-target 1 (the main body "cup") =Main body, For example, using "cup" as an example: S31, Semantic Nearest Neighbor Recall Calculate the word vector of "cup" Semantic similarity (cosine distance) with all visual regions (4 targets + 100 grids): Target 1 (cup): =0.92 (highest); Target 2 (Table): =0.35; Target 3 (Spoon): =0.28; Target 4 (Plate): =0.21; Grid (Including cup): =0.85; Candidate list record ( =“cup” =Target 1, =0.92, , Target 1 index), ( =“cup” =Grid 35, =0.85, , =35) and the top 20 highly similar entries.

[0095] S32, Attribute Verification Sub-target 2 (attribute "red") is associated with subject "cup", and the color matching score of the candidate region is calculated. : Objective 1: Color Histogram Match with "red" 0.95 → =0.95; Grid 35: Match with "red" 0.88 → =0.88; Target 2 (white): =0.12 (conflict, discard); Reserved entry: (Target 1, =0.92, =0.95), (grid 35, =0.85, =0.88).

[0096] S33, Orientation Prior Fusion The orientation cue for sub-target 4 (receptor "white table") is "up". The relative orientation of the candidate region to the table is calculated. : Target 1 (cup) coordinates [0.3, 0.2, 0.4, 0.5], table coordinates [0.1, 0.6, 0.8, 0.8], the cup is above the table → =0.90; The overlap between grid coordinate 35 and the table is low → =0.65; Overall score calculation (preset weights: semantic 0.5, attribute 0.3, location 0.2): Objective 1: =0.92×0.5+0.95×0.3+0.90×0.2=0.921; Grid 35: =0.85×0.5+0.88×0.3+0.65×0.2=0.809; S34, Anchor Point Generation Set threshold =0.85, select target 1 as the anchor point, and generate triples: , , , , ,

[0097] Write to index table A and assign anchor identifiers. , , = (entity).

[0098] S4: Local Three-Layer Scoring and Role Sensitivity Threshold Judgment Continuing with the anchor point of sub-target 1 (the main body "cup") For example: S41, Ternary Input Construction Text fragment representation = (Cup vector), multi-granularity representation of the region ( =Target 1)= Sub-target description =“Main body: cup”, Triple input= , "Main body: Cup" .

[0099] S42, Control Vector Mapping Read constraint set Control vector of the "cup" =[ ="Container" (slot 1), ="Unique" (slot 2), 0 (other slots)], with The semantics of "container" Establish alignment pointers for the "cup" category field.

[0100] S43, Three-layer decoupling score First layer (semantic backbone): (Channel-by-channel multiplication) followed by weights Dot product → (Answer "Is it a cup?": Yes); Second layer (same attributes): The "red" slot and of Matching, Gating Function Enhance consistent attributes → =0.94 (Answer "Is it red?", yes); Third layer (positional relationship): The "on the table" slot and , (Table) coordinate comparison → =0.91 (Answer: "Is it on the table?", Yes); Final compatibility (The default weights are all 1).

[0101] S44, Role Sensitivity Threshold Judgment Main character =Main body, main dimension is semantic backbone, main threshold From the threshold (Attribute / Direction) = 0.7: (Main dimension passed) =0.94≥0.7、 =0.91≥0.7 (Passed through dimension) → Decision status = "Passed"; Evidence items: , , =Target 1, =0.93, =0.94, =0.91, Status = Passed .

[0102] S5: Jump Consistency Calibration Taking the connection between sub-target 3 (action "place", depending on sub-target 1 "cup") and sub-target 4 (receptor "white table") as an example: S51, Relationship Analysis and Pairing Relationship type =“Active→Receiving” (cup→placed→on the table), the evidence for the previous jump is: (Cup), next jump candidate is target 2 (table), generate a join table: Objective 2 =Active→Received .

[0103] S52, Geometric Constraint Verification Cup coordinates [0.3, 0.2, 0.4, 0.5], table coordinates [0.1, 0.6, 0.8, 0.8], calculated center distance = 0.4 (meets "above"), occlusion ratio = 0 → geometric verification result = "Consistent", no conflict.

[0104] S53, Semantic Attribute Validation Previous hop inheritance set: Cup category = "container", next hop candidate table category = "furniture", no conflict; table color = "white" matches the text "white table" → semantic verification result = "consistent".

[0105] S54, Rule Priority Merging Necessary conditions ( =Active → Received, geometric consistency required): =“Consistent” → Passing the necessary conditions; Sufficient conditions (color, category): =“Consistent” → No deficiencies → Determine the status = “Retain”.

[0106] S55, Post-calibration evidence Post-calibration evidence entries: , , =Active → Received, State = Retained, Geometric Consistency = Yes, Attribute Consistency = Yes .

[0107] S6: Evidence Aggregation and Shared Vector Generation S61, Evidence Aggregation Evidence is aggregated by sub-objective number, with sub-objective 1 (the main subject "cup") having the following weight. ( =Main body, )=0.4 (highest priority), sub-target 3 (action "placed") =0.3, Sub-target 4 (receptor "table") =0.2, Sub-target 2 (attribute "red") =0.1.

[0108] S62, Weighted Pooling Evidence vector of sub-target 1 Sub-target 3 After pooling: (Normalized weights).

[0109] S63, Shared Vector splicing all sub-targets ,generate Secondary passage Record: .

[0110] S7: Result Output and Audit Path Generation S71, Task Results Location task: Align the region-related components with the target 1 index → ​​Output "Cup location: Target 1 (coordinates [0.3, 0.2, 0.4, 0.5])"; Search task: Compare with candidate representations in the image database. The target image (containing a red cup and a white table) has the highest similarity, and the image identifier is output.

[0111] S72, List of Evidence Triple record: , , ; Visual path: Main subject "cup" ( → Attribute "Red" → The action "placed" ( → Receptor "White Table" ( → Action “interspersed” ( → Receptor "Silver Spoon" ( → Action "towards" ( → Receptor "plate" ( ).

[0112] S73 Auditable Report The report includes: location results (cup is in target 1), evidence path (jump-by-jump alignment), key judgment criteria (such as...). (Through semantic validation), clickable Backtrack the original candidates (target 1, grid 35) and score details.

[0113] Through the above process, the text "red cup" was successfully and accurately aligned with image target 1 (red cup), "white table" with target 2 (white table), and "silver spoon" with target 3 (silver spoon). The judgment status of each sub-target was "passed", and each alignment step was supported by data (score, coordinates, matching degree) and was interpretable.

[0114] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for bidirectional alignment of text and images based on multi-hop parallel inference, characterized in that, Includes the following steps: S1. Parse the text for words / phrases / sentences and label the subject, attributes, actions, recipients, and scenes to obtain multi-granularity text feature sets. The image generation grid and target outputs are merged into a multi-scale visual feature set. And create an alignment index table from text fields to visual fields; S2. Construct a sub-target chain based on the two multi-scale features of the grid and the target, record the position and dependency of the sub-targets in the sub-target chain, and inject explicit constraints into the subject, attributes, actions, receptors and scenes; S3. Use the alignment index table to recall candidate regions of semantic nearest neighbors. Based on the sub-target dependencies and explicit constraints, perform field-level attribute verification and orientation prior fusion on the candidate regions, generate anchor points and write them to the cross-modal index. S4. Based on the generated anchor point geometry and the updated cross-modal index, perform three-layer decoupled scoring at the local anchor point, and output three-state evidence by judging through the role sensitivity threshold. S5. Using three-state evidence combined with sub-target chains and explicit constraint geometry, geometric and semantic consistency calibration is performed in the neighborhood of the anchor point according to the dependency relationship to obtain the calibrated evidence set; the merging rule of necessary condition priority and sufficient condition supplementary evidence is adopted to generate calibrated jump evidence. S6. Weight the calibrated evidence set according to the priority and dependency of the sub-target chain, and generate a shared vector and traceable meta-information; S7. Input the shared vector and traceable metadata into the task header, output the task results, and simultaneously generate an evidence list and auditable path based on the traceable metadata.

2. The text and image bidirectional alignment method based on multi-hop parallel inference according to claim 1, characterized in that, S1 includes the following steps: S11. Parse the input text by word, phrase, and sentence, and build a vector sequence. With location index Includes text type markers, color tags, action location entities. ; Segmenting sentence dependencies into phrase fragments And assign the character's main attributes, actions, and the scene. The feature sequence from word to phrase to sentence is concatenated into a multi-granularity text feature set. Explicitly reserved To support subsequent sub-target retrieval; S12. Decompose the image into two branches: grid and target. Generate a set of regions for the grid. Extracting regional features Includes a color histogram. Outline description Normalized coordinates Generate an object set for candidate targets. Extract semantic features , and attached The two outputs are merged into a multi-scale visual feature set. And record the source marker and spatial index; S13. Establish a mapping between text types and visual fields and solidify it into an alignment index table. Record a quadruple for each text item. According to Select field key (color →) Action → rel, Location → Entity → ), and provide the source grid target. Align the index table Binding to multi-granularity text feature groups With multi-scale visual feature groups In the description.

3. The text and image bidirectional alignment method based on multi-hop parallel inference according to claim 1, characterized in that, S2 includes the following steps: S21. Group multi-granularity text features The words and phrases in the text are traversed sequentially, and entity attributes, actions, and scenarios are marked by type. Categorize and index by position Bind to the original text position for each phrase / segment Determining semantic roles based on sentence dependency relations Within the same sentence range, records them as a sequence according to their order of appearance. Set cross-sentence boundaries as non-mergeable constraints; S22, Semantic sequence Sentences are assembled sequentially into sub-target chains. It adopts a fixed order of subject, attribute, action, recipient, and scene, and records the minimum necessary field for each sub-target: text location. Literal content Character Type and the dependency pointers to the previous sub-target. The missing subject's actions and the receptor are marked as a state to be completed; S23. Inject each sub-objective in the sub-objective chain into an executable set of explicit constraints. Fill in the subject and receptor category key class Same as the same key Fill the attribute with key color Or shape key To fill in the interactive prompts for actions Fill the scene with directional key bindings Or background key scene Match each key with the text position Bind to field-level key-value descriptions to form a ternary index of key, value, and position; S24, Sub-target chain Organize into a structured sequence and assign an execution sequence number. Target category hints Attribute filtering conditions with position reference Assembled into a control signal set ; Explicitly define cross-subtarget dependencies as edges This record is used to trigger subsequent association verification and local search scope limitation. As a direct input to the candidate search and filtering strategy, visual regions that are irrelevant to the current sub-target are eliminated.

4. The text and image bidirectional alignment method based on multi-hop parallel inference according to claim 1, characterized in that, S3 includes the following steps: S31. Transfer the text fragment of the current sub-target. With multi-scale visual feature groups Pair all regions one by one to generate candidate pair sequences. ,in, For each candidate pair, the main semantic comparison is performed using the cosine similarity between the word vector and the region semantic vector, covering both the grid and the target region, to obtain semantic segmentation. , will righteous Along with text position index With regional spatial index These are all included in the candidate list to cover potential alignment regions using a semantic nearest neighbor approach, providing a searchable base set for the overlay of attributes and orientation constraints. S32. Calculate attribute matching for each candidate in the candidate list. Color terms are mapped to color fields, shape terms to outline fields, and category hints to subcategory fields. Field-level consistency checks are performed, and entries lacking required fields or exhibiting obvious conflicts are directly removed, retaining only those that pass the checks. and genus Write the entry identifier as a parallel record, and specify the text position. Maintain binding with field keys; S33, Calculate the prior quantile of the retained entries. For absolute directional terms, compare with normalized coordinate fields; for relative directional terms, check the relative layout of paired regions; and define the meaning. , belonging to ,Bit The comprehensive score is obtained by weighting and merging the results using preset weights. Perform a full sort on all entries, and mark the samples that rank high in the overall sort but have conflicting attributes or positions with conflict flags. S34. Fix the first few combinations in the upper-threshold interval of the comprehensive ranking result as pairs to generate a triplet semantic attribute summary. And write it into the cross-modal index table. Assign a unique identifier to each anchor point. Preserve references to source branches and field keys.

5. The text-image bidirectional alignment method based on multi-hop parallel inference according to claim 1, characterized in that, S4 includes the following steps: S41. The cross-modal index table for the current jump... Perform batch processing, reading the text position of anchor pairs based on unique identifiers. With regional index Extract text fragments for each anchor pair. Multi-granularity representation of regions and the description of the jump target. Sequentially concatenate the data to form a ternary input. ; S42. Read the explicit constraints of the jump target from the key-value table as a control vector. Mapping to ternary inputs by field alignment; assigning separate fields to color, shape, category, orientation, and interactive prompts. Independent slots, and with , Establish alignment pointers for the corresponding fields, and... By adding an independent channel to the ternary input, we obtain ; S43, will Perform interpretable calculations and limit the calculations to a local range of anchor points, performing layer-by-layer calculations to obtain the sub-item scores and total scores of the three-layer output; S44. Generate master-slave thresholds by assigning the output sub-scores to the roles in a sensitive manner, and complete the traceability determination and database entry in a segmented process. Write the three-state results and field-level reasons into the evidence entries. S45. Summarize the valid evidence for this jump in parallel within the batch, remove duplicates by anchor point unique identifier, and maintain the reference relationship with the index table. Add jump number and timestamp to each piece of evidence, output as a structured list to the shared buffer, and retain the dependency tags and restricted retention status in the list to the entries.

6. The text-image bidirectional alignment method based on multi-hop parallel inference according to claim 5, characterized in that, The output scores are used to generate master-slave thresholds based on role sensitivity, and a segmented process is used to complete traceability determination and data entry, including: The first paragraph maps the sub-target roles to the main dimension: the subject corresponds to the semantic core, the attributes correspond to the consistent attributes, and the actions and the recipients correspond to the directional relationship; for the non-main dimensions, a threshold is set and bound to the sub-target number, forming a configuration table of threshold groups and corresponding entries; The second section performs a judgment on each anchor point in the order of primary and secondary dimensions: if the primary dimension does not meet the threshold, it is directly rejected and the rejection field is recorded; if the primary dimension passes, the secondary dimensions are checked in turn. If the primary dimension is slightly insufficient, it is marked as restricted and retained with the trigger condition. If there is a clear conflict, it is immediately removed and the conflict source key is recorded. The third stage will save the text position, region index, sub-score, threshold configuration, judgment status, and triggering conditions by writing the evidence set of the current jump with the restricted entries, so that subsequent consistency calibration can directly read and close the evidence chain according to the conditions: in: For the sub-target role; , , These are respectively: semantic core score, attribute consistency score, and location relationship score; The sensitivity coefficient for character integration; The main dimension is divided according to the role; Divide into any dimension; , Master / slave thresholds are assigned to the character. This serves as a reference value for the role-sensitive threshold used for recording. Output is a three-state determination; The above formula presents the rules in the order of primary dimension priority and secondary dimension verification, and provides explicit condition entry for restricted retention. The three-state output results and field-level reasons are written into the evidence entries: the primary field key and corresponding score for the rejected sample record that did not reach the threshold; the secondary field key and triggering condition for the insufficient restricted retention records; and all field keys and sub-items of the passed sample records. The evidence set is output as a structured list, and it is referenced one by one with the anchor unique identifier, text position and area index.

7. The text and image bidirectional alignment method based on multi-hop parallel inference according to claim 1, characterized in that, S5 includes the following steps: S51. Parse the dependencies between adjacent sub-targets in the sub-target chain into directed relationships: category, master, owner, active, passive, master, scene. The anchor point is uniquely identified by pairing the previous hop evidence set with the next hop candidate set as the two ends. With text position Establish connection keys, generate a candidate connection table, where each connection is limited to the local neighborhood of the corresponding anchor point for calibration, and sort the connection table by relation category; S52. Extract the geometric fields of paired regions from the candidate join table. Relationship hints from control vector The system reads data from the middle, compares the absolute orientation relationship with the normalized coordinate range, verifies the center distance between the two boxes, the occlusion ratio and the orientation indicator for the relative relationship, triggers neighborhood restrictions for verbs involving interaction, searches and verifies proximity or opposite signs only within the local range of the subject and the receiver, marks connections that do not meet the above geometric constraints as geometrically inconsistent, and retains the conflict source key. S53. Read the categories and attribute constraints determined in the previous hop from the evidence entries as an inheritance set, and then apply this set to the fields of the next hop candidate. The fields are checked one by one according to color, shape, material, category and quantity. Fields that must be inherited are set as strong consistency and are removed if there is a conflict. Descendable descriptions are set as subordinate attributes. If there is a deficiency, it is marked as needing to be supplemented and the triggering conditions and missing fields are registered. The consistency check results are written back to the join table and coexist with the geometric inconsistency mark to ensure that the subsequent rule merging has a complete basis. S54. Assign the join table to the table by relation type: primary, secondary, active, passive, primary. Grouping, constructing a rule-priority merger, solidifying the judgment logic based on the order of necessary prerequisites to sufficient supplementation, explicitly exposing the restricted and retained trigger conditions and insufficient fields, and obtaining a three-state judgment result; S55. Write the retained and restricted retention entries from the three-state determination results into the calibrated evidence set, and retain the anchor point identifiers. Text position Regional Index Relationship type and determination status; For restricted retention with attached trigger conditions and fields to be verified, clarify the direction to be verified in the next hop, record the removal reason key for the removed entries, merge the calibrated evidence set with the original evidence set, and generate a cross-hop evidence set.

8. The text and image bidirectional alignment method based on multi-hop parallel inference according to claim 1, characterized in that, S6 includes the following steps: S61. Aggregate the calibrated and retained jump evidence into a set according to the sub-target number. And read the role type of each sub-target. With priority sequence Map the role sensitivity weight table to sub-target weights It is used to control the contribution ratio of subject, attribute, action, receptor, and scene in the final representation, and to transform the textual features of evidence entries. With visual features Uniquely identified by anchor point Pairing into units; S62. Construct a weighted concatenation vector sum for each piece of evidence. Among them, To ensure compatibility, multiple pieces of evidence under the same sub-target are obtained through weighted pooling. The weight is determined by the overall and To jointly determine and unify the anchor points and avoid strong semantics or high priority dominance by one side, the anchor points will be identified during the pooling process. Text position Regional Index The sub-score pointers are written into the metadata index; S63. Pooling results of each sub-target Consolidate and lightweightly merge sub-target chains in order to generate a shared vector. As a unified representation across modalities; parallel output structured subchannels Record the list of valid evidence and anchor point identifiers for each sub-target. Sub-objective weights Overall compatibility With the triggering conditions, and They are bound to the same record for direct reading by retrieval, location, or question-and-answer task headers.

9. The text and image bidirectional alignment method based on multi-hop parallel inference according to claim 1, characterized in that, S7 includes the following steps: S71. Input the shared vector into the corresponding task header, select the reading rule according to the task type, and generate the target image identifier, target region identifier, and answer identifier. Then, combine the target image identifier, target region identifier, and answer identifier output from the task header with the sub-target number and anchor point identifier. Establish a reference; S72. Transcribe the valid evidence for each sub-target into a ternary record with the sub-target number. ,in For anchor point identification, For text location indexing, append records with compatibility scores. The system determines the state and triggering conditions, forming a structured evidence list. This list is then organized into a visual path according to the sub-target chain sequence and relational direction, sequentially presenting the alignment relationships of the subject, attributes, actions, and recipients or scenes. This is synchronized with the output of target image identifiers, target region identifiers, and answer identifiers. S73. Bind the target image identifier, target area identifier, answer identifier, and evidence list to the same auditable entry, and provide sub-target number and area index. Text position The system offers filtering and redirection capabilities, explicitly prompting for evidence marked with restricted retention, allowing users to view the fields to be verified and triggering conditions on the interface, and identifying them by anchor points. Returning to the original candidate, the current session is exported as a structured report, including the main result, evidence path, ternary record, and key judgment criteria.

10. A text-image bidirectional alignment system based on multi-hop parallel inference, characterized in that, It includes a preprocessing module, a target chain construction module, an anchor point generation module, a three-state evidence acquisition module, a calibration evidence generation module, a traceability information generation module, and a result generation module; The preprocessing module is used to parse the text into words / phrases / sentences and annotate the subject, attributes, actions, recipients, and scenes to obtain multi-granular text feature sets. The image generation grid and target outputs are merged into a multi-scale visual feature set. And create an alignment index table from text fields to visual fields; The target chain construction module constructs sub-target chains based on two multi-scale features: mesh and target. It records the position and dependency of sub-targets in the sub-target chains and injects explicit constraints into the subject, attributes, actions, receptors, and scene. The anchor point generation module uses the alignment index table to recall candidate regions of semantic nearest neighbors. Based on the dependencies of sub-targets and explicit constraints, it performs field-level attribute verification and orientation prior fusion on the candidate regions, generates anchor points, and writes them into the cross-modal index. The three-state evidence acquisition module performs three-layer decoupled scoring at the local anchor point based on the generated anchor point geometry and the updated cross-modal index, and outputs three-state evidence after judging by the role sensitivity threshold. The calibration evidence generation module uses three-state evidence combined with sub-target chains and explicit constraint geometry to perform geometric and semantic consistency calibration in the neighborhood of the anchor point according to the dependency relationship, and obtains the calibrated evidence set; it adopts the merging rule of necessary condition priority and sufficient condition supplementary evidence to generate calibrated jump evidence; The traceability information generation module is used to aggregate evidence by weighted pooling of the calibrated evidence set according to the priority and dependency of the sub-target chain, and generate shared vectors and traceable meta-information by weighted pooling. The results generation module is used to input shared vectors and traceable metadata into the task header, output task results, and simultaneously generate an evidence list and auditable path based on traceable metadata.

Images