A joint extraction method of multi-modal military situation data and related products

Abstract

The application discloses a kind of multi-modal military situation data joint extraction method and related products, the method includes: obtaining multi-modal military situation data;Standardization processing is carried out to multi-modal military situation data, and standardization fragment set is obtained;Standardization fragment set is aligned and is handled, and multi-modal shared representation is obtained;Based on the preset structured output syntax and preset military rule constraint graph, multi-modal shared representation is jointly decoded, and initial structured result is generated;The object result, relationship result, event result, attitude result and evidence result in initial structured result are respectively bound evidence anchor, and the structured result of binding evidence anchor is obtained;The structured result of binding evidence anchor is checked, and target structured result is obtained in the case where checking passes.Such, it can improve the structured stability of multi-modal military situation data extraction, and realize the traceability of extraction result.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method for joint extraction of multimodal military situation data and related products. Background Technology

[0002] Multimodal military situation data is widely used in intelligence processing, situation assessment, and simulation review. Efficient and accurate structured extraction of massive amounts of heterogeneous data is a crucial link in supporting command decision-making and situational awareness. However, military situation data comes from complex sources and is expressed in diverse ways. The same object may appear in different materials in various forms such as abbreviations, full names, or code names. The same event may differ in time, space, stage, and state in different modalities, posing a significant challenge to information extraction.

[0003] Currently, existing technologies mainly employ general information extraction methods based on large language models, unifying tasks such as objects, relationships, and events into a structured generation framework. However, these methods typically only apply constraints during the prompt word or post-output format checking stage, leading to issues such as missing fields, out-of-order fields, and cross-field reference errors in the output results.

[0004] Therefore, there is an urgent need for a joint extraction method for multimodal military situation data to constrain the output structure in real time during the decoding process and improve the stability of the extraction results. Summary of the Invention

[0005] This application provides a method and related products for the joint extraction of multimodal military situation data, which can improve the structured stability of multimodal military situation data extraction and achieve the traceability of extraction results, providing stable and verifiable structured data support for military situation analysis.

[0006] In a first aspect, embodiments of this application provide a method for the joint extraction of multimodal military situation data, the method comprising: Acquire multimodal military situation data, which includes target situation elements to be extracted; The multimodal military situation data is standardized to obtain a set of standardized segments; The standardized fragment set is aligned to obtain a multimodal shared representation; Based on a preset structured output syntax and a preset military rule constraint graph, the multimodal shared representation is jointly decoded to generate an initial structured result, which includes object results, relation results, event results, attitude results, and evidence results. The object result, relationship result, event result, attitude result, and evidence result in the initial structured result are respectively bound to evidence anchor points to obtain a structured result with bound evidence anchor points; The structured result of the bound evidence anchor point is verified, and if the verification is successful, the target structured result is obtained.

[0007] One feasible implementation of the method further includes: If the verification fails, the structured result of the binding evidence anchor point that failed the verification is locally re-decoded and corrected to obtain the corrected structured result; The corrected structured result is checked again. If the check passes, the corrected structured result is used as the target structured result. If the check fails, the local re-decoding correction is performed again until the check passes and the target structured result is obtained.

[0008] One feasible implementation, wherein aligning the standardized fragment set to obtain a multimodal shared representation, includes: Recall fragments related to the target situation elements from the standardized fragment set to obtain a candidate evidence fragment set; The candidate evidence fragment set is aligned to obtain the multimodal shared representation.

[0009] One feasible implementation, wherein aligning the candidate evidence fragment set to obtain the multimodal shared representation, includes: The candidate evidence fragment set is filtered and trimmed to obtain the target evidence fragment set; Feature extraction and cross-modal alignment are performed on the target evidence fragment set to obtain the multimodal shared representation.

[0010] One feasible implementation involves filtering and trimming the candidate evidence fragment set to obtain the target evidence fragment set, including: From the candidate evidence fragment set, fragments with a correlation greater than or equal to a preset correlation threshold with the target situation element are selected to obtain a preliminary set of evidence fragments; The fragments that exceed the preset context window in the initial screening evidence fragment set are trimmed to obtain the target evidence fragment set.

[0011] One feasible implementation, wherein the feature extraction and cross-modal alignment processing of the target evidence fragment set to obtain the multimodal shared representation includes: Each target evidence fragment in the target evidence fragment set is feature-encoded to obtain a feature vector corresponding to each target evidence fragment. The feature vectors corresponding to each target evidence fragment are mapped to the same representation space and fused to obtain the multimodal shared representation.

[0012] One feasible implementation involves binding the object result, relationship result, event result, attitude result, and evidence result in the initial structured result to evidence anchor points, respectively, to obtain a structured result with bound evidence anchor points, including: From the set of target evidence fragments, bind at least one target evidence fragment as an evidence anchor point to the object result, relationship result, event result, attitude result, and evidence result in the initial structured result, to obtain a structured result with bound evidence anchor points.

[0013] Secondly, embodiments of this application provide a joint extraction device for multimodal military situation data, comprising: The data acquisition module is used to acquire multimodal military situation data, which includes target situation elements to be extracted. The data standardization module is used to standardize the multimodal military situation data to obtain a set of standardized segments; The set processing module is used to align the standardized fragment set to obtain a multimodal shared representation; The result generation module is used to jointly decode the multimodal shared representation based on a preset structured output syntax and a preset military rule constraint graph to generate an initial structured result, which includes object results, relation results, event results, attitude results, and evidence results. Anchor binding module is used to bind the object result, relationship result, event result, attitude result and evidence result in the initial structured result to evidence anchors respectively, so as to obtain a structured result with bound evidence anchors; The result verification module is used to verify the structured result of the bound evidence anchor point, and obtain the target structured result if the verification is successful.

[0014] Thirdly, embodiments of this application provide an electronic device, the device including: a processor, a memory, and a system bus; The processor and the memory are connected via the system bus; The memory is used to store a program, which includes instructions that, when executed by the processor, cause the processor to perform any of the implementation steps of the above-described method for the joint extraction of multimodal military situation data.

[0015] Fourthly, embodiments of this application provide a computer-readable storage medium for storing a computer program, which, when executed by a terminal device, implements any of the implementation steps of the above-described method for jointly extracting multimodal military situation data.

[0016] As can be seen from the above technical solutions, the embodiments of this application have the following advantages: As can be seen from the above technical solution, this application provides a joint extraction method and related products for multimodal military situation data. The method includes: First, acquiring multimodal military situation data, which includes target situation elements to be extracted. Second, standardizing the multimodal military situation data to obtain a standardized fragment set, and performing feature extraction and cross-modal alignment processing on the standardized fragment set to obtain a multimodal shared representation. Subsequently, based on a preset structured output syntax and a preset military rule constraint graph, jointly decoding the multimodal shared representation to generate an initial structured result, wherein the initial structured result includes object results, relationship results, event results, attitude results, and evidence results. Finally, binding the object results, relationship results, event results, attitude results, and evidence results in the initial structured result to evidence anchors respectively to obtain a structured result bound to evidence anchors, and verifying the structured result bound to evidence anchors. If the verification passes, the target structured result is obtained.

[0017] As can be seen, this scheme, during the joint decoding process, performs joint decoding of the multimodal shared representation based on a preset structured output grammar and a preset military rule constraint graph. This ensures that the generated initial structured result is subject to the format constraints of the structured output grammar and the rule constraints of the military rule constraint graph in real time during the decoding process, thereby solving the problem of unstable output structure caused by applying constraints only at the prompt word or post-output format check stage in existing technologies. Simultaneously, the object results, relation results, event results, attitude results, and evidence results in the initial structured result are each bound to evidence anchors and verified, ensuring that each initial structured result is associated with corresponding evidence support and has a traceable evidence source, effectively improving the credibility of the extracted results. Attached Figure Description

[0018] Figure 1 A flowchart illustrating a method for jointly extracting multimodal military situational data provided in this application embodiment; Figure 2 This is a schematic diagram of the structure of a joint extraction device for multimodal military situation data provided in an embodiment of this application. Detailed Implementation

[0019] As mentioned earlier, multimodal military situational data is widely used in intelligence processing, situation assessment, and simulation debriefing. Efficient and accurate structured extraction of multi-source heterogeneous data is crucial for supporting command decisions and situational awareness. However, military situational data comes from complex sources and diverse formats. Samples often originate from news texts, official documents, webpage screenshots, reports with mixed text and images, table attachments, scanned documents, audio, video, and subtitle files. In simulation debriefing scenarios, data may also come from task logs, control seat relationships, stage status, and map grid location information exported from wargaming platforms and simulation platforms. The same object may appear in different materials using abbreviations, full names, code names, or seat names. The same event may exhibit temporal, spatial, stage, and state conflicts in different modalities. Furthermore, the same conclusion often requires support from text, images, logs, and time segments, posing significant challenges to information extraction.

[0020] Currently, existing technologies mainly fall into two categories. The first category focuses on unified information extraction and structure generation, unifying tasks such as objects, relationships, and events into a single structured generation framework. While this approach reduces the problem of inconsistent labels across tasks, it primarily targets plain text input, and structural constraints are typically only reflected in prompts or post-output format checks, lacking the ability to filter out illegal candidates in real-time during the decoding phase. When faced with long contexts, multimodal inputs, and complex referencing relationships, it is prone to issues such as missing fields, out-of-order fields, and cross-field referencing errors. The second category focuses on unified multimodal extraction and fine-grained localization, feeding text and image modalities into a unified model and providing localization results for text fragments or visual regions at output. This approach improves multimodal understanding, but its localization results are usually only considered auxiliary information, failing to incorporate the evidence itself as an independent structured result into the joint decoding objective, making it difficult to form a referential and verifiable chain of evidence.

[0021] Because structural constraints cannot take effect immediately during decoding in existing technologies, and evidence cannot be generated synchronously with conclusions, the stability and verifiability of joint extraction of multimodal military situation elements are affected.

[0022] To address the aforementioned issues, this application provides a joint extraction method and related products for multimodal military situation data. The method includes: First, acquiring multimodal military situation data, which includes target situation elements to be extracted. Second, standardizing the multimodal military situation data to obtain a standardized fragment set, and performing feature extraction and cross-modal alignment on the standardized fragment set to obtain a multimodal shared representation. Subsequently, based on a preset structured output syntax and a preset military rule constraint graph, jointly decoding the multimodal shared representation to generate an initial structured result, which includes object results, relationship results, event results, attitude results, and evidence results. Finally, binding the object results, relationship results, event results, attitude results, and evidence results in the initial structured result to evidence anchors to obtain a structured result bound to evidence anchors. The structured result bound to evidence anchors is then verified; if the verification passes, the target structured result is obtained.

[0023] As can be seen, this scheme standardizes multimodal military situational data to obtain a set of standardized fragments, and then performs feature extraction and cross-modal alignment on this set to obtain a multimodal shared representation, achieving a unified representation of multi-source heterogeneous data. During joint decoding, based on a pre-defined structured output grammar and a pre-defined military rule constraint graph, the multimodal shared representation is jointly decoded. This ensures that the generated initial structured result is subject to the format constraints of the structured output grammar and the rule constraints of the military rule constraint graph in real time during decoding, thus solving the problem of unstable output structure caused by applying constraints only at the prompt word or post-output format check stage in existing technologies. Simultaneously, the object results, relation results, event results, attitude results, and evidence results in the initial structured result are each bound to evidence anchors and verified, ensuring that each initial structured result is associated with corresponding evidence support and has a traceable evidence source, effectively improving the credibility of the extracted results.

[0024] Furthermore, this solution acquires multimodal military situational data and performs standardization, feature extraction, and cross-modal alignment to achieve a unified representation of multi-source heterogeneous data. This overcomes the limitations of existing information extraction methods, which are primarily geared towards plain text input and struggle to effectively process multimodal data. By binding object results, relationship results, event results, attitude results, and evidence results to evidence anchors, this solution ensures that the initial structured results are linked to traceable evidence sources. This overcomes the shortcomings of existing technologies where evidence location results exist only as auxiliary information and are difficult to form an effective chain of evidence.

[0025] It should be noted that the implementation subject of the joint extraction method for multimodal military situation data in this application is not limited. For example, the joint extraction method for multimodal military situation data in this application can be applied to information processing devices such as servers or terminal devices. The server can be a standalone server, a cluster server, or a cloud server. The terminal device can be an electronic device such as a smartphone, computer, personal digital assistant (PDA), or tablet computer.

[0026] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0027] Figure 1 A flowchart illustrating a method for jointly extracting multimodal military situational data, provided in an embodiment of this application. (Combined with...) Figure 1 As shown, it may include steps S101-S106.

[0028] S101: Acquire multimodal military situation data, which includes target situation elements to be extracted.

[0029] In this embodiment, the sources of multimodal military situation data are diverse, including one or more basic multi-source sample data selected from text, official documents, web pages, images, tables, scanned documents, audio, video, and subtitles. In a preferred extended scenario, simulation logs, task issuance records, and control relationship records may also be included.

[0030] It should be noted that the multimodal military situation data in this application embodiment includes target situation elements to be extracted. Target situation elements refer to five types of structured information extracted from the multimodal military situation data: objects, relationships, events, attitudes, and evidence.

[0031] Specifically, the object refers to the entity in the military situation, including combat units, military organizations, command positions, geographic grids, and mission units; the relationship refers to the association between objects, including coordination, command and control, confrontation, and pathway relationships; the event refers to the action or state change in which the object participates, including deployment events, maneuver events, strike events, mine-laying events, and mission-issuing events; the attitude refers to the position and orientation of the object towards the event or other objects, including support, opposition, hostility, incitement, panic, obedience, agreement, questioning, and neutrality; and the evidence refers to the original material location information used to support the above conclusions, including text paragraphs, image regions, video time windows, and log lines.

[0032] S102: Standardize the multimodal military situation data to obtain a set of standardized fragments.

[0033] In this embodiment of the application, since the sources of multimodal military situation data are complex and the expression methods are not uniform, it is necessary to standardize the acquired multimodal military situation data so that data from different sources and in different formats can be jointly processed and analyzed in the future.

[0034] Specifically, the acquired multimodal military situational data is standardized to obtain a set of standardized segments. Different processing methods are used for different modalities of data.

[0035] For military situation data in the form of text, official documents, and web pages, paragraph segmentation, page number retention, and paragraph numbering are performed to make each text paragraph an independent standardized segment, and to record its page number and paragraph number.

[0036] For scanned military situational data, layout detection and text recognition are performed, and the coordinates of the text boxes are preserved. Layout detection identifies text areas, image areas, and table areas in the scanned document. Text recognition then converts the text in the image into editable text content, while preserving the coordinates of each text box on the page, making each recognized text block a standardized segment.

[0037] For image-based military situational data, target region detection or region localization is performed, and the coordinates of the bounding boxes are preserved. Key target regions in the image are identified using target detection algorithms, and bounding boxes are generated for each detected target region, making each target region a standardized segment.

[0038] For tabular military situation data, the table structure is restored, preserving row and column positions and cell indices. The table's logical row and column structure is restored using a table structure recognition algorithm, recording the row number, column number, and cell index for each cell, making each table unit a standardized fragment.

[0039] For audio-based military situational data, automatic speech recognition is performed, and timestamps are retained. The audio content is converted into text using speech recognition technology, and the corresponding timestamp interval is recorded for each recognized text segment, making each speech recognition result segment a standardized segment.

[0040] For video-based military situational data, the process involves shot segmentation, keyframe extraction, and caption alignment, while preserving frame numbers or time windows. Shot segmentation divides the video into several shot units, extracts keyframes as supplementary image modalities, and aligns captions with the video timeline, making each shot segment, keyframe, or caption segment a standardized fragment.

[0041] For military situational data such as simulation logs and control records in the preferred extended scenario, the execution phases are segmented, log line numbers are assigned, control seat information is parsed, grid location information is parsed, and action names are standardized. Simulation logs are divided according to simulation phases through phase segmentation, a line number is assigned to each log line, control seat information in control relationship records is parsed, map grid location information is parsed, and action names are standardized to make each log line or control record line a standardized segment.

[0042] After the above standardization process, a set of standardized fragments is output. Each standardized fragment in the set carries at least a sample identifier, source identifier, modality type, content information, time information, spatial location information or page location information, and the original fragment index. Preferably, the standardized fragment also carries one or more of the following: stage identifier, control seat identifier, grid number, or action status identifier.

[0043] S103: Align the standardized fragment set to obtain a multimodal shared representation.

[0044] In this embodiment of the application, after obtaining the standardized fragment set, not all standardized fragments are directly sent to the subsequent decoding module. Instead, evidence candidate recall and dynamic context pruning are performed first to reduce the interference of irrelevant context on the decoding process, and then the filtered multimodal data is mapped to a unified representation space.

[0045] First, fragments related to the target situational elements are recalled from the standardized fragment set to obtain a candidate evidence fragment set. Specifically, one or more of the following are used to calculate the relevance and perform initial screening for each standardized fragment in the standardized fragment set: keyword matching score, domain dictionary hit rate, vector similarity, entity candidate overlap, temporal proximity, spatial proximity, phase consistency score, control relationship matching degree, and modal complementarity score. For example, each standardized fragment in the standardized fragment set can be scored for its relevance to the target situational elements. This relevance score is obtained by weighted summation of the above-mentioned scoring indicators. Standardized fragments with a relevance score greater than a preset recall threshold are retained as the candidate evidence fragment set.

[0046] Secondly, the candidate evidence fragment set is screened and pruned. Specifically, candidate evidence fragments with a relevance to the target situational elements greater than or equal to a preset relevance threshold are selected from the candidate evidence fragment set to obtain the initial screening evidence fragment set. At the same time, the initial screening evidence fragments that exceed a preset context window are pruned, such as by deletion, compression, or summarization, to reduce interference from irrelevant contexts on subsequent decoding. The processed fragments are then used as the target evidence fragment set.

[0047] Finally, feature extraction and cross-modal alignment are performed on the target evidence fragment set. Specifically, each target evidence fragment in the target evidence fragment set is feature-encoded into a corresponding feature vector. The feature vectors corresponding to each target evidence fragment are mapped to the same representation space and fused to obtain a multimodal shared representation. For example, in this embodiment, a cross-modal encoder can be used to directly generate a unified representation, or a text encoder, a visual encoder, a table encoder, and a log encoder can be used to encode the data separately and then aligned to the same representation space through a linear projection layer, an attention fusion layer, or a splicing mapping layer. This embodiment does not specifically limit the specific approach.

[0048] It should be noted that the multimodal shared representation in this application embodiment not only includes cross-modal semantic information, but also retains modality identifiers, page positions, regional coordinates, time order, source identifiers, and evidence anchor indexes, so as to be used for subsequent joint decoding and evidence re-indication.

[0049] S104: Based on the preset structured output syntax and the preset military rule constraint graph, perform joint decoding on the multimodal shared representation to generate initial structured results, which include object results, relation results, event results, attitude results and evidence results.

[0050] In this embodiment, the structured output syntax and the military rule constraint graph are pre-built, and they jointly constrain the output during the joint decoding process. The structured output syntax controls the field structure, field types, enumeration value range, and cross-feature reference validity of the output content. The military rule constraint graph performs military business rule-level validation on the output content, including at least one or more of the following: stage-permitted action constraints, control permission constraints, operator state transition constraints, terrain path constraints, and path access constraints.

[0051] Specifically, the structured output syntax is constructed as follows: based on the field order definition, field type definition, required field definition, optional field definition, reference relationship definition, and evidence binding definition of five elements—object, relationship, event, attitude, and evidence—a syntax constraint graph is built. The syntax constraint graph includes at least the required field constraints, field type constraints, field order constraints, optional field constraints, reference relationship constraints, and evidence binding constraints.

[0052] Preferably, the structured output syntax is implemented using a combination of a static basic syntax graph and dynamic runtime status. The static basic syntax graph predefines the paragraph order, field nodes, required attributes, optional attributes, field end markers, feature end markers, and cross-feature reference edges for five types of elements. The dynamic runtime status, during the decoding process, determines the currently reachable node set in real time based on the generated sequence, registered identifiers, current stage context, current control permission context, and optional field transition status.

[0053] Specifically, the military rule constraint diagram is constructed as follows: based on one or more rule source data from the simulation instructions, wargame operation specifications, task templates, control authority records, state transition rules, terrain path rules, and path passage rules, the original rule text, flowchart rule diagram, or structured rule table is first segmented into rule items, and then the rule subject, rule object, triggering condition, allowed action, prohibited action, state premise, and path restriction are extracted. Subsequently, these are converted into nodes and constraint edges to obtain the military rule constraint diagram.

[0054] During the joint decoding process, embodiments of this application can perform joint decoding of the multimodal shared representation by calling a large model or a multimodal large model. The decoder generates a structured sequence in the order of object result segment, relation result segment, event result segment, attitude result segment, and evidence result segment, and finally obtains the initial structured result.

[0055] Specifically, in each decoding step In the middle, the decoder calculates the next word to be output based on the currently generated structured sequence and multimodal shared representation. The probability distribution is used to obtain the base score. Based on this, format constraints of the structured output syntax are added. and the rule constraint items of the military rule constraint diagram Obtain the constrained score .

[0056] ;

[0057] If a candidate term does not satisfy any of the field type constraints, field order constraints, enumeration value constraints, or cross-feature reference validity constraints defined in the structured output syntax, then that candidate term is masked in the current decoding step. If a candidate term does not satisfy any of the phase-permitted action constraints, control authority constraints, state transition constraints, or path access constraints defined in the military rule constraint diagram, that candidate term will also be masked. In other words, in this embodiment of the application, only candidate tokens that pass both format constraints and rule constraints can be used as the output of the current decoding step.

[0058] Through the above methods, the embodiments of this application enable the generated initial structured results to meet the preset structured format requirements and military business rule requirements at the moment of generation, without the need for format repair or rule filtering after generation.

[0059] It should be noted that the above constraint processing includes three methods: masking, reordering, and truncation. Masking is used to directly exclude candidates that violate hard constraints such as required fields, field types, citation validity, stage-permitted actions, or control permissions; reordering is used to readjust the priority among valid candidates according to the sufficiency of evidence, modal coverage, or rule matching degree; truncation is used to retain candidates that are ranked higher and have complete fields when the number of candidate structures, the number of evidence citations, or the length of the output sequence exceeds a preset limit.

[0060] Furthermore, the initial structured results in this application embodiment include object results, relationship results, event results, attitude results, and evidence results. Specifically, object results include at least object type, object name, alias, attribute fields, and object identifier; relationship results include at least starting object, target object, relationship type, and relationship identifier; event results include at least event type, participants, time, location, status, stage, and event identifier; attitude results include at least attitude subject, attitude object, attitude type, attitude intensity, time range, and attitude identifier; and evidence results include at least evidence type, source location field, anchor point location field, and evidence identifier.

[0061] S105: Bind the object results, relationship results, event results, attitude results, and evidence results in the initial structured results to evidence anchors respectively to obtain the structured results with bound evidence anchors.

[0062] In this embodiment of the application, after obtaining the initial structured results, it is necessary to bind at least one evidence anchor point to the object results, relationship results, event results, attitude results and evidence results in the initial structured results, so that each result has traceable original evidence support.

[0063] Specifically, at least one target evidence fragment is bound as an evidence anchor from the target evidence fragment set to the object result, relationship result, event result, attitude result, and evidence result in the initial structured result. The target evidence fragment set is the set of evidence fragments obtained after the aforementioned screening and trimming process. An evidence anchor is a localization unit that can point back to the specific location of the original sample, including but not limited to text paragraphs, text sentences, text boxes of character recognition results, image region boxes, table cells, subtitle fragments, audio time intervals, video time windows, and local regions of keyframes. In inference and analysis scenarios, evidence anchors can also be map grids, task log lines, stage markers, and control relationship nodes.

[0064] The first text paragraph, image bounding box, or log line in an object's result is used as the anchor point for evidence. For example, if an object's result originates from a text description, that text paragraph is used as the anchor point for the object's result; if it originates from an object detection result in an image, the corresponding image region bounding box is used as the anchor point for evidence.

[0065] The outcome of a relationship simultaneously binds the associated fragments of the objects at both ends of the relationship and the evidentiary fragments indicating the establishment of the relationship as evidentiary anchors. For example, for the outcome of a command and control relationship, on the one hand, it binds the original fragments in which the commanding object and the controlled object each appear, and on the other hand, it binds the task issuance record or control relationship record that clearly records the control relationship as evidentiary anchors.

[0066] The event outcome is simultaneously linked to the event trigger fragment, time fragment, location fragment, and participant fragment as evidence anchors. For example, for a strike event, the following fragments are linked: a text paragraph or event trigger fragment describing the strike action, a log line or timestamp recording the time of the event, a text or location fragment indicating the location of the event, and the fragment containing the participants involved in the event.

[0067] Attitude outcomes are anchored to the original text paragraph or subtitle segment where the attitude is expressed. For example, if an attitude conclusion originates from a statement of position in a text or an opinion expressed in a video subtitle, then that text paragraph or subtitle segment is used as the anchor of evidence.

[0068] The evidence result is bound to its corresponding original data location information as the evidence anchor. For example, when the evidence result itself describes a certain log line, the line number and timestamp of that log line are used as its evidence anchor.

[0069] It should be noted that the structured results of binding evidence anchors in this embodiment can generate referential evidence location information, which includes at least one of the following: page number, segment number, coordinate frame, frame number, timestamp, grid number, and log line number. For the structured results of binding evidence anchors, multiple evidence anchors of different modalities can be bound to form a multi-chain of evidence. For example, regarding the time of movement in the event outcome, both image evidence anchors and log evidence anchors can be bound simultaneously, with the two anchors corroborating each other and enhancing the credibility of the conclusion.

[0070] S106: Verify the structured results of the bound evidence anchors. If the verification passes, obtain the target structured results.

[0071] In this embodiment of the application, after obtaining the structured result of the binding evidence anchor point, it is necessary to verify the structured result of the binding evidence anchor point in order to verify its completeness, consistency and logical rationality.

[0072] This application embodiment can verify the structured results bound to evidence anchors in terms of field completeness, multimodal evidence consistency, and temporal consistency. Specifically, regarding field completeness, it verifies the existence of required fields in each structured result of the structured results bound to the evidence anchors. Regarding multimodal evidence consistency, it verifies whether there are contradictions or inconsistencies between different modal evidences bound to the same structured result. Regarding temporal consistency, it verifies whether the chronological order of each event result is reasonable, and whether the temporal logic within the same event result is self-consistent.

[0073] If the verification passes, the structured result bound to the evidence anchor points is output as the target structured result. If the verification fails, at least one of the following is output: a problem field, a subset of problem evidence, a problem rule subgraph, or a problem event subgraph. A local rollback and re-decoding is triggered to correct the structured result bound to the evidence anchor points that failed the verification, resulting in a corrected structured result. The corrected structured result is then verified again. If the verification passes, the corrected structured result is used as the target structured result; if the verification fails, local re-decoding correction continues until the verification passes and the target structured result is obtained.

[0074] The local re-decoding correction includes one or more of field-level rollback, subgraph-level rollback, and local context-level rollback. When performing local re-decoding correction, the fields that have passed the verification are locked in a read-only state, and only the fields that have not passed the verification and their associated evidence fragments and rule subgraphs are given write permissions to prevent the correction process from rewriting the already correct field content.

[0075] Based on the above steps S101-S105, this application provides a joint extraction method and related products for multimodal military situation data. The method includes: First, acquiring multimodal military situation data, which includes target situation elements to be extracted. Second, standardizing the multimodal military situation data to obtain a standardized fragment set, and performing feature extraction and cross-modal alignment processing on the standardized fragment set to obtain a multimodal shared representation. Subsequently, based on a preset structured output syntax and a preset military rule constraint graph, jointly decoding the multimodal shared representation to generate an initial structured result, wherein the initial structured result includes object results, relationship results, event results, attitude results, and evidence results. Finally, binding the object results, relationship results, event results, attitude results, and evidence results in the initial structured result to evidence anchors to obtain a structured result bound to evidence anchors, and verifying the structured result bound to evidence anchors. If the verification passes, the target structured result is obtained. As can be seen, this scheme standardizes multimodal military situational data to obtain a set of standardized fragments, and then performs feature extraction and cross-modal alignment on this set to obtain a multimodal shared representation, achieving a unified representation of multi-source heterogeneous data. During joint decoding, based on a pre-defined structured output grammar and a pre-defined military rule constraint graph, the multimodal shared representation is jointly decoded. This ensures that the generated initial structured result is subject to the format constraints of the structured output grammar and the rule constraints of the military rule constraint graph in real time during decoding, thus solving the problem of unstable output structure caused by applying constraints only at the prompt word or post-output format check stage in existing technologies. Simultaneously, the object results, relation results, event results, attitude results, and evidence results in the initial structured result are each bound to evidence anchors and verified, ensuring that each initial structured result is associated with corresponding evidence support and has a traceable evidence source, effectively improving the credibility of the extracted results.

[0076] Furthermore, Figure 2 This is a schematic diagram of a multimodal military situation data joint extraction device provided in an embodiment of this application. (Combined with...) Figure 2 As shown, the multimodal military situation data joint extraction device 200 provided in this application embodiment may include: The data acquisition module 201 is used to acquire multimodal military situation data, which includes target situation elements to be extracted. Data standardization module 202 is used to standardize the multimodal military situation data to obtain a set of standardized segments; The set processing module 203 is used to perform alignment processing on the standardized fragment set to obtain a multimodal shared representation; The result generation module 204 is used to jointly decode the multimodal shared representation based on a preset structured output syntax and a preset military rule constraint graph to generate an initial structured result, which includes object results, relation results, event results, attitude results and evidence results. Anchor binding module 205 is used to bind the object result, relationship result, event result, attitude result and evidence result in the initial structured result to evidence anchors respectively, so as to obtain a structured result with bound evidence anchors; The result verification module 206 is used to verify the structured result of the bound evidence anchor point, and obtain the target structured result if the verification is passed.

[0077] Optionally, the joint extraction device 200 for multimodal military situation data may include: The result correction module is used to perform local re-decoding correction on the structured result of the binding evidence anchor point that failed the verification in the case of failure to pass the verification, so as to obtain the corrected structured result. The re-verification module is used to re-verify the corrected structured result. If the verification passes, the corrected structured result is used as the target structured result. If the verification fails, the local re-decoding correction is continued until the verification passes and the target structured result is obtained.

[0078] Optionally, the collection processing module 203 may include: The candidate fragment module is used to recall fragments related to the target situation elements from the standardized fragment set to obtain a candidate evidence fragment set. The candidate alignment module is used to align the set of candidate evidence fragments to obtain the multimodal shared representation.

[0079] Optionally, the candidate alignment module may include: The candidate pruning module is used to filter and prune the set of candidate evidence fragments to obtain the set of target evidence fragments; The shared representation module is used to perform feature extraction and cross-modal alignment processing on the target evidence fragment set to obtain the multimodal shared representation.

[0080] Optionally, the candidate cropping module is specifically used for: From the candidate evidence fragment set, fragments with a correlation greater than or equal to a preset correlation threshold with the target situation element are selected to obtain a preliminary set of evidence fragments; The fragments that exceed the preset context window in the initial screening evidence fragment set are trimmed to obtain the target evidence fragment set.

[0081] Optionally, the shared representation module is specifically used for: Each target evidence fragment in the target evidence fragment set is feature-encoded to obtain a feature vector corresponding to each target evidence fragment. The feature vectors corresponding to each target evidence fragment are mapped to the same representation space and fused to obtain the multimodal shared representation.

[0082] Optionally, the anchor point binding module 205 is specifically used for: From the set of target evidence fragments, bind at least one target evidence fragment as an evidence anchor point to the object result, relationship result, event result, attitude result, and evidence result in the initial structured result, to obtain a structured result with bound evidence anchor points.

[0083] Furthermore, embodiments of this application also provide an electronic device, including: a processor, a memory, and a system bus; The processor and the memory are connected via the system bus; The memory is used to store one or more programs, the one or more programs including instructions that, when executed by the processor, cause the processor to perform any of the implementation steps of the above-described method for joint extraction of multimodal military situation data.

[0084] Furthermore, embodiments of this application also provide a computer-readable storage medium for storing a computer program, which, when executed by a terminal device, implements any of the implementation steps of the above-described method for jointly extracting multimodal military situation data.

[0085] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application. It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on describing the differences from other embodiments. The same or similar parts between the various embodiments can be referred to mutually.

[0086] The system disclosed in the embodiments is described in a relatively simple manner because it corresponds to the method disclosed in the embodiments. For relevant details, please refer to the method section.

[0087] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0088] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for joint extraction of multimodal military situation data, characterized in that, The method includes: Acquire multimodal military situation data, which includes target situation elements to be extracted; The multimodal military situation data is standardized to obtain a set of standardized segments; The standardized fragment set is aligned to obtain a multimodal shared representation; Based on a preset structured output syntax and a preset military rule constraint graph, the multimodal shared representation is jointly decoded to generate an initial structured result, which includes object results, relation results, event results, attitude results, and evidence results. The object result, relationship result, event result, attitude result, and evidence result in the initial structured result are respectively bound to evidence anchor points to obtain a structured result with bound evidence anchor points; The structured result of the bound evidence anchor point is verified, and if the verification is successful, the target structured result is obtained.

2. The method according to claim 1, characterized in that, The method further includes: If the verification fails, the structured result of the binding evidence anchor point that failed the verification is locally re-decoded and corrected to obtain the corrected structured result; The corrected structured result is checked again. If the check passes, the corrected structured result is used as the target structured result. If the check fails, the local re-decoding correction is performed again until the check passes and the target structured result is obtained.

3. The method according to claim 1, characterized in that, The alignment process of the standardized fragment set to obtain a multimodal shared representation includes: Recall fragments related to the target situation elements from the standardized fragment set to obtain a candidate evidence fragment set; The candidate evidence fragment set is aligned to obtain the multimodal shared representation.

4. The method according to claim 3, characterized in that, The alignment process of the candidate evidence fragment set to obtain the multimodal shared representation includes: The candidate evidence fragment set is filtered and trimmed to obtain the target evidence fragment set; Feature extraction and cross-modal alignment are performed on the target evidence fragment set to obtain the multimodal shared representation.

5. The method according to claim 4, characterized in that, The process of filtering and trimming the candidate evidence fragment set to obtain the target evidence fragment set includes: From the candidate evidence fragment set, fragments with a correlation greater than or equal to a preset correlation threshold with the target situation element are selected to obtain a preliminary set of evidence fragments; The fragments that exceed the preset context window in the initial set of evidence fragments are trimmed to obtain the target set of evidence fragments.

6. The method according to claim 4, characterized in that, The step of performing feature extraction and cross-modal alignment on the target evidence fragment set to obtain the multimodal shared representation includes: Each target evidence fragment in the target evidence fragment set is feature-encoded to obtain a feature vector corresponding to each target evidence fragment. The feature vectors corresponding to each target evidence fragment are mapped to the same representation space and fused to obtain the multimodal shared representation.

7. The method according to claim 4, characterized in that, The step of binding the object result, relationship result, event result, attitude result, and evidence result in the initial structured result to evidence anchors to obtain a structured result with bound evidence anchors includes: From the set of target evidence fragments, bind at least one target evidence fragment as an evidence anchor point to the object result, relationship result, event result, attitude result, and evidence result in the initial structured result, to obtain a structured result with bound evidence anchor points.

8. A joint extraction device for multimodal military situation data, characterized in that, include: The data acquisition module is used to acquire multimodal military situation data, which includes target situation elements to be extracted. The data standardization module is used to standardize the multimodal military situation data to obtain a set of standardized segments; The set processing module is used to align the standardized fragment set to obtain a multimodal shared representation; The result generation module is used to jointly decode the multimodal shared representation based on a preset structured output syntax and a preset military rule constraint graph to generate an initial structured result, which includes object results, relation results, event results, attitude results, and evidence results. Anchor binding module is used to bind the object result, relationship result, event result, attitude result and evidence result in the initial structured result to evidence anchors respectively, so as to obtain a structured result with bound evidence anchors; The result verification module is used to verify the structured result of the bound evidence anchor point, and obtain the target structured result if the verification is successful.

9. An electronic device, characterized in that, The device includes: a processor, a memory, and a system bus; The processor and the memory are connected via the system bus; The memory is used to store a program, the program including instructions that, when executed by the processor, cause the processor to perform the steps of the joint extraction method for multimodal military situation data according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program, which, when executed by a terminal device, implements the steps of the joint extraction method for multimodal military situation data as described in any one of claims 1-7.

Images