An electronic information intelligent processing method based on multi-modal data fusion

Abstract

The application discloses an electronic information intelligent processing method based on multi-modal data fusion, and relates to the technical field of electronic information processing, comprising the following steps: performing multi-dimensional path evaluation processing on an effective reasoning path set, generating an optimal path scheme, extracting path semantic features of the optimal path scheme and performing semantic structure reconstruction to generate a path semantic representation sequence; performing deviation inversion analysis on the path semantic representation sequence to generate a feedback deviation record set, and correcting corresponding path decision rule parameters in the path semantic representation sequence through deviation mapping based on the feedback deviation record set to generate a processing rule set. Through deviation driving adjustment processing by combining the feedback deviation record with the semantic relation table, the application realizes the conversion of multi-modal data from scattered expression to unified semantic structure, and improves the self-adaptive ability and long-term stability of the electronic information processing method.

Description

Technical Field

[0001] This invention relates to the field of electronic information processing technology, and in particular to an intelligent electronic information processing method based on multimodal data fusion. Background Technology

[0002] With the rapid development of the Internet of Things, mobile Internet, and intelligent sensing technologies, electronic information processing is gradually evolving from being driven by a single data source to collaborative processing of multi-source heterogeneous data. In practical applications, different modalities of data differ in their information representation, temporal distribution, and semantic structure. Therefore, multimodal data fusion processing technology has become an important research direction in the field of electronic information. Existing technologies typically employ feature-level fusion, decision-level fusion, or deep learning models to uniformly process multimodal data, achieving information complementarity and enhanced representation.

[0003] In existing technical solutions, the following shortcomings exist in multimodal data fusion processing: Existing methods mostly focus on the fusion processing of the data layer or feature layer, lacking a unified modeling of the semantic structure of multimodal data. In particular, a stable structured expression mechanism has not been formed in terms of object orientation, behavioral logic, and temporal evolution relationships, resulting in subsequent reasoning processes relying on implicit associations and making it difficult to achieve interpretable path construction and filtering. Existing technologies generally lack dynamic correction mechanisms based on execution feedback. The processing process is usually a one-way reasoning flow, failing to analyze and reverse correct the differences between the reasoning results and the actual execution results. This makes it difficult for processing rules to be updated adaptively, and the problem of accumulated reasoning bias is prone to occur in variable environments. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides an intelligent processing method for electronic information based on multimodal data fusion to solve the problems of insufficient semantic structure modeling of multimodal data and lack of dynamic correction mechanism based on execution feedback.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] This invention provides an intelligent processing method for electronic information based on multimodal data fusion, comprising: collecting multimodal electronic information data and performing cross-modal consistency constraint processing to generate a multimodal standard dataset; calculating the modal participation evaluation index of the multimodal standard dataset and prioritizing it to generate a modal combination sequence; extracting the semantic anchors of the modal combination sequence and performing cross-modal association organization processing to form a semantic relationship table; performing constraint screening by analyzing the association connectivity and temporal consistency of the semantic relationship table to output a candidate inference path set; reading erroneous inference path records from the historical processing process and comparing them with the candidate inference path set to generate a valid inference path set; performing multidimensional path evaluation processing on the valid inference path set to generate a preferred path scheme; extracting the path semantic features of the preferred path scheme and reconstructing its semantic structure to generate a path semantic representation sequence; performing deviation inversion analysis on the path semantic representation sequence to generate a feedback deviation record set; and, based on the feedback deviation record set, correcting the corresponding path decision rule parameters in the path semantic representation sequence through deviation mapping to generate a processing rule set.

[0008] As a preferred embodiment of the intelligent electronic information processing method based on multimodal data fusion described in this invention, the specific steps for generating the multimodal standard dataset are as follows:

[0009] Multimodal electronic information data is processed by time-series mapping according to a unified time base to establish the time correspondence between different modal data and generate cross-modal time-aligned sequences.

[0010] The semantic identifiers of each modality in the cross-modal time-aligned sequence are parsed, and consistency constraint matching is performed to generate a multimodal standard dataset.

[0011] As a preferred embodiment of the electronic information intelligent processing method based on multimodal data fusion described in this invention, the specific steps for generating the modal combination sequence are as follows:

[0012] Calculate the current validity value, historical credibility value, and task relevance value of each modality in the multimodal specification data according to a unified time segment;

[0013] Perform joint mapping and priority sorting on the current validity value, historical credibility value, and task relevance value to generate a modality combination sequence.

[0014] As a preferred embodiment of the electronic information intelligent processing method based on multimodal data fusion described in this invention, the specific steps for forming the semantic relationship table are as follows:

[0015] Extract semantic anchors representing object orientation, state orientation, behavior orientation, and time orientation from each modal data of the modal combination sequence to form a semantic anchor sequence;

[0016] The semantic anchors of different modal data in the semantic anchor sequence are matched according to temporal adjacency, semantic consistency, and object correspondence to generate a cross-modal anchor correspondence table.

[0017] Perform path concatenation and hierarchical organization processing on semantic anchors with corresponding relationships in the cross-modal anchor correspondence table to generate semantic association chains;

[0018] Semantic anchors are used as semantic nodes, and the relationships in the semantic association chain are used as node connection relationships. Node-based structured mapping is then performed to generate a semantic relationship table.

[0019] As a preferred embodiment of the electronic information intelligent processing method based on multimodal data fusion described in this invention, the specific steps for outputting the candidate inference path set are as follows:

[0020] Perform association connectivity analysis on each semantic node and its corresponding node connection relationship in the semantic relation table to generate a semantic connectivity relation table;

[0021] Semantic nodes with reachable relationships in the semantic connectivity table are sequentially linked according to the direction of association extension, and temporal consistency is checked according to a unified time order to output a set of candidate reasoning paths.

[0022] As a preferred embodiment of the electronic information intelligent processing method based on multimodal data fusion described in this invention, the specific steps for generating an effective inference path set are as follows:

[0023] Read the records of erroneous reasoning paths in the historical processing, and perform unified path structure encoding processing on each historical erroneous reasoning path and candidate reasoning path according to the semantic anchor order, the connection of the relationship and the time order to generate a path structure encoding sequence;

[0024] The differences in semantic anchor arrangement, relational connection, and temporal distribution among historical erroneous reasoning paths and candidate reasoning paths in the path structure encoding sequence are analyzed and suppressed and filtered to generate a set of effective reasoning paths.

[0025] As a preferred embodiment of the electronic information intelligent processing method based on multimodal data fusion described in this invention, the specific steps for generating the preferred path scheme are as follows:

[0026] The path integrity, semantic consistency, temporal coherence, and historical adaptability of each valid reasoning path in the valid reasoning path set are evaluated separately, and a path evaluation index sequence is generated.

[0027] Based on the path evaluation index sequence, a joint mapping and sorting process is performed on each effective inference path in the effective inference path set to generate the optimal path scheme.

[0028] As a preferred embodiment of the electronic information intelligent processing method based on multimodal data fusion described in this invention, the specific steps for generating the path semantic representation sequence are as follows:

[0029] Through semantic parsing, the event status information, correlation information, abnormal location description information and change trend information corresponding to each effective reasoning path in the preferred path scheme are extracted to generate a path semantic description sequence.

[0030] The path semantic description sequence is associated, reorganized, and rearranged, and then consistency adjustments are performed according to the time evolution order to generate a path semantic representation sequence.

[0031] As a preferred embodiment of the electronic information intelligent processing method based on multimodal data fusion described in this invention, the specific steps for generating the feedback deviation record set are as follows:

[0032] Acquire the status feedback data, event response data, and result confirmation data during the electronic information execution process corresponding to the preferred path scheme, and perform time alignment and semantic correspondence processing to generate an execution feedback sequence;

[0033] The path semantic representation sequence and the execution feedback sequence are matched according to their time position and semantic direction to generate a semantic comparison sequence;

[0034] Based on the semantic comparison sequence, the difference between the path semantic representation sequence and the execution feedback sequence is reversed and analyzed for the source of the difference, generating a feedback deviation record set.

[0035] As a preferred embodiment of the electronic information intelligent processing method based on multimodal data fusion described in this invention, the specific steps for generating the processing rule set are as follows:

[0036] Extract the path selection weight parameters, semantic relationship influence parameters, and time evolution constraint parameters corresponding to each effective reasoning path in the path semantic representation sequence, and perform parameterized organization processing according to the semantic position order to generate a path decision rule parameter sequence;

[0037] Based on the feedback deviation record set, the path selection weight parameters, semantic relationship influence parameters, and time evolution constraint parameters corresponding to the path decision rule parameter sequence are extracted, and deviation-driven adjustment processing is performed to generate a processing rule sequence.

[0038] The beneficial effects of this invention are as follows: By using semantic anchors as semantic nodes and performing path concatenation and structured mapping to form a semantic relationship table, the multimodal data is transformed from a dispersed expression to a unified semantic structure, enabling object orientation, behavioral logic, and temporal evolution relationships to be organized in an explicit structural form, thereby improving the interpretability and structural consistency of the reasoning process; by performing difference reverse analysis based on semantic comparison sequences and combining feedback deviation records for deviation-driven adjustment processing, the source of the difference between the reasoning result and the actual execution result can be located and the propagation path can be traced, enabling the processing rules to be dynamically updated with execution feedback, thereby improving the adaptability and long-term stability of electronic information processing methods. Attached Figure Description

[0039] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart of an intelligent electronic information processing method based on multimodal data fusion.

[0041] Figure 2 A schematic diagram for constructing a semantic relation table.

[0042] Figure 3 This is a schematic diagram of a multimodal semantic reasoning and bias feedback structure.

[0043] Figure 4 This is a schematic diagram of path semantic reconstruction and rule correction. Detailed Implementation

[0044] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0045] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0046] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0047] Reference Figures 1-4 This is one embodiment of the present invention, which provides an intelligent processing method for electronic information based on multimodal data fusion, comprising the following steps:

[0048] S1. Collect multimodal electronic information data and perform cross-modal consistency constraint processing to generate a multimodal standard dataset.

[0049] S1.1 Perform time-series mapping processing on multimodal electronic information data according to a unified time reference, establish the time correspondence between different modal data, and generate cross-modal time-aligned sequences.

[0050] It should be noted that by unifying the access of multi-source data interfaces in the electronic information processing scenario, structured business data, semi-structured log data, unstructured text data, image data, and interactive behavior data are acquired synchronously. During the acquisition process, the data source identifier, collection time identifier, and data content structure corresponding to each type of data are recorded in a unified manner, and a time index association relationship is established according to the generation order of each type of data to form a multimodal electronic information data set.

[0051] The system uniformly reads the collection time identifiers corresponding to various types of data in multimodal electronic information data, and performs time mapping processing on various types of data according to a unified time benchmark, transforming data at different time scales to the same time axis. On the unified time axis, the system matches and organizes the correspondence between various types of data at the same time position or adjacent time positions, so that structured business data, log data, text data, image data and interactive behavior data form a one-to-one correspondence or interval correspondence in the time dimension, thereby establishing cross-modal time correspondence and generating cross-modal time alignment sequences.

[0052] S1.2. Parse the semantic identifiers of each modality data in the cross-modal time-aligned sequence and perform consistency constraint matching to generate a multimodal standard dataset.

[0053] It should be noted that semantic identifier parsing is performed on the different modal data corresponding to each time position in the cross-modal time alignment sequence. Data content representing object identifiers, state descriptions, and behavioral events is extracted from each modal data, and corresponding semantically valid tags are generated. Character normalization, name correspondence mapping, and encoding format unification are performed on the extracted object identifiers to generate a unified object identifier sequence. The unified object identifiers of different modal data are matched at the same time position to determine data combinations pointing to the same object. In the data combinations that determine the object correspondence, the state values ​​of each modal data are extracted and mapped to a unified state category set. The consistency of the mapped state values ​​is compared, and data combinations with consistent state values ​​are selected and state consistency tags are generated. Behavioral events are sorted according to time order, and the logical consistency of the sequential relationship between behavioral events is checked. Data combinations with no conflict in behavioral order are selected and behavior consistency tags are generated. Data combinations that simultaneously have semantically valid tags, state consistency tags, and behavior consistency tags are associated and organized to generate a multimodal standard dataset.

[0054] S2. Calculate the modality participation evaluation index of the multimodal standard dataset and prioritize it, generate modality combination sequences, extract the semantic anchors of the modality combination sequences and perform cross-modal association organization processing to form a semantic relationship table.

[0055] S2.1 Calculate the current validity value, historical credibility value, and task relevance value of each modality data in the multimodal specification data according to a unified time segment.

[0056] It should be noted that the ratio of the number of data items with semantically valid tags to the number of data items participating in time alignment is used as the content completeness value, and the ratio of the number of data items with both state consistency tags and behavior consistency tags to the number of data items participating in time alignment is used as the cross-modal consistency value. The current validity value, historical credibility value, and task relevance value of each modality data in the multimodal specification data are calculated.

[0057] The expression for calculating the current validity value is:

[0058] ;

[0059] The expression for calculating the historical reliability value is:

[0060] ;

[0061] The expression for calculating the task relevance value is:

[0062] ;

[0063] in, Indicates the first Each modality in the time interval The current valid value within, Indicates the first The historical reliability value of each modality Indicates the first The task relevance values ​​for each modality are taken as follows. Indicates the first Each modality in the time interval The content completeness value is determined by the following parameters: Indicates the first Each modality in the time interval The cross-modal consistency value within the range, Indicates the first Each modality in the time interval The time offset within, Indicates time segment The average value of the time offset of all modes within the module. Represents a very small positive number. Indicates the number of historical time segments. Indicates the first Each modality in the time interval The extent of semantic change within, Indicates time segment The average magnitude of semantic changes across all modalities. Indicates the first The number of semantic tags involved in task matching in each modality. Indicates the first The first of the modalities A semantic identifier, This represents the semantic expression of the task currently being processed.

[0064] It should also be noted that modality refers to the data category formed by classifying multimodal electronic information data according to the data source form and data expression method. Each modality corresponds to a data expression form. Specifically, modality includes structured business data modality, log data modality, text data modality, image data modality, and interactive behavior data modality.

[0065] The semantic change magnitude refers to the degree of difference between the sets of semantic tags corresponding to the same modality at adjacent time positions. It is obtained by taking the sum of the number of semantic tags added and the number of semantic tags that disappeared at the current time position relative to the previous time position, and the ratio of the total number of semantic tags at the current time position.

[0066] The time offset refers to the time difference between the original timestamp of a data item and the unified time reference time. It is obtained by obtaining the difference between the timestamp of a data item in a cross-modal time alignment sequence and the reference time at the corresponding time axis position.

[0067] Task semantic expression refers to the semantic set formed after semantic parsing of the input information of the current electronic information processing task. It is obtained by extracting object identification fields, state description fields and behavior instruction fields from the task input, forming object sets, state sets and behavior sets respectively, and then combining them.

[0068] S2.2 Perform joint mapping and priority sorting on the current validity value, historical credibility value, and task relevance value to generate a modal combination sequence.

[0069] It should be noted that, within a unified time period, the current validity value, historical reliability value, and task relevance value corresponding to each modality data are read and subjected to interval normalization to ensure comparability of different values ​​within the same numerical range. The normalized three categories of values ​​are used as the evaluation features of each modality data. Based on the relative magnitude relationships among the modality data in the three evaluation features, a dominance relationship determination process is performed on all modality data. Data that are not lower than other modalities in all three evaluation features and are superior to other modalities in at least one feature are classified into the first priority layer. Modality data in the first priority layer are then removed from the entire modality data set. The remaining modal data are then processed again to determine dominance relationships until all modal data are assigned to their respective priority levels, thus forming a complete priority hierarchy. Within the first priority level, the modal data are arranged sequentially according to the magnitude of their current validity values, and the degree of difference between adjacent modal data in the three evaluation features is calculated. The statistical median of this degree of difference among all adjacent modal data is used as the continuity determination criterion, and adjacent modal data whose degree of difference is not higher than the continuity determination criterion are divided into the same continuous segment. Modal data within the same continuous segment are combined and organized to generate a modal combination sequence.

[0070] S2.3 Extract semantic anchors representing object orientation, state orientation, behavior orientation, and time orientation from each modal data of the modal combination sequence to form a semantic anchor sequence.

[0071] It should be noted that, for each modal data in the modal combination sequence, the corresponding data items are read in chronological order within a unified time segment. The semantic identifiers in each data item are parsed, and the fields representing object entities are extracted as object pointing information, the fields representing running status or attribute values ​​are extracted as status pointing information, and the fields representing operation events or behavior types are extracted as behavior pointing information. The time position of the data item on a unified time axis is used as time pointing information. The extracted object pointing information, status pointing information, behavior pointing information, and time pointing information are combined and encapsulated so that the same data item corresponds to a complete set of semantic anchors. All semantic anchors are arranged in a unified chronological order, while maintaining the chronological relationship between each semantic anchor, to generate a semantic anchor sequence.

[0072] S2.4 Match the semantic anchors of different modal data in the semantic anchor sequence according to temporal adjacency, semantic consistency, and object correspondence to generate a cross-modal anchor correspondence table.

[0073] It should be noted that, following a unified timeline, the semantic anchors corresponding to the modal data at the same or adjacent time positions in the semantic anchor sequence are read one by one. Semantic anchors with the same time position are identified as a temporal adjacency candidate set. For cases where there are no identical time positions, one or more semantic anchors with the smallest temporal position difference among the semantic anchors are identified as a temporal adjacency candidate set. In the temporal adjacency candidate set, semantic anchors with the same object identifier or consistent correspondence after name mapping are associated and marked to form an object correspondence relationship. Among the semantic anchors that satisfy the object correspondence relationship, the consistency comparison of state pointing information and behavior pointing information is further performed. Semantic anchors with the same state category and no conflict in behavior type are matched and marked to form a semantic pointing consistency relationship. Semantic anchors that simultaneously satisfy the temporal adjacency relationship, object correspondence relationship, and semantic pointing consistency relationship are combined and organized. The corresponding semantic anchors are summarized in the form of associated records, and the correspondence relationship and source modal information between each semantic anchor are recorded to generate a cross-modal anchor correspondence table.

[0074] S2.5 Perform path concatenation and hierarchical organization processing on semantic anchors with corresponding relationships in the cross-modal anchor correspondence table to generate semantic association chains.

[0075] It should be noted that, following a unified timeline, each semantic anchor in the cross-modal anchor correspondence table is read sequentially. Using the temporal information of the semantic anchors as the basis for priority, semantic anchors with consecutive or adjacent temporal positions are concatenated in chronological order to form an initial time series path. Within this initial time series path, semantic anchors with the same object identifier are merged and organized, using object-oriented information as the main thread, to form object-dimensional path segments. Within each object-dimensional path segment, semantic anchors are sequentially connected based on the order of behavioral information, and a sequential relationship is established for semantic anchors with behavioral connections. After completing the path organization for both object and behavioral dimensions, the path segments corresponding to different objects are arranged hierarchically in chronological order. Path segments within the same time period but with different objects are grouped into the same level, and hierarchical relationships are established for path segments extending across time periods. Finally, the semantic anchor paths, after time concatenation, object merging, and hierarchical organization, are integrated as a whole to generate a semantic association chain.

[0076] S2.6. Take the semantic anchor as the semantic node, the association relationship in the semantic association chain as the node connection relationship, and perform node-based structured mapping processing to generate a semantic relationship table.

[0077] It should be noted that, for each semantic anchor point in the semantic association chain, the corresponding object pointing information, state pointing information, behavior pointing information, and time pointing information are read respectively, and each semantic anchor point is registered as a semantic node. The object identifier, state category, behavior type, and time position are recorded as the attribute information of the semantic node. For the sequential relationship between adjacent semantic anchor points in the semantic association chain, the connection relationship formed by time sequence and the association relationship formed by behavior connection are extracted, and node connection records are established in the form of semantic node pairs. At the same time, the connection direction and relationship type between nodes are identified. All semantic nodes are organized according to a unified structure, so that each semantic node records its corresponding predecessor and successor node information. The semantic nodes and their corresponding connection relationships are summarized and organized to generate a semantic relationship table.

[0078] S3. Constraint filtering is performed by analyzing the association connectivity and temporal consistency of the semantic relationship table, outputting a set of candidate reasoning paths, reading the records of erroneous reasoning paths in the historical processing, and comparing the path similarity with the set of candidate reasoning paths to generate a set of valid reasoning paths.

[0079] S3.1 Perform association connectivity analysis on each semantic node and its corresponding node connection relationship in the semantic relation table to generate a semantic connectivity relation table.

[0080] It should be noted that, according to the node connection relationship, the predecessor and successor nodes directly associated with each semantic node in the semantic relationship table are read one by one. Starting from the current semantic node, the association search is expanded level by level along the node connection direction. Semantic nodes that can be reached through continuous node connection relationships are progressively expanded and recorded to form a set of connected nodes for the corresponding semantic nodes. In the set of connected nodes, the connection paths between each semantic node are organized, and the node connection order and corresponding connection direction information from the starting semantic node to each connected semantic node are recorded to form the reachable path relationship between nodes. The set of connected nodes and path relationships corresponding to each semantic node are summarized and organized, and registered according to a unified structure to generate a semantic connectivity table.

[0081] S3.2. Connect the semantic nodes with reachable relationships in the semantic connectivity table step by step according to the direction of association extension, and perform temporal consistency verification according to a unified time order to output a set of candidate reasoning paths.

[0082] It should be noted that, for each semantic node in the semantic connectivity table, successor semantic nodes are selected one by one according to the association extension direction indicated by the node connection relationship. Starting from the current semantic node, the connection is extended level by level along the connection direction. Semantic nodes that can form continuous connection relationships are connected in sequence according to the connection order to construct a node sequence. During the construction of the node sequence, when there are multiple successor semantic nodes, the extension process is continued along different connection directions to form multiple branch node sequences. After the node sequence is formed, the time pointing information of each semantic node in the sequence is compared one by one according to a unified time axis order. When the time position of the successor semantic node is not earlier than that of the preceding semantic node, the corresponding node connection relationship is retained. When there is a case where the time position is earlier than that of the preceding semantic node, the corresponding node connection relationship is deleted and the extension in that direction is terminated. The corresponding node connection relationship is marked as a time-series invalid connection relationship. The node sequence that satisfies the time sequence constraint is defined as a candidate reasoning path, and all candidate reasoning paths are summarized and organized to generate a candidate reasoning path set.

[0083] S3.3 Read the records of erroneous reasoning paths in the historical processing, and perform unified path structure encoding processing on each historical erroneous reasoning path and candidate reasoning path according to the semantic anchor order, the connection of the relationship and the time position, to generate a path structure encoding sequence.

[0084] It should be noted that the erroneous reasoning path records stored during the historical processing are read, and the semantic node arrangement order, the connection order of the association relationship between adjacent semantic nodes, and the time position of each semantic node on a unified time axis are obtained for each historical erroneous reasoning path and candidate reasoning path. At the same time, the semantic anchor point identifier corresponding to each semantic node is extracted as the node content identifier. Using the sequential arrangement of semantic nodes as the path position benchmark, the connection order of association relationship as the path connection benchmark, and the time position as the path time sequence benchmark, the semantic node sequence, relationship connection sequence, and time position sequence corresponding to each candidate reasoning path are sequentially combined and encoded. During the encoding process, the corresponding semantic anchor point identifier, relationship type identifier, and time position identifier are recorded for each position, so that each position in the same path corresponds to a determined anchor point identifier, relationship identifier, and time position identifier, thereby generating a path structure encoding sequence.

[0085] It should also be noted that the erroneous reasoning path record originates from the process of comparing the differences between the execution results of candidate reasoning paths and the actual execution results during historical processing. After each round of electronic information processing, the semantic representation of the path corresponding to the candidate reasoning path is compared with the actual execution feedback results item by item. When there are cases of mismatched object identifiers, inconsistent state values, or inconsistent behavior execution results, the corresponding candidate reasoning path is determined to be an erroneous reasoning path and recorded. During the recording process, the semantic node arrangement order, the connection order of the association relationship between adjacent semantic nodes, and the time position information corresponding to each semantic node in the current candidate reasoning path are extracted, and the path position that produces the difference and the corresponding difference type are recorded, thus forming the erroneous reasoning path record.

[0086] S3.4 Analyze the differences in semantic anchor arrangement, relational connection and temporal distribution of each historical erroneous reasoning path and candidate reasoning path in the path structure encoding sequence, and perform suppression and screening to generate a set of effective reasoning paths.

[0087] It should be noted that, for each candidate inference path in the path structure encoding sequence, the corresponding semantic node arrangement encoding, the semantic anchor point identifier encoding, the relation connection encoding, and the time sequence encoding are read one by one, and aligned with the path structure encoding sequence of historical erroneous inference paths according to the path start position and encoding order; after alignment, the semantic anchor point identifiers at each corresponding position are compared to identify semantic anchor point replacement, missing, and misalignment; the relation connection encodings at each corresponding position are compared to identify relation breaks, reverse connections, and jump connections; and the time sequence at each corresponding position is compared to identify time reversal and time intervals. The analysis considers cases of abnormal increases and time jumps. After obtaining the differences at each location, the segments where adjacent locations are all marked as different are identified as difference segments. The difference segments are then analyzed step by step according to the path extension direction. When the length of a difference segment is not less than the median of the length distribution of all difference segments in the current candidate inference path, or when the number of difference positions in the difference segment is greater than the average number of positions in all candidate inference paths, the corresponding candidate inference path is marked as an erroneous similar path and suppression processing is performed. Candidate inference paths that are not marked as erroneous similar paths are retained and summarized to generate a set of valid inference paths.

[0088] It should also be noted that the median was chosen as the criterion for determining the length of the difference segment because the distribution of the difference segment length is easily affected by a few extremely long segments, while the median can reflect the central level of the overall distribution and is not affected by extreme values, thus more stably identifying representative difference segments. The average value was chosen as the criterion for determining the number of difference positions because the average value can reflect the overall level of the overall difference scale and is used to measure the relative proportion of the current difference segment in the entire path or alignment result, thereby determining whether the difference has reached the point where candidate inference paths need to be suppressed.

[0089] S4. Perform multi-dimensional path evaluation processing on the effective reasoning path set to generate the preferred path scheme, extract the path semantic features of the preferred path scheme and reconstruct the semantic structure to generate a path semantic representation sequence.

[0090] S4.1 Evaluate the path integrity, semantic consistency, temporal coherence, and historical adaptability of each valid reasoning path in the valid reasoning path set, and generate a path evaluation index sequence.

[0091] It should be noted that, for each valid reasoning path in the valid reasoning path set, the corresponding semantic node arrangement order, semantic anchor point identifier, relationship connection order, and temporal position information are read one by one. Based on the continuous connection of semantic nodes in each valid reasoning path, the connection relationship between each adjacent semantic node in the valid reasoning path is checked one by one. When there is no connection interruption or missing node connection relationship in the valid reasoning path, it is marked as a complete path; otherwise, the position of connection interruption is recorded as the basis for determining path completeness. Based on the semantic anchor point identifier corresponding to each semantic node in the valid reasoning path, the object pointing information, state pointing information, and behavior pointing information between adjacent semantic nodes are compared one by one. When there is no object conflict, state contradiction, or behavior disconnection between adjacent semantic nodes, it is marked as semantically consistent; otherwise, the position of conflict is recorded as the basis for determining semantic consistency. Based on the temporal position of each semantic node in the valid reasoning path, the temporal order between adjacent semantic nodes is checked one by one. When there is no time reversal or time jump anomaly in the valid reasoning path, it is marked as temporally coherent; otherwise, the position of time anomaly is recorded as the basis for determining temporal coherence.

[0092] Based on the historical error reasoning path records, the corresponding positions of the current valid reasoning path and the historical error reasoning path in the path structure encoding sequence are compared. If the current path does not show the same difference distribution pattern as the historical error path in terms of semantic anchor arrangement, relation connection, and temporal order, the corresponding valid reasoning path is marked as historically adapted. Otherwise, the positions with matching relationships are recorded as the basis for historical adaptability judgment. The judgment results and corresponding position distributions of path integrity, semantic consistency, temporal coherence, and historical adaptability are summarized and organized to generate a path evaluation index sequence.

[0093] S4.2 Based on the path evaluation index sequence, perform joint mapping and sorting processing on each effective inference path in the effective inference path set to generate the optimal path scheme.

[0094] It should be noted that, for each valid inference path in the path evaluation index sequence, the corresponding path integrity judgment result, semantic consistency judgment result, temporal coherence judgment result, and historical adaptability judgment result are read respectively, and a path evaluation representation composed of the four types of judgment results is constructed for each valid inference path. Based on the path evaluation representation, the judgment results among all valid inference paths are compared item by item. When a valid inference path is not inferior to another valid inference path in all four types of judgment results (path integrity, semantic consistency, temporal coherence, and historical adaptability) and is superior to another valid inference path in at least one type of judgment result, the current valid inference path is considered valid. Effective reasoning paths are identified as priority paths. Effective reasoning paths not identified as priority paths by other paths are identified as the first preferred path set. Within the first preferred path set, effective reasoning paths satisfying the path integrity judgment, semantic consistency judgment, temporal coherence judgment, and historical adaptability judgment are retained sequentially. When only one effective reasoning path remains after each selection, it is identified as the preferred path scheme. When multiple effective reasoning paths still exist after each selection, the effective reasoning path with the fewest semantic nodes is selected as the preferred path scheme.

[0095] S4.3. Through semantic parsing, extract the event status information, correlation information, abnormal location description information and change trend information corresponding to each effective reasoning path in the preferred path scheme, and generate a path semantic description sequence.

[0096] It should be noted that, for the effective reasoning path determined in the preferred path scheme, the semantic anchor identifiers and object pointing information, state pointing information, behavior pointing information, and time pointing information corresponding to each semantic node are read one by one according to the arrangement order of the semantic nodes; based on the state pointing information in the semantic anchor identifiers, the state values ​​corresponding to each semantic node are extracted and organized in chronological order to form an event state information sequence; based on the node connection relationship and its relationship type identifier between adjacent semantic nodes, the association method and connection order between nodes are extracted to form an association relationship information sequence; based on the connection interruption position and semantic conflict position recorded in the path evaluation index sequence... The process involves identifying time anomaly locations and historical matching locations, matching each semantic node to extract corresponding semantic nodes and their relationships, and generating an anomaly location description information sequence by combining semantic anchor point identifiers. Based on the arrangement order of semantic nodes on the time axis, the state values ​​of adjacent semantic nodes are compared one by one to identify the direction and magnitude of state changes, and then organized in chronological order to form a change trend information sequence. The event state information sequence, relationship information sequence, anomaly location description information sequence, and change trend information sequence are combined and organized according to their corresponding positions so that each time location corresponds to complete semantic description information, generating a path semantic description sequence.

[0097] S4.4 Perform association organization and structural rearrangement on the path semantic description sequence, and perform consistency adjustment according to the time evolution order to generate the path semantic representation sequence.

[0098] It should be noted that, for the semantic description content such as event state information, correlation information, abnormal position description information, and change trend information corresponding to each time position in the path semantic description sequence, the semantic association is organized according to the sequential dependency of semantic association. The semantic description content describing the same object is merged into the same semantic branch, and the semantic description content describing different objects but having correlation relationships is connected across branches according to the correlation direction to form a path semantic association structure. The semantic description content in the path semantic description sequence is structurally rearranged according to the unified expression order of "event state - correlation relationship - abnormal position description - change trend" to ensure that the semantic description content corresponding to the same time position is consistent in arrangement. The semantic description content between the time positions is consistently adjusted according to the time evolution order. The object state and correlation relationship that have appeared in the previous time position and continue in the next time position are inherited and retained. The object state, correlation relationship, abnormal position description, and change trend that newly appear in the next time position are supplemented and registered. The semantic description content with expression conflicts between the time positions is uniformly adjusted according to the rule that the later time covers the earlier time and the explicit description covers the implicit description, thus generating a path semantic representation sequence.

[0099] S5. Perform deviation inversion analysis on the path semantic representation sequence to generate a feedback deviation record set. Based on the feedback deviation record set, correct the corresponding path decision rule parameters in the path semantic representation sequence through deviation mapping to generate a processing rule set.

[0100] S5.1 Obtain the status feedback data, event response data, and result confirmation data during the electronic information execution process corresponding to the preferred path scheme, and perform time alignment and semantic correspondence processing to generate an execution feedback sequence.

[0101] It should be noted that, for the electronic information processing process corresponding to the preferred path scheme, status feedback data, event response data, and result confirmation data generated during the execution process are collected according to a unified time base. Status feedback data characterizes the status changes of each processing object during execution, event response data characterizes the execution triggers and response results corresponding to each processing behavior, and result confirmation data characterizes the completion status and result identifiers of each processing objective. The original timestamps in each type of data are mapped using a unified time base, aligning data from different sources according to a unified time axis to ensure that each time position corresponds to a complete data record. After time alignment, object identifiers, status values, and behavioral events are extracted from the status feedback data, event response data, and result confirmation data, and matched one by one with the object pointing information, status pointing information, and behavioral pointing information corresponding to the semantic nodes in the preferred path scheme. Data items with consistent semantic pointing or established correspondences are associated and organized and arranged in chronological order, ensuring that each time position corresponds to a combination of status feedback, event response, and result confirmation data, generating an execution feedback sequence.

[0102] S5.2 Match the path semantic representation sequence with the execution feedback sequence according to time position and semantic direction to generate a semantic comparison sequence.

[0103] It should be noted that the object pointing information in the path semantic representation sequence is compared one by one with the object identifiers in the execution feedback sequence. Data items with the same object identifier or those that correspond after name mapping are established as object correspondences. After establishing the object correspondences, the state values ​​in the path semantic representation sequence are compared with the state feedback values ​​in the execution feedback sequence. Data items with the same state value are marked as consistent, and data items with different state values ​​are recorded as state differences. At the same time, the behavior pointing information in the path semantic representation sequence is compared with the event response results in the execution feedback sequence. When the event response result is consistent with the behavior pointing information, the corresponding data item is marked as consistent, and when the event response result is inconsistent with the behavior pointing information, the corresponding data item is recorded as a behavior difference. After completing the item-by-item correspondence comparison of objects, states, and behaviors, the path semantic description information and execution feedback data at the same time position are combined and organized according to the correspondence, and the state difference and behavior difference information are recorded in the combination result to generate a semantic comparison sequence.

[0104] S5.3 Based on the semantic comparison sequence, reversely analyze the difference between the path semantic representation sequence and the execution feedback sequence, and perform attribution analysis on the source of the difference to generate a feedback deviation record set.

[0105] It should be noted that, for the state and behavioral differences at each time point in the semantic comparison sequence, the difference markers at each time point are read one by one in a unified chronological order, and the data items with differences are determined as the starting point for difference analysis. Starting from the starting point of difference analysis, the corresponding semantic nodes and relationships are traced back step by step in reverse chronological order, and the object pointing information, state pointing information, and behavioral pointing information at the corresponding time point are read. Combined with the relationship connection in the path semantic representation sequence, the semantic evolution process before the difference occurs is traced and analyzed. During the backtracking process, when a change in the state value at a certain time point is detected to have a direct correlation with the difference results at subsequent time points, the analysis is performed. When there is a discrepancy between the correlation relationship, or when the information pointing to a certain behavior is inconsistent with the event response result in the execution feedback, and this inconsistency continues to have an impact at subsequent time points, the corresponding time point is identified as the source of the discrepancy. The semantic nodes corresponding to the source of the discrepancy are classified and categorized. When the source of the discrepancy is caused by a change in state value, it is marked as a source of state deviation. When the source of the discrepancy is caused by an inconsistency in behavior execution, it is marked as a source of behavior deviation. When the source of the discrepancy is caused by an error in the correspondence between objects, it is marked as a source of object deviation. The identified source of the discrepancy, the corresponding semantic node information, the type of discrepancy, and its propagation range in the path are summarized and organized to generate a feedback deviation record set.

[0106] S5.4 Extract the path selection weight parameters, semantic relationship influence parameters, and time evolution constraint parameters corresponding to each effective reasoning path in the path semantic representation sequence, and perform parameterized organization processing according to the semantic position order to generate a path decision rule parameter sequence.

[0107] It should be noted that, according to the arrangement order of semantic nodes, the event state information, correlation information, abnormal position description information, and change trend information of each effective reasoning path in the path semantic representation sequence at each time position are read one by one; the difference in state value between adjacent time positions is obtained one by one, and the state change amplitude corresponding to each semantic node is calculated. When the state change amplitude of a certain semantic node in consecutive time positions is not greater than the maximum value of the change amplitude of its adjacent time positions, the average value of the state change amplitude of the corresponding semantic node position in that consecutive time position is taken as the path selection weight parameter; for each correlation in the path semantic representation sequence, the number of times it appears in the effective reasoning path and the number of semantic nodes involved in the connection are counted, and the product of the number of occurrences of the correlation and the number of connected semantic nodes is taken as the semantic relationship influence parameter; for the state change direction in the path semantic representation sequence, the time segments in which the state change direction is consistent in consecutive time positions are divided according to the time order, and the number of time positions corresponding to each time segment is taken as the time evolution constraint parameter; according to the time order of semantic nodes in the path, the path selection weight parameter, semantic relationship influence parameter, and time evolution constraint parameter are aligned and organized at the corresponding semantic positions to generate the path decision rule parameter sequence.

[0108] S5.5 Based on the feedback deviation record set, extract the path selection weight parameters, semantic relationship influence parameters and time evolution constraint parameters corresponding to the path decision rule parameter sequence, and perform deviation-driven adjustment processing to generate a processing rule sequence.

[0109] It should be noted that, for each source of difference and its corresponding type of deviation in the feedback deviation record set, the corresponding semantic node positions and propagation ranges are read one by one in a unified chronological order. The corresponding path selection weight parameters, semantic relationship influence parameters, and temporal evolution constraint parameters are then located in the path decision rule parameter sequence. Based on the deviation type, targeted adjustments are made to the corresponding parameters to ensure that the corresponding path selection weight parameter is not greater than the maximum value of the path selection weight parameter at its adjacent time position, and that the influence parameter corresponding to the association relationship is not greater than the maximum value of the influence parameters of other related relationships connected to it. When the source of difference is an object deviation, the temporal evolution constraint parameters of the corresponding time segment are adjusted so that the time segment is no longer considered a continuous constraint segment. The adjusted path selection weight parameters, semantic relationship influence parameters, and temporal evolution constraint parameters are reorganized according to the chronological order of the semantic nodes in the path, so that each semantic position corresponds to the updated parameter values, generating a processing rule sequence.

[0110] In summary, this invention achieves the transformation of multimodal data from scattered representations to a unified semantic structure by using semantic anchors as semantic nodes and performing path concatenation and structured mapping to form a semantic relationship table. This enables object orientation, behavioral logic, and temporal evolution relationships to be organized in an explicit structural form, improving the interpretability and structural consistency of the reasoning process. Furthermore, by performing difference-based reverse analysis based on semantic comparison sequences and combining feedback deviation records for deviation-driven adjustment, this invention enables the source location and propagation path tracing of differences between reasoning results and actual execution results. This allows processing rules to be dynamically updated with execution feedback, enhancing the adaptability and long-term stability of electronic information processing methods.

[0111] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for intelligent processing of electronic information based on multimodal data fusion, characterized in that, include: Collect multimodal electronic information data and perform cross-modal consistency constraint processing to generate a multimodal standard dataset; The following steps are taken to calculate the modality participation evaluation index of the multimodal standard dataset and prioritize it, generate modality combination sequences, extract semantic anchors from the modality combination sequences and perform cross-modal association organization processing to form a semantic relationship table: Extract semantic anchors representing object orientation, state orientation, behavior orientation, and time orientation from each modal data of the modal combination sequence to form a semantic anchor sequence; The semantic anchors of different modal data in the semantic anchor sequence are matched according to temporal adjacency, semantic consistency, and object correspondence to generate a cross-modal anchor correspondence table. Perform path concatenation and hierarchical organization processing on semantic anchors with corresponding relationships in the cross-modal anchor correspondence table to generate semantic association chains; Semantic anchors are used as semantic nodes, and the relationships in the semantic association chain are used as node connection relationships. Node-based structured mapping is then performed to generate a semantic relationship table. Constraint filtering is performed by analyzing the association connectivity and temporal consistency of the semantic relationship table to output a candidate inference path set. Error inference path records from the historical processing are read and compared with the candidate inference path set to generate a valid inference path set. The specific steps are as follows: Read the records of erroneous reasoning paths in the historical processing, and perform unified path structure encoding processing on each historical erroneous reasoning path and candidate reasoning path according to the semantic anchor order, the connection of the relationship and the time order to generate a path structure encoding sequence; The differences in semantic anchor arrangement, relational connection and temporal distribution of each historical erroneous reasoning path and candidate reasoning path in the path structure encoding sequence are analyzed and suppressed and filtered to generate a set of effective reasoning paths. Perform multi-dimensional path evaluation processing on the effective reasoning path set to generate the optimal path scheme, extract the path semantic features of the optimal path scheme and reconstruct the semantic structure to generate a path semantic representation sequence; A deviation inversion analysis is performed on the path semantic representation sequence to generate a feedback deviation record set. Based on the feedback deviation record set, the corresponding path decision rule parameters in the path semantic representation sequence are corrected through deviation mapping to generate a processing rule set.

2. The electronic information intelligent processing method based on multimodal data fusion as described in claim 1, characterized in that, The specific steps for generating the multimodal standard dataset are as follows: Multimodal electronic information data is processed by time-series mapping according to a unified time base to establish the time correspondence between different modal data and generate cross-modal time-aligned sequences. The semantic identifiers of each modality in the cross-modal time-aligned sequence are parsed, and consistency constraint matching is performed to generate a multimodal standard dataset.

3. The electronic information intelligent processing method based on multimodal data fusion as described in claim 2, characterized in that, The specific steps for generating the modal combination sequence are as follows: Calculate the current validity value, historical credibility value, and task relevance value of each modality in the multimodal specification data according to a unified time segment; Perform joint mapping and priority sorting on the current validity value, historical credibility value, and task relevance value to generate a modality combination sequence.

4. The electronic information intelligent processing method based on multimodal data fusion as described in claim 1, characterized in that, The specific steps for generating the candidate inference path set are as follows: Perform association connectivity analysis on each semantic node and its corresponding node connection relationship in the semantic relation table to generate a semantic connectivity relation table; Semantic nodes with reachable relationships in the semantic connectivity table are sequentially linked according to the direction of association extension, and temporal consistency is checked according to a unified time order to output a set of candidate reasoning paths.

5. The electronic information intelligent processing method based on multimodal data fusion as described in claim 1, characterized in that, The specific steps for generating the preferred path scheme are as follows: The path integrity, semantic consistency, temporal coherence, and historical adaptability of each valid reasoning path in the valid reasoning path set are evaluated separately, and a path evaluation index sequence is generated. Based on the path evaluation index sequence, a joint mapping and sorting process is performed on each effective inference path in the effective inference path set to generate the optimal path scheme.

6. The electronic information intelligent processing method based on multimodal data fusion as described in claim 1, characterized in that, The specific steps for generating the path semantic representation sequence are as follows: Through semantic parsing, the event status information, correlation information, abnormal location description information and change trend information corresponding to each effective reasoning path in the preferred path scheme are extracted to generate a path semantic description sequence. The path semantic description sequence is associated, reorganized, and rearranged, and then consistency adjustments are performed according to the time evolution order to generate a path semantic representation sequence.

7. The electronic information intelligent processing method based on multimodal data fusion as described in claim 5 or 6, characterized in that, The specific steps for generating the feedback deviation record set are as follows: Acquire the status feedback data, event response data, and result confirmation data during the electronic information execution process corresponding to the preferred path scheme, and perform time alignment and semantic correspondence processing to generate an execution feedback sequence; The path semantic representation sequence and the execution feedback sequence are matched according to their time position and semantic direction to generate a semantic comparison sequence; Based on the semantic comparison sequence, the difference between the path semantic representation sequence and the execution feedback sequence is reversed and analyzed for the source of the difference, generating a feedback deviation record set.

8. The electronic information intelligent processing method based on multimodal data fusion as described in claim 7, characterized in that, The specific steps for generating the processing rule set are as follows: Extract the path selection weight parameters, semantic relationship influence parameters, and time evolution constraint parameters corresponding to each effective reasoning path in the path semantic representation sequence, and perform parameterized organization processing according to the semantic position order to generate a path decision rule parameter sequence; Based on the feedback deviation record set, the path selection weight parameters, semantic relationship influence parameters, and time evolution constraint parameters corresponding to the path decision rule parameter sequence are extracted, and deviation-driven adjustment processing is performed to generate a processing rule sequence.

Images