A structured data intelligent extraction method based on multi-modal feature fusion

Abstract

The application discloses a kind of structured data intelligent extraction method based on multi-modal feature fusion, method includes multi-modal data acquisition, data preliminary processing, multi-modal fusion alignment, data analysis model construction and obtains event structured representation.The application relates to the technical field of public safety event structured data intelligent extraction, specifically refers to a kind of structured data intelligent extraction method based on multi-modal feature fusion, the original data is obtained by data acquisition;Using joint semantic field design combined with memory prototype retrieval mechanism, solve the semantic gap problem between heterogeneous multi-source data, at the same time, the stability of the model under the condition of partial modal loss or data noise interference is improved;Cascade residual decoder model is used as data analysis model, the overall consistency and accuracy of the final structured output result are improved by explicitly modeling the dependency between tasks.

Description

Technical Field

[0001] This invention relates to the field of intelligent extraction technology of structured data from public safety incidents, specifically to an intelligent extraction method for structured data based on multimodal feature fusion. Background Technology

[0002] The structured data intelligent extraction method based on multimodal feature fusion refers to the automated intelligent extraction and processing of structured data by integrating data features from different modalities and utilizing advanced algorithms to achieve a structured representation of complex public safety events. This method can improve the comprehensiveness and accuracy of data processing, uncover potential correlations and deeper information between data points, and automatically output structured information containing key elements such as event type, time, and location, thereby enhancing the speed and efficiency of event response in the public safety field.

[0003] However, traditional intelligent extraction methods for structured data suffer from several problems. Firstly, they lack the ability to model complex nonlinear relationships between modalities, failing to achieve true semantic alignment, resulting in poor information fusion and a high risk of information redundancy. Secondly, they lack the ability to leverage collective wisdom and historical experience, exhibiting poor generalization ability for novel patterns unseen in training data or scenarios where some modalities are missing, leading to unstable performance in real-world complex environments. Thirdly, traditional intelligent extraction methods ignore the strong inherent logical connections between output fields, potentially causing prediction results to violate common sense. Furthermore, independent prediction heads cannot share and utilize each other's decoded information, easily leading to contradictory structured outputs due to independent error accumulation. Summary of the Invention

[0004] To address the above issues and overcome the shortcomings of existing technologies, this invention provides a structured data intelligent extraction method based on multimodal feature fusion. Traditional structured data intelligent extraction methods suffer from several drawbacks: a lack of ability to model complex nonlinear relationships between modalities, inability to achieve true semantic alignment leading to poor information fusion results, easy information redundancy, and a lack of ability to leverage collective wisdom and historical experience. Furthermore, they exhibit poor generalization ability to novel patterns unseen in training data or scenarios lacking certain modalities, resulting in unstable performance in real-world complex environments. This solution creatively employs a joint semantic field design combined with a memory prototype retrieval mechanism to map information from different modalities into a common semantic space for weighted fusion, thus resolving the semantic barriers between heterogeneous multi-source data. This solution addresses the problem of introducing prior information retrieval, which dynamically references historical experience data to supplement and enhance the representation of current events, improving the model's stability under partial modality loss or data noise interference. It also addresses the technical issues of traditional intelligent extraction methods for structured data, which ignore the strong inherent logical relationships between output fields, leading to prediction results that may violate common sense, and the inability of independent prediction heads to share and utilize each other's decoded information, resulting in contradictory structured outputs due to independent error accumulation. This solution creatively adopts a cascaded residual decoder model as the data analysis model. By explicitly modeling the dependencies between tasks, it avoids information conflicts and logical inconsistencies that may arise from parallel decoding, thereby improving the overall consistency and accuracy of the final structured output.

[0005] The technical solution adopted by this invention is as follows: This invention provides a structured data intelligent extraction method based on multimodal feature fusion, which includes the following steps:

[0006] Step S1: Multimodal data acquisition;

[0007] Step S2: Preliminary data processing;

[0008] Step S3: Multimodal fusion and alignment;

[0009] Step S4: Data analysis model construction;

[0010] Step S5: Obtain the structured representation of the event.

[0011] Further, in step S1, the multimodal data acquisition is used to collect the raw data required to realize the structured representation of public safety events. Specifically, through data acquisition, an event extraction raw dataset is obtained. The event extraction raw dataset specifically includes a past event extraction raw dataset and a real-time event extraction raw dataset. Both the past event extraction raw dataset and the real-time event extraction raw dataset contain event text modal data, event video modal data, event image modal data, event table modal data, event geographic location data, and event environmental data. The past event extraction raw dataset also contains public safety event type labels and public safety event occurrence scenario type labels.

[0012] Further, in step S2, the preliminary data processing is used to perform preliminary processing on the collected raw data, specifically including the following steps:

[0013] Step S21: Text modality processing, used to extract core semantic representations related to the event from unstructured text information. Specifically, it obtains deep word embeddings through the BERT model and uses a self-attention weighted pooling mechanism to highlight the role of keywords and suppress irrelevant information to obtain text modality features.

[0014] Step S22: Video modality processing, used to extract the most representative spatiotemporal features of the event content from the long video stream. Specifically, it reduces redundancy by sampling keyframes, captures spatiotemporal information using a 3D convolutional network, and integrates the global temporal context through a temporal segmentation network to obtain video modality features.

[0015] Step S23: Image modality processing, used to extract global scene information and local salient detail information from event-related images. Specifically, it obtains the global representation of the image through the ViT model, and uses class activation mapping technology to locate key regions and extract their local features. The global representation and local features are then weighted and fused to obtain image modality features.

[0016] Step S24: Table modality processing, used to mine the row and column relationships within structured table data and extract effective feature representations. Specifically, the original data is cleaned and normalized, and each column of data is encoded using a Transformer encoder. Then, the dependencies between columns are captured through a cross-column attention mechanism to obtain table modality features.

[0017] Step S25: Location and environment data processing, which is used to transform numerical latitude and longitude coordinates and environmental parameters into high-dimensional features with semantic information. Specifically, coordinates are converted into semantic features through geocoding, environmental parameters are standardized, and the two are concatenated and fused and projected through a learnable embedding layer to obtain spatiotemporal background features.

[0018] The event extraction of the past original dataset undergoes preliminary data processing and dataset segmentation to obtain a fused training set and a fused test set. The event extraction of the real-time original dataset undergoes preliminary data processing to obtain a real-time dataset to be fused.

[0019] Furthermore, in step S3, the multimodal fusion alignment is used to achieve semantic alignment and fusion between multimodal features. Specifically, it involves designing a joint semantic field to uniformly represent the information of each modality, and combining it with a memory prototype retrieval mechanism to obtain multimodal fusion features.

[0020] The multimodal fusion alignment specifically includes the following steps:

[0021] Step S31: Joint semantic field construction, used to create a unified mathematical framework for representing and fusing multimodal information, includes the following steps:

[0022] Step S311: Forward propagation of the field generation network is used to convert the features of each mode into the parameters of the corresponding Gaussian field subfield. Specifically, the field generation network corresponding to each mode maps the features of each mode to the set of parameters of the Gaussian field subfield, thus obtaining the set of parameters of each mode subfield.

[0023] Step S312: Subfield morphology instantiation, used to instantiate the specific influence form of each modality in the semantic space based on parameters. Specifically, it calculates the spatial distribution of each modality in the semantic space based on the parameter set of each modality subfield using a Gaussian function, and obtains the Gaussian subfield of each modality.

[0024] Step S313: Joint field superposition, used to fuse multiple modal Gaussian subfields into a unified joint field. Specifically, it involves superimposing all modal Gaussian subfields together by weighted summation to obtain a multimodal joint Gaussian field.

[0025] Step S32: Memory Prototype Retrieval, used to retrieve the information fragment most relevant to the current multimodal joint Gaussian field from prior knowledge to enhance and complete the multimodal fusion representation. The steps include:

[0026] Step S321: Memory bank construction, specifically, using CNN as the projection network, extracting all modal features from the past original dataset after the initial data processing and mapping them to the same semantic space, and using K-means++ clustering algorithm to perform clustering operation, the obtained cluster centers are used as prototypes to form the memory bank;

[0027] Step S322: Memory bank query, used to evaluate the correlation between the multimodal joint Gaussian field and the prototypes in the memory bank. Specifically, by substituting each prototype in the memory bank into the multimodal joint Gaussian field function, the calculated function value is used as the degree of correlation between the corresponding prototype and the multimodal joint Gaussian field.

[0028] Step S323: Prototype aggregation, used to fuse prototype information into an aggregated representation, specifically, to obtain prototype aggregation weights based on the correlation between the prototype and the multimodal joint Gaussian field, and to perform a weighted summation of the prototypes in the memory bank to obtain prototype aggregation features;

[0029] Step S33: Adaptive fusion generation, used to integrate multimodal joint Gaussian field information with prototype aggregated features, includes the following steps:

[0030] Step S331: Global field feature extraction, used to extract a fixed-size global feature from the field function. Specifically, it obtains the global field feature through an independent multilayer perceptron combined with a multi-head self-attention mechanism, based on a multimodal joint Gaussian field parameter set.

[0031] Step S332: Information fusion, used to fuse global information with prototype aggregated features, specifically by concatenating global field features with prototype aggregated features and obtaining multimodal fusion features after linear mapping;

[0032] Step S34: Modality missing simulation training, used to simulate the scenario of modality missing in real scene. Specifically, during the training phase, the input features of random modalities are set to zero, forcing the model to make predictions and inferences based on the remaining modalities.

[0033] Step S35: Multimodal fusion, specifically, involves integrating the joint semantic field construction, memory prototype retrieval, adaptive fusion generation, and modality missing simulation training into a multimodal fusion model. The model is trained and its performance is verified based on the fusion training set and the fusion test set. The real-time dataset to be fused, the fusion training set, and the fusion test set are used as inputs to the multimodal fusion model for multimodal fusion alignment to obtain the real-time dataset to be extracted, the extraction training set, and the extraction test set.

[0034] Furthermore, in step S4, the data analysis model construction is used to construct the model required to realize the structured representation of public safety events, specifically by constructing a cascaded residual decoder model as the data analysis model;

[0035] The construction of the data analysis model specifically includes the following steps:

[0036] Step S41: Shared base decoding, used to generate a common base hidden state for all specific decoding heads. Specifically, the multimodal fusion features are linearly projected twice and a gating mechanism is applied to obtain the base hidden state.

[0037] Step S42: Cascaded residual decoding, used for step-by-step reasoning, to assist subsequent decoding based on the decoded results;

[0038] Step S43: Obtain structured output, specifically by post-processing the output results of all output headers to obtain the final structured representation of the public safety event;

[0039] Step S44: Construct and train the model. Specifically, the cascaded residual decoder model is constructed by integrating the shared basic decoding, the cascaded residual decoding, and the acquisition of structured output. The model is trained and its performance is verified based on the extracted training set and the extracted test set to obtain the cascaded residual decoder model, which serves as the data analysis model.

[0040] Furthermore, in step S5, obtaining the structured representation of the event specifically involves using the real-time dataset to be extracted as input to the data analysis model, performing real-time structured data extraction of public safety events, and obtaining the structured representation result of the public safety events output by the data analysis model.

[0041] The beneficial effects achieved by adopting the above solution are as follows:

[0042] (1) In view of the technical problems of traditional structured data intelligent extraction methods, such as lack of modeling ability for complex nonlinear relationships between modalities, inability to achieve true semantic alignment, poor information fusion effect, easy formation of information redundancy, lack of ability to utilize collective wisdom and historical experience, poor generalization ability for novel patterns or scenarios with missing modalities in training data, resulting in unstable performance in real complex environments, this solution creatively adopts joint semantic field design combined with memory prototype retrieval mechanism to map information of different modalities to a common semantic space for weighted fusion, solve the semantic gap problem between heterogeneous multi-source data, and introduce prior information retrieval, which can dynamically refer to historical experience data to complete and enhance the representation of current events, and improve the stability of the model under partial modal missingness or data noise interference.

[0043] (2) In view of the technical problems of traditional intelligent extraction methods of structured data, which ignore the strong internal logical relationship between output fields, resulting in prediction results that may violate common sense, and independent prediction heads cannot share and utilize the information they have decoded, and are prone to generating contradictory structured outputs due to independent accumulation of errors, this solution creatively adopts a cascaded residual decoder model as a data analysis model. By explicitly modeling the dependency relationship between tasks, it avoids information conflicts and logical inconsistencies that may be generated by parallel decoding, thereby improving the overall consistency and accuracy of the final structured output results. Attached Figure Description

[0044] Figure 1 A flowchart illustrating a structured data intelligent extraction method based on multimodal feature fusion provided by this invention;

[0045] Figure 2 This is a flowchart illustrating the preliminary data processing in step S2.

[0046] Figure 3 This is a flowchart illustrating the process of multimodal fusion and alignment in step S3.

[0047] Figure 4 A flowchart illustrating the process of building the data analysis model for step S4.

[0048] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. Detailed Implementation

[0049] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0050] In the description of this invention, it should be understood that the terms "upper", "lower", "front", "rear", "left", "right", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0051] Example 1, see Figure 1 This invention provides a structured data intelligent extraction method based on multimodal feature fusion, which includes the following steps:

[0052] Step S1: Multimodal data acquisition;

[0053] Step S2: Preliminary data processing;

[0054] Step S3: Multimodal fusion and alignment;

[0055] Step S4: Data analysis model construction;

[0056] Step S5: Obtain the structured representation of the event.

[0057] Example 2, see Figure 1This embodiment is based on the above embodiment. In step S1, the multimodal data acquisition is used to collect the raw data required to realize the structured representation of public safety events. Specifically, through data acquisition, an event extraction raw dataset is obtained. The event extraction raw dataset specifically includes a past event extraction raw dataset and a real-time event extraction raw dataset. Both the past event extraction raw dataset and the real-time event extraction raw dataset contain event text modal data, event video modal data, event image modal data, event table modal data, event geographic location data, and event environmental data. The past event extraction raw dataset also contains public safety event type labels and public safety event occurrence scenario type labels.

[0058] The event text modal data specifically includes publicly released notification text data from relevant departments, public feedback text data obtained through questionnaires, and publicly reported text data from news media.

[0059] The event video modal data specifically includes surveillance video data of the event-related area and video data taken by the public at the scene;

[0060] The event image modal data specifically includes panoramic image data of the event area and close-up image data of the event area;

[0061] The event table modal data specifically refers to the event statistics table data from relevant departments.

[0062] The event geographic location data specifically includes event area geographic coordinate data and event area environmental GIS data;

[0063] The event environment data specifically includes meteorological data of the event area and environmental sensor data of the event area.

[0064] Example 3, see Figure 1 and Figure 2 This embodiment is based on the above embodiment. In step S2, the preliminary data processing is used to perform preliminary processing on the collected raw data, specifically including the following steps:

[0065] Step S21: Text modality processing, used to extract core semantic representations related to the event from unstructured text information. Specifically, it obtains deep word embeddings through the BERT model and uses a self-attention weighted pooling mechanism to highlight the role of keywords and suppress irrelevant information to obtain text modality features.

[0066] Step S22: Video modality processing, used to extract the most representative spatiotemporal features of the event content from the long video stream. Specifically, it reduces redundancy by sampling keyframes, captures spatiotemporal information using a 3D convolutional network, and integrates the global temporal context through a temporal segmentation network to obtain video modality features.

[0067] Step S23: Image modality processing, used to extract global scene information and local salient detail information from event-related images. Specifically, it obtains the global representation of the image through the ViT model, and uses class activation mapping technology to locate key regions and extract their local features. The global representation and local features are then weighted and fused to obtain image modality features.

[0068] Step S24: Table modality processing, used to mine the row and column relationships within structured table data and extract effective feature representations. Specifically, the original data is cleaned and normalized, and each column of data is encoded using a Transformer encoder. Then, the dependencies between columns are captured through a cross-column attention mechanism to obtain table modality features.

[0069] Step S25: Location and environment data processing, which is used to transform numerical latitude and longitude coordinates and environmental parameters into high-dimensional features with semantic information. Specifically, coordinates are converted into semantic features through geocoding, environmental parameters are standardized, and the two are concatenated and fused and projected through a learnable embedding layer to obtain spatiotemporal background features.

[0070] The event extraction of the past original dataset undergoes preliminary data processing and dataset segmentation to obtain a fused training set and a fused test set. The event extraction of the real-time original dataset undergoes preliminary data processing to obtain a real-time dataset to be fused.

[0071] Example 4, see Figure 1 and Figure 3 This embodiment is based on the above embodiment. In step S3, the multimodal fusion alignment is used to realize the semantic alignment and fusion between multimodal features. Specifically, it is to design a joint semantic field to uniformly represent the information of each modality, and combine the memory prototype retrieval mechanism to obtain multimodal fusion features.

[0072] The multimodal fusion alignment specifically includes the following steps:

[0073] Step S31: Joint semantic field construction, used to create a unified mathematical framework for representing and fusing multimodal information, includes the following steps:

[0074] Step S311: Forward propagation of the field generation network is used to convert the features of each modality into the parameters of the corresponding Gaussian field subfield. Specifically, the field generation network corresponding to each modality maps the features of each modality to a set of Gaussian field subfield parameters, thus obtaining the parameter set of each modality subfield. The field generation network is specifically a multilayer perceptron. The forward propagation of the field generation network uses the following formula:

[0075] ;

[0076] In the formula, This represents the set of subfield parameters for mode a. This represents the function that generates the network for the field corresponding to mode a. Let 'a' represent the modality feature, 'a' represent the modality index, 'Text' represent the text modality, 'Video' represent the video modality, 'Image' represent the image modality, and 'Table' represent the table modality.

[0077] Step S312: Subfield morphology instantiation, used to instantiate the specific influence form of each modality in the semantic space based on parameters. Specifically, it calculates the spatial distribution of each modality in the semantic space based on the parameter set of each modality subfield using a Gaussian function, and obtains the Gaussian subfield of each modality.

[0078] Step S313: Joint field superposition, used to fuse multiple modal Gaussian subfields into a unified joint field. Specifically, this involves superimposing all modal Gaussian subfields together using a weighted summation method to obtain a multimodal joint Gaussian field. The superposition weights of each modal Gaussian subfield are obtained by an independent multilayer perceptron based on the parameter sets of each modal subfield. The formula used for joint field superposition is as follows:

[0079] ;

[0080] In the formula, This represents the superposition weights of the Gaussian subfields of mode a. This represents the running function of the multilayer perceptron used to obtain the superposition weights of the Gaussian subfields of each modality. Represents a multimodal joint Gaussian field. Represents the Gaussian subfield of mode a;

[0081] Step S32: Memory Prototype Retrieval, used to retrieve the information fragment most relevant to the current multimodal joint Gaussian field from prior knowledge to enhance and complete the multimodal fusion representation. The steps include:

[0082] Step S321: Memory bank construction, specifically, using CNN as the projection network, extracting all modal features from the past original dataset after the initial data processing and mapping them to the same semantic space, and using K-means++ clustering algorithm to perform clustering operation, the obtained cluster centers are used as prototypes to form the memory bank;

[0083] Step S322: Memory bank query, used to evaluate the correlation between the multimodal joint Gaussian field and the prototypes in the memory bank. Specifically, this involves substituting each prototype in the memory bank into the multimodal joint Gaussian field function, and calculating the function value as the degree of correlation between the corresponding prototype and the multimodal joint Gaussian field. The formula used is as follows:

[0084] ;

[0085] In the formula, This indicates the degree of correlation between the b-th prototype in the memory bank and the multimodal joint Gaussian field. This represents the multimodal joint Gaussian field function. This represents the b-th prototype in the memory bank;

[0086] Step S323: Prototype aggregation, used to fuse prototype information into an aggregated representation. Specifically, it obtains prototype aggregation weights based on the correlation between the prototype and the multimodal joint Gaussian field, and performs a weighted summation on the prototypes in the memory to obtain the prototype aggregation features. The formula used is as follows:

[0087] ;

[0088] In the formula, Sp represents the prototype aggregation weight, and Sp represents the relevance vector. This represents the softmax function. Indicates prototype aggregation features;

[0089] Step S33: Adaptive fusion generation, used to integrate multimodal joint Gaussian field information with prototype aggregated features, includes the following steps:

[0090] Step S331: Global field feature extraction, used to extract a fixed-size global feature from the field function. Specifically, it uses an independent multilayer perceptron with a multi-head self-attention mechanism to obtain the global field feature based on the multimodal joint Gaussian field parameter set. The formula used is as follows:

[0091] ;

[0092] In the formula, Represents global field characteristics. This represents the operating function of a multilayer perceptron incorporating a multi-head self-attention mechanism. Represents the set of parameters of the multimodal joint Gaussian field;

[0093] Step S332: Information fusion, used to fuse global information with prototype aggregated features, specifically by concatenating global field features with prototype aggregated features and obtaining multimodal fusion features after linear mapping;

[0094] Step S34: Modality missing simulation training, used to simulate the scenario of modality missing in real scene. Specifically, during the training phase, the input features of random modalities are set to zero, forcing the model to make predictions and inferences based on the remaining modalities.

[0095] Step S35: Multimodal fusion, specifically, involves integrating the joint semantic field construction, memory prototype retrieval, adaptive fusion generation, and modality missing simulation training into a multimodal fusion model. The model is trained and its performance is verified based on the fusion training set and the fusion test set. The real-time dataset to be fused, the fusion training set, and the fusion test set are used as inputs to the multimodal fusion model for multimodal fusion alignment to obtain the real-time dataset to be extracted, the extraction training set, and the extraction test set.

[0096] By performing the above operations, this solution addresses the technical problems of traditional intelligent extraction methods for structured data. These problems include a lack of ability to model complex nonlinear relationships between modalities, an inability to achieve true semantic alignment, poor information fusion results, a high risk of information redundancy, a lack of ability to utilize collective wisdom and historical experience, poor generalization ability for novel patterns or scenarios with missing modalities in the training data, and unstable performance in real complex environments. This solution creatively adopts a joint semantic field design combined with a memory prototype retrieval mechanism to map information from different modalities to a common semantic space for weighted fusion, solving the semantic gap problem between heterogeneous multi-source data. At the same time, the introduction of prior information retrieval enables dynamic reference to historical experience data to complete and enhance the representation of current events, improving the stability of the model under conditions of missing modalities or data noise interference.

[0097] Example 5, see Figure 1 and Figure 4 This embodiment is based on the above embodiment. In step S4, the data analysis model is constructed to build the model required to realize the structured representation of public safety events. Specifically, it is to construct a cascaded residual decoder model as a data analysis model.

[0098] The construction of the data analysis model specifically includes the following steps:

[0099] Step S41: Shared base decoding, used to generate a common base hidden state for all specific decoder heads. Specifically, the multimodal fusion features are linearly projected twice and a gating mechanism is applied to obtain the base hidden state. The formula used is as follows:

[0100] ;

[0101] In the formula, This represents the features of the first linear projection. This represents the second linear projection feature. This represents the weights of the first linear projection. Indicates the weights of the second linear projection. This represents the bias term of the first linear projection. This represents the bias term of the second linear projection. Indicates multimodal fusion features, Indicates the basic hidden state. This represents the sigmoid function. This represents element-wise multiplication.

[0102] Step S42: Cascaded residual decoding, used for stepwise inference, assists subsequent decoding based on the decoded results. The steps include:

[0103] Step S421: Design of the main inference head, which is used to first decode the relatively independent and easily predictable public safety event time information. Specifically, the time decoding head is used as the main inference head. Based on the basic hidden state, spatiotemporal background features are embedded through residual connections to predict the public safety event time information. The time decoding head specifically includes two fully connected layers and one output layer.

[0104] Step S422: Event type header design, used to decode public safety event types. Specifically, the public safety event time information is embedded, combined with the basic hidden state through residual connection, and the public safety event type is predicted using the event type header. The event type header specifically includes two fully connected layers and one output layer.

[0105] Step S423: Event scene head design, used for finally decoding the public safety event occurrence scene type. Specifically, the public safety event time information and public safety event type are embedded, and combined with the basic hidden state through residual connection. The public safety event occurrence scene type is predicted by using the event scene head. The event scene head specifically includes two fully connected layers and one output layer.

[0106] Step S43: Obtain structured output, specifically by post-processing the output results of all output headers to obtain the final structured representation of the public safety event;

[0107] Step S44: Construct and train the model. Specifically, the cascaded residual decoder model is constructed by integrating the shared basic decoding, the cascaded residual decoding, and the acquisition of structured output. The model is trained and its performance is verified based on the extracted training set and the extracted test set to obtain the cascaded residual decoder model, which serves as the data analysis model.

[0108] By performing the above operations, this solution addresses the technical problems of traditional intelligent extraction methods for structured data, which ignore the strong inherent logical relationships between output fields, leading to prediction results that may violate common sense, and the inability of independent prediction heads to share and utilize each other's decoded information, resulting in contradictory structured outputs due to independent accumulation of errors. This solution creatively adopts a cascaded residual decoder model as the data analysis model. By explicitly modeling the dependencies between tasks, it avoids information conflicts and logical inconsistencies that may arise from parallel decoding, thereby improving the overall consistency and accuracy of the final structured output results.

[0109] Example 6, see Figure 1 This embodiment is based on the above embodiment. In step S5, obtaining the structured representation of the event specifically involves using the real-time dataset to be extracted as the input of the data analysis model, performing real-time structured data extraction of public safety events, and obtaining the structured representation result of the public safety events output by the data analysis model.

[0110] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0111] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention.

[0112] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the invention, such designs should fall within the protection scope of the present invention.

Claims

1.A method for intelligent extraction of structured data based on multi-modal feature fusion, characterized in that: The method comprises the following steps: Step S1: multi-modal data acquisition, by performing data acquisition, obtaining an event extraction original data set, the event extraction original data set specifically comprising an event extraction past original data set and an event extraction real-time original data set, the event extraction past original data set and the event extraction real-time original data set both containing event text modal data, event video modal data, event image modal data, event table modal data, event geographic location data and event environment data, the event extraction past original data set further containing a public security event type label and a public security event occurrence scene type label; Step S2: data preliminary processing, performing preliminary processing on the collected original data to obtain a real-time to-be-fused data set, a fusion training set and a fusion test set; Step S3: multi-modal fusion alignment, used for realizing semantic alignment and fusion between multi-modal features, specifically comprising the following steps: Step S31: joint semantic field construction; Step S32: memory prototype retrieval; Step S33: adaptive fusion generation; Step S34: modal missing simulation training; Step S35: multi-modal fusion. The joint semantic field construction is used for creating a unified mathematical framework to represent and fuse multi-modal information, and the steps comprise the following steps: 2.The method of claim 1, wherein: Step S311: field generation network forward propagation, used for converting the features of each modality into the parameters of the corresponding Gaussian field subfield, specifically comprising the following steps: Step S3111: field generation network forward propagation, used for converting the features of each modality into the parameters of the corresponding Gaussian field subfield, specifically comprising the following steps: ​ ​ ​ ​ 3.The method of claim 2, wherein: ​ ​ Step S312: sub-field mode instantiation, for instantiating the specific influence form of each mode in the semantic space based on the parameters, specifically, calculating the spatial distribution of each mode in the semantic space based on the parameter set of each mode sub-field to obtain the Gaussian sub-field of each mode through a Gaussian function; Step S313: joint field superposition, for fusing the Gaussian sub-fields of multiple modes into a unified joint field, specifically, superimposing the Gaussian sub-fields of all modes together through weighted summation to obtain a multi-modal joint Gaussian field; The memory prototype retrieval is used to retrieve the information segment most relevant to the current multi-modal joint Gaussian field from the prior knowledge to enhance and complete the multi-modal fusion representation, and the steps include: Step S321: memory bank construction, specifically, using CNN as a projection network to map all modal features in the past original data set to the same semantic space after preliminary processing of the event, and using the K-means++ clustering algorithm for clustering operation, and the obtained clustering center is used as a prototype to form a memory bank; Step S322: memory bank query, for evaluating the relevance of the multi-modal joint Gaussian field and the prototype in the memory bank, specifically, by substituting each prototype in the memory bank into the multi-modal joint Gaussian field function, the function value calculated is used as the relevance degree of the corresponding prototype and the multi-modal joint Gaussian field; Step S323: prototype aggregation, for fusing the prototype information into an aggregated representation, specifically, obtaining the prototype aggregation weight based on the relevance of the prototype and the multi-modal joint Gaussian field, and performing weighted summation on the prototypes in the memory bank to obtain the prototype aggregation feature; The adaptive fusion generation is used to integrate the multi-modal joint Gaussian field information and the prototype aggregation feature, and the steps include: Step S331: global field feature extraction, for extracting a fixed-size global feature from the field function, specifically, through a multi-layer perceptron combined with a multi-head self-attention mechanism, a global field feature is obtained based on the parameter set of the multi-modal joint Gaussian field; Step S332: information fusion, for fusing the global information and the prototype aggregation feature, specifically, the global field feature and the prototype aggregation feature are spliced, and after linear mapping, a multi-modal fusion feature is obtained; The modal missing simulation training is used to simulate the scene of modal missing in the real scene, specifically, in the training phase, the input features of the random modal are set to zero, forcing the model to make prediction and reasoning based on the remaining modal; The multi-modal fusion is specifically a multi-modal fusion model integrated by the joint semantic field construction, the memory prototype retrieval, the adaptive fusion generation and the modal missing simulation training, the model is trained based on the fusion training set and the fusion test set and the performance is verified, the real-time data set to be fused, the fusion training set and the fusion test set are used as the input of the multi-modal fusion model, the multi-modal fusion is aligned, and the real-time data set to be extracted, the extraction training set and the extraction test set are obtained. 4.The method of claim 1, wherein: The data preliminary processing specifically includes the following steps: Step S21: text modality processing, for extracting core semantic representation related to the event from unstructured text information, specifically, obtaining deep word embedding through a BERT model, and using a self-attention weighted pooling mechanism to highlight the role of key words and suppress irrelevant information to obtain text modality features; Step S22: video modality processing, for extracting compact spatiotemporal features that best represent the content of the event from lengthy video streams, specifically, reducing redundancy through key frame sampling, then using a 3D convolutional network to capture spatiotemporal information, and integrating global temporal context through a temporal segmentation network to obtain video modality features; Step S23: image modality processing, for extracting global scene information and local salient detail information from event-related images, specifically, obtaining global representation of the image through a ViT model, and using class activation mapping technology to locate key areas and extract local features, and then weighting and fusing global representation and local features to obtain image modality features; Step S24: table modality processing, for mining the internal row-column relationship of structured table data and extracting effective feature representation, specifically, processing raw data through data cleaning and normalization, and using a Transformer encoder to encode each column of data, then capturing the dependency between columns through a cross-column attention mechanism to obtain table modality features; Step S25: location and environmental data processing, for converting numerical latitude and longitude coordinates and environmental parameters into high-dimensional features with semantic information, specifically, converting coordinates into semantic features through geocoding, standardizing environmental parameters, and then concatenating the two and passing them through a learnable embedding layer for fusion and projection to obtain spatiotemporal context features; The event extraction past original dataset is subjected to the data preliminary processing, and after dataset segmentation, a fusion training set and a fusion test set are obtained, and the event extraction real-time original dataset is subjected to the data preliminary processing to obtain a real-time to-be-fused dataset.

Images