Event analysis method, system and device based on task generation and multiple modes

Abstract

The invention discloses an event analysis method, system and device based on task generation and multiple modes, which are used for quickly and accurately mastering the development trend of an event. The method comprises the steps that structured event information is generated through multi-modal data, and an event relation is obtained; constructing an event graph; extracting a multi-modal feature, and obtaining a multi-modal emotion relationship vector by using a cross-modal attention mechanism; based on a preset graph neural network, combining the event graph to obtain an emotion feature vector; calculating a comparative learning loss value, and obtaining a target emotion feature based on the comparative learning loss value; generating a target event atlas by using a cross-modal attention mechanism; extracting relation characteristics, and combining the attention weight and a preset time sequence model to obtain an event prediction map; extracting a causal chain and reasoning, and obtaining a trigger condition and an emotion factor after reasoning; and combining the event prediction map and the target event map with the node feature sequence, the trigger condition and the emotion factor to obtain an event analysis result.

Description

Technical Field

[0001] The present application relates to the field of computer data analysis, and in particular to a method, system and device for event analysis based on generation tasks and multimodality. Background Art

[0002] With the rise of the wave of smart city construction, the scale and complexity of cities are constantly expanding, and the governance demands for security, services and refined management are becoming increasingly strong. Urban event situation monitoring and analysis have become key supporting factors. All departments hope to combine new technologies to govern urban operations and respond to governance needs in a timely manner. However, driven by mobile Internet and Internet of Things technologies, urban event data presents complex characteristics such as polymorphism, heterogeneity, spatiotemporal and social characteristics, and the formats of urban event data are diverse, which brings huge challenges to data fusion analysis.

[0003] In the prior art, event analysis is usually achieved using feature fusion technology based on deep learning. First, the BERT model is used to extract the semantic features of text data, convert the text data into vector representation, capture the contextual information and semantic relationships in the text, and then a convolutional neural network is used to extract the visual features of the image, and an audio processing model is used to extract the acoustic features of the audio. After the extraction is completed, the extracted features of different modalities are fused, and the fused features are input into the regression model for analysis and prediction to obtain the event analysis results.

[0004] However, event analysis achieved by feature fusion technology based on deep learning only integrates the basic features of each modality. When faced with complex and changeable urban events, it is difficult to accurately grasp the direction and emotional dynamics of events, which in turn affects the scientificity and timeliness of urban governance decisions, resulting in missing the best time to deal with emergency events and expanding the scope of event impact. Summary of the invention

[0005] In order to solve the above technical problems, the present application provides a method, system and device for event analysis based on generation tasks and multimodality.

[0006] The technical solution provided in this application is described below: The first aspect of the present application provides an event analysis method based on generation tasks and multimodality, the method comprising: Acquiring multimodal data, wherein the multimodal data includes text data, image data, and audio data; A generation task is set based on a preset generation model, the multimodal data is input into the generation task, structured event information is generated, and relationship extraction and semantic analysis are performed on the structured event information to obtain event relationships; Constructing an event graph, taking the structured event information as nodes of the event graph, and connecting the nodes according to the event relationships; Extracting text features, image features, and audio features of the multimodal data, and fusing the text features, the image features, and the audio features using a cross-modal attention mechanism to obtain a multimodal sentiment relationship vector; Aggregating the multimodal sentiment relationship vector based on a preset graph neural network, and obtaining a sentiment feature vector in combination with the event graph; Constructing positive sample pairs and negative sample pairs according to the emotional feature vector and the multimodal data, calculating contrastive learning loss values ​​based on the positive sample pairs and the negative sample pairs using a preset contrast function, and aligning and optimizing the emotional feature vector and the multimodal data using the contrastive learning loss value to obtain a target emotional feature; The emotion feature vector is transmitted to the node of the event graph, and the attention weight of the emotion feature vector is adjusted by using the cross-modal attention mechanism to generate a target event graph; The preset graph neural network is used to extract the relationship features in the target event graph, and the node feature sequence is obtained by combining the attention weight, and the node feature sequence is input into the preset time series model to capture the dynamic evolution law, and the event prediction graph is obtained according to the dynamic evolution law; Extracting the causal chain in the event prediction graph, performing causal relationship reasoning based on the node feature sequence and the causal chain, and obtaining trigger conditions and emotional factors after reasoning; The event prediction graph is matched with the target event graph, and the event analysis result is calculated by combining the node feature sequence, the trigger condition and the emotional factor.

[0007] Optionally, extracting text features, image features, and audio features of the multimodal data, and fusing the text features, the image features, and the audio features using a cross-modal attention mechanism to obtain a multimodal sentiment relationship vector includes: Extracting text semantics and text sentiment from the multimodal data using a preset language macro model, and obtaining text features based on the text semantics and the text sentiment; Extracting image scenes and image expressions from the multimodal data using ResNet or Vision Transformer, and obtaining image features based on the image scenes and the image expressions; Extracting audio intonation, audio volume, and audio rhythm from the multimodal data using a Wav2Vec audio model, and obtaining audio features based on the audio intonation, the audio volume, and the audio rhythm; The target weights of the text features, the image features, and the audio features are calculated using a cross-modal attention mechanism, and the target weights are combined with the text features, the image features, and the audio features to obtain a multimodal sentiment relationship vector.

[0008] Optionally, the generating task is set based on the preset generating large model, the multimodal data is input into the generating task, and the generating structured event information comprises: Setting a generation task based on a preset generation model, inputting the multimodal data into the generation task for training, and obtaining a structured description; Under the preset parameters of the preset large model, the multimodal data and the corresponding structured description are combined into sample pairs, and the target probability of the structured event information generated by the input multimodal data is calculated based on the sample pairs; According to the target probability, the preset parameters are adjusted in combination with a cross entropy loss function, and structured event information is generated after the adjustment.

[0009] Optionally, the preset parameters are adjusted according to the target probability in combination with a cross entropy loss function, and structured event information is generated after the adjustment, which is represented by the following formula: ; in, Represents structured event information, represents the amount of the multimodal data, represents the target probability, represents the preset parameters, Indicates The multimodal data of the input; Indicates The structured description of the output.

[0010] Optionally, after matching the event prediction graph with the target event graph, combining the node feature sequence, the trigger condition and the emotional factor, and calculating the event analysis result, the method further includes: The data structure of the event analysis result is converted into a target data structure, and a preset visualization tool is used to generate a visualization analysis result from the converted event analysis result.

[0011] Optionally, the causal chain in the event prediction graph is extracted, and causal relationship reasoning is performed based on the node feature sequence and the causal chain. After reasoning, trigger conditions and emotional factors are obtained, which are expressed by the following formula: ; in, represents the trigger condition, Indicates the emotional factors, express The trigger condition corresponding to the time step, represents the node feature sequence, represents the weight matrix parameters, Represents the causal chain.

[0012] Optionally, the contrastive learning loss value is expressed by the following formula: ; in, represents the contrast loss value, represents the sentiment feature vector of the positive sample pair, Represents the sentiment feature vector of the negative sample pair, Represents adjustable parameters, represents the amount of the multimodal data, represents the similarity between the sentiment feature vectors of the positive sample pair and the negative sample pair, Represents traversal indicator variables.

[0013] The second aspect of the present application provides an event analysis system based on generation tasks and multimodality, the system comprising: An acquisition unit, configured to acquire multimodal data, wherein the multimodal data includes text data, image data, and audio data; A first generation unit is used to set a generation task based on a preset generation model, input the multimodal data into the generation task, generate structured event information, and perform relationship extraction and semantic analysis on the structured event information to obtain event relations; A construction unit, used to construct an event graph, use the structured event information as nodes of the event graph, and connect the nodes according to the event relationship; A fusion unit, used to extract text features, image features and audio features of the multimodal data, and fuse the text features, the image features and the audio features using a cross-modal attention mechanism to obtain a multimodal sentiment relationship vector; An aggregation unit, used to aggregate the multimodal sentiment relationship vector based on a preset graph neural network, and obtain a sentiment feature vector in combination with the event graph; A first calculation unit is used to construct a positive sample pair and a negative sample pair according to the emotional feature vector and the multimodal data, calculate a contrastive learning loss value based on the positive sample pair and the negative sample pair using a preset contrast function, and align and optimize the emotional feature vector and the multimodal data by using the contrastive learning loss value to obtain a target emotional feature; A second generating unit is used to transmit the emotion feature vector to the node of the event graph, adjust the attention weight of the emotion feature vector by using the cross-modal attention mechanism, and generate a target event graph; A capture unit, used to extract the relationship features in the target event graph using the preset graph neural network, obtain a node feature sequence in combination with the attention weight, and input the node feature sequence into a preset time series model to capture the dynamic evolution law, and obtain an event prediction graph according to the dynamic evolution law; An inference unit, used to extract the causal chain in the event prediction graph, perform causal relationship inference based on the node feature sequence and the causal chain, and obtain trigger conditions and emotional factors after inference; The second calculation unit is used to match the event prediction graph with the target event graph, and calculate the event analysis result by combining the node feature sequence, the trigger condition and the emotional factor.

[0014] Optionally, the fusion unit is specifically used for: Extracting text semantics and text sentiment from the multimodal data using a preset language macro model, and obtaining text features based on the text semantics and the text sentiment; Extracting image scenes and image expressions from the multimodal data using ResNet or Vision Transformer, and obtaining image features based on the image scenes and the image expressions; Extracting audio intonation, audio volume, and audio rhythm from the multimodal data using a Wav2Vec audio model, and obtaining audio features based on the audio intonation, the audio volume, and the audio rhythm; The target weights of the text features, the image features, and the audio features are calculated using a cross-modal attention mechanism, and the target weights are combined with the text features, the image features, and the audio features to obtain a multimodal sentiment relationship vector.

[0015] Optionally, the first generating unit is specifically used for: Setting a generation task based on a preset generation model, inputting the multimodal data into the generation task for training, and obtaining a structured description; Under the preset parameters of the preset large model, the multimodal data and the corresponding structured description are combined into sample pairs, and the target probability of the structured event information generated by the input multimodal data is calculated based on the sample pairs; According to the target probability, the preset parameters are adjusted in combination with a cross entropy loss function, and structured event information is generated after the adjustment.

[0016] Optionally, the preset parameters are adjusted according to the target probability in combination with a cross entropy loss function, and structured event information is generated after the adjustment, which is represented by the following formula: ; in, Represents structured event information, represents the amount of the multimodal data, represents the target probability, represents the preset parameters, Indicates The multimodal data of the input; Indicates The structured description of the output.

[0017] Optionally, a third generating unit is further included, which is specifically used to: The data structure of the event analysis result is converted into a target data structure, and a preset visualization tool is used to generate a visualization analysis result from the converted event analysis result.

[0018] Optionally, the causal chain in the event prediction graph is extracted, and causal relationship reasoning is performed based on the node feature sequence and the causal chain. After reasoning, trigger conditions and emotional factors are obtained, which are expressed by the following formula: ; in, represents the trigger condition, Indicates the emotional factors, express The trigger condition corresponding to the time step, represents the node feature sequence, represents the weight matrix parameters, Represents the causal chain.

[0019] Optionally, the contrastive learning loss value is expressed by the following formula: ; in, represents the contrast loss value, represents the sentiment feature vector of the positive sample pair, Represents the sentiment feature vector of the negative sample pair, Represents adjustable parameters, represents the amount of the multimodal data, represents the similarity between the sentiment feature vectors of the positive sample pair and the negative sample pair, Represents traversal indicator variables.

[0020] A third aspect of the present application provides an event analysis device based on generation tasks and multi-modality, the device comprising: Processor, memory, input-output unit, and bus; The processor is connected to the memory, the input and output unit, and the bus; The memory stores a program, and the processor calls the program to execute the first aspect and any optional method in the first aspect.

[0021] A fourth aspect of the present application provides a computer-readable storage medium, on which a program is stored. When the program is executed on a computer, the program executes the first aspect and any optional method in the first aspect.

[0022] It can be seen from the above technical solutions that this application has the following advantages: The event analysis method based on generation tasks and multimodality solves the limitation of existing deep learning-based feature fusion technology that only fuses the basic features of each modality by using multimodal data, large model generation tasks and cross-modal attention mechanism, and realizes more accurate urban event analysis. First, the text, image, audio and video features of multimodal data are extracted, and the weights are dynamically allocated using the cross-modal attention mechanism to form a multimodal emotional relationship vector, which can more comprehensively and deeply explore the potential associations between data and improve the ability to capture event features. Then, by generating structured event information and constructing event graphs, the multimodal emotional relationship vectors are aggregated in combination with the preset graph neural network to generate emotional feature vectors, which closely combines the emotional information of the event with the structure and relationship, and more accurately grasps the event trend and emotional dynamics.

[0023] After generating the sentiment feature vector, the contrastive learning loss value calculated by the preset contrastive loss function is used to optimize the sentiment feature vector and multimodal data. Then, the sentiment feature vector is input into the node as the node attribute of the event graph. The cross-modal attention mechanism is used to track the development and changes of events in real time, and the weight of the sentiment feature vector is dynamically adjusted to generate the target event graph. Then, the target event graph is processed by the preset graph neural network to extract the relationship features of the nodes. In the extraction process, the graph neural network can deeply explore the potential associations between nodes and provide reliable relationship information for subsequent analysis. After extracting the relationship features, the node feature sequence is obtained by combining the attention weights, and the node feature sequence is input into the preset time series model. After receiving the node feature sequence, the time series model can capture the dynamic evolution law of the event in the time series through iterative calculation, and generate the event prediction graph according to the dynamic evolution law, so as to predict the future development trend of the event. Finally, the event prediction graph is fused with the target event graph, and the event analysis result is obtained by combining the node feature sequence. The event analysis results obtained in this application comprehensively consider the current status and future trends of the event, and can provide a more comprehensive and accurate basis for decision-making when facing the complex and changeable situations of urban events, which will help improve the scientificity and timeliness of urban governance decisions, avoid missing the best time to deal with emergency events, and effectively control the scope of the impact of the event. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solution in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 A schematic diagram of an embodiment of a method for event analysis based on generation tasks and multimodality provided in this application; Figure 2 A flowchart of another embodiment of the event analysis method based on generation tasks and multimodality provided by the present application; Figure 3 A schematic diagram of the structure of an embodiment of an event analysis system based on generation tasks and multi-modality provided by the present application; Figure 4 A schematic diagram of the structure of an embodiment of an event analysis device based on task generation and multimodality provided in the present application. DETAILED DESCRIPTION

[0026] The present application provides an event analysis method based on generation tasks and multi-modality, which can quickly and accurately grasp the development trend of events. It should be noted that the event analysis method based on generation tasks and multi-modality of the present application is applied to a terminal.

[0027] See also Figure 1 The present application first provides an embodiment of an event analysis method based on generation tasks and multi-modality, the embodiment comprising: S101, acquiring multimodal data, where the multimodal data includes text data, image data, and audio data; In this embodiment, in order to comprehensively obtain multimodal data of urban event situations, it is necessary to screen out the source platforms of text data, image data and audio data respectively. Comprehensive acquisition of multimodal data can provide a richer source of information for subsequent analysis.

[0028] Text data is mainly obtained by crawling text information related to urban events from various news websites, social media platforms, and government bulletin boards through web crawler technology. The text information covers descriptions of events, discussions among the public, and other content; image data is mainly obtained by real-time key frames captured by surveillance cameras and pictures related to urban events on social platforms. The captured key frames and related pictures on social platforms can reflect the on-site situation when urban events occur; audio data is mainly obtained by setting up recording equipment in public places and audio from on-site interviews, which can capture sound information on the scene, such as people's shouts, alarms, etc., to assist in the judgment of events from multiple dimensions. Through multiple channels and methods, it is ensured that rich and comprehensive multimodal data are obtained, which improves the accuracy and comprehensiveness of urban situation monitoring and avoids the limitations of single-modal data.

[0029] S102, setting a generation task based on a preset generation model, inputting multimodal data into the generation task, generating structured event information, and performing relationship extraction and semantic analysis on the structured event information to obtain event relationships; When setting a generation task based on the preset generation big model, the acquired multimodal data is preprocessed, which includes removing noise, filtering irrelevant information, and processing missing values. The multimodal data after preprocessing is then input into the generation task, and the structured event information is generated by the preset generation big model in the generation task.

[0030] After generating structured event information, the preset large model will also perform relationship extraction and semantic analysis on the generated structured event information to identify the causal relationship, time sequence, etc. between events. After the identification is completed, the event relationship is obtained. Relationship extraction and semantic analysis of structured event information further explore the internal connection between events, making the understanding of the city situation more in-depth and comprehensive.

[0031] S103, constructing an event graph, using structured event information as nodes of the event graph, and connecting the nodes according to event relationships; When constructing an event graph, the processed structured event information is used as the node of the event graph. Each node represents a specific event and contains detailed information about the event, such as event type, key elements, etc. Then, based on the event relationship obtained through relationship extraction and semantic analysis, relationship connections are established between the nodes. The constructed event graph can quickly understand the relationship and overall situation between events.

[0032] In practical applications, the event graph can be formalized as a directed graph ,in is a collection of event nodes, is the causal relationship between events, such as nodes and , yes Because of this, there is an edge , indicating that from point to causal relationship.

[0033] S104, extracting text features, image features, and audio features of the multimodal data, and fusing the text features, image features, and audio features using a cross-modal attention mechanism to obtain a multimodal sentiment relationship vector; In this embodiment, text features, image features, and audio features are extracted for the acquired multimodal data, respectively, and features of different modal data are extracted respectively, so as to fully mine the key information in each modal data.

[0034] After feature extraction, the cross-modal attention mechanism is used for feature fusion. The cross-modal attention mechanism allows features of different modalities to refer to each other. For example, text features will refer to the information carried by image features and audio features during the fusion process. When analyzing urban safety events, the text feature is "the atmosphere on the scene is tense", the image feature shows a chaotic scene, and the audio hears noisy sounds. The cross-modal attention mechanism will combine the information of these different modalities and finally obtain a multimodal sentiment relationship vector that integrates information from multiple modalities. The use of the cross-modal attention mechanism breaks the information barrier between different modalities, allowing features of different modalities to complement and fuse with each other. The fused multimodal sentiment relationship vector contains rich information from multiple data sources, greatly improving the understanding and representation of the sentiment relationship of multimodal data. Compared with the feature analysis of a single modality, it can more comprehensively and accurately capture the sentiment and semantic information behind the data, providing a solid data foundation for subsequent sentiment analysis and event understanding.

[0035] S105, aggregating the multimodal sentiment relationship vector based on a preset graph neural network, and obtaining a sentiment feature vector in combination with the event graph; The preset graph neural network takes the multimodal sentiment relationship vector as input, and uses the network structure and algorithm in the preset graph neural network to model the sentiment relationship between different modalities represented by the multimodal sentiment relationship vector. In the modeling process, the preset graph neural network regards the multimodal sentiment relationship vector as nodes and edges in the graph, and aggregates the multimodal sentiment relationship vector through graph convolution. The preset graph neural network can effectively aggregate the multimodal sentiment relationship vector and mine the hidden relationships and patterns between different modal emotions.

[0036] Subsequently, the aggregation results are combined with the constructed event graph, which contains rich event information and the relationship between events. By integrating the aggregated multimodal sentiment relationship vector into the event graph, the event graph not only contains the information of the event itself, but also incorporates the sentiment information reflected by the multimodal data, thereby obtaining the sentiment feature vector.

[0037] In the actual analysis of urban security incidents, the event map records information such as the time, location, and relevant personnel of the incident, while the multimodal sentiment relationship vector reflects the emotional attitude of the masses towards the incident. After combining the two, the sentiment feature vector can more comprehensively reflect the event and the emotional state related to the event, and more deeply understand the various situations of incidents in the city, which helps to discover potential social problems and risks in advance, and provide a more comprehensive and valuable reference basis for urban governance and decision-making.

[0038] S106, constructing positive sample pairs and negative sample pairs according to the emotional feature vector and the multimodal data, calculating contrastive learning loss values ​​based on the positive sample pairs and the negative sample pairs using a preset contrast function, and aligning and optimizing the emotional feature vector and the multimodal data through the contrastive learning loss values ​​to obtain target emotional features; By constructing positive and negative sample pairs, we can learn the differences between different emotion categories and the similarities of multimodal data under the same emotion category. First, we combine multimodal data of the same emotion category to form positive sample pairs. For example, in urban safety incidents, all text, image, and audio data expressing the emotion of "worry" are paired with each other to form positive sample pairs. Then, we combine multimodal data of different emotion categories to form negative sample pairs. For example, multimodal data expressing the emotion of "anger" and multimodal data expressing the emotion of "calmness" are combined to form negative sample pairs.

[0039] After constructing the positive sample pairs and negative sample pairs, the contrastive learning loss value is calculated using the preset contrast function, which can quantify the degree of difference between different sample pairs, and then the sentiment feature vector and multimodal data are aligned and optimized according to the contrastive learning loss value. During the optimization process, the parameters are continuously adjusted so that the multimodal features of the same sentiment category are closer in the feature space, and the multimodal features of different sentiment categories are farther away in the feature space, and finally the target sentiment feature is obtained. The target sentiment feature obtained accurately reflects the actual sentiment in the multimodal data, which can improve the accuracy of sentiment analysis and further provide more reliable results for the analysis of the emotional attitudes of the masses in urban safety incidents.

[0040] Specifically, the contrastive learning loss value is expressed by the following formula: ; in, represents the contrast loss value, Represents the sentiment feature vector of the positive sample pair, Represents the sentiment feature vector of the negative sample pair, Represents adjustable parameters, represents the number of multimodal data, Represents the similarity between the sentiment feature vectors of the positive sample pair and the negative sample pair, Represents traversal indicator variables.

[0041] S107, transmitting the emotion feature vector to the node of the event graph, adjusting the attention weight of the emotion feature vector using the cross-modal attention mechanism, and generating a target event graph; In this embodiment, the obtained emotional feature vector is transmitted to each node of the event graph. The nodes of the event graph represent different event elements, such as subject, time, place, behavior, etc. The emotional feature vector carries the emotional information contained in the multimodal data. After the emotional feature vector is integrated into the node, the information dimension of the event graph is enriched, expanding from a single event structure description to a comprehensive representation including emotional information.

[0042] Subsequently, the cross-modal attention mechanism will dynamically allocate attention weights according to the importance of different modal data to event understanding and emotional expression. For example, when the information conveyed by the image is more important for emotional judgment, the cross-modal attention mechanism will increase the weight of the emotional feature vector corresponding to the image feature; when the text is more important, the weight of the emotional feature vector corresponding to the text feature will be increased accordingly. Through this dynamic adjustment, the emotional information of key modalities can be flexibly highlighted according to the characteristics of different events, so that each node in the event map can fuse multi-modal emotional information with the most appropriate weight, and finally generate the target event map.

[0043] S108, using a preset graph neural network to extract relationship features in the target event graph, combining the attention weight to obtain a node feature sequence, and inputting the node feature sequence into a preset time series model to capture dynamic evolution laws, and obtaining an event prediction graph according to the dynamic evolution laws; In this embodiment, the preset graph neural network first traverses and calculates the nodes and edges in the target event graph, identifies and extracts the causal relationship, temporal relationship, and semantic association between events, and then integrates the node features in combination with the attention weights adjusted by the cross-modal attention mechanism to obtain a node feature sequence.

[0044] The obtained node feature sequence is then input into the preset time series model, which analyzes and models the data that changes over time. By processing the node feature sequence, the dynamic evolution law of the event in the time dimension is captured. Finally, based on the captured dynamic evolution law, an event prediction map is generated. The event prediction map not only contains information about the current event, but also predicts and presents possible future events based on historical data and dynamic laws. It can understand the possible development direction and changes of the event in advance and prevent potential crises.

[0045] S109, extracting the causal chain in the event prediction graph, performing causal relationship reasoning based on the node feature sequence and the causal chain, and obtaining the triggering conditions and emotional factors after reasoning; The event prediction graph contains rich event information and the prediction relationship between them. After sorting out the causal relationship between events in the event prediction graph, the causal chain is extracted. The causal chain can clearly present the causal logic between events. After extraction, causal reasoning is performed based on the node feature sequence and the causal chain. In the reasoning process, the event attributes, emotional information contained in the node feature sequence, and the logical relationship presented by the causal chain can be comprehensively considered.

[0046] Through causal reasoning, we finally get trigger conditions and emotional factors. Trigger conditions clarify the prerequisites and key factors that lead to an event, while emotional factors reveal how emotions drive the evolution of events during the development of events, and the impact of different emotions on the outcome of events.

[0047] Specifically, the causal chain in the event prediction graph is extracted, and causal relationship reasoning is performed based on the node feature sequence and the causal chain. After reasoning, the trigger conditions and emotional factors are obtained, which are expressed by the following formula: ; in, Indicates the trigger condition. Indicates emotional factors, express The corresponding trigger condition in the time step, Node feature sequence, represents the weight matrix parameters, Represents a cause-effect chain.

[0048] S110, matching the event prediction graph with the target event graph, combining the node feature sequence, trigger conditions and emotional factors, and calculating the event analysis result.

[0049] In this embodiment, the process of matching the event prediction map with the target event map will compare the nodes, edges and attribute information in the two maps, find the similarities and differences between the two maps for matching and fusion. Subsequently, a comprehensive analysis and calculation is performed in combination with the node feature sequence, trigger conditions and emotional factors. In the process of calculation, full use is made of various information resources to make the analysis results more comprehensive, in-depth and accurate. The node feature sequence contains the detailed features and emotional information of the event, the trigger condition clarifies the premise of the event, and the emotional factor includes the impact of emotion on the event.

[0050] After the calculation is completed, the event analysis results obtained provide detailed decision-making basis for urban governance departments, which can clearly understand the development dynamics of the event, potential risks, and public emotional tendencies, etc., facilitating the formulation of more scientific, reasonable and effective response strategies, improving the efficiency and effectiveness of urban governance, and better ensuring the safety and stable development of the city.

[0051] The event analysis method based on generation tasks and multimodality solves the limitation of existing deep learning-based feature fusion technology that only fuses the basic features of each modality by using multimodal data, large model generation tasks and cross-modal attention mechanism, and realizes more accurate urban event analysis. First, the text, image, audio and video features of multimodal data are extracted, and the weights are dynamically allocated using the cross-modal attention mechanism to form a multimodal emotional relationship vector, which can more comprehensively and deeply explore the potential associations between data and improve the ability to capture event features. Then, by generating structured event information and constructing event graphs, the multimodal emotional relationship vectors are aggregated in combination with the preset graph neural network to generate emotional feature vectors, which closely combines the emotional information of the event with the structure and relationship, and more accurately grasps the event trend and emotional dynamics.

[0052] After generating the sentiment feature vector, the contrastive learning loss value calculated by the preset contrastive loss function is used to optimize the sentiment feature vector and multimodal data. Then, the sentiment feature vector is input into the node as the node attribute of the event graph. The cross-modal attention mechanism is used to track the development and changes of events in real time, and the weight of the sentiment feature vector is dynamically adjusted to generate the target event graph. Then, the target event graph is processed by the preset graph neural network to extract the relationship features of the nodes. In the extraction process, the graph neural network can deeply explore the potential associations between nodes and provide reliable relationship information for subsequent analysis. After extracting the relationship features, the node feature sequence is obtained by combining the attention weights, and the node feature sequence is input into the preset time series model. After receiving the node feature sequence, the time series model can capture the dynamic evolution law of the event in the time series through iterative calculation, and generate the event prediction graph according to the dynamic evolution law, so as to predict the future development trend of the event. Finally, the event prediction graph is fused with the target event graph, and the event analysis result is obtained by combining the node feature sequence. The event analysis results obtained in this embodiment comprehensively consider the current status and future trends of the event, and can provide a more comprehensive and accurate basis for decision-making when faced with complex and changeable situations of urban events, which helps to improve the scientificity and timeliness of urban governance decisions, avoid missing the best time to deal with emergency events, and thus effectively control the scope of impact of the event.

[0053] See also Figure 2 , Figure 2 Another embodiment of an event analysis method based on generation tasks and multimodality provided by the present application includes: S201, acquiring multimodal data, where the multimodal data includes text data, image data, and audio data; In this embodiment, step S201 is similar to step S101 in the above embodiment and will not be described again here.

[0054] S202, setting a generation task based on a preset generation large model, inputting multimodal data into the generation task for training, and obtaining a structured description; When setting a generation task based on the preset generation big model, first of all, according to the needs of urban situation monitoring and the characteristics of multimodal data, clarify the goal of the generation task, design input prompt words and output templates, and then preprocess the acquired multimodal data. The preprocessing includes removing noise, filtering irrelevant information and processing missing values. After the preprocessing is completed, the input prompt words and multimodal data are input into the generation task, and the preset generation big model in the generation task analyzes and trains the input prompt words and multimodal data.

[0055] During the training process, the preset large-scale model will perform semantic understanding and analysis on text data to extract key information; perform feature recognition on image data to identify objects and human actions in the scene; and perform content analysis on audio data to determine the meaning of the sound. After the training is completed, a structured description is obtained according to the output template. For example, in an urban fire incident, the preset large-scale model in the generation task learns and trains the text, fire scene images, and rescue audio to generate a structured description containing the time, location, cause, development of the fire, and rescue operations. This structured description has higher accuracy and completeness, and is more comprehensive than the description generated by single-modal data.

[0056] Specifically, the input prompt words are designed to enable the preset large generation model to more accurately understand the goal of the generation task and generate structured event information. The input prompt word examples are as follows: “Please extract event information from the following text and generate structured event information, including event type, time, location, participants, victims, results, and causal relationships.”; The output template for generating a task needs to clearly define the event type and elements, which can be described using JSON structure or natural language: The following is an example of the JSON structure: { "Event Type": "Urban Security Event", "event": { "name": "fire", "Location": "Huangpu District, Shanghai", "Time": "October 10, 2023", "People involved": ["Residents", "Firefighters"], "Behavior": ["Fire", "Rescue"], "Reason": "Line short circuit" } } An example of a natural language description is as follows: "On October 10, 2023, a fire broke out in Huangpu District, Shanghai. A short circuit caused a fire in the residents' home, and firefighters rushed to the scene to carry out rescue."

[0057] S203, under the preset parameters of the currently preset large model generation, the multimodal data and the corresponding structured description are combined into sample pairs, and the target probability of the structured event information generated by the input multimodal data is calculated based on the sample pairs; Under the preset parameters of the current preset generation model, each group of acquired multimodal data is paired with the structured description one by one into sample pairs, and then the target probability of the structured description generated by the input multimodal data is calculated based on the sample pairs. In the process of calculating the target probability, the preset generation model will process and analyze the multimodal data according to the preset parameters, such as word vector conversion of text, feature extraction and matching of images, and feature analysis of audio, etc., and then calculate the percentage of structured event information obtained from the processed multimodal data based on the sample pairs. The calculated percentage result is the target probability of generating structured event information from multimodal data.

[0058] The target probability can reflect the preset generative big model's ability to understand and generate different multimodal data, making it easier to discover the advantages and disadvantages of the preset generative big model, so as to improve and optimize the preset generative big model in a targeted manner.

[0059] S204, adjusting the preset parameters according to the target probability and the cross entropy loss function, generating structured event information after the adjustment, and performing relationship extraction and semantic analysis on the structured event information to obtain event relationships; In this embodiment, the preset parameters are adjusted according to the calculated target probability in combination with the cross entropy loss function, wherein the cross entropy loss function is mainly used to measure the degree of difference between the model prediction result and the actual result. First, the calculated target probability is obtained, and the loss value of generating structured event information is calculated by the cross entropy loss function. According to the size of the loss value, the preset parameters of the preset generation model are adjusted by back propagation using the optimization algorithm, wherein the optimization algorithm can use the stochastic gradient descent algorithm, combined with the cross entropy loss function to adjust the preset parameters, which can continuously optimize the preset generation model, so that the structured event information generated by the preset generation model is more accurate and reliable.

[0060] Specifically, according to the target probability, the preset parameters are adjusted in combination with the cross entropy loss function, and the structured event information is generated after the adjustment, which is expressed by the following formula: ; in, Represents structured event information, represents the amount of multimodal data, represents the target probability, Indicates the preset parameters. Indicates Input multimodal data; Indicates A structured description of the output.

[0061] During the adjustment process, the preset large model will update its parameters in the direction of reducing the loss value, so that the structured event information generated by the preset large model in the generation task is closer to the real label, and the structured event information is generated after the adjustment is completed. Subsequently, the structured event information is identified and the relationship between different elements in the event information is extracted, and the semantic analysis is performed to deeply understand the meaning of the event information, and finally the event relationship is obtained.

[0062] S205, constructing an event graph, using structured event information as nodes of the event graph, and connecting the nodes according to event relationships; In this embodiment, step S205 is similar to step S103 in the above embodiment and will not be described again here.

[0063] S206, extracting text semantics and text sentiment from the multimodal data using a preset language macro model, and obtaining text features based on the text semantics and text sentiment; In this embodiment, the text data in the multimodal data is first input into the preset language model, which performs lexical analysis on the text data, splits the text data into words or phrases, and performs part-of-speech tagging on each word, and then performs syntactic analysis to parse the grammatical structure of the sentence and determine the relationship between the subject, predicate, object and other sentence components. Through lexical analysis and syntactic analysis, the preset language model further understands the key information and subject content in the text data, and then extracts the text semantics.

[0064] While extracting the text semantics, the sentiment analysis module in the preset language model is used to judge the emotional tendency of the text data to determine whether the text data expresses positive, negative or neutral emotions. For example, it can be determined whether the text is praising the city’s safety measures or expressing dissatisfaction with city safety incidents. After the judgment is completed, the judgment result is extracted as the text sentiment.

[0065] The extracted text semantics and text sentiment are feature encoded and converted into numerical text features. For example, the text semantics is mapped to a point in a high-dimensional vector space, the text sentiment is represented by a specific numerical value and integrated into the vector, and finally the text features are obtained.

[0066] This embodiment extracts text semantics and text sentiment and obtains text features through a preset language model, which can deeply mine the rich information contained in the text part of multimodal data. The extraction of text semantics helps to accurately understand people's descriptions and views on urban safety incidents and grasp the key points and core content of the incidents; text sentiment allows urban governance departments to understand the public's attitude towards the incident.

[0067] S207, extracting image scenes and image expressions from the multimodal data using ResNet or Vision Transformer, and obtaining image features based on the image scenes and image expressions; In this embodiment, ResNet or Vision Transformer is used to extract image scenes and image expressions in multimodal data, which can effectively extract key information from the image part of the multimodal data and obtain representative image features. The extraction of image scenes can intuitively display the environment and background of urban security incidents and quickly understand the on-site situation of the incidents. The extraction of image expressions can reflect people’s emotional state in the incidents, providing important clues for understanding the public’s reaction to the incidents.

[0068] When ResNet is used for extraction, the image data in the multimodal data needs to be processed by adjusting the image size, normalizing the pixel values, etc., so that the image data meets the input requirements of the ResNet model, and then the processed image data is input into the ResNet model, and the image scene and image expression in the image data are extracted through the convolution layer and pooling layer in the ResNet model, and finally the extracted image scene and image expression are converted into image features; when Vision Transformer is used for extraction, the image can be directly divided into multiple small blocks, and each small block is mapped to a vector sequence and input into the Vision Transformer model. The vector sequence is processed through the self-attention mechanism of the Vision Transformer model to capture the relationship between different small blocks, thereby identifying the image scene and image expression, and finally the extracted image scene and image expression are converted into image features. Converting image scene and expression information into image features facilitates fusion with the features of text data and audio data in multimodal data, thereby more comprehensively analyzing urban events.

[0069] In the process of using ResNet or Vision Transformer for extraction, the ResNet or Vision Transformer model will capture detailed information in the image data, such as buildings, roads, people and other elements in the image data, and then identify the scene depicted by the image data, and judge the expression of the person in the image, such as fear, surprise, calmness, etc., by analyzing the facial features of the person.

[0070] S208, extracting audio intonation, audio volume, and audio rhythm from the multimodal data using a Wav2Vec audio model, and obtaining audio features based on the audio intonation, audio volume, and audio rhythm; In this embodiment, the audio data in the multimodal data is first preprocessed, and the preprocessing includes removing noise interference in the audio, improving the clarity and quality of the audio, and ensuring that the format of the audio data meets the input requirements of the Wav2Vec audio model. The preprocessed audio data is then input into the Wav2Vec audio model, and the Wav2Vec audio model analyzes the preprocessed audio data frame by frame, extracts the audio intonation in the audio data through the convolution layer, pooling layer, and fully connected layer of the Wav2Vec audio model, and measures the volume of the audio data at the same time to understand the strength of the sound, and analyzes the rhythm characteristics of the audio to obtain the audio rhythm. Audio intonation can convey the speaker's emotional state and attitude, the volume can reflect the urgency or importance of the event, and the audio rhythm can represent the dynamic changes in the development of the event.

[0071] After the extraction is completed, feature encoding is performed based on the extracted audio intonation, audio volume, and audio rhythm, and converted into numerical audio features. For example, for audio at the scene of an urban safety incident, the Wav2Vec audio model can be used to extract the tone of the crowd's shouting, the volume of the alarm, and the rhythm of the rescue operation, and this information can be converted into audio features.

[0072] This embodiment uses the Wav2Vec audio model to extract audio features, which can not only provide an in-depth understanding of urban events from an audio perspective, but also supplement information that cannot be fully expressed by text and images, helping urban governance departments to more accurately judge the atmosphere of the event scene, people's emotional fluctuations, and the development trend of the event, providing strong support for the formulation of more effective response strategies.

[0073] S209. Calculate the target weights of text features, image features, and audio features using a cross-modal attention mechanism, and combine the target weights with the text features, image features, and audio features to obtain a multimodal sentiment relationship vector.

[0074] In this embodiment, the cross-modal attention mechanism is used to calculate the target weights of text features, image features, and audio features, which can effectively explore the intrinsic connections between different modal data, and can dynamically allocate the proportion of different modal features in the overall information, highlight key information, and avoid a certain modal data from causing excessive interference to the overall analysis due to incomplete or inaccurate information.

[0075] First, the extracted text features, image features, and audio features are input into the cross-modal attention mechanism model, which will compare and analyze the features of these three different modalities. For text features and image features, the cross-modal attention mechanism model will find the connection points between text features and image features, such as whether the scene elements mentioned in the text features are reflected in the image features, so as to measure the correlation between text features and image features; for text features and audio features, the cross-modal attention mechanism model will analyze the relationship between the emotions expressed by the text features and the audio intonation, volume, etc. in the audio features; for image features and audio features, the cross-modal attention mechanism model will explore whether the image scene in the image features is consistent with the audio rhythm and sound atmosphere of the audio features. Through these comparative analyses, the cross-modal attention mechanism model can calculate the target weight of each modal feature in the overall information expression, and finally weight the target weight with the corresponding text features, image features, and audio features to obtain a multimodal sentiment relationship vector.

[0076] The multimodal emotional relationship vector fully combines the characteristic advantages of text, image and audio modes. It is no longer limited to the information of a single mode. It contains more comprehensive and richer emotional and relationship information, which can more accurately reflect the real situation and inherent emotional tendencies of the incident. It helps urban governance departments to deeply analyze incidents from multiple dimensions, better grasp the public's emotional attitudes and the potential relationship between incidents, so as to formulate response measures that are more in line with actual needs and further improve the decision-making quality and effectiveness of urban safety management.

[0077] S210, aggregating the multimodal sentiment relationship vector based on a preset graph neural network, and obtaining a sentiment feature vector in combination with the event graph; S211, constructing positive sample pairs and negative sample pairs according to the sentiment feature vector and the multimodal data, calculating contrastive learning loss values ​​based on the positive sample pairs and the negative sample pairs using a preset contrast function, and aligning and optimizing the sentiment feature vector and the multimodal data through the contrastive learning loss values ​​to obtain target sentiment features; S212, transmitting the emotion feature vector to the node of the event graph, adjusting the attention weight of the emotion feature vector using the cross-modal attention mechanism, and generating a target event graph; S213, using a preset graph neural network to extract relationship features in the target event graph, combining the attention weight to obtain a node feature sequence, and inputting the node feature sequence into a preset time series model to capture dynamic evolution laws, and obtaining an event prediction graph according to the dynamic evolution laws; S214, extracting the causal chain in the event prediction graph, performing causal relationship reasoning based on the node feature sequence and the causal chain, and obtaining the triggering conditions and emotional factors after reasoning; S215, matching the event prediction graph with the target event graph, combining the node feature sequence, trigger conditions and emotional factors, and calculating the event analysis result.

[0078] In this embodiment, steps S210 to S215 are similar to steps S105 to S110 in the aforementioned embodiment, and are not described again here.

[0079] S216: Convert the data structure of the event analysis result into a target data structure, and use a preset visualization tool to generate a visualization analysis result from the converted event analysis result.

[0080] In this embodiment, the data structure of the event analysis results is first analyzed in detail to clarify the existing data organization form, data elements and the relationship between the elements of the event analysis results. According to the actual needs of the target data structure, the data conversion rules are designed. For example, the original data structure is a hierarchical structure, and the target data structure is a key-value pair structure that is easy to query quickly, then the original data needs to be reorganized and mapped in the form of key-value pairs. Subsequently, the data in the event analysis results is converted into the target data structure according to the data conversion rules. After the conversion is completed, the preset visualization tool is used, and the appropriate visualization chart type is selected according to the different dimensions and needs of the event analysis results, such as a bar chart for comparing the frequency of occurrence of different events, a line chart showing the development trend of events, and a map for intuitively presenting the distribution of the locations where events occurred, etc., and finally a visualization analysis result is generated.

[0081] The use of preset visualization tools to generate visual analysis results improves the readability and comprehensibility of event analysis results. Visual charts can present complex and abstract event analysis data in an intuitive graphical form, enabling quick understanding of the event's key information, development trends, and the relationship between different factors.

[0082] The following is a detailed description of the event analysis system based on generation tasks and multi-modality provided by this application. Figure 3 , Figure 3 Another embodiment of the event analysis system based on generation tasks and multimodality provided by the present application includes: An acquisition unit 301 is used to acquire multimodal data, where the multimodal data includes text data, image data, and audio data; The first generation unit 302 is used to set a generation task based on a preset generation large model, input the multimodal data into the generation task, generate structured event information, and perform relationship extraction and semantic analysis on the structured event information to obtain event relationships; A construction unit 303 is used to construct an event graph, use the structured event information as nodes of the event graph, and connect the nodes according to event relationships; A fusion unit 304 is used to extract text features, image features and audio features of the multimodal data, and fuse the text features, image features and audio features using a cross-modal attention mechanism to obtain a multimodal sentiment relationship vector; Aggregation unit 305, used to aggregate the multimodal sentiment relationship vector based on a preset graph neural network, and obtain a sentiment feature vector in combination with the event graph; A first calculation unit 306 is used to construct positive sample pairs and negative sample pairs according to the emotion feature vector and the multimodal data, calculate a contrastive learning loss value based on the positive sample pairs and the negative sample pairs using a preset contrast function, and align and optimize the emotion feature vector and the multimodal data through the contrastive learning loss value to obtain a target emotion feature; The second generating unit 307 is used to transmit the emotion feature vector to the node of the event graph, adjust the attention weight of the emotion feature vector by using the cross-modal attention mechanism, and generate the target event graph; A capture unit 308 is used to extract the relationship features in the target event graph using a preset graph neural network, obtain a node feature sequence in combination with the attention weight, and input the node feature sequence into a preset time series model to capture the dynamic evolution law, and obtain an event prediction graph according to the dynamic evolution law; The reasoning unit 309 is used to extract the causal chain in the event prediction graph, perform causal relationship reasoning based on the node feature sequence and the causal chain, and obtain the triggering conditions and emotional factors after reasoning; The second calculation unit 310 is used to match the event prediction graph with the target event graph, and calculate the event analysis result by combining the node feature sequence, trigger conditions and emotional factors.

[0083] Optionally, the fusion unit 304 is specifically configured to: Use the preset language model to extract text semantics and text sentiment from multimodal data, and obtain text features based on text semantics and text sentiment; Use ResNet or Vision Transformer to extract image scenes and image expressions from multimodal data, and obtain image features based on the image scenes and image expressions; Use the Wav2Vec audio model to extract audio intonation, audio volume, and audio rhythm from multimodal data, and obtain audio features based on the audio intonation, audio volume, and audio rhythm; The cross-modal attention mechanism is used to calculate the target weights of text features, image features, and audio features. The target weights are combined with text features, image features, and audio features to obtain a multimodal sentiment relationship vector.

[0084] Optionally, the first generating unit 302 is specifically configured to: The generation task is set based on the preset large generation model, and the multimodal data is input into the generation task for training to obtain a structured description; Under the preset parameters of the currently preset large model, the multimodal data and the corresponding structured description are combined into sample pairs, and the target probability of the structured event information generated by the input multimodal data is calculated based on the sample pairs; According to the target probability, the preset parameters are adjusted in combination with the cross entropy loss function, and structured event information is generated after adjustment.

[0085] Optionally, the preset parameters are adjusted according to the target probability and the cross entropy loss function, and structured event information is generated after the adjustment, which is expressed by the following formula: ; in, Represents structured event information, represents the number of multimodal data, represents the target probability, Indicates the preset parameters. Indicates Input multimodal data; Indicates A structured description of the output.

[0086] Optionally, a third generating unit 311 is further included, which is specifically used to: The data structure of the event analysis results is converted into a target data structure, and a preset visualization tool is used to generate a visual analysis result from the converted event analysis results.

[0087] Optionally, extract the causal chain in the event prediction graph, perform causal relationship reasoning based on the node feature sequence and the causal chain, and obtain the triggering conditions and emotional factors after reasoning, which are expressed by the following formula: ; in, Indicates the trigger condition. Indicates emotional factors, express The corresponding trigger condition in the time step, represents the node feature sequence, represents the weight matrix parameters, Represents a cause-effect chain.

[0088] Optionally, the contrastive learning loss value is expressed as follows: ; in, represents the contrast loss value, Represents the sentiment feature vector of the positive sample pair, Represents the sentiment feature vector of the negative sample pair, Represents adjustable parameters, represents the number of multimodal data, Represents the similarity between the sentiment feature vectors of the positive sample pair and the negative sample pair, Represents traversal indicator variables.

[0089] This application also provides an event analysis device based on generation tasks and multi-modality, see Figure 4 , Figure 4 An embodiment of an event analysis device based on generation tasks and multimodality provided by the present application includes: Processor 401, memory 402, input and output unit 403, bus 404; The processor 401 is connected to the memory 402, the input and output unit 403 and the bus 404; The memory 402 stores a program, and the processor 401 calls the program to execute any of the above methods.

[0090] The present application also relates to a computer-readable storage medium on which a program is stored, wherein when the program is run on a computer, the computer is caused to execute any of the above methods.

[0091] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0092] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0093] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0094] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0095] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

Claims

1. An event analysis method based on generation tasks and multimodality, characterized in that: The method comprises: Acquiring multimodal data, wherein the multimodal data includes text data, image data, and audio data; A generation task is set based on a preset generation model, the multimodal data is input into the generation task, structured event information is generated, and relationship extraction and semantic analysis are performed on the structured event information to obtain event relationships; Constructing an event graph, taking the structured event information as nodes of the event graph, and connecting the nodes according to the event relationships; Extracting text features, image features, and audio features of the multimodal data, and fusing the text features, the image features, and the audio features using a cross-modal attention mechanism to obtain a multimodal sentiment relationship vector; Aggregating the multimodal sentiment relationship vector based on a preset graph neural network, and obtaining a sentiment feature vector in combination with the event graph; Constructing positive sample pairs and negative sample pairs according to the emotional feature vector and the multimodal data, calculating contrastive learning loss values ​​based on the positive sample pairs and the negative sample pairs using a preset contrast function, and aligning and optimizing the emotional feature vector and the multimodal data using the contrastive learning loss value to obtain a target emotional feature; The emotion feature vector is transmitted to the node of the event graph, and the attention weight of the emotion feature vector is adjusted by using the cross-modal attention mechanism to generate a target event graph; The preset graph neural network is used to extract the relationship features in the target event graph, and the node feature sequence is obtained by combining the attention weight, and the node feature sequence is input into the preset time series model to capture the dynamic evolution law, and the event prediction graph is obtained according to the dynamic evolution law; Extracting the causal chain in the event prediction graph, performing causal relationship reasoning based on the node feature sequence and the causal chain, and obtaining trigger conditions and emotional factors after reasoning; The event prediction graph is matched with the target event graph, and the event analysis result is calculated by combining the node feature sequence, the trigger condition and the emotional factor.

2. The method according to claim 1, characterized in that The extracting of text features, image features and audio features of the multimodal data, and fusing the text features, the image features and the audio features using a cross-modal attention mechanism to obtain a multimodal sentiment relationship vector includes: Extracting text semantics and text sentiment from the multimodal data using a preset language macro model, and obtaining text features based on the text semantics and the text sentiment; Extracting image scenes and image expressions from the multimodal data using ResNet or Vision Transformer, and obtaining image features based on the image scenes and the image expressions; Extracting audio intonation, audio volume, and audio rhythm from the multimodal data using a Wav2Vec audio model, and obtaining audio features based on the audio intonation, the audio volume, and the audio rhythm; The target weights of the text features, the image features, and the audio features are calculated using a cross-modal attention mechanism, and the target weights are combined with the text features, the image features, and the audio features to obtain a multimodal sentiment relationship vector.

3. The method according to claim 1, characterized in that The step of setting a generation task based on a preset generation large model, inputting the multimodal data into the generation task, and generating structured event information includes: Setting a generation task based on a preset generation model, inputting the multimodal data into the generation task for training, and obtaining a structured description; Under the preset parameters of the preset large model, the multimodal data and the corresponding structured description are combined into sample pairs, and the target probability of the structured event information generated by the input multimodal data is calculated based on the sample pairs; According to the target probability, the preset parameters are adjusted in combination with a cross entropy loss function, and structured event information is generated after the adjustment.

4. The method according to claim 3, characterized in that The preset parameters are adjusted according to the target probability in combination with the cross entropy loss function, and structured event information is generated after the adjustment, which is expressed by the following formula: ； in, Represents structured event information, represents the amount of the multimodal data, represents the target probability, represents the preset parameters, Indicates The multimodal data of the input; Indicates The structured description of the output.

5. The method according to claim 1, characterized in that After matching the event prediction graph with the target event graph, combining the node feature sequence, the trigger condition and the emotional factor to calculate the event analysis result, the method further includes: The data structure of the event analysis result is converted into a target data structure, and a preset visualization tool is used to generate a visualization analysis result from the converted event analysis result.

6. The method according to claim 1, characterized in that The causal chain in the event prediction graph is extracted, and causal relationship reasoning is performed based on the node feature sequence and the causal chain. After reasoning, trigger conditions and emotional factors are obtained, which are expressed by the following formula: ； in, represents the trigger condition, Indicates the emotional factors, express The trigger condition corresponding to the time step, represents the node feature sequence, represents the weight matrix parameters, Represents the causal chain.

7. The method according to any one of claims 1 to 6, characterized in that The contrastive learning loss value is expressed by the following formula: ； in, represents the contrast loss value, represents the sentiment feature vector of the positive sample pair, Represents the sentiment feature vector of the negative sample pair, Represents adjustable parameters, represents the amount of the multimodal data, represents the similarity between the sentiment feature vectors of the positive sample pair and the negative sample pair, Represents traversal indicator variables.

8. An event analysis system based on generation tasks and multimodality, characterized in that: The system comprises: An acquisition unit, configured to acquire multimodal data, wherein the multimodal data includes text data, image data, and audio data; A first generation unit is used to set a generation task based on a preset generation model, input the multimodal data into the generation task, generate structured event information, and perform relationship extraction and semantic analysis on the structured event information to obtain event relations; A construction unit, used to construct an event graph, use the structured event information as nodes of the event graph, and connect the nodes according to the event relationship; A fusion unit, used to extract text features, image features and audio features of the multimodal data, and fuse the text features, the image features and the audio features using a cross-modal attention mechanism to obtain a multimodal sentiment relationship vector; An aggregation unit, used to aggregate the multimodal sentiment relationship vector based on a preset graph neural network, and obtain a sentiment feature vector in combination with the event graph; A first calculation unit is used to construct a positive sample pair and a negative sample pair according to the emotional feature vector and the multimodal data, calculate a contrastive learning loss value based on the positive sample pair and the negative sample pair using a preset contrast function, and align and optimize the emotional feature vector and the multimodal data by using the contrastive learning loss value to obtain a target emotional feature; A second generating unit is used to transmit the emotion feature vector to the node of the event graph, adjust the attention weight of the emotion feature vector by using the cross-modal attention mechanism, and generate a target event graph; A capture unit, used to extract the relationship features in the target event graph using the preset graph neural network, obtain a node feature sequence in combination with the attention weight, and input the node feature sequence into a preset time series model to capture the dynamic evolution law, and obtain an event prediction graph according to the dynamic evolution law; An inference unit, used to extract the causal chain in the event prediction graph, perform causal relationship inference based on the node feature sequence and the causal chain, and obtain trigger conditions and emotional factors after inference; The second calculation unit is used to match the event prediction graph with the target event graph, and calculate the event analysis result by combining the node feature sequence, the trigger condition and the emotional factor.

9. An event analysis device based on generation tasks and multi-modality, characterized in that: The device comprises: Processor, memory, input-output unit, and bus; The processor is connected to the memory, the input and output unit, and the bus; The memory stores a program, and the processor calls the program to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a program stored thereon, wherein the program, when executed on a computer, performs the method according to any one of claims 1 to 7.

Images