Dynamic generation method for visual communication emergency information based on multi-modal perception

Abstract

The invention relates to a multi-modal perception visual communication emergency information dynamic generation method. The method comprises the following steps: acquiring visual environment state data and non-visual environment state data; according to the visual environment state data and the non-visual environment state data, based on a cross-modal feature fusion model, obtaining a fused multi-modal feature map; according to the fused multi-modal feature graph, based on a dynamic generative adversarial network, obtaining a fused multi-modal feature trend graph; when the fused multi-modal feature trend chart meets a preset condition, emergency reminding information is generated; the emergency reminding information is used for generating dynamic visual display parameters. By adopting the method, cross-modal semantic alignment can be optimized, and the accuracy of the generated emergency information is improved.

Description

Technical Field

[0001] The present invention belongs to the field of multimodal perception, and in particular relates to a method for dynamically generating multimodal perception visually conveyed emergency information. Background Art

[0002] With the rapid development of sensor technology, multimodal sensing technology has gradually become a research hotspot. By fusing multiple sensors (such as cameras, infrared sensors, and sound sensors) to obtain data from different modalities, multimodal sensing technology can more comprehensively perceive complex environments and generate emergency alerts based on environmental changes.

[0003] In the existing multimodal perception technology, multi-dimensional modal perception data is obtained, feature data is extracted using a deep learning model, and the feature data of the multi-dimensional modalities are fused through cross-modal alignment and mapping technology to obtain a fused modal feature map, and emergency information is generated based on the fused modal feature map.

[0004] However, current multimodal perception technology has deviations in cross-modal semantic mapping. Data of different modalities (such as images, sounds, temperature, etc.) are often difficult to fully align at the semantic level, resulting in semantic inconsistencies in the generated visual communication information, leading to a lack of accuracy in emergency information. Summary of the Invention

[0005] Based on this, it is necessary to provide a dynamic generation method of multimodal perception visual communication emergency information that can improve the accuracy of emergency information in response to the above technical problems.

[0006] In a first aspect, the present application provides a method for dynamically generating visually conveyed emergency information, comprising:

[0007] Acquire visual environment status data and non-visual environment status data;

[0008] According to the visual environment state data and the non-visual environment state data, a fused multimodal feature map is obtained based on the cross-modal feature fusion model;

[0009] According to the fused multimodal feature graph, a fused multimodal feature trend graph is obtained based on a dynamic generative adversarial network;

[0010] When the fused multimodal feature trend graph meets the preset conditions, emergency reminder information is generated; the emergency reminder information is used to generate dynamic visual display parameters.

[0011] Furthermore, according to the visual environment state data and the non-visual environment state data, based on the cross-modal feature fusion model, a fused multimodal feature map is obtained, including:

[0012] Segment the visual environment state data and the non-visual environment state data into elements to obtain corresponding visual node data and non-visual node data;

[0013] Combining visual node data and non-visual node data into a multimodal feature set;

[0014] The multimodal feature set is input into the cross-modal feature fusion model to obtain a fused multimodal feature map.

[0015] Furthermore, the multimodal feature set is input into the cross-modal feature fusion model to obtain a fused multimodal feature map, including:

[0016] Use the following formula to calculate the visual fusion feature vector:

[0017]

[0018] in, is the visual fusion feature vector of node j in the visual node data, σ is the Sigmoid activation function, is a node set of non-visual node data, is the influence weight of node j in the visual node data on node i in the non-visual node data, W nv is the transformation matrix of non-visual node data, It is the i-node data in the non-visual node data;

[0019] Use the following formula to calculate the non-visual fusion feature vector:

[0020]

[0021] in, is the non-visual fusion feature vector of node i in the non-visual node data, σ is the Sigmoid activation function, K(j) is the node set of the visual node data, is the influence weight of node i in non-visual node data on node j in visual node data, W v is the transformation matrix of the visual node data, It is the j-node data in the visual node data;

[0022] Based on the cross-modal attention weight, the visual fusion feature vector and the non-visual fusion feature vector are combined to form a fused multimodal feature map;

[0023] Among them, the cross-modal attention weight is calculated by the following formula:

[0024]

[0025] in is the influence weight of node j in n-node data on node i in m-node data, φ is the activation function, W q is the learnable parameter matrix of n-node data, is the j-node data among the n-node data, W k is the learnable parameter matrix of m-node data, is the i-node data in the m-node data, and d is the multimodal feature dimension.

[0026] Furthermore, based on the fused multimodal feature graph and the dynamic generative adversarial network, a fused multimodal feature trend graph is obtained, including:

[0027] According to the fusion multimodal feature graph, based on the time series analysis algorithm, the multimodal prediction change characteristics of the current moment are obtained to form a multimodal prediction change feature set;

[0028] According to the multimodal prediction change characteristics at the current moment, the prediction model generation parameters are updated to obtain dynamic generation parameters;

[0029] According to the dynamic generation parameters and the fused multimodal feature map, a fused multimodal feature trend map is obtained based on the dynamic generative adversarial network.

[0030] Furthermore, according to the multimodal prediction change characteristics at the current moment, the prediction model generation parameters are updated to obtain dynamic generation parameters, including:

[0031] The dynamic generation parameters are calculated using the following formula:

[0032]

[0033] in, It is a dynamically generated parameter. is the dynamically generated parameter at the last moment, η is the learning rate, is a dynamically generated parameter gradient operator, is the loss function, H is the multimodal prediction change feature, and t is the supervision time signal.

[0034] Furthermore, according to the dynamic generation parameters and the fused multimodal feature graph, a fused multimodal feature trend graph is obtained based on the dynamic generation adversarial network, including:

[0035] According to the dynamic generation parameters, the equation of environmental physical parameter change is obtained;

[0036] According to the environmental physical parameter change equation and the fusion multi-modal feature map, the environmental vector is formed;

[0037] According to the environment vector and dynamic generation parameters, a fusion multimodal feature trend map is obtained based on the dynamic generative adversarial network.

[0038] Furthermore, the method further comprises:

[0039] According to the emergency reminder information, the emergency reminder information entropy is obtained based on information entropy calculation;

[0040] If the emergency reminder information entropy is less than the preset information entropy threshold, obtain the multimodal prediction change feature set to obtain the multimodal prediction change feature of the previous moment;

[0041] The updated dynamic generation parameters are calculated based on the multimodal prediction change characteristics at the previous moment and the multimodal prediction change characteristics at the current moment.

[0042] In a second aspect, the present application also provides a multimodal perception visual communication emergency information dynamic generation device, comprising:

[0043] A data acquisition module, used to acquire visual environment status data and non-visual environment status data;

[0044] A fusion feature module is used to obtain a fused multimodal feature map based on the visual environment state data and the non-visual environment state data and a cross-modal feature fusion model;

[0045] The feature prediction module is used to obtain a fused multimodal feature trend graph based on the fused multimodal feature graph and a dynamic generative adversarial network;

[0046] The information generation module is used to generate emergency reminder information when the fusion multimodal feature trend graph meets the preset conditions; the emergency reminder information is used to generate dynamic visual display parameters.

[0047] In a third aspect, the present application also provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements any multimodal perception visual communication emergency information dynamic generation method described in the first aspect of the present application.

[0048] In a fourth aspect, the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any multimodal perception visual communication emergency information dynamic generation method described in the first aspect of the present application.

[0049] The present application provides a method for dynamically generating emergency information through visual communication of multimodal perception, which obtains visual environment status data and non-visual environment status data; obtains a fused multimodal feature map based on a cross-modal feature fusion model according to the visual environment status data and the non-visual environment status data; obtains a fused multimodal feature trend map based on a dynamic generative adversarial network according to the fused multimodal feature map; generates emergency reminder information when the fused multimodal feature trend map meets preset conditions; the emergency reminder information is used to generate dynamic visual display parameters, set reasonable feature extraction and fusion models, optimize cross-modal semantic alignment, and combine multimodal update data to increase the accuracy of the generated emergency information. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0051] Figure 1 A flowchart of a method for dynamically generating emergency information using multimodal perception and visual communication provided by an exemplary embodiment of the present application;

[0052] Figure 2 This is a structural block diagram of a multimodal perception visual communication emergency information dynamic generation device provided as an exemplary embodiment of this application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0054] The method for dynamically generating emergency information through multimodal perception and visual communication provided in the embodiments of the present application can be applied to application scenarios in which visual emergency information can be dynamically generated through multimodal information of real-time perception of on-site images and audio.

[0055] In one embodiment, Figure 1 As shown, a method for dynamically generating emergency information with multimodal perception and visual communication is provided. This embodiment uses the method applied to a terminal as an example. It is understandable that the method can also be applied to a server, or to a system including a terminal and a server, and implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0056] Step S101: Acquire visual environment status data and non-visual environment status data.

[0057] Specifically, the terminal acquires visual environment status data and non-visual environment status data. Schematically, the visual environment status data and non-visual environment status data may be obtained via sensors. Optionally, the visual environment status data may include image data. Schematically, the non-visual environment status data may include temperature data and sound data.

[0058] Step S102 : obtaining a fused multimodal feature map based on a cross-modal feature fusion model according to the visual environment state data and the non-visual environment state data.

[0059] Specifically, the visual environment state data and the non-visual environment state data are preprocessed to obtain visual environment state data and non-visual environment state data at the same scale. Based on a cross-modal feature fusion model, feature extraction and fusion processing are performed on the visual environment state data and the non-visual environment state data to obtain a fused multimodal feature map. Illustratively, the preprocessing may include feature segmentation. Optionally, the cross-modal feature fusion model may be a dynamic multimodal graph attention network. Illustratively, the fused multimodal feature map may include visual and non-visual features of the environment at the current moment.

[0060] Step S103: According to the fused multimodal feature graph, a fused multimodal feature trend graph is obtained based on a dynamic generative adversarial network.

[0061] Specifically, the terminal summarizes the variation patterns of the multimodal features in the fused multimodal feature graph and further analyzes and predicts the environmental state based on a dynamic generative adversarial network to obtain a fused multimodal feature trend graph. In principle, the terminal can calculate the variation patterns of the multimodal features using a time series analysis algorithm. Alternatively, the dynamic generative adversarial network can be constructed based on the user's understanding of the physical laws of a specific application scenario.

[0062] Step S104: When the fused multimodal feature trend graph meets the preset conditions, emergency reminder information is generated; the emergency reminder information is used to generate dynamic visual display parameters.

[0063] Specifically, the terminal determines whether the fused multimodal feature trend graph meets preset conditions. If the conditions are met, it generates an emergency alert message. This information is then used to generate dynamic visual display parameters to support the visualization of emergency information. Illustratively, the preset conditions can be user-defined based on emergency situation criteria, and may include temperature thresholds, combustion intensity, and impact distance range.

[0064] This embodiment obtains visual environment status data and non-visual environment status data, performs multimodal data fusion on the visual environment status data and non-visual environment status data, and realizes real-time perception and trend prediction of environmental status based on a dynamic generative adversarial network, optimizes the cross-modal semantic alignment of the fusion of visual environment status data and non-visual environment status data, and increases the accuracy of the generated emergency information.

[0065] In one embodiment, a fused multimodal feature map is obtained based on the visual environment state data and the non-visual environment state data and a cross-modal feature fusion model, including:

[0066] S201 , performing element segmentation on the visual environment state data and the non-visual environment state data to obtain corresponding visual node data and non-visual node data.

[0067] Specifically, the terminal uses superpixel segmentation technology to segment the visual environment state data into superpixel regions to obtain visual node data; the terminal uses supervoxel segmentation technology to decompose the non-visual environment state data into non-visual node data for subsequent feature extraction and fusion processing. Schematically, the superpixel segmentation technology can be SLIC (Simple Linear Iterative Clustering algorithm). Optionally, the supervoxel segmentation technology can be selected according to the data type of the non-visual environment state data.

[0068] S202: Combining visual node data and non-visual node data into a multimodal feature set.

[0069] Specifically, the visual node data and the non-visual node data are combined into a multimodal feature set. Schematically, the multimodal feature set can be composed in the order of visual node data-non-visual node data.

[0070] S203: Input the multimodal feature set into a cross-modal feature fusion model to obtain a fused multimodal feature map.

[0071] Specifically, a multimodal feature set is input into a cross-modal feature fusion model, and through feature extraction and weighted combination, a corresponding fused multimodal feature map is obtained. In principle, the intensity of feature extraction and the weighted combination coefficient can be set based on the user's focus on the emergency situation, adjusting the focus of the fused multimodal feature map.

[0072] This embodiment decomposes visual and non-visual data into finer-grained node data through factor segmentation and feature set construction, and integrates them into a multimodal feature set, which is then input into a cross-modal feature fusion model to generate a fused multimodal feature map, achieving refined processing of complex environmental data and efficient integration of multimodal information, providing a more reliable basis for subsequent prediction of multimodal feature trends.

[0073] In one embodiment, a multimodal feature set is input into a cross-modal feature fusion model to obtain a fused multimodal feature map, including:

[0074] Use the following formula to calculate the visual fusion feature vector:

[0075]

[0076] in, is the visual fusion feature vector of node j in the visual node data, σ is the Sigmoid activation function, is a node set of non-visual node data, is the influence weight of node j in the visual node data on node i in the non-visual node data, W nv is the transformation matrix of non-visual node data, It is the i-node data in the non-visual node data;

[0077] Use the following formula to calculate the non-visual fusion feature vector:

[0078]

[0079] in, is the non-visual fusion feature vector of node i in the non-visual node data, σ is the Sigmoid activation function, K(j) is the node set of the visual node data, is the influence weight of node i in non-visual node data on node j in visual node data, W v is the transformation matrix of the visual node data, It is the j-node data in the visual node data.

[0080] Specifically, the cross-modal feature fusion model is based on the activation function. By calculating the influence weight and setting the transformation matrix, the visual node data and the non-visual node data are calculated to obtain the corresponding visual fusion feature vector and non-visual fusion feature vector. and non-visual fusion feature vector The fusion features of visual node data and non-visual node data can be represented respectively. Optionally, the activation function can be a Sigmoid function. Schematically, the node set of non-visual node data It can be the set of all nodes in the non-visual node data, and the node set K(j) of the visual node data can be the set of all nodes in the visual node data. Optionally, the influence weight of the j node in the visual node data on the i node in the non-visual node data is The influence weight of node i in non-visual node data on node j in visual node data It can be calculated by cross-modal attention weights. Schematically, the transformation matrix W of non-visual node data nv , the transformation matrix W of non-visual node data v The size can be set according to user needs. It can be any i-node data in non-visual node data. It can be any j-node data in the visual node data.

[0081] Based on the cross-modal attention weight, the visual fusion feature vector and the non-visual fusion feature vector are combined to form a fused multimodal feature map;

[0082] Among them, the cross-modal attention weight is calculated by the following formula:

[0083]

[0084] in is the influence weight of node j in n-node data on node i in m-node data, φ is the activation function, W q is the learnable parameter matrix of n-node data, is the j-node data among the n-node data, W k is the learnable parameter matrix of m-node data, is the i-node data in the m-node data, and d is the multimodal feature dimension.

[0085] Specifically, the formula is based on the activation function and calculates the influence weight according to the multimodal feature dimension, the learnable parameter matrix of the two types of node data, and the node data. In principle, the terminal can reasonably allocate m and n node data to continue the calculation, and calculate the influence weight of node j in the visual node data on node i in the non-visual node data. The influence weight of node i in non-visual node data on node j in visual node data Optionally, the activation function φ can be a ReLU function. Schematically, W q It can be a learnable parameter matrix of n-node data, W k It can be a learnable parameter matrix of m-node data. The size of the matrix can be set according to user needs, and different modal features can be mapped to the same space. It can be the j-node data in the n-node data. It can be the i-node data in the m-node data. Schematically, the multimodal feature dimension d can be set according to the user's computing needs, and the amount of computation can be reduced by dimensionality reduction.

[0086] This embodiment constructs a cross-modal feature fusion model, reasonably sets the dimensions of influence weights, calculates visual fusion feature vectors and non-visual fusion feature vectors based on the influence weights, forms a fused multimodal feature map, maps different modal features to the same space, and improves cross-modal semantic alignment.

[0087] In one embodiment, a fused multimodal feature trend graph is obtained based on a dynamic generative adversarial network according to the fused multimodal feature graph, including:

[0088] S401 : According to the fused multimodal feature graph, based on the time series analysis algorithm, the multimodal prediction change features at the current moment are obtained to form a multimodal prediction change feature set.

[0089] Specifically, the terminal uses a time series analysis algorithm to analyze and predict the multimodal feature changes at the current moment based on the fused multimodal feature graph, obtains multimodal predicted change features, and combines these features into a multimodal predicted change feature set. Schematically, the time series analysis algorithm can be an ARIMA model, which learns to fuse the multimodal feature graph and predict the multimodal predicted change features at the next time point.

[0090] S402: Update the prediction model generation parameters according to the multimodal prediction change characteristics at the current moment to obtain dynamic generation parameters.

[0091] Specifically, based on the multimodal prediction change characteristics at the current moment, the prediction model generation parameters are updated according to the dynamic generation parameter update formula to obtain the dynamic generation parameters. Schematically, the dynamic generation parameters can be the generator update parameters in the generative adversarial network.

[0092] S403: According to the dynamic generation parameters and the fused multimodal feature graph, a fused multimodal feature trend graph is obtained based on a dynamic generative adversarial network.

[0093] Specifically, based on the dynamic generation parameters and the fused multimodal feature graph, the generator is updated with updated parameters based on a dynamic generative adversarial network, and a fused multimodal feature trend graph is calculated. Schematically, the dynamic generative adversarial network can include a physical encoder and a dynamic evolution generator, which can be used to quantify the physical laws in the usage scenario and generate the fused multimodal feature trend graph, respectively.

[0094] This embodiment calculates multimodal prediction change features through a time series analysis algorithm to form a multimodal prediction change feature set. The generator update parameters of the dynamic generative adversarial network are updated based on the features, thereby improving the accuracy of the real-time information generated by the dynamic generative adversarial network.

[0095] In one embodiment, based on the multimodal prediction change characteristics at the current moment, the prediction model generation parameters are updated to obtain the dynamically generated parameters, including:

[0096] Dynamic generation parameters are calculated using the following formula:

[0097]

[0098] in, It is a dynamically generated parameter. is the dynamically generated parameter at the last moment, η is the learning rate, is a dynamically generated parameter gradient operator, is the loss function, H is the multimodal prediction change feature, and t is the supervision time signal.

[0099] Specifically, based on the loss calculation of the dynamic generation parameters at the previous moment, the multimodal prediction change characteristics and the supervision moment signal, the model parameters are optimized by gradient descent to calculate the dynamic generation parameters at the current moment. It can be the parameter updated by the generator at the current moment. Optionally, the parameter is dynamically generated at the previous moment. It can be the last time the generator was updated. In principle, the learning rate η can control the magnitude of the parameter update and determine the speed of model learning. Optionally, the parameter gradient operator is dynamically generated. The loss function can be expressed as Parameters The sensitivity of is used to guide the direction of parameter update. Schematically, the loss function Measure the gap between the current multimodal prediction change feature H and the supervision time signal t and the model prediction result. Optionally, the multimodal prediction change feature H can be calculated based on a time series analysis algorithm. Illustratively, the supervision time signal t can be set based on the user's requirements for the emergency message generation time.

[0100] This embodiment constructs a dynamic generation parameter update formula to update the dynamic evolution generator parameters in real time, thereby improving the accuracy of the dynamic generation of emergency information of the adversarial network.

[0101] In one embodiment, according to the dynamic generation parameters and the fused multimodal feature graph, a fused multimodal feature trend graph is obtained based on a dynamic generative adversarial network, including:

[0102] S601, obtaining an environmental physical parameter variation equation based on dynamically generated parameters;

[0103] S602, forming an environment vector based on the environmental physical parameter variation equation and the fused multi-modal feature map;

[0104] S603 , obtaining a fused multimodal feature trend graph based on a dynamic generative adversarial network according to the environment vector and the dynamically generated parameters.

[0105] Specifically, the terminal determines the physical laws that need to be used in the application environment based on the dynamically generated parameters, obtains the corresponding environmental physical parameter change equation in the physical encoder, combines the environmental physical parameter change equation and the fused multimodal feature map into an environmental vector, inputs the environmental vector into the dynamic evolution generator in the dynamic generative adversarial network, updates the generator parameters with the dynamic generated parameters at the current moment, and obtains a fused multimodal feature trend map. Schematically, the environmental physical parameter change equation can be stored in the terminal in advance according to user needs. Optionally, the dynamic evolution generator can use a lightweight U-Net architecture, including a dynamic convolution layer. Schematically, the fused multimodal feature trend map can include prediction effect data of visual data and non-visual data of the input model in the environment within a preset time.

[0106] This embodiment integrates physical laws and updates generator parameters in real time, and calculates a fused multimodal feature trend graph based on the fused multimodal feature graph, so that the dynamic generative adversarial network can respond quickly and accurately to changes in complex environments and provide intuitive and reliable visual information.

[0107] In one embodiment, the method for dynamically generating multimodal perception visually communicated emergency information provided by the embodiment of the present application may further include the following steps:

[0108] S701, obtaining emergency reminder information entropy based on information entropy calculation according to the emergency reminder information;

[0109] S702: If the emergency reminder information entropy is less than a preset information entropy threshold, a multimodal prediction change feature set is obtained to obtain the multimodal prediction change feature at the previous moment;

[0110] S703 : Calculate updated dynamic generation parameters based on the multimodal prediction change characteristics at the previous moment and the multimodal prediction change characteristics at the current moment.

[0111] Specifically, the terminal measures the uncertainty of emergency alert information by calculating its information entropy. If the information entropy is less than a preset threshold, indicating that the emergency information prediction accuracy does not meet user requirements, the terminal then obtains the multimodal prediction change characteristics of the previous moment. Finally, combining the multimodal prediction change characteristics of the previous and current moments, the dynamic generation parameters are updated, and the fused multimodal feature trend graph is recalculated. In principle, the information entropy threshold can be set based on the user's requirements for emergency information generation accuracy.

[0112] This embodiment determines the accuracy of the generated emergency information through information entropy calculation, and adds more multimodal prediction change features to the dynamic generation parameter update formula based on the accuracy, thereby improving the accuracy of the dynamic generation parameters, better adapting to environmental changes, and improving the accuracy of the prediction.

[0113] In the above-mentioned multimodal perception-based dynamic generation method of visually conveyed emergency information, the acquired visual environment status data and non-visual environment status data are segmented, and then a fused multimodal feature graph is generated through a cross-modal feature fusion model. Based on the time series analysis algorithm and the dynamic generative adversarial network, the multimodal feature changes at the current moment are predicted, dynamic generation parameters and a fused multimodal feature trend graph are generated, and the information entropy is set to determine the accuracy of the generated emergency information. More multimodal prediction change features calculated based on the time series analysis algorithm are used to update the parameters of the dynamic generative adversarial network generator, so as to better improve the cross-modal semantic alignment while ensuring real-time data update, thereby increasing the accuracy of the generated emergency information.

[0114] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0115] Based on the same inventive concept, the embodiments of the present application also provide a multimodal perception visual communication emergency information dynamic generation device for implementing the multimodal perception visual communication emergency information dynamic generation method involved above. The implementation solution provided by this device is similar to the implementation solution described in the above method. Therefore, the specific limitations in the embodiments of one or more multimodal perception visual communication emergency information dynamic generation devices provided below can be referred to the limitations of the multimodal perception visual communication emergency information dynamic generation method above, and will not be repeated here.

[0116] In an exemplary embodiment, Figure 2 As shown, a multi-modal perception visual communication emergency information dynamic generation device 200 is provided, comprising:

[0117] A data acquisition module 201 is used to acquire visual environment status data and non-visual environment status data;

[0118] A fusion feature module 202 is configured to obtain a fused multimodal feature map based on the visual environment state data and the non-visual environment state data and a cross-modal feature fusion model;

[0119] The feature prediction module 203 is used to obtain a fused multimodal feature trend graph based on the fused multimodal feature graph and a dynamic generative adversarial network;

[0120] The information generation module 204 is used to generate emergency reminder information when the fused multimodal feature trend graph meets the preset conditions; the emergency reminder information is used to generate dynamic visual display parameters.

[0121] Furthermore, the fusion feature module also includes:

[0122] An element segmentation unit is used to segment the visual environment state data and the non-visual environment state data into elements to obtain corresponding visual node data and non-visual node data;

[0123] A feature combination unit, used to combine visual node data and non-visual node data into a multimodal feature set;

[0124] The multimodal fusion unit is used to input the multimodal feature set into the cross-modal feature fusion model to obtain a fused multimodal feature map.

[0125] Furthermore, the multimodal fusion unit is also used to:

[0126] Use the following formula to calculate the visual fusion feature vector:

[0127]

[0128] in, is the visual fusion feature vector of node j in the visual node data, σ is the Sigmoid activation function, is a node set of non-visual node data, is the influence weight of node j in the visual node data on node i in the non-visual node data, W nv is the transformation matrix of non-visual node data, It is the i-node data in the non-visual node data;

[0129] Use the following formula to calculate the non-visual fusion feature vector:

[0130]

[0131] in, is the non-visual fusion feature vector of node i in the non-visual node data, σ is the Sigmoid activation function, K(j) is the node set of the visual node data, is the influence weight of node i in non-visual node data on node j in visual node data, W v is the transformation matrix of the visual node data, It is the j-node data in the visual node data;

[0132] Based on the cross-modal attention weight, the visual fusion feature vector and the non-visual fusion feature vector are combined to form a fused multimodal feature map;

[0133] Among them, the cross-modal attention weight is calculated by the following formula:

[0134]

[0135] in is the influence weight of node j in n-node data on node i in m-node data, φ is the activation function, W q is the learnable parameter matrix of n-node data, is the j-node data among the n-node data, W k is the learnable parameter matrix of m-node data, is the i-node data in the m-node data, and d is the multimodal feature dimension.

[0136] Furthermore, the feature prediction module also includes:

[0137] A feature set generation unit is used to obtain the multimodal prediction change features at the current moment based on the fused multimodal feature graph and a time series analysis algorithm to form a multimodal prediction change feature set;

[0138] A dynamic generation parameter calculation unit is used to update the prediction model generation parameters according to the multimodal prediction change characteristics at the current moment to obtain the dynamic generation parameters;

[0139] The fused multimodal feature trend graph calculation unit is used to obtain the fused multimodal feature trend graph based on the dynamic generation adversarial network according to the dynamic generation parameters and the fused multimodal feature graph.

[0140] Furthermore, the dynamically generated parameter calculation unit is further used to:

[0141] Dynamic generation parameters are calculated using the following formula:

[0142]

[0143] in, It is a dynamically generated parameter. is the dynamically generated parameter at the last moment, η is the learning rate, is a dynamically generated parameter gradient operator, is the loss function, H is the multimodal prediction change feature, and t is the supervision time signal.

[0144] Furthermore, the fusion multimodal feature trend graph calculation unit is also used to:

[0145] According to the dynamic generation parameters, the equation of environmental physical parameter change is obtained;

[0146] According to the environmental physical parameter change equation and the fusion multi-modal feature map, the environmental vector is formed;

[0147] According to the environment vector and dynamic generation parameters, a fusion multimodal feature trend map is obtained based on the dynamic generative adversarial network.

[0148] Furthermore, the device further comprises:

[0149] An information entropy calculation module is used to obtain emergency reminder information entropy based on information entropy calculation according to the emergency reminder information;

[0150] An information entropy comparison module is used to obtain a multimodal prediction change feature set and obtain the multimodal prediction change feature of the previous moment if the emergency reminder information entropy is less than a preset information entropy threshold;

[0151] The update generation parameter module is used to calculate the updated dynamic generation parameters based on the multimodal prediction change characteristics of the previous moment and the multimodal prediction change characteristics of the current moment.

[0152] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements a module of the nutrition decision support system for gastric cancer patients undergoing chemotherapy after surgery as described above.

[0153] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0154] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the components described as separate parts may or may not be physically separated, and the parts displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0155] The above-described embodiments merely represent several implementation methods of the embodiments of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the concept of the embodiments of the present application, and these modifications and improvements fall within the scope of protection of the embodiments of the present application.

Claims

1. A multimodal perception visual communication emergency information dynamic generation method, characterized by: The method comprises: Acquire visual environment status data and non-visual environment status data; Obtaining a fused multimodal feature map based on a cross-modal feature fusion model according to the visual environment state data and the non-visual environment state data; According to the fused multimodal feature graph, a fused multimodal feature trend graph is obtained based on a dynamic generative adversarial network; When the fused multimodal feature trend graph meets preset conditions, emergency reminder information is generated; the emergency reminder information is used to generate dynamic visual display parameters.

2. The method according to claim 1, characterized in that The step of obtaining a fused multimodal feature map based on a cross-modal feature fusion model according to the visual environment state data and the non-visual environment state data includes: Segmenting the visual environment state data and the non-visual environment state data into elements to obtain corresponding visual node data and non-visual node data; Combining the visual node data and the non-visual node data into a multimodal feature set; The multimodal feature set is input into the cross-modal feature fusion model to obtain the fused multimodal feature map.

3. The method according to claim 2, characterized in that Inputting the multimodal feature set into the cross-modal feature fusion model to obtain the fused multimodal feature graph includes: Use the following formula to calculate the visual fusion feature vector: in, is the visual fusion feature vector of node j in the visual node data, σ is the Sigmoid activation function, is a node set of non-visual node data, is the influence weight of node j in the visual node data on node i in the non-visual node data, W nv is the transformation matrix of non-visual node data, It is the j-node data in the non-visual node data; Use the following formula to calculate the non-visual fusion feature vector: in, is the non-visual fusion feature vector of node i in the non-visual node data, σ is the Sigmoid activation function, K(j) is the node set of the visual node data, is the influence weight of node i in non-visual node data on node j in visual node data, W v is the transformation matrix of the visual node data, It is the j-node data in the visual node data; Combining the visual fusion feature vector and the non-visual fusion feature vector into the fused multimodal feature map based on the cross-modal attention weight; The cross-modal attention weight is calculated using the following formula: in is the influence weight of node j in n-node data on node i in m-node data, φ is the activation function, W q is the learnable parameter matrix of n-node data, is the j-node data among the n-node data, W k is the learnable parameter matrix of m-node data, is the i-node data in the m-node data, and d is the multimodal feature dimension.

4. The method according to claim 1, wherein The method of obtaining a fused multimodal feature trend graph based on a dynamic generative adversarial network according to the fused multimodal feature graph includes: According to the fused multimodal feature graph, based on a time series analysis algorithm, a multimodal prediction change feature at the current moment is obtained to form a multimodal prediction change feature set; According to the multimodal prediction change characteristics at the current moment, the prediction model generation parameters are updated to obtain dynamic generation parameters; According to the dynamic generation parameters and the fused multimodal feature graph, based on a dynamic generative adversarial network, the fused multimodal feature trend graph is obtained.

5. The method according to claim 4, characterized in that The updating of the prediction model generation parameters according to the multimodal prediction change characteristics at the current moment to obtain the dynamically generated parameters includes: Dynamic generation parameters are calculated using the following formula: in, It is a dynamically generated parameter. is the dynamically generated parameter at the last moment, η is the learning rate, is a dynamically generated parameter gradient operator, is the loss function, H is the multimodal prediction change feature, and t is the supervision time signal.

6. The method according to claim 4, characterized in that The step of obtaining the fused multimodal feature trend graph based on a dynamic generative adversarial network according to the dynamically generated parameters and the fused multimodal feature graph includes: Obtaining an environmental physical parameter variation equation based on the dynamically generated parameters; An environment vector is formed according to the environmental physical parameter change equation and the fused multimodal feature map; According to the environment vector and the dynamically generated parameters, based on a dynamic generative adversarial network, the fused multimodal feature trend graph is obtained.

7. The method according to claim 4, characterized in that The method also includes: According to the emergency reminder information, based on information entropy calculation, the emergency reminder information entropy is obtained; If the emergency reminder information entropy is less than a preset information entropy threshold, obtaining the multimodal prediction change feature set to obtain the multimodal prediction change feature at the previous moment; An updated dynamic generation parameter is calculated based on the multimodal prediction change characteristics at the previous moment and the multimodal prediction change characteristics at the current moment.

8. A multimodal perception visual communication emergency information dynamic generation device, characterized by: The device comprises: A data acquisition module, used to acquire visual environment status data and non-visual environment status data; A fusion feature module, configured to obtain a fused multimodal feature map based on the visual environment state data and the non-visual environment state data and a cross-modal feature fusion model; A feature prediction module, configured to obtain a fused multimodal feature trend graph based on the fused multimodal feature graph and a dynamic generative adversarial network; The information generation module is used to generate emergency reminder information when the fused multimodal feature trend graph meets preset conditions; the emergency reminder information is used to generate dynamic visual display parameters.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images