Entity state recognition method and device based on multi-modal time sequence fusion and medium
By employing a multimodal temporal fusion method that combines text and numerical data, and utilizing temporal convolutional networks and gated attention mechanisms, the problem of inaccurate entity state recognition is solved, achieving higher recognition accuracy and robustness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HANGZHOU YUNXIN ZHICE TECH CO LTD
- Filing Date
- 2026-01-27
- Publication Date
- 2026-05-12
AI Technical Summary
In existing technologies, entity state recognition methods rely solely on descriptive text or numerical state indicator data, which fails to accurately capture the inherent relationships between the dynamic changes and inherent attributes of entities, leading to inaccurate recognition.
A multimodal temporal fusion method is adopted. The descriptive text is encoded by a pre-trained language model to generate a text semantic feature vector, which is then concatenated with the time-series data of numerical state indicators. The fusion is performed using a temporal convolutional network and a gated attention mechanism to generate a comprehensive state feature vector. Finally, the state level classification and trend prediction of the entity are output.
It improves the accuracy and adaptability of entity state recognition, bridges the semantic gap, and enhances the robustness of the model under different entities and states.
Smart Images
Figure CN122020376A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of electronic digital data processing technology, and in particular to a method, device and medium for entity state recognition based on multimodal temporal fusion. Background Technology
[0002] In today's digital management, accurate and timely identification and prediction of the status of various entities (such as financial applications and software systems) is crucial for risk control. The status information of such entities is usually contained in two different types of data: one is text describing the inherent attributes or functions of the entity (such as application descriptions, function introductions, etc.), and the other is numerical status indicator data generated by the entity during operation and recorded in chronological order (such as the number of failed interface calls, operation frequency, etc.).
[0003] Currently, entity state recognition methods primarily rely solely on the aforementioned descriptive text or numerical state indicator data. For example, in financial risk identification tasks, domain-specific pre-trained models with relatively small parameter sizes (such as FinBERT) are used to identify the risk state of descriptive text for financial applications. Compared to pre-trained large language models based on the Transformer architecture, these domain-specific pre-trained models, after fine-tuning on financial corpora, can more accurately capture the risk semantics in the text. However, this method does not consider the temporal data generated during the operation of financial applications and cannot capture changes in the risk state of financial applications in a timely manner based on the dynamic data of entities. Alternatively, models such as Recurrent Neural Networks (RNNs), Long Short-Term Memory Networks (LSTMs), or standard Temporal Convolutional Networks (TCNs) are used to analyze numerical state indicator data. While this method can capture temporal dynamics, it ignores the textual descriptive information of the entity itself, resulting in an inability to understand the intrinsic relationship between the entity's inherent attributes and its current behavior.
[0004] The aforementioned methods for entity state identification that rely solely on descriptive text or numerical state indicator data have the drawback of inaccurate entity state identification. Improving the accuracy of entity state identification is an urgent problem to be solved. Summary of the Invention
[0005] The purpose of this invention is to provide a method, device, and medium for entity state recognition based on multimodal temporal fusion, so as to improve the accuracy of entity state recognition.
[0006] According to a first aspect of the present invention, an entity state recognition method based on multimodal temporal fusion is provided, the method comprising the following steps: Obtain the descriptive text of the target entity and the time-series data of its numerical status indicators.
[0007] The descriptive text of the target entity is encoded using a pre-trained language model to generate a text semantic feature vector.
[0008] The text semantic feature vector is concatenated with the numerical state index time series data to obtain a multimodal time series feature sequence.
[0009] The multimodal temporal feature sequence is input into a temporal convolutional network, and the temporal fusion features output by the temporal convolutional network are fused with the text semantic feature vector through gating attention to generate a comprehensive state feature vector.
[0010] Based on the comprehensive state feature vector, the state level classification and state trend prediction results of the target entity are output.
[0011] Furthermore, the temporal convolutional network includes at least two parallel branches with different combinations of dilation coefficients, and the temporal fusion feature output by the temporal convolutional network includes the temporal fusion feature output by each parallel branch.
[0012] Furthermore, the temporal convolutional network includes a first branch and a second branch, wherein the geometric mean of the dilation coefficient of the first branch is greater than the geometric mean of the dilation coefficient of the second branch by a preset multiple.
[0013] Furthermore, the step of performing gated attention fusion of the temporal fusion features output by the temporal convolutional network and the text semantic feature vector to generate a comprehensive state feature vector includes: Temporal global pooling is performed on the temporal fusion features output by each parallel branch to obtain the pooled features of each branch.
[0014] The pooling features of each branch are concatenated with the text semantic feature vector to obtain the features to be fused.
[0015] The weights of each component in the feature to be fused are obtained by using a gated attention mechanism, and the weights of each component are summed according to the weights to output the comprehensive state feature vector.
[0016] Furthermore, the output of the state level classification and state trend prediction results of the target entity based on the comprehensive state feature vector includes: The comprehensive state feature vector is input into a multi-task learning framework; the multi-task learning framework includes a first fully connected layer, a first activation function, a second fully connected layer, and a second activation function.
[0017] The classification probability distribution of the current risk level is output through the first fully connected layer and the first activation function.
[0018] By using a second fully connected layer and a second activation function, the predicted probability of an increase in risk level within a specified future time window is output.
[0019] Furthermore, the step of concatenating the text semantic feature vector with the numerical state index time-series data to obtain a multimodal time-series feature sequence includes: The text semantic feature vector is copied and expanded so that it is aligned with the time series data of the numerical state index in the time dimension.
[0020] The copied and expanded text semantic feature vectors are concatenated with the corresponding time-based numerical state index data to form the multimodal features for each time period.
[0021] The multimodal temporal feature sequence is formed from the multimodal features at all times.
[0022] Furthermore, after obtaining the pooling features of each branch, before concatenating the pooling features of each branch with the text semantic feature vector, the method further includes: using a linear transformation layer to project the pooling features output by each parallel branch and the text semantic feature vector onto a unified feature dimension.
[0023] The step of concatenating the pooling features of each branch with the text semantic feature vector to obtain the feature to be fused is replaced by concatenating the projected features corresponding to the pooling features of each branch with the projected features corresponding to the text semantic feature vector to obtain the feature to be fused.
[0024] Furthermore, the entity is a financial application, and the pre-trained language model is FinBERT.
[0025] According to a second aspect of the present invention, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the above-described entity state recognition method based on multimodal temporal fusion.
[0026] According to a third aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described entity state recognition method based on multimodal temporal fusion.
[0027] Compared with the prior art, the present invention has at least the following beneficial effects: This invention acquires the descriptive text of the target entity and its numerical state indicators in temporal sequence. Through a two-stage fusion architecture of temporal convolutional network (TCN) and gated attention, the text semantic vector is first embedded into each time step, enabling the TCN to perceive the entity's context when analyzing temporal patterns. Subsequently, at the high-level feature level, deep temporal features and the original text semantics are dynamically fused again through a gated attention mechanism. This collaborative analysis mechanism effectively bridges the semantic gap, improving recognition accuracy compared to single-modality models. Furthermore, the gated attention mechanism can automatically calculate and allocate weights between text features and each temporal feature component based on the specific entity data and state pattern. This means the model does not rigidly process information but intelligently determines which information is more important in a specific scenario, thereby enhancing the model's adaptability and robustness across different entities and states. Attached Figure Description
[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0029] Figure 1 The flowchart shows the entity state recognition method based on multimodal temporal fusion provided in Embodiment 1 of the present invention. Figure 2 A flowchart illustrating the steps of concatenating the text semantic feature vector with the time-series data of the numerical state index, as provided in Embodiment 1 of the present invention; Figure 3 This is a flowchart of the steps for performing gated attention fusion of the temporal fusion features output by the temporal convolutional network and the text semantic feature vector, as provided in Embodiment 1 of the present invention. Detailed Implementation
[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0031] Example 1: According to this embodiment, as Figure 1 As shown, an entity state recognition method based on multimodal temporal fusion is provided, the method comprising the following steps: S100, Obtain the description text of the target entity and the time series data of the numerical status indicators of the target entity.
[0032] In one specific implementation, the descriptive text of the target entity is natural language text such as the official description of the target entity, product manual, or user agreement. The source of the time-series data of the numerical status indicators of the target entity can be application runtime logs, transaction system records, user behavior databases, or system monitoring platforms, etc. Status indicator data can be collected by setting a preset sampling frequency. Status indicators are parameters that characterize the status, and status indicators can be such as the number of interface calls, the number of logins, transaction frequency, or error rate.
[0033] As a specific implementation method, the target entity is a financial application. The descriptive text includes an application description (text used to explain the application's functional positioning, operation process, and data output type), the application name, and the application's unique identifier. This type of data is used to support the intelligent classification of applications based on FinBERT and to define the data source attributes for risk identification.
[0034] S200 uses a pre-trained language model to encode the descriptive text of the target entity, generating a text semantic feature vector.
[0035] Those skilled in the art will know that the process of encoding the descriptive text of a target entity based on a pre-trained language model is existing technology and will not be described in detail here.
[0036] In one specific implementation, the target entity is a financial application, and the pre-trained language model is FinBERT. As another specific implementation, FinBERT is fine-tuned end-to-end using manually annotated financial risk corpora, model parameters are optimized through backpropagation, and a dynamic learning rate scheduling strategy is employed to enable the model to converge quickly and achieve optimal performance in the financial risk identification task.
[0037] As a specific implementation method, before encoding the descriptive text of the target entity using a pre-trained language model, the descriptive text of the target entity is preprocessed. This preprocessing includes word segmentation, stop word removal, and terminology standardization. Word segmentation refers to using domain-specific word segmentation tools to segment the descriptive text of the target entity, ensuring accurate splitting of specialized terms and avoiding semantic fragmentation caused by general word segmentation tools. Stop word removal refers to filtering words in the text that have no actual risk relevance based on a domain-specific stop word library, reducing data redundancy and improving processing efficiency. Terminology standardization refers to establishing a terminology mapping table to uniformly convert synonymous and heterogeneous terms into standard expressions, eliminating feature bias caused by terminology differences.
[0038] S300, the text semantic feature vector is concatenated with the numerical state index time series data to obtain a multimodal time series feature sequence.
[0039] As a specific implementation, the step of concatenating the text semantic feature vector with the numerical state index time-series data to obtain a multimodal time-series feature sequence includes, for example: Figure 2 As shown: S310, the text semantic feature vector is copied and extended so that the text semantic feature vector is aligned with the time series data of the numerical state index in the time dimension.
[0040] In this embodiment, the alignment operation copies the static text semantic feature vector n times so that the multimodal features at each subsequent time point include the same text background information; where n is the length of the time series data. For example, if the numerical state index time series data includes numerical state index data corresponding to 20 time points, then n=20.
[0041] S320 concatenates the copied and expanded text semantic feature vector with the corresponding time-based numerical state index data to form the multimodal features for each time period.
[0042] In one specific implementation, the text semantic feature vector is F, the length of the numerical state indicator time series data is n, and the numerical state indicator time series data includes the numerical state indicator data corresponding to the i-th time point as T. i Then the multimodal feature corresponding to the i-th time is (F, T) i ), T i =(T i,1 ,T i,2 ,…,T i,j ,…,T i,m ,), T i,j Let j be the data for the j-th state indicator corresponding to the i-th time, where j ranges from 1 to m, m is the number of state indicators, and i ranges from 1 to n.
[0043] As a specific implementation method, considering the potential for large differences in the magnitude of numerical state indicators, the time series data of the numerical state indicators is normalized before concatenation (e.g., by Z-Score standardization or maximum-minimum normalization), and then the text semantic feature vector is concatenated with the normalized time series data of the numerical state indicators.
[0044] S330, the multimodal temporal feature sequence is formed by the multimodal features of all times.
[0045] In this embodiment, the multimodal time series feature sequence includes multimodal features at all times, and the multimodal time series feature sequence also maintains the sequentiality of the time series data of numerical state indicators, that is, it retains the time information.
[0046] S400, the multimodal temporal feature sequence is input into a temporal convolutional network, and the temporal fusion feature output by the temporal convolutional network is fused with the text semantic feature vector by gating attention to generate a comprehensive state feature vector.
[0047] In this embodiment, the multimodal time series feature sequence is a matrix, where each row represents a time period and each column represents a feature.
[0048] In one specific implementation, the temporal convolutional network includes at least two parallel branches with different combinations of dilation coefficients. The temporal fusion features output by the temporal convolutional network include the temporal fusion features output by each parallel branch. Each parallel branch is an independent network with the same temporal convolutional network architecture but different combinations of dilation coefficients. Each branch has its own independent and trainable network parameters, which are optimized separately during training to learn and capture temporal patterns at different time scales. For example, the temporal convolutional network includes a first branch and a second branch. The geometric mean of the dilation coefficient of the first branch is greater than the geometric mean of the dilation coefficient of the second branch by a preset multiple. The preset multiple is an empirical value, such as 2, 5, or 10. Thus, the first branch, using a larger dilation coefficient, can identify trend changes; the second branch, using a smaller dilation coefficient, can capture rapid changes. Both branches examine the network from different perspectives, capturing different types of behavioral patterns, which can improve the accuracy of subsequent outputs of target entity state level classification and state trend prediction results. For example, the expansion coefficient combination of branch one is [1,2,3,4], using a relatively small expansion coefficient. It is mainly used to focus on close correlations and abrupt changes over several consecutive days, and is more sensitive to recent changes, enabling it to detect short-term abnormal signals. The expansion coefficient combination of branch two is [1,4,8,16], using a larger expansion coefficient. It is mainly used to analyze macro trends and long-term dependencies, and can identify deep-seated, gradual risk patterns. As a specific implementation method, the optimal values of the relevant hyperparameters of the temporal convolutional network are determined through experimental verification.
[0049] In this embodiment, after each branch analysis is completed, a time-series feature sequence will be output, which includes the high-level feature representation corresponding to each time point after in-depth understanding and processing of the corresponding branch.
[0050] As a specific implementation, the step of performing gated attention fusion of the temporal fusion features output by the temporal convolutional network and the text semantic feature vector to generate a comprehensive state feature vector includes, for example: Figure 3 As shown: S410, perform time-series global pooling on the time-series fusion features output by each parallel branch to obtain the pooled features of each branch.
[0051] In this embodiment, the temporal fusion features output by each branch are maximized or averaged along the time dimension, compressing the temporal fusion features output by each branch into a fixed-length vector while retaining the most salient features. For example, for any parallel branch, its output temporal feature sequence includes features for 20 days, with each day's feature being a 256-dimensional feature. By performing temporal global pooling, the maximum value of each dimension is extracted, compressing the temporal feature sequence into a 256-dimensional feature vector, i.e., pooled features.
[0052] S420, the pooling features of each branch are concatenated with the text semantic feature vector to obtain the features to be fused.
[0053] As a specific implementation, after obtaining the pooling features of each branch, and before concatenating the pooling features of each branch with the text semantic feature vector, the method further includes: using a linear transformation layer to project the pooling features output by each parallel branch and the text semantic feature vector to a unified feature dimension; and replacing the step of concatenating the pooling features of each branch with the text semantic feature vector to obtain the feature to be fused with: concatenating the projected features corresponding to the pooling features of each branch with the projected features corresponding to the text semantic feature vector to obtain the feature to be fused. For example, the pooling features of each branch are all 256-dimensional, and the text semantic feature vector is 768-dimensional. By using a linear transformation layer to transform the pooling features of each branch and the text semantic feature vector to the same dimension (e.g., 512-dimensional), all features are in a semantic space and have versatility; concatenating the features transformed to the same dimension (i.e., the projected features) end to end yields the feature to be fused.
[0054] S430: The weights of each component in the feature to be fused are obtained using a gated attention mechanism, and the weights of each component are summed according to the weights to output the comprehensive state feature vector.
[0055] In this embodiment, the process of obtaining the weights of each component in the feature to be fused using a gated attention mechanism includes: generating a query vector and a key-value vector for each component, and calculating the dot product similarity between the query vector and all key vectors to obtain the attention score for each component; obtaining the credibility gating value for each component; and normalizing the score obtained by combining the attention score and the credibility gating value for each component using Softmax to obtain the weight for each component. The combination of the attention score and the credibility gating value can be achieved by multiplying the attention score and the credibility gating value. Those skilled in the art will understand that the process of obtaining the query vector and the key-value vector is prior art and will not be elaborated here. As an optional specific implementation, the credibility gating value for each component is obtained through a trainable gating network (fully connected layer + sigmoid). The input of this gating network is the component, and the output is the corresponding credibility score (i.e., the credibility gating value).
[0056] In this embodiment, the comprehensive state feature vector is obtained by multiplying each component by its corresponding weight and then summing the results.
[0057] S500, based on the comprehensive state feature vector, output the state level classification and state trend prediction results of the target entity.
[0058] As a specific implementation, the step of outputting the state level classification and state trend prediction results of the target entity based on the comprehensive state feature vector includes: inputting the comprehensive state feature vector into a multi-task learning framework; the multi-task learning framework includes a first fully connected layer, a first activation function, a second fully connected layer, and a second activation function; outputting the classification probability distribution of the current risk level through the first fully connected layer and the first activation function; and outputting the prediction result (i.e., the state trend prediction result) of whether the risk level will increase within a specified future time window through the second fully connected layer and the second activation function. The first activation function is a Softmax activation function, and the second activation function is a Sigmoid activation function. Optionally, the specified future time window is a preset future period, such as one day or one week.
[0059] As a specific implementation method, the status levels include: low risk, medium risk, high risk, and extremely high risk.
[0060] This embodiment acquires the descriptive text of the target entity and its numerical state indicators in time series. Through a two-stage fusion architecture of temporal convolutional network (TCN) and gated attention, the text semantic vector is first embedded into each time step, enabling the TCN to perceive the entity's context when analyzing temporal patterns. Subsequently, at the high-level feature level, deep temporal features and the original text semantics are dynamically fused again through a gated attention mechanism. This collaborative analysis mechanism effectively bridges the semantic gap, improving recognition accuracy compared to single-modal models. Furthermore, the gated attention mechanism can automatically calculate and allocate weights between text features and each temporal feature component based on the specific entity data and state pattern. This means the model does not rigidly process information but intelligently determines which information is more important in a specific scenario, thereby enhancing the model's adaptability and robustness across different entities and states.
[0061] Example 2: This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following steps: Obtain the descriptive text of the target entity and the time-series data of its numerical status indicators.
[0062] The descriptive text of the target entity is encoded using a pre-trained language model to generate a text semantic feature vector.
[0063] The text semantic feature vector is concatenated with the numerical state index time series data to obtain a multimodal time series feature sequence.
[0064] The multimodal temporal feature sequence is input into a temporal convolutional network, and the temporal fusion features output by the temporal convolutional network are fused with the text semantic feature vector through gating attention to generate a comprehensive state feature vector.
[0065] Based on the comprehensive state feature vector, the state level classification and state trend prediction results of the target entity are output.
[0066] Example 3: This embodiment provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it performs the following steps: Obtain the descriptive text of the target entity and the time-series data of its numerical status indicators.
[0067] The descriptive text of the target entity is encoded using a pre-trained language model to generate a text semantic feature vector.
[0068] The text semantic feature vector is concatenated with the numerical state index time series data to obtain a multimodal time series feature sequence.
[0069] The multimodal temporal feature sequence is input into a temporal convolutional network, and the temporal fusion features output by the temporal convolutional network are fused with the text semantic feature vector through gating attention to generate a comprehensive state feature vector.
[0070] Based on the comprehensive state feature vector, the state level classification and state trend prediction results of the target entity are output.
[0071] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0072] While specific embodiments of the invention have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of the invention. It should also be understood that various modifications can be made to the embodiments without departing from the scope and spirit of the invention. The scope of the invention is defined by the appended claims.
Claims
1. A method for entity state recognition based on multimodal temporal fusion, characterized in that, The method includes the following steps: Obtain the descriptive text of the target entity and the time-series data of its numerical status indicators; The text describing the target entity is encoded using a pre-trained language model to generate a text semantic feature vector. The text semantic feature vector is concatenated with the numerical state index time series data to obtain a multimodal time series feature sequence; The multimodal temporal feature sequence is input into a temporal convolutional network, and the temporal fusion feature output by the temporal convolutional network is fused with the text semantic feature vector through gating attention to generate a comprehensive state feature vector. Based on the comprehensive state feature vector, the state level classification and state trend prediction results of the target entity are output.
2. The entity state recognition method based on multimodal temporal fusion according to claim 1, characterized in that, The temporal convolutional network contains at least two parallel branches with different combinations of dilation coefficients, and the temporal fusion feature output by the temporal convolutional network includes the temporal fusion feature output by each parallel branch.
3. The entity state recognition method based on multimodal temporal fusion according to claim 2, characterized in that, The temporal convolutional network includes a first branch and a second branch, wherein the geometric mean of the dilation coefficient of the first branch is greater than the geometric mean of the dilation coefficient of the second branch by a preset multiple.
4. The entity state recognition method based on multimodal temporal fusion according to claim 2, characterized in that, The step of fusing the temporal fusion features output by the temporal convolutional network with the text semantic feature vector using gated attention to generate a comprehensive state feature vector includes: Temporal global pooling is performed on the temporal fusion features output by each parallel branch to obtain the pooled features of each branch; The pooling features of each branch are concatenated with the text semantic feature vector to obtain the features to be fused. The weights of each component in the feature to be fused are obtained by using a gated attention mechanism, and the weights of each component are summed according to the weights to output the comprehensive state feature vector.
5. The entity state recognition method based on multimodal temporal fusion according to claim 1, characterized in that, The output of the target entity's state level classification and state trend prediction results based on the comprehensive state feature vector includes: The comprehensive state feature vector is input into a multi-task learning framework; the multi-task learning framework includes a first fully connected layer, a first activation function, a second fully connected layer, and a second activation function; The classification probability distribution of the current risk level is output through the first fully connected layer and the first activation function; By using a second fully connected layer and a second activation function, the predicted probability of an increase in risk level within a specified future time window is output.
6. The entity state recognition method based on multimodal temporal fusion according to claim 1, characterized in that, The step of concatenating the text semantic feature vector with the numerical state index time series data to obtain a multimodal time series feature sequence includes: The text semantic feature vector is copied and expanded so that it is aligned with the time-series data of the numerical state index in the time dimension. The copied and expanded text semantic feature vectors are concatenated with the corresponding time-based numerical state index data to form the multimodal features for each time period. The multimodal temporal feature sequence is formed from the multimodal features at all times.
7. The entity state recognition method based on multimodal temporal fusion according to claim 4, characterized in that, After obtaining the pooling features of each branch, and before concatenating the pooling features of each branch with the text semantic feature vector, the method further includes: using a linear transformation layer to project the pooling features output by each parallel branch and the text semantic feature vector onto a unified feature dimension. The step of concatenating the pooling features of each branch with the text semantic feature vector to obtain the feature to be fused is replaced by concatenating the projected features corresponding to the pooling features of each branch with the projected features corresponding to the text semantic feature vector to obtain the feature to be fused.
8. The entity state recognition method based on multimodal temporal fusion according to claim 1, characterized in that, The entity is a financial application, and the pre-trained language model is FinBERT.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the entity state recognition method based on multimodal temporal fusion as described in any one of claims 1 to 8.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the entity state recognition method based on multimodal temporal fusion as described in any one of claims 1 to 8.