Multi-modal financial question-answering system and method based on AI

Abstract

The invention discloses an AI-based multi-modal financial question and answer system and method, and relates to the technical field of financial question and answer data processing, and the system comprises a multi-modal input analysis module, a question and answer analysis module, an emotion perception module, a question and answer dynamic evaluation module, and a financial dynamic decision module. The emotion change of the user can be perceived in real time by analyzing the voice emotion characteristics and the text emotion keyword density, the emotion of the user may reflect important information such as anxiety to financial conditions and satisfaction to answers in a financial question and answer scene, task urgency degree evaluation is dynamically performed on question and answer, and based on various dynamic evaluation indexes, the user experience is improved. An evaluation strategy is adjusted in real time according to user behaviors and questioning characteristics, requirements of different users are better adapted, financial decision analysis is performed based on a task evaluation index and an emotion early warning index in combination with user identity permission, and the urgency degree of tasks, user emotion and permission of the users in financial decision are considered, so that the user experience is improved. And limitation of single-factor decision making is avoided.

Description

Technical Field

[0001] This invention relates to the field of Dendrobium officinale processing technology, and in particular to an AI-based multimodal financial question-answering system and method. Background Technology

[0002] Machine learning and deep learning are core technologies of artificial intelligence, capable of automatically learning patterns and rules from massive amounts of data to make predictions and decisions. In the financial field, machine learning algorithms can be used for tasks such as financial data classification, prediction, and anomaly detection. For example, algorithms such as decision trees and support vector machines can be used to classify a company's financial situation and determine whether the company faces financial risks; neural network models can be used to predict stock prices and financial indicators, providing a reference for investment decisions. Convolutional neural networks (CNNs) and recurrent neural networks (RNNs) and their variants (such as LSTM and GRUs) in deep learning have unique advantages in processing image and sequence data, and can be applied to scenarios such as financial statement image recognition and time series financial data analysis.

[0003] Most existing AI financial question-answering systems are based on a simple "question-answer" matching model. While they can handle multimodal input, they have significant limitations: Providing answers with the same depth and content to all users fails to differentiate between beginners and experts, potentially leading to information overload or information shortage. Lack of context awareness: Unable to perceive the user's emotional state and the true urgency of the problem, potentially giving inappropriate standard answers in high-risk situations, or even exacerbating the psychological risk to the questioner; To address the aforementioned technical shortcomings, a solution is proposed. Summary of the Invention

[0004] The purpose of this invention is to: by analyzing the intensity fluctuations of voice emotion features and the density of emotional keywords in text, the system can perceive users' emotional changes in real time. In financial Q&A scenarios, users' emotions may reflect important information such as anxiety about their financial situation and satisfaction with the answers. Using users' historical average waiting time, the number of questions asked per unit time, and the keyword library as dynamic evaluation standards, the system assesses the urgency of the Q&A process. Based on multiple dynamic evaluation indicators, the system can adjust its evaluation strategy in real time according to user behavior and question characteristics to better adapt to the needs of different users. Based on the task evaluation index and the emotion warning index, combined with user identity and permissions, the system performs financial decision analysis, taking into account the urgency of the task, user emotions, and user permissions in financial decision-making, thus avoiding the limitations of single-factor decision-making.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: an AI-based multimodal financial question-answering system, including a multimodal input parsing module, a question-answering analysis module, an emotion perception module, a question-answering dynamic evaluation module, and a financial dynamic decision-making module; The multimodal input parsing module is used to extract and parse multimodal data features from the target question-and-answer data, including extracting speech emotion features, text keywords, contextual logical association features, and image recognition results from the target question-and-answer data; The question-answering analysis module is used to perform deep semantic analysis and question type identification on the target question-answering data. It identifies the semantic type based on text keywords and key features of context logic to determine the question type of the target question-answering data. It analyzes the correlation between questions through context logic association features, verifies the question type based on the correlation, and extracts and verifies key parameters by combining image recognition features to generate the target question type. The emotion perception module is used to build a time-series model of user emotional state based on deep learning. By analyzing the intensity fluctuation of voice emotion features and the density of text emotion keywords, it calculates the emotion stability index. When the stability index is detected to be lower than the critical value, an emotion warning index is generated. The question-and-answer dynamic evaluation module acquires target question-and-answer data for question-and-answer operation analysis. It uses the user's historical average waiting time, the number of questions asked per unit time, and the keyword library as dynamic evaluation targets to assess the urgency of the question-and-answer dynamics and generate a task evaluation index. The financial dynamic decision-making module analyzes financial decisions based on task evaluation index and sentiment warning index, combined with user identity and permissions. It generates financial answers and interaction strategies by selecting actions from the answer candidate set in the financial knowledge base and dynamically optimizing indicators based on data.

[0006] Furthermore, multimodal data feature extraction and parsing are performed on the target question-and-answer data. The specific process is as follows: The speech fundamental frequency is obtained through audio frame processing technology, and identification is performed based on the audio standard emotional fundamental frequency threshold. Abnormal turning points of the speech fundamental frequency are marked to form a speech emotional feature vector. Deep semantic analysis of the text content is performed, and financial professional terms, business scenario keywords and sentiment polarity words are extracted through entity recognition technology to generate text keyword feature vectors; Based on the text content, causal relationship analysis is performed, and causal, parallel, and conditional relationships are analyzed and compared according to the problem logic chain diagram to obtain the contextual logical association characteristics. By locating key elements such as table regions, seals, and handwritten annotations in the image, character recognition and structure extraction are performed on the text region to generate table structure features, numerical distribution patterns, and key data identifiers, thus obtaining the image recognition results.

[0007] Furthermore, semantic type identification is performed based on text keywords and key features of contextual logic to determine the question type of the target data in the question-answering process. The specific process is as follows: Obtain a knowledge base in the financial field, define question type tags, including factual questions, analytical questions, and causal questions, and form classification criteria; The text keyword feature vector and the context logic association feature vector are fused together, and combined with the speech emotion features and image recognition results to form a multimodal feature fusion vector; The multimodal feature fusion vector is compared with the question type label to identify the question type. The semantic similarity between the question and the standard question library is calculated to obtain the initial identification type. Verify the initial identification results of the question type, combine the contextual logical features, verify the consistency between the question type identification results and the overall text semantic logic, generate a visual keyword distribution and question type distribution, and generate an interpretable analysis report; Extract structured parameters from the image, check the scope of keywords in the question and the image content, and finally generate a question type label and verification report, including main type and subtype features.

[0008] Furthermore, a time-series model of user sentiment states based on deep learning is constructed, and the specific process is as follows: S300. Based on speech emotion features, perform sentiment analysis, using a sliding window as a unit, average the sentiment probability within the window to generate a smooth, time-series-arranged speech emotion intensity vector, define a time window and calculate the ratio of the number of negative sentiment keywords appearing in the window to the total number of words, as the negative sentiment density of the text. S301. Establish a unified timeline and use timestamps to align the voice emotion intensity vector and text emotion density to form synchronized multimodal emotion time-series data. S302. Using the aligned speech emotion intensity vector and text emotion density as input sample data, construct an emotion state temporal model; Within a short time window, the standard deviation of instantaneous emotional intensity is calculated as a measure of short-term fluctuations. Combined with the density of negative emotions in the text, a stability index is obtained, which is then used as the output dimension for constructing a temporal model of emotional state.

[0009] Furthermore, when the stability index is detected to be below a critical value, an emotion warning index is generated. The specific process is as follows: S400, the stability index threshold, is a threshold used to judge the degree of user emotional stability. Based on emotional data analysis and actual scenario needs, a reasonable stability index threshold is set, denoted as T. S401. Calculate the emotional warning index based on the difference between the emotional stability index and the critical value, in combination with the warning value. The emotional warning index = T * 0.5 - S * 0.5; S402. Compare the calculated emotional stability index S with the set critical value of the stability index. If S < T, it is determined that the user's mood is extremely unstable, and the risk value R is added; if S ≥ T, it is determined that the user's mood is in a relatively stable state, and the Q&A continues to be asked.

[0010] Furthermore, evaluate the task urgency of the Q&A dynamics to generate a task evaluation index. The specific process is as follows: Extract the duration data from when the user initiated a Q&A request to when a response was obtained, calculate the user's historical average waiting time, obtain the number of questions the user initiated within the time period, and calculate the number of questions asked per unit time. Perform keyword matching on the content of the user's current question. Natural language processing techniques can be used to extract the keywords in the question and compare them with the keyword library to obtain the matching keywords; Based on the user's historical average waiting time, the number of questions asked per unit time, and the number of matching keywords, perform dynamic evaluation calculations to obtain the task evaluation index. The task evaluation index = P * 0.4 + T * 0.4; Classify the tasks into different emergency levels according to the magnitude of the task urgency index.

[0011] Furthermore, based on the dynamic analysis indicators of data optimization, generate financial answers and interaction strategies. The specific process is as follows: Based on the task evaluation index and the emotional warning index, combined with the user's identity permissions, make financial decisions, perform financial permission analysis and restrictions, and obtain the question of the target Q&A data from the financial index library to get a multi-granularity answer candidate set; Calculate the semantic matching degree between each answer and the user's question, and select the answer with the highest matching degree as the financial answer. The dynamic analysis indicators of data optimization are style decision-making and information presentation; Based on the emotional warning index value, task urgency, and permission level, generate financial answers and the generated financial answers and interaction strategies.

[0012] Furthermore, the risk warning and compliance execution module is used to construct a three-dimensional risk monitoring matrix to achieve active warning and automatic response to high-risk situations; By monitoring the three-dimensional space of "user permissions - sentiment index - urgency" in real time, when an abnormal pattern of "high-privilege users + high negative sentiment + high urgency" concurrently accessing core sensitive financial data is detected, a multi-level early warning rule chain is triggered. A high-risk log is generated using question-and-answer risk verification, and a structured alert is sent to the regulator through a question-and-answer encrypted channel. At the same time, a predefined crisis response protocol is activated, and finally, the output content is reviewed for compliance and risk calibration through a security verification layer.

[0013] The AI-based multimodal financial question answering method, applied to the AI-based multimodal financial question answering system according to any one of claims 1-8, includes the following steps: Step one is used to extract and parse multimodal data features from the target question-and-answer data, including extracting speech emotion features, text keywords, contextual logical association features, and image recognition results from the target question-and-answer data; Step two is used to perform deep semantic analysis and question type identification on the target question and answer data. Semantic type identification is performed based on text keywords and key features of context logic to determine the question type of the target question and answer data. The correlation between questions is analyzed through context logic association features, and the question type is verified based on the correlation. Key parameters are extracted and verified by combining image recognition features to generate the target question type data. Step 3 is used to construct a time-series model of user emotional state based on deep learning. By analyzing the intensity fluctuation of voice emotional features and the density of text emotional keywords, an emotional stability index is calculated. When the stability index is detected to be lower than the critical value, an emotional warning index is generated. Step 4: Obtain target question and answer data for question and answer operation analysis. Using the user's historical average waiting time, the number of questions asked per unit time, and the keyword library as dynamic evaluation targets, assess the urgency of the question and answer dynamics and generate a task evaluation index. Step 5: Based on the task evaluation index and sentiment warning index, and combined with user identity and permissions, conduct financial decision analysis. By selecting actions from the answer candidate set in the financial knowledge base, and dynamically optimizing the indicators based on data, generate financial answers and interaction strategies.

[0014] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: This AI-based multimodal financial question-answering system and method can gain insight into users' emotional tendencies when asking questions through voice emotion features, text keywords help quickly locate the core of the question, contextual logical association features can understand the background and coherence of the question, and image recognition results can supplement visual information, making the system's understanding of the question more comprehensive and three-dimensional. A time-series model of user emotional state is constructed, and by analyzing the intensity fluctuations of voice emotion features and the density of text emotion keywords, real-time changes in user emotions can be perceived. In financial question-answering scenarios, user emotions may reflect important information such as anxiety about financial situation and satisfaction with the answer. Using users' historical average waiting time, the number of questions per unit time, and the keyword library as dynamic evaluation standards, the system assesses the urgency of the question-answering dynamics. Based on multiple dynamic evaluation indicators, the system can adjust the evaluation strategy in real time according to user behavior and question characteristics to better adapt to the needs of different users. Financial decision analysis is performed based on task evaluation index and emotion warning index, combined with user identity and permissions, taking into account the urgency of the task, user emotions, and user permissions in financial decision-making, avoiding the limitations of single-factor decision-making. Attached Figure Description

[0015] Figure 1 A schematic diagram of the system steps of the present invention is shown; Figure 2 A schematic diagram of the method flow structure of the present invention is shown. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Example 1:

[0018] like Figure 1 As shown, the AI-based multimodal financial question-answering system includes a multimodal input parsing module, a question-answering analysis module, an emotion perception module, a question-answering dynamic evaluation module, and a financial dynamic decision-making module. The multimodal input parsing module is used to extract and parse multimodal data features from the target question-and-answer data, including extracting speech emotion features, text keywords, contextual logical association features, and image recognition results from the target question-and-answer data; The question-answering analysis module is used to perform deep semantic analysis and question type identification on the target question-answering data. It identifies the semantic type based on text keywords and key features of context logic to determine the question type of the target question-answering data. It analyzes the correlation between questions through context logic association features, verifies the question type based on the correlation, and extracts and verifies key parameters by combining image recognition features to generate the target question type. The emotion perception module is used to build a time-series model of user emotional state based on deep learning. By analyzing the intensity fluctuation of voice emotion features and the density of text emotion keywords, it calculates the emotion stability index. When the stability index is detected to be lower than the critical value, an emotion warning index is generated. The question-and-answer dynamic evaluation module acquires target question-and-answer data for question-and-answer operation analysis. It uses the user's historical average waiting time, the number of questions asked per unit time, and the keyword library as dynamic evaluation targets to assess the urgency of the question-and-answer dynamics and generate a task evaluation index. The financial dynamic decision-making module analyzes financial decisions based on task evaluation index and sentiment warning index, combined with user identity and permissions. It generates financial answers and interaction strategies by selecting actions from the answer candidate set in the financial knowledge base and dynamically optimizing indicators based on data.

[0019] The specific process of multimodal data feature extraction and parsing of the target question-and-answer data is as follows: The speech fundamental frequency is obtained through audio frame processing technology, and identification is performed based on the audio standard emotional fundamental frequency threshold. Abnormal turning points of the speech fundamental frequency are marked to form a speech emotional feature vector. Deep semantic analysis of the text content is performed, and financial professional terms, business scenario keywords and sentiment polarity words are extracted through entity recognition technology to generate text keyword feature vectors; Based on the text content, causal relationship analysis is performed, and causal, parallel, and conditional relationships are analyzed and compared according to the problem logic chain diagram to obtain the contextual logical association characteristics. By locating key elements such as table regions, seals, and handwritten annotations in the image, character recognition and structure extraction are performed on the text region to generate table structure features, numerical distribution patterns, and key data identifiers, thus obtaining the image recognition results.

[0020] Semantic type identification is performed based on text keywords and key features of contextual logic to determine the question type of the target question-answering data. The specific process is as follows: Obtain a knowledge base in the financial field, define question type tags, including factual questions, analytical questions, and causal questions, and form classification criteria; The text keyword feature vector and the context logic association feature vector are fused together, and combined with the speech emotion features and image recognition results to form a multimodal feature fusion vector; The multimodal feature fusion vector is compared with the question type label to identify the question type. The semantic similarity between the question and the standard question library is calculated to obtain the initial identification type. Verify the initial identification results of the question type, combine the contextual logical features, verify the consistency between the question type identification results and the overall text semantic logic, generate a visual keyword distribution and question type distribution, and generate an interpretable analysis report; Extract structured parameters from the image, check the scope of keywords in the question and the image content, and finally generate a question type label and verification report, including main type and subtype features.

[0021] Specifically, this includes: establishing a mapping relationship between semantic features of question-and-answer data and question types; building an association model between semantic features and question types using natural language processing techniques; extracting multimodal features from the target question-and-answer data as model input, including extracting speech sentiment features, text keywords, contextual logical association features, and image recognition results from the target question-and-answer data; selecting the semantic feature combination with the highest relevance to the question type as model input through feature importance analysis; and verifying the feature redundancy of the model through confusion matrix analysis and feature co-occurrence; applying the trained classification model to the target question-and-answer data, batch identifying the question type of each data point, and outputting a structured dataset containing the original text, semantic feature vectors, and question type labels. The specific process of constructing a time-series model of user sentiment state based on deep learning is as follows: S300. Based on speech emotion features, perform sentiment analysis, using a sliding window as a unit, average the sentiment probability within the window to generate a smooth, time-series-arranged speech emotion intensity vector, define a time window and calculate the ratio of the number of negative sentiment keywords appearing in the window to the total number of words, as the negative sentiment density of the text. S301. Establish a unified timeline and use timestamps to align the voice emotion intensity vector and text emotion density to form synchronized multimodal emotion time-series data. S302. Using the aligned speech emotion intensity vector and text emotion density as input sample data, construct an emotion state temporal model; Within a short time window, the standard deviation of instantaneous emotional intensity is calculated as a measure of short-term fluctuations. Combined with the density of negative emotions in the text, a stability index is obtained, which is then used as the output dimension for constructing a temporal model of emotional state.

[0022] When the stability index is detected to be below a critical value, an emotion warning index is generated. The specific process is as follows: S400. The stability index critical value is a threshold for judging the emotional stability of users. According to the emotional data analysis and actual scenario requirements, a reasonable stability index critical value is set, denoted as T. S401. Calculate the emotional warning index according to the gap between the emotional stability index and the critical value, combined with the warning value. The emotional warning index = T * 0.5 - S * 0.5. S402. Compare the calculated emotional stability index S with the set stability index critical value. When S < T, it is determined that the user's emotion is extremely unstable, and the risk value R is added; if S ≥ T, it is determined that the user's emotion is in a relatively stable state, and the question and answer continue to be asked.

[0023] Evaluate the task urgency of the question and answer dynamics to generate a task evaluation index. The specific process is as follows: Extract the duration data from when the user initiated a question and answer request to when a response was obtained in the past, calculate the user's historical average waiting time, obtain the number of times the user initiated questions within a time period, and calculate the number of questions asked per unit time. Perform keyword matching on the content of the user's current question. Natural language processing technology can be used to extract the keywords in the question and compare them with the keyword library to obtain the matching keywords. Based on the user's historical average waiting time, the number of questions asked per unit time, and the number of matching keywords, perform dynamic evaluation calculations to obtain the task evaluation index. The task evaluation index = P * 0.4 + T * 0.4. According to the size of the task urgency index, the tasks are divided into different urgency levels.

[0024] Based on the data optimization analysis index dynamics, generate financial answers and interaction strategies. The specific process is as follows: Based on the task evaluation index and the emotional warning index, combined with the user's identity permissions, make financial decisions, perform financial permission analysis and restrictions, and obtain the question of the target question and answer data from the financial index library to get a multi-granularity answer candidate set. Calculate the semantic matching degree between each answer and the user's question, and select the answer with the highest matching degree as the financial answer. The data optimization analysis index dynamics are style decision-making and information presentation. Based on the emotional warning index value, task urgency, and permission level, generate financial answers and generate financial answers and interaction strategies.

[0025] The risk warning and compliance execution module is used to construct a three-dimensional risk monitoring matrix to achieve active warning and automatic response for high-risk situations. By monitoring the three-dimensional space of "user permissions - sentiment index - urgency" in real time, when an abnormal pattern of "high-privilege users + high negative sentiment + high urgency" concurrently accessing core sensitive financial data is detected, a multi-level early warning rule chain is triggered. A high-risk log is generated using question-and-answer risk verification, and a structured alert is sent to the regulator through a question-and-answer encrypted channel. At the same time, a predefined crisis response protocol is activated, and finally, the output content is reviewed for compliance and risk calibration through a security verification layer.

[0026] The problem logic chain graph is a mapping diagram of causal relationships, parallel relationships, and conditional relationships constructed by converting historical text into semantic vectors and performing correlation analysis.

[0027] A multimodal financial question-answering system and methodology based on AI leverages voice emotion features to understand users' emotional tendencies when asking questions. Text keywords help quickly pinpoint the core of the question, contextual logical association features help understand the background and coherence of the question, and image recognition results supplement visual information, making the system's understanding of the question more comprehensive and three-dimensional. A time-series model of user emotional states is constructed. By analyzing the intensity fluctuations of voice emotion features and the density of text emotion keywords, the system can perceive changes in user emotions in real time. In financial question-answering scenarios, user emotions may reflect important information such as anxiety about financial situations and satisfaction with answers. An emotion stability index is calculated and a threshold is set. When the stability index is detected to be below the threshold, an emotion warning index is generated. This helps the system detect abnormal user emotions in advance and take corresponding measures, such as adjusting answer strategies and promptly reassuring users, thus improving user experience. The system uses the user's historical average waiting time, the number of questions per unit time, and the keyword library as dynamic evaluation standards to assess the urgency of the question-answering dynamics. Based on multiple dynamic evaluation indicators, the system can adjust the evaluation strategy in real time according to user behavior and question characteristics to better adapt to the needs of different users. Based on the task evaluation index and the emotion warning index, combined with user identity and permissions, financial decision analysis is performed, making decisions more comprehensive and scientific. It takes into account the urgency of the task, user emotions, and user authority in financial decision-making, thus avoiding the limitations of single-factor decision-making.

[0028] Example 2:

[0029] like Figure 2 As shown, the AI-based multimodal financial question answering method, applied to the AI-based multimodal financial question answering system according to any one of claims 1-8, includes the following steps: Step one is used to extract and parse multimodal data features from the target question-and-answer data, including extracting speech emotion features, text keywords, contextual logical association features, and image recognition results from the target question-and-answer data; Step two is used to perform deep semantic analysis and question type identification on the target question and answer data. Semantic type identification is performed based on text keywords and key features of context logic to determine the question type of the target question and answer data. The correlation between questions is analyzed through context logic association features, and the question type is verified based on the correlation. Key parameters are extracted and verified by combining image recognition features to generate the target question type data. Step 3 is used to construct a time-series model of user emotional state based on deep learning. By analyzing the intensity fluctuation of voice emotional features and the density of text emotional keywords, an emotional stability index is calculated. When the stability index is detected to be lower than the critical value, an emotional warning index is generated. Step 4: Obtain target question and answer data for question and answer operation analysis. Using the user's historical average waiting time, the number of questions asked per unit time, and the keyword library as dynamic evaluation targets, assess the urgency of the question and answer dynamics and generate a task evaluation index. Step 5: Based on the task evaluation index and sentiment warning index, and combined with user identity and permissions, conduct financial decision analysis. By selecting actions from the answer candidate set in the financial knowledge base, and dynamically optimizing the indicators based on data, generate financial answers and interaction strategies.

[0030] The size of the interval and threshold is set to facilitate comparison. The size of the threshold depends on the amount of sample data and the number of bases set by those skilled in the art for each set of sample data; as long as it does not affect the ratio between the parameter and the quantized value.

[0031] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation. In the two embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of modules is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection between the shown or discussed components may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms. The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. An AI-based multimodal financial question-answering system, characterized in that: It includes a multimodal input parsing module, a question-and-answer analysis module, an emotion perception module, a question-and-answer dynamic evaluation module, and a financial dynamic decision-making module; The multimodal input parsing module is used to extract and parse multimodal data features from the target question-and-answer data, including extracting speech emotion features, text keywords, contextual logical association features, and image recognition results from the target question-and-answer data; The question-answering analysis module is used to perform deep semantic analysis and question type identification on the target question-answering data. It identifies the semantic type based on text keywords and key features of context logic to determine the question type of the target question-answering data. It analyzes the correlation between questions through context logic association features, verifies the question type based on the correlation, and extracts and verifies key parameters by combining image recognition features to generate the target question type. The emotion perception module is used to build a time-series model of user emotional state based on deep learning. By analyzing the intensity fluctuation of voice emotion features and the density of text emotion keywords, it calculates the emotion stability index. When the stability index is detected to be lower than the critical value, an emotion warning index is generated. The question-and-answer dynamic evaluation module acquires target question-and-answer data for question-and-answer operation analysis. It uses the user's historical average waiting time, the number of questions asked per unit time, and the keyword library as dynamic evaluation targets to assess the urgency of the question-and-answer dynamics and generate a task evaluation index. The financial dynamic decision-making module analyzes financial decisions based on task evaluation index and sentiment warning index, combined with user identity and permissions. It selects actions from the answer candidate set in the financial knowledge base, and generates financial answers and interaction strategies based on data optimization analysis indicators. The specific process of multimodal data feature extraction and parsing of the target question-and-answer data is as follows: The speech fundamental frequency is obtained through audio frame processing technology, and identification is performed based on the audio standard emotional fundamental frequency threshold. Abnormal turning points of the speech fundamental frequency are marked to form a speech emotional feature vector. Deep semantic analysis of the text content is performed, and financial professional terms, business scenario keywords and sentiment polarity words are extracted through entity recognition technology to generate text keyword feature vectors; Based on the text content, causal relationship analysis is performed, and causal, parallel, and conditional relationships are analyzed and compared according to the problem logic chain diagram to obtain the contextual logical association characteristics. By locating key elements such as table regions, seals, and handwritten annotations in the image, character recognition and structure extraction are performed on the text region to generate table structure features, numerical distribution patterns, and key data identifiers, thus obtaining the image recognition results.

2. The AI-based multimodal financial question-answering system according to claim 1, characterized in that, Semantic type identification is performed based on text keywords and key features of contextual logic to determine the question type of the target question-answering data. The specific process is as follows: Obtain a knowledge base in the financial field, define question type tags, including factual questions, analytical questions, and causal questions, and form classification criteria; The text keyword feature vector and the contextual logical association feature vector are fused together, and combined with the speech emotion features and image recognition results to form a multimodal feature fusion vector; Compare and identify the problem type of the multimodal feature fusion vector with the problem type of the problem type label, calculate the semantic similarity between the problem and the standard problem library, and obtain the initial recognition type; Verify the initial recognition type result of the problem type, combine the context logical features, check whether the problem type recognition result is consistent with the overall text semantic logic, generate a visual keyword distribution and problem type distribution, and generate an interpretable analysis report; Extract structured parameters from the picture, check the range of the keyword problem in the problem and the picture content, and finally the problem type label and verification report, including the main type and subtype features.

3. The AI-based multimodal financial question-answering system according to claim 1, characterized in that, Construct a user emotional state time series model based on deep learning, and the specific process is as follows: S300. Conduct emotional analysis based on voice emotional features. Taking the sliding window as the unit, average the emotional probabilities within the window to generate a smooth voice emotional intensity vector arranged in time series. Define a time window and calculate the ratio of the number of negative emotional keywords appearing within the window to the total number of words as the text negative emotional density; S301. Establish a unified time axis, and use timestamps to align the voice emotional intensity vector and the text emotional density to form synchronous multimodal emotional time series data; S302. Use the aligned voice emotional intensity vector and text emotional density as input sample data to construct an emotional state time series model; Within a short time window, calculate the standard deviation of the instantaneous emotional intensity as a measure of short-term fluctuations, and combine it with the number of text negative emotional densities to obtain a stability index. Use the obtained stability index as the output dimension of the constructed emotional state time series model.

4. The AI-based multimodal financial question-answering system according to claim 1, characterized in that, When it is detected that the stability index is lower than the critical value, generate an emotional warning index, and the specific process is as follows: S400. The stability index critical value is a threshold for judging the stability of the user's mood. According to the emotional data analysis and actual scenario requirements, set a reasonable stability index critical value, denoted as T; S401. According to the gap degree between the emotional stability index and the critical value, calculate the emotional warning index in combination with the warning value. The emotional warning index = T * 0.5 - S * 0.5; S402. Compare the calculated emotional stability index S with the set stability index critical value. When S < T, it is determined that the user's mood is extremely unstable, and a risk value R is added; if S ≥ T, it is determined that the user's mood is in a relatively stable state, and the question and answer continue to be asked.

5. The AI-based multimodal financial question-answering system according to claim 1, characterized in that, Evaluate the task urgency of the question and answer dynamics and generate a task evaluation index. The specific process is as follows: Extract the duration data from when the user发起 a question and answer request to when a response is obtained, calculate the user's historical average waiting time, obtain the number of questions the user发起 within the time period, and calculate the number of questions发起 per unit time; Perform keyword matching on the content of the user's current question. Natural language processing techniques can be used to extract the keywords in the question and compare them with the keyword library to obtain the matching keywords; Based on the user's historical average waiting time, the number of questions发起 per unit time, and the number of matching keywords, conduct dynamic evaluation calculations to obtain a task evaluation index. The task evaluation index = P * 0.4 + T * 0.4; Tasks are categorized into different urgency levels based on their urgency index.

6. The AI-based multimodal financial question-answering system according to claim 1, characterized in that, Based on dynamic data optimization analysis indicators, the generated financial answers and interaction strategies are as follows: Based on the task evaluation index and sentiment warning index, financial decisions are made in combination with user identity and permissions. Financial permission analysis and restrictions are implemented, and questions from the target question-and-answer data are obtained from the financial index library to obtain a multi-granularity set of answer candidates. Calculate the semantic matching degree between each answer and the user's question, select the answer with the highest matching degree as the financial answer, and dynamically use data optimization analysis indicators for style decisions and information presentation; Based on the emotional warning index value, task urgency and permission level, financial answers are generated, along with the generated financial answers and interaction strategies.

7. The AI-based multimodal financial question-answering system according to claim 1, characterized in that, The risk warning and compliance execution module is used to build a three-dimensional risk monitoring matrix to achieve proactive warning and automatic response in high-risk situations; By monitoring the three-dimensional space of "user permissions - sentiment index - urgency" in real time, when an abnormal pattern of "high-privilege user + high negative sentiment + high urgency" concurrently accessing core sensitive financial data is detected, a multi-level early warning rule chain is triggered. A high-risk log is generated using question-and-answer risk verification, and a structured alert is sent to the regulator through a question-and-answer encrypted channel. At the same time, a predefined crisis response protocol is activated, and finally, the output content is reviewed for compliance and risk calibration through a security verification layer.

8. An AI-based multimodal financial question-answering method, applied to the AI-based multimodal financial question-answering system according to any one of claims 1-7, characterized in that, Includes the following steps: Step one is used to extract and parse multimodal data features from the target question-and-answer data, including extracting speech emotion features, text keywords, contextual logical association features, and image recognition results from the target question-and-answer data; Step two is used to perform deep semantic analysis and question type identification on the target question and answer data. Semantic type identification is performed based on text keywords and key features of context logic to determine the question type of the target question and answer data. The correlation between questions is analyzed through context logic association features, and the question type is verified based on the correlation. Key parameters are extracted and verified by combining image recognition features to generate the target question type data. Step 3 is used to construct a time-series model of user emotional state based on deep learning. By analyzing the intensity fluctuation of voice emotional features and the density of text emotional keywords, an emotional stability index is calculated. When the stability index is detected to be lower than the critical value, an emotional warning index is generated. Step 4: Obtain target question and answer data for question and answer operation analysis. Using the user's historical average waiting time, the number of questions asked per unit time, and the keyword library as dynamic evaluation targets, assess the urgency of the question and answer dynamics and generate a task evaluation index. Step 5: Based on the task evaluation index and sentiment warning index, and combined with user identity and permissions, conduct financial decision analysis. By selecting actions from the answer candidate set in the financial knowledge base, and dynamically optimizing the indicators based on data, generate financial answers and interaction strategies.

Images