Time sequence task decision generation method and device, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, provides a time sequence task decision generation method, device and equipment and a medium, can be applied to financial and medical health care service scenes, can extract multi-modal time sequence characteristics in input data, and provides a comprehensive data basis for subsequent processing. Performing cross-modal time sequence alignment on the multi-modal time sequence features to realize accurate matching of the multi-modal data in the time dimension; time sequence memory enhancement is performed on the multi-mode time sequence alignment feature sequence, so that the capability of capturing a long-time-span dependency relationship can be enhanced, and the problem of information forgetting during processing of long-time-sequence data is solved; and inputting the multi-modal time sequence enhanced feature sequence into a time sequence inference network, and inputting the output feature into a decision-making layer to obtain decision-making data, so that complex interaction information between different time steps and different modals can be deeply mined, and a corresponding decision-making result is output according to an actual task. And the decision-making accuracy and applicability of the model in the time sequence task are improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and medium for generating decisions for time-series tasks. Background Technology

[0002] In the current VLA (Vision-Language-Action) domain, there are still significant shortcomings in handling tasks with temporal characteristics.

[0003] Specifically, when traditional models process long-term data (such as long video content understanding and complex multi-step task instruction execution), they often struggle to learn dependencies over long time spans due to the vanishing or exploding gradient problem, leading to the forgetting of early information. For example, when a robot in a financial hall performs a multi-step assembly task, subsequent actions may result in operational errors because it forgets the key assembly sequence in the earlier instructions.

[0004] Furthermore, most existing multimodal fusion methods only focus on information fusion at the same point in time, ignoring the dynamic correlation between visual, linguistic, and action modalities over time. For example, in video caption generation tasks in medical teaching scenarios, the model cannot accurately align the temporal progression of actions in the video with the temporal cues of the linguistic descriptions, resulting in generated captions that may not match the visuals in terms of temporal logic.

[0005] In addition, existing models lack the ability to adapt to temporal changes and are difficult to adjust their strategies when faced with changes in task flow or dynamic environmental interference, which severely limits the application effect of VLA tasks in temporal scenarios. Summary of the Invention

[0006] In view of the above, it is necessary to provide a method, apparatus, device and medium for decision generation of time-series tasks, in order to solve the problem of poor processing effect of decision generation tasks with time-series characteristics in the VLA field.

[0007] A decision generation method for a time-series task, the decision generation method for the time-series task comprising:

[0008] In response to the decision generation instructions for the target time series task, multimodal time series features are extracted from the input data;

[0009] Cross-modal temporal alignment is performed on the multimodal temporal features to obtain a multimodal temporal alignment feature sequence;

[0010] Temporal memory enhancement is performed on the multimodal temporal alignment feature sequence to obtain a multimodal temporal enhancement feature sequence;

[0011] The multimodal temporal enhancement feature sequence is input into the temporal inference network, and the output features of the temporal inference network are obtained.

[0012] The output features are input into the decision layer corresponding to the target time-series task to obtain the target decision data of the target time-series task.

[0013] A decision generation apparatus for a time-series task, the decision generation apparatus for a time-series task comprising:

[0014] The extraction unit is used to extract multimodal temporal features from the input data in response to the decision generation instructions for the target temporal task;

[0015] An alignment unit is used to perform cross-modal temporal alignment on the multimodal temporal features to obtain a multimodal temporal aligned feature sequence;

[0016] An enhancement unit is used to perform temporal memory enhancement on the multimodal temporal alignment feature sequence to obtain a multimodal temporal enhancement feature sequence;

[0017] The inference unit is used to input the multimodal temporal enhancement feature sequence into the temporal inference network and obtain the output features of the temporal inference network.

[0018] The decision unit is used to input the output features into the decision layer corresponding to the target time-series task to obtain the target decision data of the target time-series task.

[0019] A computer device, the computer device comprising:

[0020] Memory, storing at least one instruction; and

[0021] The processor executes instructions stored in the memory to implement the decision generation method for the timing task.

[0022] A computer-readable storage medium storing at least one instruction, which is executed by a processor in a computer device to implement a decision generation method for a timing task.

[0023] As can be seen from the above technical solutions, this invention can extract multimodal temporal features from input data, providing a comprehensive data foundation for subsequent processing; cross-modal temporal alignment of multimodal temporal features yields multimodal temporal alignment feature sequences, achieving accurate matching of multimodal data in the time dimension; temporal memory enhancement of the multimodal temporal alignment feature sequences yields multimodal temporal enhancement feature sequences, which enhances the ability to capture long-term dependencies, thereby solving the information forgetting problem when processing long-term temporal data; inputting the multimodal temporal enhancement feature sequences into a temporal inference network and inputting the output features into the decision layer yields target decision data for the target temporal task, enabling in-depth mining of complex interaction information between different time steps and different modalities, and outputting corresponding decision results according to the actual task, thus improving the model's decision accuracy and applicability in temporal tasks. Attached Figure Description

[0024] Figure 1 This is a flowchart of a preferred embodiment of the decision generation method for time-series tasks of the present invention.

[0025] Figure 2 This is a functional block diagram of a preferred embodiment of the decision generation device for time-series tasks of the present invention.

[0026] Figure 3 This is a schematic diagram of the structure of a computer device that is a preferred embodiment of the decision generation method for implementing time-series tasks according to the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0028] like Figure 1 The diagram shown is a flowchart of a preferred embodiment of the decision generation method for time-series tasks according to the present invention. The order of the steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements.

[0029] The decision generation method for the time-series task is applied to one or more computer devices. The computer device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0030] The computer device can be any electronic product that can interact with the user, such as a personal computer, tablet computer, smartphone, personal digital assistant (PDA), game console, interactive network television (IPTV), smart wearable device, etc.

[0031] The computer equipment may also include network equipment and / or user equipment. The network equipment includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0032] The server can be a standalone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0033] Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0034] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0035] The network in which the computer device is located includes, but is not limited to, the Internet, wide area network, metropolitan area network, local area network, and virtual private network (VPN).

[0036] S10, in response to the decision generation instruction for the target time series task, extracts multimodal time series features from the input data.

[0037] In this embodiment, the target time-series task can be a control task for a robot in a bank lobby in the financial field, or a task for generating instructional videos in the medical and health field.

[0038] In this embodiment, the extraction of multimodal temporal features from the input data includes:

[0039] The video data in the input data is split into a continuous sequence of video frames; spatial features of each frame and temporal features between adjacent frames are extracted using a 3D convolutional neural network to obtain an initial visual feature sequence; the multi-head attention mechanism in the temporal Transformer is used to capture the visual feature dependencies over a long temporal range in the initial visual feature sequence to obtain a visual temporal feature vector; and / or

[0040] The input text data is segmented and part-of-speech tagged to obtain a word sequence; a bidirectional long short-term memory network is used to capture the semantic dependencies between the preceding and following contexts of the word sequence to obtain a hidden state sequence; the hidden state sequence is input into an attention mechanism layer to obtain a language temporal feature vector; and / or

[0041] The action data in the input data is input into the gating loop unit to obtain the action timing feature vector.

[0042] For example, in high-frequency trading monitoring videos in financial scenarios, visual temporal feature extraction can process real-time trading interface video streams and capture temporal changes in visual elements such as candlestick charts and trading volume; linguistic temporal feature extraction can analyze the speech-to-text content of analysts' real-time comments and extract key semantics; and action temporal feature extraction can monitor the timing of traders' mouse and keyboard operations to determine whether the operations are compliant.

[0043] For example, in live surgical videos in medical settings, visual temporal feature extraction can be used to extract visual temporal features such as the movement of surgical instruments and tissue changes; linguistic temporal feature extraction can be used to process the text information of doctors' intraoperative dialogues and instructions; and action temporal feature extraction can be used to analyze the timing of the robotic arm movements of surgical robots, providing support for surgical teaching and quality assessment.

[0044] Through the above embodiments, visual temporal feature vectors, language temporal feature vectors, and action temporal feature vectors can be extracted from the input data respectively, and used as the multimodal temporal features, providing a comprehensive data foundation for subsequent processing.

[0045] S11, perform cross-modal temporal alignment on the multimodal temporal features to obtain a multimodal temporal alignment feature sequence.

[0046] In this embodiment, the step of performing cross-modal temporal alignment on the multimodal temporal features to obtain a multimodal temporal aligned feature sequence includes:

[0047] Obtain the first modal temporal feature sequence and the second modal temporal feature sequence corresponding to any two modalities in the multimodal temporal features;

[0048] An alignment score matrix is ​​constructed between the first modality temporal feature sequence and the second modality temporal feature sequence using a multilayer perceptron; wherein each element in the alignment score matrix represents the alignment score between each first modality temporal feature and each second modality temporal feature;

[0049] Obtain the alignment path from the first element to the last element in the alignment score matrix; wherein the sum of the alignment scores of all elements traversed by the alignment path is the highest.

[0050] Unify the temporal dimension of the alignment paths across all modalities to obtain the temporal feature sequence of each modality after temporal alignment;

[0051] The temporal feature sequences of each modality after temporal alignment are concatenated according to a unified timestamp to obtain the multimodal temporal alignment feature sequence.

[0052] For example, in the correlation analysis between financial news and stock market trends, cross-modal time-series alignment can accurately align the time cues of the language descriptions in news reports with the visual time changes of stock market K-line charts and the time sequences of capital inflows and outflows, thereby more clearly analyzing the timeliness and extent of the impact of news events on the stock market. In the emergency rescue process in medical scenarios, the vital sign curves of the monitor (visual), the doctor's instructions (verbal), and the nurse's rescue operations (actions) can be time-series aligned, which facilitates the post-event analysis of the time matching degree of each link in the rescue process, thereby optimizing the rescue process.

[0053] Through the above embodiments, the temporal correlation of multimodal data can be quantified using matrix, and the optimal matching path can be found using dynamic programming. Ultimately, visual, linguistic, and action modalities are aligned on the time axis, achieving accurate matching of visual, linguistic, and action modalities in the time cues. This breaks through the limitations of traditional models in the time dimension of fusion, thereby solving the problem of temporal misalignment of multimodal data and improving the quality and logical coherence of multimodal information fusion.

[0054] S12, perform temporal memory enhancement on the multimodal temporal alignment feature sequence to obtain a multimodal temporal enhancement feature sequence.

[0055] In this embodiment, before performing temporal memory enhancement on the multimodal temporally aligned feature sequence, the method further includes:

[0056] Create multiple memory slots and initialize each memory slot as a zero vector;

[0057] Obtain the current time corresponding to each memory slot, and extract the splicing features of the current time corresponding to each memory slot from the historical multimodal temporal alignment feature sequence;

[0058] Calculate the matching degree between the splicing features of the current moment corresponding to each memory slot and the splicing features of the previous moment corresponding to the previous memory slot, as the degree of association between each memory slot and historical memory;

[0059] The softmax function is used to normalize the correlation between each memory slot and the historical memory, thus obtaining the weight corresponding to each memory slot.

[0060] Based on the weight corresponding to each memory slot, the splicing feature of the current time corresponding to each memory slot is fused with the splicing feature of the previous time corresponding to the previous memory slot to obtain the updated feature of the current time corresponding to each memory slot.

[0061] Each memory slot is updated based on the update characteristics of the current moment corresponding to each memory slot, thus obtaining a temporal memory unit based on the memory network.

[0062] The degree of association can be calculated using the cosine similarity function.

[0063] Through the above embodiments, it is possible to construct a temporal memory unit based on a memory network, which facilitates temporal enhancement during subsequent reasoning.

[0064] Specifically, the step of fusing the concatenation features of the current time step corresponding to each memory slot with the concatenation features of the previous time step corresponding to the previous memory slot according to the weights corresponding to each memory slot to obtain the update features of the current time step corresponding to each memory slot includes:

[0065] Calculate the product of the weight corresponding to each memory slot and the concatenated feature corresponding to the previous memory slot at the previous time step to obtain the first feature corresponding to each memory slot;

[0066] Calculate the difference between 1 and the weight corresponding to each memory slot to obtain the second weight corresponding to each memory slot;

[0067] Calculate the product of the second weight corresponding to each memory slot and the concatenated feature corresponding to the current time step for each memory slot to obtain the second feature corresponding to each memory slot;

[0068] The sum of the first feature corresponding to each memory slot and the second feature corresponding to each memory slot is calculated to obtain the updated feature corresponding to each memory slot at the current time.

[0069] In this embodiment, the step of performing temporal memory enhancement on the multimodal temporally aligned feature sequence to obtain a multimodal temporally enhanced feature sequence includes:

[0070] For each currently spliced ​​feature in the multimodal temporal alignment feature sequence, calculate the degree of association between the current spliced ​​feature and each memory slot;

[0071] Based on the correlation between the current splicing feature and each memory slot, the splicing features corresponding to all memory slots are weighted and aggregated to obtain the multimodal temporal enhancement feature sequence.

[0072] For example, in financial credit risk assessment tasks, it is necessary to analyze long-term time-series information such as a company's long-term operating data, changes in financial statements (visual), communication records between the bank and the company (verbal), and the company's cash flow operations (actions). This embodiment uses time-series memory enhancement to record a company's early key financial indicators and operating behaviors to help more accurately assess the company's credit risk.

[0073] For example, in the long-term monitoring of chronic disease patients in medical settings, this embodiment can store information such as examination images (visual), symptom descriptions (verbal), and changes in medication and lifestyle habits (actions) at different times. This information can be combined with historical data to more accurately determine the trend of disease progression and formulate personalized treatment plans in subsequent diagnoses.

[0074] The above embodiments can solve the problem of information forgetting when processing long-term time-series data, effectively store and utilize historical information, and enhance the ability to capture long-term dependencies.

[0075] S13, input the multimodal temporal enhancement feature sequence into the temporal inference network, and obtain the output features of the temporal inference network.

[0076] In this embodiment, the step of inputting the multimodal temporal enhancement feature sequence into the temporal inference network and obtaining the output features of the temporal inference network includes:

[0077] The multimodal temporal enhancement feature sequence is input into the temporal inference network composed of the Transformer architecture, and the multi-head self-attention mechanism is used to model the dependency relationship of the multimodal temporal enhancement feature sequence in the time dimension to obtain the output feature.

[0078] In the above embodiments, the multi-head self-attention mechanism can model the dependencies of multimodal features in the time dimension and mine complex interaction information.

[0079] S14, the output features are input to the decision layer corresponding to the target time-series task to obtain the target decision data of the target time-series task.

[0080] In this embodiment, different task types can correspond to different decision-making layers.

[0081] For example, for robot control tasks, a fully connected neural network can be combined with a reinforcement learning policy network to build the corresponding decision layer; for video content tasks, a corresponding decision layer can be built based on the Transformer decoder.

[0082] Through the above embodiments, after deeply mining the complex interaction information between different time steps and different modalities, the corresponding decision layer can be called according to the specific task to output the corresponding decision data, thereby improving the accuracy and applicability of decision-making in time-series tasks.

[0083] For example, in a financial scenario, after the temporal inference network outputs multimodal aligned features such as market conditions (visual), policy news (linguistic), and historical trading actions, the decision layer predicts and outputs the optimal trading timing and amount, thereby improving trading returns.

[0084] For example, in a medical and health setting, during surgical robot-assisted surgery, after the temporal inference network processes alignment features such as surgical field images (vision), doctor's instructions (language), and robot's historical actions (actions), the decision layer controls the robot to perform precise surgical actions, thereby improving surgical accuracy.

[0085] As can be seen from the above technical solutions, this invention can extract multimodal temporal features from input data, providing a comprehensive data foundation for subsequent processing; cross-modal temporal alignment of multimodal temporal features yields multimodal temporal alignment feature sequences, achieving accurate matching of multimodal data in the time dimension; temporal memory enhancement of the multimodal temporal alignment feature sequences yields multimodal temporal enhancement feature sequences, which enhances the ability to capture long-term dependencies, thereby solving the information forgetting problem when processing long-term temporal data; inputting the multimodal temporal enhancement feature sequences into a temporal inference network and inputting the output features into the decision layer yields target decision data for the target temporal task, enabling in-depth mining of complex interaction information between different time steps and different modalities, and outputting corresponding decision results according to the actual task, thus improving the model's decision accuracy and applicability in temporal tasks.

[0086] like Figure 2 The diagram shown is a functional block diagram of a preferred embodiment of the decision generation device for time-series tasks according to the present invention. The decision generation device 11 for time-series tasks includes an extraction unit 110, an alignment unit 111, an enhancement unit 112, a reasoning unit 113, and a decision unit 114. The module / unit referred to in this invention refers to a series of computer program segments that can be executed by a processor and perform a fixed function, and are stored in memory. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0087] The extraction unit 110 is used to extract multimodal temporal features from the input data in response to the decision generation instruction for the target temporal task.

[0088] In this embodiment, the target time-series task can be a control task for a robot in a bank lobby in the financial field, or a task for generating instructional videos in the medical and health field.

[0089] In this embodiment, the extraction unit 110 extracts multimodal temporal features from the input data, including:

[0090] The video data in the input data is split into a continuous sequence of video frames; spatial features of each frame and temporal features between adjacent frames are extracted using a 3D convolutional neural network to obtain an initial visual feature sequence; the multi-head attention mechanism in the temporal Transformer is used to capture the visual feature dependencies over a long temporal range in the initial visual feature sequence to obtain a visual temporal feature vector; and / or

[0091] The input text data is segmented and part-of-speech tagged to obtain a word sequence; a bidirectional long short-term memory network is used to capture the semantic dependencies between the preceding and following contexts of the word sequence to obtain a hidden state sequence; the hidden state sequence is input into an attention mechanism layer to obtain a language temporal feature vector; and / or

[0092] The action data in the input data is input into the gating loop unit to obtain the action timing feature vector.

[0093] For example, in high-frequency trading monitoring videos in financial scenarios, visual temporal feature extraction can process real-time trading interface video streams and capture temporal changes in visual elements such as candlestick charts and trading volume; linguistic temporal feature extraction can analyze the speech-to-text content of analysts' real-time comments and extract key semantics; and action temporal feature extraction can monitor the timing of traders' mouse and keyboard operations to determine whether the operations are compliant.

[0094] For example, in live surgical videos in medical settings, visual temporal feature extraction can be used to extract visual temporal features such as the movement of surgical instruments and tissue changes; linguistic temporal feature extraction can be used to process the text information of doctors' intraoperative dialogues and instructions; and action temporal feature extraction can be used to analyze the timing of the robotic arm movements of surgical robots, providing support for surgical teaching and quality assessment.

[0095] Through the above embodiments, visual temporal feature vectors, language temporal feature vectors, and action temporal feature vectors can be extracted from the input data respectively, and used as the multimodal temporal features, providing a comprehensive data foundation for subsequent processing.

[0096] The alignment unit 111 is used to perform cross-modal temporal alignment on the multimodal temporal features to obtain a multimodal temporal alignment feature sequence.

[0097] In this embodiment, the alignment unit 111 performs cross-modal temporal alignment on the multimodal temporal features to obtain a multimodal temporal aligned feature sequence including:

[0098] Obtain the first modal temporal feature sequence and the second modal temporal feature sequence corresponding to any two modalities in the multimodal temporal features;

[0099] An alignment score matrix is ​​constructed between the first modality temporal feature sequence and the second modality temporal feature sequence using a multilayer perceptron; wherein each element in the alignment score matrix represents the alignment score between each first modality temporal feature and each second modality temporal feature;

[0100] Obtain the alignment path from the first element to the last element in the alignment score matrix; wherein the sum of the alignment scores of all elements traversed by the alignment path is the highest.

[0101] Unify the temporal dimension of the alignment paths across all modalities to obtain the temporal feature sequence of each modality after temporal alignment;

[0102] The temporal feature sequences of each modality after temporal alignment are concatenated according to a unified timestamp to obtain the multimodal temporal alignment feature sequence.

[0103] For example, in the correlation analysis between financial news and stock market trends, cross-modal time-series alignment can accurately align the time cues of the language descriptions in news reports with the visual time changes of stock market K-line charts and the time sequences of capital inflows and outflows, thereby more clearly analyzing the timeliness and extent of the impact of news events on the stock market. In the emergency rescue process in medical scenarios, the vital sign curves of the monitor (visual), the doctor's instructions (verbal), and the nurse's rescue operations (actions) can be time-series aligned, which facilitates the post-event analysis of the time matching degree of each link in the rescue process, thereby optimizing the rescue process.

[0104] Through the above embodiments, the temporal correlation of multimodal data can be quantified using matrix, and the optimal matching path can be found using dynamic programming. Ultimately, visual, linguistic, and action modalities are aligned on the time axis, achieving accurate matching of visual, linguistic, and action modalities in the time cues. This breaks through the limitations of traditional models in the time dimension of fusion, thereby solving the problem of temporal misalignment of multimodal data and improving the quality and logical coherence of multimodal information fusion.

[0105] The enhancement unit 112 is used to perform temporal memory enhancement on the multimodal temporal alignment feature sequence to obtain a multimodal temporal enhancement feature sequence.

[0106] In this embodiment, before performing temporal memory enhancement on the multimodal temporal alignment feature sequence, the enhancement unit 112 creates multiple memory slots and initializes each memory slot as a zero vector.

[0107] Obtain the current time corresponding to each memory slot, and extract the splicing features of the current time corresponding to each memory slot from the historical multimodal temporal alignment feature sequence;

[0108] Calculate the matching degree between the splicing features of the current moment corresponding to each memory slot and the splicing features of the previous moment corresponding to the previous memory slot, as the degree of association between each memory slot and historical memory;

[0109] The softmax function is used to normalize the correlation between each memory slot and the historical memory, thus obtaining the weight corresponding to each memory slot.

[0110] Based on the weight corresponding to each memory slot, the splicing feature of the current time corresponding to each memory slot is fused with the splicing feature of the previous time corresponding to the previous memory slot to obtain the updated feature of the current time corresponding to each memory slot.

[0111] Each memory slot is updated based on the update characteristics of the current moment corresponding to each memory slot, thus obtaining a temporal memory unit based on the memory network.

[0112] The degree of association can be calculated using the cosine similarity function.

[0113] Through the above embodiments, it is possible to construct a temporal memory unit based on a memory network, which facilitates temporal enhancement during subsequent reasoning.

[0114] Specifically, the step of fusing the concatenation features of the current time step corresponding to each memory slot with the concatenation features of the previous time step corresponding to the previous memory slot according to the weights corresponding to each memory slot to obtain the update features of the current time step corresponding to each memory slot includes:

[0115] Calculate the product of the weight corresponding to each memory slot and the concatenated feature corresponding to the previous memory slot at the previous time step to obtain the first feature corresponding to each memory slot;

[0116] Calculate the difference between 1 and the weight corresponding to each memory slot to obtain the second weight corresponding to each memory slot;

[0117] Calculate the product of the second weight corresponding to each memory slot and the concatenated feature corresponding to the current time step for each memory slot to obtain the second feature corresponding to each memory slot;

[0118] The sum of the first feature corresponding to each memory slot and the second feature corresponding to each memory slot is calculated to obtain the updated feature corresponding to each memory slot at the current time.

[0119] In this embodiment, the enhancement unit 112 performs temporal memory enhancement on the multimodal temporal alignment feature sequence to obtain a multimodal temporal enhancement feature sequence including:

[0120] For each currently spliced ​​feature in the multimodal temporal alignment feature sequence, calculate the degree of association between the current spliced ​​feature and each memory slot;

[0121] Based on the correlation between the current splicing feature and each memory slot, the splicing features corresponding to all memory slots are weighted and aggregated to obtain the multimodal temporal enhancement feature sequence.

[0122] For example, in financial credit risk assessment tasks, it is necessary to analyze long-term time-series information such as a company's long-term operating data, changes in financial statements (visual), communication records between the bank and the company (verbal), and the company's cash flow operations (actions). This embodiment uses time-series memory enhancement to record a company's early key financial indicators and operating behaviors to help more accurately assess the company's credit risk.

[0123] For example, in the long-term monitoring of chronic disease patients in medical settings, this embodiment can store information such as examination images (visual), symptom descriptions (verbal), and changes in medication and lifestyle habits (actions) at different times. This information can be combined with historical data to more accurately determine the trend of disease progression and formulate personalized treatment plans in subsequent diagnoses.

[0124] The above embodiments can solve the problem of information forgetting when processing long-term time-series data, effectively store and utilize historical information, and enhance the ability to capture long-term dependencies.

[0125] The inference unit 113 is used to input the multimodal temporal enhancement feature sequence into the temporal inference network and obtain the output features of the temporal inference network.

[0126] In this embodiment, the inference unit 113 inputs the multimodal temporal enhancement feature sequence into the temporal inference network and obtains the output features of the temporal inference network, including:

[0127] The multimodal temporal enhancement feature sequence is input into the temporal inference network composed of the Transformer architecture, and the multi-head self-attention mechanism is used to model the dependency relationship of the multimodal temporal enhancement feature sequence in the time dimension to obtain the output feature.

[0128] In the above embodiments, the multi-head self-attention mechanism can model the dependencies of multimodal features in the time dimension and mine complex interaction information.

[0129] The decision unit 114 is used to input the output features into the decision layer corresponding to the target time-series task to obtain the target decision data of the target time-series task.

[0130] In this embodiment, different task types can correspond to different decision-making layers.

[0131] For example, for robot control tasks, a fully connected neural network can be combined with a reinforcement learning policy network to build the corresponding decision layer; for video content tasks, a corresponding decision layer can be built based on the Transformer decoder.

[0132] Through the above embodiments, after deeply mining the complex interaction information between different time steps and different modalities, the corresponding decision layer can be called according to the specific task to output the corresponding decision data, thereby improving the accuracy and applicability of decision-making in time-series tasks.

[0133] For example, in a financial scenario, after the temporal inference network outputs multimodal aligned features such as market conditions (visual), policy news (linguistic), and historical trading actions, the decision layer predicts and outputs the optimal trading timing and amount, thereby improving trading returns.

[0134] For example, in a medical and health setting, during surgical robot-assisted surgery, after the temporal inference network processes alignment features such as surgical field images (vision), doctor's instructions (language), and robot's historical actions (actions), the decision layer controls the robot to perform precise surgical actions, thereby improving surgical accuracy.

[0135] As can be seen from the above technical solutions, this invention can extract multimodal temporal features from input data, providing a comprehensive data foundation for subsequent processing; cross-modal temporal alignment of multimodal temporal features yields multimodal temporal alignment feature sequences, achieving accurate matching of multimodal data in the time dimension; temporal memory enhancement of the multimodal temporal alignment feature sequences yields multimodal temporal enhancement feature sequences, which enhances the ability to capture long-term dependencies, thereby solving the information forgetting problem when processing long-term temporal data; inputting the multimodal temporal enhancement feature sequences into a temporal inference network and inputting the output features into the decision layer yields target decision data for the target temporal task, enabling in-depth mining of complex interaction information between different time steps and different modalities, and outputting corresponding decision results according to the actual task, thus improving the model's decision accuracy and applicability in temporal tasks.

[0136] like Figure 3 The diagram shown is a schematic representation of the structure of a computer device that implements a decision generation method for time-series tasks according to a preferred embodiment of the present invention.

[0137] The computer device 1 may include a memory 12, a processor 13, and a bus (the arrow in the figure represents the bus), and may also include a computer program stored in the memory 12 and executable on the processor 13, such as a decision generation program for a timing task.

[0138] Those skilled in the art will understand that the schematic diagram is merely an example of computer device 1 and does not constitute a limitation on computer device 1. Computer device 1 can be either a bus topology or a star topology. Computer device 1 may also include more or fewer other hardware or software than shown in the diagram, or different component arrangements. For example, computer device 1 may also include input / output devices, network access devices, etc.

[0139] It should be noted that the computer device 1 described is merely an example. Other existing or future electronic products that are adaptable to this invention should also be included within the scope of protection of this invention and are incorporated herein by reference.

[0140] The memory 12 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the computer device 1, such as a portable hard drive of the computer device 1. In other embodiments, the memory 12 can be an external storage device of the computer device 1, such as a plug-in portable hard drive, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., equipped on the computer device 1. Furthermore, the memory 12 can include both internal and external storage units of the computer device 1. The memory 12 can be used not only to store application software and various types of data installed on the computer device 1, such as the code of a decision generation program for a time-series task, but also to temporarily store data that has been output or will be output.

[0141] In some embodiments, the processor 13 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 13 is the control unit of the computer device 1, connecting various components of the computer device 1 via various interfaces and lines. It executes programs or modules stored in the memory 12 (e.g., decision generation programs for timing tasks) and calls data stored in the memory 12 to perform various functions of the computer device 1 and process data.

[0142] The processor 13 executes the operating system of the computer device 1 and various installed applications. The processor 13 executes the applications to implement the steps in the decision generation method embodiments of the various time-series tasks described above, for example... Figure 1 The steps are shown.

[0143] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete the present invention. The one or more modules / units may be a series of computer-readable instruction segments capable of performing specific functions, which describe the execution process of the computer program in the computer device 1. For example, the computer program may be divided into an extraction unit 110, an alignment unit 111, an enhancement unit 112, a reasoning unit 113, and a decision-making unit 114.

[0144] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, a computer device, or a network device, etc.) or processor to execute the decision generation method for the timing task described in the various embodiments of the present invention.

[0145] If the modules / units integrated in the computer device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware devices. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above.

[0146] The computer program includes computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory, etc.

[0147] Furthermore, the computer-readable storage medium may primarily include a stored program area and a stored data area, wherein the stored program area may store the operating system, an application program required for at least one function, etc.; and the stored data area may store data created based on the use of blockchain nodes, etc.

[0148] The blockchain referred to in this invention is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0149] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, in... Figure 3 The bus is represented by only one straight line, but this does not mean that there is only one bus or one type of bus. The bus is configured to enable communication between the memory 12 and at least one processor 13, etc.

[0150] Although not shown, the computer device 1 may also include a power supply (such as a battery) to power various components. Preferably, the power supply can be logically connected to the at least one processor 13 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The computer device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0151] Furthermore, the computer device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish a communication connection between the computer device 1 and other computer devices.

[0152] Optionally, the computer device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), and optionally, a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the computer device 1 and to display a visual user interface.

[0153] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0154] It will be understood by those skilled in the art that Figure 3 The structure shown does not constitute a limitation on the computer device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0155] Combination Figure 1 The memory 12 in the computer device 1 stores multiple instructions to implement a decision generation method for a time-series task, and the processor 13 can execute the multiple instructions to achieve:

[0156] In response to the decision generation instructions for the target time series task, multimodal time series features are extracted from the input data;

[0157] Cross-modal temporal alignment is performed on the multimodal temporal features to obtain a multimodal temporal alignment feature sequence;

[0158] Temporal memory enhancement is performed on the multimodal temporal alignment feature sequence to obtain a multimodal temporal enhancement feature sequence;

[0159] The multimodal temporal enhancement feature sequence is input into the temporal inference network, and the output features of the temporal inference network are obtained.

[0160] The output features are input into the decision layer corresponding to the target time-series task to obtain the target decision data of the target time-series task.

[0161] Specifically, the processor 13's implementation method for the above instructions can be found in [reference needed]. Figure 1 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0162] It should be noted that all data involved in this case was legally obtained. Software tools or components not belonging to this company that appear in the embodiments of this application are merely illustrative examples and do not represent actual use.

[0163] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0164] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0165] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0166] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0167] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0168] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0169] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices described in this invention can also be implemented by a single unit or device through software or hardware. Terms such as "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0170] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A decision generation method for a time-series task, characterized in that, The decision generation method for the time-series task includes: In response to the decision generation instructions for the target time series task, multimodal time series features are extracted from the input data; Cross-modal temporal alignment is performed on the multimodal temporal features to obtain a multimodal temporal alignment feature sequence; Temporal memory enhancement is performed on the multimodal temporal alignment feature sequence to obtain a multimodal temporal enhancement feature sequence; The multimodal temporal enhancement feature sequence is input into the temporal inference network, and the output features of the temporal inference network are obtained. The output features are input into the decision layer corresponding to the target time-series task to obtain the target decision data of the target time-series task.

2. The decision generation method for time-series tasks as described in claim 1, characterized in that, The extraction of multimodal temporal features from the input data includes: The video data in the input data is split into a continuous sequence of video frames; spatial features of each frame and temporal features between adjacent frames are extracted using a 3D convolutional neural network to obtain an initial visual feature sequence; the multi-head attention mechanism in the temporal Transformer is used to capture the visual feature dependencies over a long temporal range in the initial visual feature sequence to obtain a visual temporal feature vector; and / or The input text data is segmented and part-of-speech tagged to obtain a word sequence; a bidirectional long short-term memory network is used to capture the semantic dependencies between the preceding and following contexts of the word sequence to obtain a hidden state sequence; the hidden state sequence is input into an attention mechanism layer to obtain a language temporal feature vector; and / or The action data in the input data is input into the gating loop unit to obtain the action timing feature vector.

3. The decision generation method for time-series tasks as described in claim 1, characterized in that, The step of performing cross-modal temporal alignment on the multimodal temporal features to obtain a multimodal temporal aligned feature sequence includes: Obtain the first modal temporal feature sequence and the second modal temporal feature sequence corresponding to any two modalities in the multimodal temporal features; An alignment score matrix is ​​constructed between the first modality temporal feature sequence and the second modality temporal feature sequence using a multilayer perceptron; wherein each element in the alignment score matrix represents the alignment score between each first modality temporal feature and each second modality temporal feature; Obtain the alignment path from the first element to the last element in the alignment score matrix; wherein the sum of the alignment scores of all elements traversed by the alignment path is the highest. Unify the temporal dimension of the alignment paths across all modalities to obtain the temporal feature sequence of each modality after temporal alignment; The temporal feature sequences of each modality after temporal alignment are concatenated according to a unified timestamp to obtain the multimodal temporal alignment feature sequence.

4. The decision generation method for time-series tasks as described in claim 1, characterized in that, Before performing temporal memory enhancement on the multimodal temporally aligned feature sequence, the method further includes: Create multiple memory slots and initialize each memory slot as a zero vector; Obtain the current time corresponding to each memory slot, and extract the splicing features of the current time corresponding to each memory slot from the historical multimodal temporal alignment feature sequence; Calculate the matching degree between the splicing features of the current moment corresponding to each memory slot and the splicing features of the previous moment corresponding to the previous memory slot, as the degree of association between each memory slot and historical memory; The softmax function is used to normalize the correlation between each memory slot and the historical memory, thus obtaining the weight corresponding to each memory slot. Based on the weight corresponding to each memory slot, the splicing feature of the current time corresponding to each memory slot is fused with the splicing feature of the previous time corresponding to the previous memory slot to obtain the updated feature of the current time corresponding to each memory slot. Each memory slot is updated based on the update characteristics of the current moment corresponding to each memory slot, thus obtaining a temporal memory unit based on the memory network.

5. The decision generation method for time-series tasks as described in claim 4, characterized in that, The step of fusing the concatenated features of the current time step corresponding to each memory slot with the concatenated features of the previous time step corresponding to the previous memory slot according to the weights corresponding to each memory slot to obtain the updated features of the current time step corresponding to each memory slot includes: Calculate the product of the weight corresponding to each memory slot and the concatenated feature corresponding to the previous memory slot at the previous time step to obtain the first feature corresponding to each memory slot; Calculate the difference between 1 and the weight corresponding to each memory slot to obtain the second weight corresponding to each memory slot; Calculate the product of the second weight corresponding to each memory slot and the concatenated feature corresponding to the current time step for each memory slot to obtain the second feature corresponding to each memory slot; The sum of the first feature corresponding to each memory slot and the second feature corresponding to each memory slot is calculated to obtain the updated feature corresponding to each memory slot at the current time.

6. The decision generation method for time-series tasks as described in claim 4, characterized in that, The step of performing temporal memory enhancement on the multimodal temporally aligned feature sequence to obtain the multimodal temporally enhanced feature sequence includes: For each currently spliced ​​feature in the multimodal temporal alignment feature sequence, calculate the degree of association between the current spliced ​​feature and each memory slot; Based on the correlation between the current splicing feature and each memory slot, the splicing features corresponding to all memory slots are weighted and aggregated to obtain the multimodal temporal enhancement feature sequence.

7. The decision generation method for time-series tasks as described in claim 1, characterized in that, The step of inputting the multimodal temporal enhancement feature sequence into the temporal inference network and obtaining the output features of the temporal inference network includes: The multimodal temporal enhancement feature sequence is input into the temporal inference network composed of the Transformer architecture, and the multi-head self-attention mechanism is used to model the dependency relationship of the multimodal temporal enhancement feature sequence in the time dimension to obtain the output feature.

8. A decision generation device for a time-series task, characterized in that, The decision generation device for the time-series task includes: The extraction unit is used to extract multimodal temporal features from the input data in response to the decision generation instructions for the target temporal task; An alignment unit is used to perform cross-modal temporal alignment on the multimodal temporal features to obtain a multimodal temporal aligned feature sequence; An enhancement unit is used to perform temporal memory enhancement on the multimodal temporal alignment feature sequence to obtain a multimodal temporal enhancement feature sequence; The inference unit is used to input the multimodal temporal enhancement feature sequence into the temporal inference network and obtain the output features of the temporal inference network. The decision unit is used to input the output features into the decision layer corresponding to the target time-series task to obtain the target decision data of the target time-series task.

9. A computer device, characterized in that, The computer device includes: Memory, storing at least one instruction; and The processor executes instructions stored in the memory to implement the decision generation method for a timing task as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, which is executed by a processor in a computer device to implement the decision generation method for a timing task as described in any one of claims 1 to 7.

Images