Data sequence generation method and device based on joint modeling, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, can be applied to business scenes such as robot agent decision making, financial science and technology and medical health, and discloses a data sequence generation method, device and equipment based on joint modeling and a medium. Encoding is performed as a sequence of potential visual tokens and a sequence of action tokens, a unified potential representation is generated, a prediction sequence is generated using a visual diffusion decoder and an action diffusion decoder, a random mask is applied, a loss value is determined, parameters are optimized, loop training is performed, and a target future action sequence and / or a target future video frame sequence is generated. According to the method, the visual information and the action information are fused, the unified potential representation is constructed, the visual diffusion decoder and the action diffusion decoder are jointly optimized, error transmission and information loss caused by independent prediction are avoided, and the prediction accuracy under multi-modal data and the overall reasoning efficiency of the system are improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a data sequence generation method, device, equipment and storage medium based on joint modeling. Background Art

[0002] The joint prediction of complex time series data is a widespread problem in robotic agent decision-making, healthcare analytics, and FinTech scenarios. The need for effective fusion and efficient reasoning of multimodal information is particularly pressing. Existing technologies for joint prediction of data sequences, particularly the collaborative modeling of action sequences and visual information, remain significantly limited. This directly impacts the accuracy, robustness, and real-time performance of systems in complex, multi-task environments.

[0003] In the field of robotic agents, mainstream methods are mainly divided into two categories: one is the action prediction method represented by Diffusion Policy and OpenVLA, which relies on historical action data to infer future actions. Although it has good performance in single tasks or restricted scenarios, it lacks effective use of visual information, resulting in limitations in the model in environmental perception and context understanding. Especially when facing complex visual scenes or dynamic changes in the environment, it is easy to overfit historical actions and lack environmental adaptability; the other method is represented by PAD, UniPi, etc., which first generates video sequences and then uses video information to assist action prediction. Although it partially alleviates the problem of lack of visual perception, this type of method generally has the risk of cascading video generation errors, and the inference process is time-consuming, which makes it difficult to meet real-time control or decision-making scenarios that require high-timeliness responses.

[0004] In the healthcare business, existing data analysis methods for dynamic patient monitoring, vital sign prediction, and clinical decision-making support also have similar problems. Action prediction models often rely solely on historical behavioral data (such as equipment operation records and patient position adjustment information) to infer state evolution. It is difficult to fully consider visual data such as medical images and real-time monitoring videos, which reduces the ability to perceive the evolution of complex diseases and the accuracy of predictions. Although the method based on video generation to assist diagnosis and treatment analysis can improve scene understanding by combining multi-source data, it is limited by the accuracy and efficiency of the multi-step image generation process. In actual applications, there are problems with slow reasoning speed and unstable generation quality, making it difficult to adapt to high-real-time scenarios such as intensive care and surgical assistance.

[0005] In the fintech sector, typical intelligent risk control, customer behavior prediction, and multi-dimensional data modeling systems face similar challenges. Traditional action prediction methods focus on analyzing structured data such as transaction behavior and operation sequences, lacking effective integration of image recognition and video surveillance information, limiting the models' ability to capture dynamic risk in complex financial scenarios. Some technical approaches that use video synthesis or image generation to assist in risk control decision-making, relying on multi-level, staged reasoning, suffer from high computational resource consumption and high system response latency, hindering their practical deployment in core business scenarios such as high-frequency trading and real-time compliance monitoring.

[0006] In addition, existing methods generally lack the ability to generalize across multiple tasks under a unified architecture. Most systems are optimized for a single task and a single data type, lacking the ability to jointly predict both visual and motion information while adapting to different data structures and business needs. This results in the need for frequent retraining or adjustment of models when switching between multiple tasks and multiple scenarios, severely restricting the promotion and application of intelligent systems across scenarios and businesses, as well as cost control. Summary of the Invention

[0007] The main purpose of the present invention is to provide a data sequence generation method, device, equipment and storage medium based on joint modeling, aiming to solve the technical problems in the existing technology that the action prediction method lacks visual information fusion, the video generation method has low reasoning efficiency and easy error transmission, and cannot achieve efficient joint prediction of visual information and action information.

[0008] To achieve the above objectives, the present invention provides a data sequence generation method based on joint modeling, comprising:

[0009] Obtain training samples including historical image sequences, historical action sequences, real future video frame sequences, and real future action sequences;

[0010] Applying a random mask to at least one of the historical image sequence or the historical action sequence to obtain a masked historical sequence;

[0011] encoding a historical image sequence in the masked history sequence into a latent visual token sequence, and encoding a historical action sequence in the masked history sequence into an action token sequence;

[0012] Processing the latent visual token sequence and the action token sequence through a joint modeling module to generate a unified latent representation;

[0013] Based on the unified latent representation, using a visual diffusion decoder to generate a predicted future video frame sequence, and using an action diffusion decoder to generate a predicted future action sequence;

[0014] applying a random mask to at least one of the predicted future video frame sequence or the predicted future action sequence to obtain a masked prediction sequence;

[0015] Determine a loss value based on the difference between the future video frame sequence in the masked prediction sequence and the true future video frame sequence, and the difference between the future action sequence in the masked prediction sequence and the true future action sequence;

[0016] Optimizing parameters of the joint modeling module, parameters of the visual diffusion decoder, and parameters of the motion diffusion decoder based on the loss value;

[0017] Repeat the steps of applying random masking to the history sequence, encoding, processing to generate a unified latent representation, generating a prediction sequence, applying random masking to the prediction sequence, determining the loss value and optimizing the parameters until the preset training termination condition is met, thereby obtaining the trained joint modeling module, visual diffusion decoder and action diffusion decoder;

[0018] The trained joint modeling module, visual diffusion decoder, and action diffusion decoder are used to process reasoning tasks and generate target future action sequences and / or target future video frame sequences.

[0019] Furthermore, to achieve the above-mentioned purpose, the present invention provides a data sequence generation device based on joint modeling, comprising:

[0020] A training data preparation module is used to obtain training samples including historical image sequences, historical action sequences, real future video frame sequences, and real future action sequences;

[0021] a historical sequence masking module, configured to apply a random mask to at least one of the historical image sequence or the historical action sequence to obtain a masked historical sequence;

[0022] a history sequence encoding module, configured to encode a history image sequence in the masked history sequence into a potential visual token sequence, and to encode a history action sequence in the masked history sequence into an action token sequence;

[0023] a joint representation generation module, configured to process the latent visual token sequence and the action token sequence through a joint modeling module to generate a unified latent representation;

[0024] a prediction sequence generation module, configured to generate a predicted future video frame sequence using a visual diffusion decoder and a predicted future action sequence using an action diffusion decoder based on the unified latent representation;

[0025] A prediction sequence masking module, configured to apply a random mask to at least one of the predicted future video frame sequence or the predicted future action sequence to obtain a masked prediction sequence;

[0026] a loss calculation module, configured to determine a loss value based on a difference between a future video frame sequence in the masked predicted sequence and a true future video frame sequence, and a difference between a future action sequence in the masked predicted sequence and a true future action sequence;

[0027] a parameter optimization module, configured to optimize the parameters of the joint modeling module, the parameters of the visual diffusion decoder, and the parameters of the motion diffusion decoder based on the loss value;

[0028] A training control module is configured to repeatedly execute the steps of applying random masking to the historical sequence, encoding, processing to generate a unified latent representation, generating a prediction sequence, applying random masking to the prediction sequence, determining a loss value, and optimizing parameters until a preset training termination condition is met, thereby obtaining a trained joint modeling module, a visual diffusion decoder, and an action diffusion decoder;

[0029] The inference generation module is used to process the inference task using the trained joint modeling module, visual diffusion decoder and action diffusion decoder to generate target future action sequences and / or target future video frame sequences.

[0030] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and a data sequence generation program based on joint modeling stored in the memory and runnable on the processor, and when the data sequence generation program based on joint modeling is executed by the processor, the steps of the data sequence generation method based on joint modeling as described above are implemented.

[0031] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a data sequence generation program based on joint modeling is stored. When the data sequence generation program based on joint modeling is executed by a processor, the steps of the data sequence generation method based on joint modeling as described above are implemented.

[0032] Beneficial effect: The present invention relates to the field of artificial intelligence technology and can be applied to business scenarios such as robot intelligent body decision-making, financial technology and medical health. A data sequence generation method, device, equipment and medium based on joint modeling are disclosed, including: obtaining training samples including historical image sequences, historical action sequences, real future video frame sequences and real future action sequences, applying a random mask to at least one of the historical image sequences or historical action sequences to obtain a masked historical sequence, encoding the historical image sequence in the masked historical sequence into a potential visual token sequence, encoding the historical action sequence in the masked historical sequence into an action token sequence, processing the potential visual token sequence and the action token sequence through a joint modeling module to generate a unified potential representation, based on the unified Latent representation, using a visual diffusion decoder to generate a predicted future video frame sequence, using a motion diffusion decoder to generate a predicted future action sequence, applying a random mask to at least one of the predicted future video frame sequence or the predicted future action sequence to obtain a masked predicted sequence, determining a loss value based on the difference between the masked predicted sequence and the real future video frame sequence and the real future action sequence, optimizing the parameters of the joint modeling module, the visual diffusion decoder, and the motion diffusion decoder based on the loss value, repeating the above operation to obtain a trained joint modeling module, visual diffusion decoder, and motion diffusion decoder, using the trained joint modeling module, visual diffusion decoder, and motion diffusion decoder to process the reasoning task and generate a target future action sequence and / or a target future video frame sequence. The present invention constructs a unified latent representation by fusing visual information and motion information, jointly optimizes the visual diffusion decoder and the motion diffusion decoder, avoids error transmission and information loss caused by separate predictions, and improves the prediction accuracy under multimodal data and the overall reasoning efficiency of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0034] Figure 1 A schematic diagram of an application environment of a data sequence generation method based on joint modeling in one embodiment of the present invention;

[0035] Figure 2 This is a flow chart of an embodiment of a method for generating a data sequence based on joint modeling according to the present invention;

[0036] Figure 3 This is a functional module diagram of a preferred embodiment of the data sequence generation device based on joint modeling of the present invention;

[0037] Figure 4 A schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0038] Figure 5FIG. 2 is another structural diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0040] The data sequence generation method based on joint modeling provided by the embodiment of the present invention can be applied in the following Figure 1 In an application environment, a user terminal communicates with a server terminal through a network. The server terminal can obtain training samples including historical image sequences, historical action sequences, real future video frame sequences, and real future action sequences through the user terminal, apply a random mask to at least one of the historical image sequences or the historical action sequences to obtain a masked historical sequence, encode the historical image sequence in the masked historical sequence into a latent visual token sequence, encode the historical action sequence in the masked historical sequence into an action token sequence, process the latent visual token sequence and the action token sequence through a joint modeling module to generate a unified latent representation, generate a predicted future video frame sequence based on the unified latent representation using a visual diffusion decoder, and generate a predicted future video frame sequence using an action diffusion decoder. Generate a predicted future action sequence, apply a random mask to at least one of the predicted future video frame sequence or the predicted future action sequence to obtain a masked predicted sequence, determine a loss value based on the difference between the masked predicted sequence and the real future video frame sequence or the real future action sequence, optimize the parameters of the joint modeling module, the visual diffusion decoder, and the action diffusion decoder based on the loss value, repeat the above operation to obtain a trained joint modeling module, visual diffusion decoder, and action diffusion decoder, use the trained joint modeling module, visual diffusion decoder, and action diffusion decoder to process the reasoning task, and generate a target future action sequence and / or a target future video frame sequence. The present invention fuses visual information and action information to construct a unified potential representation, jointly optimizes the visual diffusion decoder and the action diffusion decoder, avoids error transmission and information loss caused by separate predictions, and improves the prediction accuracy and overall reasoning efficiency of the system under multimodal data. The user end can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server end can be implemented using an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0041] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of a data sequence generation method based on joint modeling provided by the present invention. It should be noted that although a logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0042] like Figure 2 As shown, the data sequence generation method based on joint modeling proposed in the present invention includes the following steps:

[0043] S10, obtaining training samples including historical image sequences, historical action sequences, real future video frame sequences, and real future action sequences;

[0044] In this embodiment, the training samples include historical image sequences, historical action sequences, real future video frame sequences, and real future action sequences. The historical image sequence refers to a data set that reflects the visual information of the target object over a period of time in the past. It can include a video frame sequence, an image frame set, or other data continuously collected based on visual information. The source is usually a camera, a visual sensor, a medical imaging device, or an environmental monitoring device. The temporal continuity of the image sequence is not limited to a fixed frame rate. It can be collected at a fixed time interval or dynamically collected according to system trigger conditions. The format of the image data is not limited to specific forms such as RGB, grayscale, or depth map. Multimodal information fusion can be used to form an image sequence in combination with different business scenarios. The historical action sequence refers to the time series data of the target object's action, behavior, or operation trajectory within a corresponding time period. The source can be a sensor device, a motion capture system, an operation log, or a human action recognition system. The type of action data can be position coordinates, speed, angle, control instructions, posture parameters, or a combination of information. The historical action sequence corresponds to or is synchronized with the historical image sequence on the time axis and is used to jointly describe the dynamic state of the object.

[0045] The real future video frame sequence refers to the visual data of the target object that actually occurs in the time window after the historical time period. It is usually obtained through real-time acquisition, delayed monitoring or historical records. The video frame sequence can be a multi-frame continuous image or video stream. The visual information content includes the target object itself, environmental background, interactive scene, etc. The real future action sequence refers to the action data actually performed by the target object in the same time window. The data format, source and representation method are consistent with the historical action sequence. The future video frame sequence and the future action sequence are continuous with the historical sequence in time order, but the data structure is not required to be completely consistent. Combined with different system requirements, it can adapt to different dimensions, precision and information types.

[0046] In actual implementation, training samples are typically acquired based on existing data collection platforms. In the healthcare sector, historical image sequences can be derived from surgical videos, rehabilitation monitoring, or image sequences captured by medical imaging equipment. Historical action sequences can be derived from patient physiological monitoring, motion capture equipment, or rehabilitation training data. Real-future video frame sequences and real-future action sequences are acquired through real-time tracking or historical label data recording, combined with simultaneous acquisition of multi-source data from clinical monitoring systems, rehabilitation assistive devices, and other sources. In the fintech sector, historical image sequences can be records of interface changes during user operation of self-service terminals or smart devices. Historical action sequences can be user operation instructions, interaction paths, or behavioral data. Real-future video frame sequences and real-future action sequences are acquired through subsequent actual system records, including transaction behavior, terminal operation data, or surveillance image information.

[0047] A multi-channel data acquisition system can be configured to simultaneously acquire historical image sequences and historical motion sequences. Visual and motion data use a unified timestamp alignment mechanism to ensure temporal correlation. Image sensors and motion capture units can be integrated within data acquisition terminals to form a real-time edge data acquisition system, reducing data synchronization latency. Video frame sequences can be transmitted using compressed encoding, and motion sequences can be encapsulated using standardized protocols to ensure data consistency and availability. Real-world video frame sequences and real-world motion sequences can be acquired through delayed observation or generated through high-fidelity scene reconstruction in a simulation environment. This is suitable for offline training in scenarios lacking real-time data. For healthcare scenarios, training samples can be sourced from a joint data platform of multiple hospitals, combining historical patient monitoring records and follow-up data. Data types encompass medical imaging, rehabilitation movements, vital sign monitoring, and other sources, increasing sample diversity and coverage. For fintech scenarios, training samples can be based on data collected from bank smart devices, remote transaction terminals, or mobile app operations, combined with user behavior logs, interface change records, and system feedback to form a comprehensive dataset of visual and motion sequences.

[0048] Example description: In the application scenario of robotic agents, historical image sequences can come from the visual perception data of the robot during autonomous cruising, the continuous image input of the SLAM system, or the visual information output by multimodal sensor fusion. Historical action sequences can include robot chassis control instructions, robotic arm execution trajectories, or environmental interaction action records. Real future video frame sequences can refer to the future visual prediction results of the robot during environmental exploration, path planning, or task execution. Real future action sequences can be the subsequent decision-making actions or control instruction sequences dynamically generated by the robot for the task environment. By obtaining training samples with the above information structure, the robot's perception prediction, action generation, and multi-task collaborative decision-making capabilities in complex dynamic environments can be effectively improved.

[0049] In the medical and health business scenario, for the rehabilitation training process, the historical image sequence comes from the continuously captured images of the rehabilitation monitoring equipment, the historical action sequence comes from the motion capture device or somatosensory device worn by the patient, the real future video frame sequence is obtained through the real-time video recording of the rehabilitation assessment phase, and the real future action sequence is obtained through the system's tracking and recording of the patient's actual exercise performance. Combined with the complete training samples, it can realize the movement prediction and abnormal warning for the patient's rehabilitation process, and assist in the formulation of personalized rehabilitation plans.

[0050] In the financial technology business scenario, for smart terminal trading systems, the historical image sequence comes from the screenshot sequence of the user operation interface, the historical action sequence is the command data or interaction path entered by the user, and the real future video frame sequence is obtained by the system recording the subsequent interface changes during the user operation process. The real future action sequence is the user's subsequent actual operation instructions. Combined with this type of training samples, the trading system's ability to predict user operation behavior can be improved, and potential anomaly detection and process optimization can be achieved.

[0051] This embodiment forms a complete temporal data structure by synchronously acquiring historical image sequences, historical action sequences, real future video frame sequences, and real future action sequences. This avoids the problem of decreased prediction accuracy caused by training a single data type, realizes the fusion modeling of visual information and action information, and improves the prediction stability and adaptability in complex dynamic environments.

[0052] S20, applying a random mask to at least one of the historical image sequence or the historical action sequence to obtain a masked historical sequence;

[0053] In this embodiment, at least one of the historical image sequence or the historical action sequence refers to the random masking operation that can be performed on the historical image sequence alone, the historical action sequence alone, or both at the same time. The historical image sequence corresponds to a data set of collected image frames or video frames, containing visual information reflecting the target object on a time-continuous axis. The source can be a fixed camera device, a mobile visual sensor, a medical imaging device, or an image capture module based on a computer vision system. The data type can be a single-frame static image, a continuous dynamic image, or a video clip. The format of the image data is not limited to a specific type and can include RGB images, depth maps, infrared images, medical images, etc. The historical action sequence refers to a data set of the target object's behavior within a corresponding time period. The source can be an inertial sensor, a motion recognition system, an interaction log, a control instruction record, or other motion capture device. The data content can cover position changes, posture angles, speed information, instruction sequences, or behavior tags. The action data can be either continuous numerical data or discrete state labels.

[0054] Applying random masking refers to performing masking or replacement operations on some data positions in historical image sequences or historical action sequences based on a randomly generated masking strategy. Random masking can determine the specific data segments or data dimensions that need to be masked by generating random position indications or probability distributions. Masking methods may include data deletion, data zeroing, replacing data with specific placeholders, or introducing noise data. The choice of mask position is not limited to a fixed ratio, fixed time window, or fixed spatial area. The range, ratio, and distribution pattern of the mask can be flexibly adjusted according to specific task requirements, model training strategies, or system configurations. The masked historical sequence refers to a new sequence data structure consisting of the unmasked part and the masked part after the random masking operation is performed. This sequence retains some real historical information while artificially introducing information loss to enhance data diversity and the model's adaptability to uncertainty.

[0055] Random masking can be performed on historical image sequences alone, selecting portions of image frames for masking or content replacement to simulate training for situations where visual information is missing. Random masking can be performed on historical action sequences alone, masking some action data points, state labels, or control instructions, improving the model's robustness in scenarios with incomplete action data. Random masking can also be performed on both historical image and action sequences simultaneously to create more complex scenarios with missing data and enhance joint modeling capabilities. Mask positions can be generated using random sampling based on a uniform distribution, probability mapping based on a normal distribution, or by dynamically adjusting mask probabilities based on the task scenario. Masking can be applied to any time period or spatial location in the sequence, without restrictions on the continuity or distribution density of the mask.

[0056] In healthcare business scenarios, we can randomly mask some video frames or motion data points in image monitoring data and motion capture data for rehabilitation training to simulate image frame loss, sensor data anomalies, or information missing during actual monitoring, thereby improving the system's motion prediction and health status assessment capabilities in environments with incomplete information.

[0057] In the financial technology business scenario, we can randomly mask some user operation interface change information or behavioral instruction data for smart terminal operation data to simulate interface freezes, data loss or transmission interruptions during system interaction, and improve the system's ability to predict user behavior in complex business scenarios when information is missing.

[0058] Example: In robotic agent applications, historical image sequences can be derived from continuous image frames captured by the robot's onboard visual sensors, depth cameras, or environmental monitoring equipment. Historical motion sequences can include the robot's joint control instructions, trajectory position parameters, operational behavior logs, or high-level decision-making instruction sequences. By performing random masking operations on historical image sequences or motion sequences, it is possible to simulate the incomplete information encountered by the robot in complex environments due to occlusion, visual interference, sensor failure, or communication anomalies. Random masking of visual data can obscure portions of image frames or visual regions, improving the system's robustness in environmental perception under varying lighting conditions, occlusion interference, or unstable image acquisition. Random masking of motion data can obscure portions of control parameters or execution status information, enhancing the system's dynamic decision-making and action generation capabilities in the presence of missing control signals, delayed feedback, or incomplete motion information. This improves the robot agent's adaptability, stability, and multi-tasking capabilities in non-ideal, dynamically changing environments.

[0059] In the medical and health business scenario, for the rehabilitation monitoring system, the historical image sequence comes from the video frame data captured by the rehabilitation equipment, and the historical motion sequence comes from the motion data of the somatosensory device worn by the patient. Random masking operations can be used to mask part of the monitoring images or motion data, thereby enhancing the system's stability in the event of video frame loss or motion capture abnormalities, improving the ability to continuously predict the patient's motion trends, and assisting medical personnel in judging the progress of rehabilitation.

[0060] In the financial technology business scenario, for smart terminal systems, the historical image sequence comes from the video or screenshot data of the user operation interface, and the historical action sequence is the operation instructions entered by the user. Random masking operations can be used to mask some interface changes or operation path data, thereby enhancing the system's behavior inference capabilities in the event of network delays and data transmission loss, and ensuring the security and continuity of the transaction operation process.

[0061] This embodiment introduces data missing and uncertainty factors by performing random masking operations on historical image sequences or historical action sequences, thereby enhancing the diversity and complexity of sequence data, improving the robustness and adaptability of the system in environments with incomplete information, sensor anomalies, or data missing, effectively alleviating the overfitting risk brought about by the model's dependence on complete data, and promoting the generalization effect of prediction results in multiple tasks and across scenarios.

[0062] S30, encoding a historical image sequence in the masked historical sequence into a potential visual token sequence, and encoding a historical action sequence in the masked historical sequence into an action token sequence;

[0063] In this embodiment, encoding the historical image sequence in the masked historical sequence into a latent visual token sequence belongs to a feature extraction and conversion process for image-related information. The historical image sequence specifically refers to continuous multi-frame image data related to visual information in the masked historical sequence. The source of the image sequence may include the image input of the environmental perception system, the visual information obtained by the camera device, or the data stream collected by the visual sensor, wherein the image data retained after the mask operation is used for subsequent encoding processing. Encoding into a latent visual token sequence means that the image sequence is converted into a set of structured sequence representations with information compression and semantic abstraction capabilities through a preset encoding mechanism. The latent visual token sequence can reduce the data dimension and extract key visual content on the basis of maintaining the temporal characteristics of the original image, so as to facilitate subsequent joint modeling or multimodal fusion processing.

[0064] Encoding historical action sequences in the masked history sequence into action token sequences targets action information processing within a time series structure. Historical action sequences refer to the continuous action data stream related to system behavior, operational instructions, or environmental interactions retained after masking. Encoding into action token sequences involves mapping action data sequences into a set of expressive, compact, and semantically distinct discrete token sequences through structured encoding methods. These action token sequences reflect the dynamics of system behavior, possessing temporal dependencies and decision-making logic relevance, facilitating the subsequent generation of unified representations and joint analysis in multi-task reasoning scenarios.

[0065] In the field of robotic agents, historical image sequences can include environmental visual information from autonomous robot navigation, local image sequences captured during robotic arm operation, or visual perception data from multimodal fusion systems. The encoding process can extract spatial structural features, target position relationships, or scene semantic information from image sequences through visual encoding modules or deep convolutional neural networks, generating latent visual token sequences that reduce data volume while retaining the necessary visual expression. Historical action sequences can originate from robot chassis motion control commands, robotic arm path execution data, or environmental interaction instructions. The encoding process can map action sequences into action token sequences that express the system's action intentions and trajectory trends through temporal modeling modules, sequence embedding structures, or self-attention mechanisms.

[0066] In healthcare scenarios, historical image sequences can be multi-frame medical images from ultrasound imaging, endoscopy, continuous monitoring images, or bedside monitoring systems. The encoding process uses a medical image feature extraction model to extract key anatomical structures, lesion areas, or clinical indicator information as a potential visual token sequence, facilitating downstream clinical decision-making. Historical action sequences can be surgical operation records, medical equipment control signals, or nursing process instructions. The encoding process uses sequence compression or semantic mapping methods to encode operational behavior patterns into action token sequences, facilitating multimodal diagnosis and treatment analysis based on visual information.

[0067] In the FinTech sector, historical image sequences can be derived from image data from remote identity verification processes, visual information from intelligent teller machine (ATM) interfaces, or business operation videos. Through image feature compression and scene semantic extraction, identity characteristics, operational behaviors, and interface change information are encoded into a sequence of latent visual tokens. Historical action sequences can include system transaction records, customer operation sequences, or risk response instructions. The encoding process, based on time-dependent modeling and sequence abstraction, maps complex financial operational processes into action token sequences, improving the efficiency of behavior prediction and risk control strategy generation.

[0068] This embodiment encodes the masked historical image sequence and historical action sequence into a latent visual token sequence and an action token sequence respectively. This can effectively compress the original data size, retain the necessary temporal structure and semantic information, achieve structural unification of different modal information, reduce downstream modeling complexity, and improve the multimodal fusion expression capability, which is conducive to enhancing the system's perception and understanding, behavior prediction and joint decision-making capabilities in complex environments.

[0069] S40, processing the latent visual token sequence and the action token sequence by a joint modeling module to generate a unified latent representation;

[0070] In this embodiment, the latent visual token sequence and the action token sequence are processed by the joint modeling module to generate a unified latent representation, which is a specific operation of multi-source information fusion and joint expression generation. The joint modeling module refers to an integrated structure capable of simultaneously receiving and processing inputs of different modalities, usually combining sequence splicing, position encoding, feature interaction and structural fusion mechanisms. The joint modeling module may include a sequence modeling structure based on the Transformer architecture, a fusion attention layer or other network frameworks with multimodal fusion capabilities. The latent visual token sequence is a set of visual information features obtained by the pre-order encoding operation and expressed in a sequence structure. The action token sequence is a serialized expression that expresses the dynamic features of the action information. The two have temporal consistency and information relevance. The latent visual token sequence and the action token sequence are input into the joint modeling module, indicating that different modal sequences are introduced at the same time. The joint modeling module fully captures the semantic relationship and dynamic dependency between the two types of information through sequence splicing, structural reorganization or multi-layer feature interaction mechanisms. Generating a unified latent representation means obtaining an intermediate expression with structural unity, information fusion, and semantic integrity through the above-mentioned joint processing. The unified latent representation can reflect the system's ability to jointly understand visual and motion information, facilitating subsequent unified driving of prediction, reasoning, or decision-making tasks.

[0071] In the field of robotic agents, the joint modeling module can include a sequence fusion structure based on a multi-head self-attention mechanism. The latent visual token sequence can come from environmental perception images or multi-view fusion visual input, and the action token sequence can come from autonomous navigation paths, robotic arm operation trajectories or system control instructions. The joint modeling module integrates the two types of token sequences in the time dimension through sequence splicing, marks the sequence identities of different data sources through position encoding, and then realizes deep interaction between visual and action information through a multi-layer Transformer structure to generate a unified latent representation. This representation reflects environmental structure information, system dynamic characteristics and historical behavior patterns, which facilitates subsequent multi-task reasoning.

[0072] In the medical and health business scenario, the joint modeling module can include a feature fusion structure for multimodal medical data. The potential visual token sequence can come from a continuous medical image sequence, such as ultrasound and endoscopic examination images, and the action token sequence can come from a surgical operation sequence, equipment control sequence or nursing process. The joint modeling module integrates visual and action information through feature splicing and structural fusion mechanisms to generate a unified potential representation, thereby improving the system's ability to understand complex medical processes and scenarios.

[0073] In the field of financial technology business, the joint modeling module can include a sequence processing structure for multi-source business data. The potential visual token sequence can come from the video sequence of the customer identity authentication process and the image data of the business operation interface. The action token sequence can come from the system transaction sequence, customer operation behavior data or the operation instruction sequence output by the intelligent risk control system. The joint modeling module generates a unified potential representation through sequence reorganization, semantic mapping and feature interaction mechanism, which facilitates the system to realize multimodal joint reasoning in risk identification, compliance review and intelligent recommendation tasks.

[0074] This embodiment generates a unified latent representation by inputting the latent visual token sequence and the action token sequence into the joint modeling module, which can break the limitation of single modal information, improve the expression consistency and semantic fusion ability of visual information and action information, enhance the system's comprehensive understanding of complex environments, multi-source information and dynamic scenes, reduce the modeling difficulty of downstream prediction and reasoning tasks, and improve the system's overall multi-task adaptability and information expression integrity.

[0075] S50, based on the unified potential representation, using a visual diffusion decoder to generate a predicted future video frame sequence, and using an action diffusion decoder to generate a predicted future action sequence;

[0076] In this embodiment, based on a unified latent representation, a visual diffusion decoder is used to generate a predicted future video frame sequence, and an action diffusion decoder is used to generate a predicted future action sequence, involving an operational chain of multimodal information decoding and diffusion generation. The unified latent representation is derived from the previous information fusion process and has the ability to express integrated visual features and action feature information, reflecting the system's joint understanding of environmental structure, historical dynamics, and behavioral patterns. The visual diffusion decoder is a sequence generation structure designed for video data generation tasks. It usually adopts a diffusion model architecture and generates image sequence expressions that meet temporal coherence and visual feature consistency through progressive noise reconstruction or random process inverse simulation. Predicting future video frame sequences means inferring the visual state changes of future scenes based on current information. The visual diffusion decoder receives the unified latent representation as conditional input and gradually constructs a visual sequence with clarity, continuity, and scene consistency through multi-step noise sampling, dynamic feature fusion, and high-dimensional image restoration.

[0077] The action diffusion decoder is a structure designed for action sequence generation tasks. Combining a diffusion mechanism with sequence modeling techniques, it possesses the ability to predict dynamic features and restore behavioral patterns. Predicting future action sequences refers to the system inferring future action directions or operational trends based on known information during reasoning or decision-making. Based on a unified latent representation, the action diffusion decoder utilizes time-stepped noise injection, sequence structure dynamic modeling, and reverse noise restoration techniques to gradually generate action sequence outputs with temporal logic, behavioral rationality, and dynamic continuity. The visual diffusion decoder and action diffusion decoder are independent of each other, addressing the needs of visual prediction and action reasoning, respectively. They achieve information consistency and joint prediction capabilities by sharing a unified latent representation.

[0078] In the field of robotic agents, the visual diffusion decoder can include a multi-level image reconstruction network. Based on the diffusion principle, it generates a predicted future video frame sequence corresponding to the dynamic changes of the environment through multiple rounds of noise injection and iterative denoising mechanism. The action diffusion decoder combines sequence information such as the motion trajectory of the robotic arm and autonomous navigation instructions, and uses time-stepped noise injection and dynamic inversion mechanism to generate a predicted future action sequence to guide the robotic system to adjust the path, track the target or perform complex tasks.

[0079] In the medical and health business field, the visual diffusion decoder can process continuous medical image sequences, such as intraoperative endoscopic images and real-time monitoring images, and generate future video frame sequences through diffusion, assisting medical staff in observing potential change trends in advance. The action diffusion decoder combines surgical instrument operation records and nursing process sequences to generate predicted future action sequences, assisting in the intelligent control of medical equipment or optimization of surgical processes.

[0080] In fintech business scenarios, the visual diffusion decoder can generate predicted future video frame sequences based on changes in the user operation interface and transaction terminal image information, and infer interface interaction trends or abnormal images in advance. The action diffusion decoder combines customer operation behavior sequences and intelligent risk control system strategy outputs to generate predicted future action sequences, which are used to evaluate potential risk operations in advance, optimize risk control decision-making processes, and improve system response speed and security protection levels.

[0081] This embodiment uses a unified latent representation to generate predicted future video frame sequences and predicted future action sequences using a visual diffusion decoder and an action diffusion decoder, respectively. This enables parallel prediction and dynamic generation of visual and action information, improves the system's scene understanding, behavioral reasoning, and future situation prediction capabilities in complex environments, enhances the system's decision-making efficiency and overall robustness driven by multimodal information, reduces prediction errors caused by a single information source, and optimizes the generation quality and timing rationality under the joint prediction structure.

[0082] S60, applying a random mask to at least one of the predicted future video frame sequence or the predicted future action sequence to obtain a masked prediction sequence;

[0083] In this embodiment, a random mask is applied to at least one of the predicted future video frame sequence or the predicted future action sequence to obtain the masked predicted sequence, involving sequence data processing operations based on uncertainty simulation and information shielding mechanisms. The predicted future video frame sequence is a visual image sequence restored by the visual diffusion decoder in a multi-step generation process. It contains scene state information at the future moment and exhibits spatial continuity and visual consistency. The predicted future action sequence is generated by the action diffusion decoder and reflects the system's future behavioral trends inferred based on historical states and potential information, exhibiting temporal logic and operational rationality.

[0084] Random masking is a technique for simulating missing information. It aims to simulate scenarios with incomplete data or information interference by randomly selecting portions of sequence data for masking, thereby improving the model's interference tolerance and generalization capabilities. Masking is not limited to a specific data type and can be applied to at least one of the following: predicting future video frame sequences or predicting future action sequences. This allows for flexible adaptation to different task requirements or training strategies.

[0085] The masked prediction sequence refers to the sequence form after partial data information is masked on the basis of the original complete prediction sequence. This sequence retains some observable information while introducing uncertainty and missing features, thereby enhancing the model's adaptability to information loss, noise interference or structural changes.

[0086] In the field of robotic intelligence, the predicted future video frame sequence corresponds to the visual prediction image generated by the robot's environmental perception system, and the predicted future action sequence corresponds to the operation trend instructions output by the robot's autonomous control system. Applying a random mask can include: randomly selecting several frames or several areas in the pixel matrix of the predicted future video frame sequence, setting them as mask mark values, and simulating a scenario where visual information is missing; it can also randomly block some action nodes in the time steps of the predicted future action sequence to simulate the situation of control signal interruption or lack of behavior pattern, thereby improving the stability of the robot system under environmental uncertainty and task abnormalities.

[0087] In the medical and health business field, the predicted future video frame sequences may include continuous medical image sequences or remote monitoring images, and the predicted future action sequences may include medical equipment operation trends or medical staff behavior predictions. Random mask operations can simulate image loss, occlusion or signal interruption in video sequences to improve the system's fault tolerance to equipment failure and image noise; it can also simulate operator behavior interruption, information loss or system delay in action sequences to optimize the stability and safety of intelligent assistance systems.

[0088] In the fintech business scenario, the predicted future video frame sequence includes changes in the transaction terminal interface and dynamic information of the intelligent interactive system interface. The predicted future action sequence includes customer operation behavior trends and intelligent risk control strategy outputs. Applying random masks can randomly block some interactive elements or interface areas in the interface sequence to simulate system anomalies or incomplete information. It can also randomly miss some action nodes in the operation behavior sequence to enhance the system's risk control and decision-making stability when facing incomplete data and abnormal operations.

[0089] By applying a random mask to at least one of the predicted future video frame sequences or predicted future action sequences, this embodiment can effectively simulate information loss, uncertainty, and data anomalies in real application scenarios, improve the robustness and generalization ability of the system in the visual prediction and action reasoning process, enhance the adaptability to data loss, noise interference, and structural changes, reduce system stability issues caused by incomplete information or prediction errors, and optimize the overall performance and practical value of the multimodal joint prediction system.

[0090] S70, determining a loss value based on a difference between the future video frame sequence in the masked prediction sequence and the real future video frame sequence, and a difference between the future action sequence in the masked prediction sequence and the real future action sequence;

[0091] In this embodiment, the loss value is determined based on the difference between the future video frame sequence in the masked prediction sequence and the real future video frame sequence, as well as the difference between the future action sequence in the masked prediction sequence and the real future action sequence. This is a result evaluation operation in the multi-source comparison and deviation measurement process. The masked prediction sequence is a partially information-missing state data formed by random masking operations based on the predicted future video frame sequence and the predicted future action sequence, and contains the uncertainty and information loss characteristics of the simulated environment. The real future video frame sequence is the actual observed video data obtained by the system, and the real future action sequence is the actual operation or behavior data obtained by the system. Together, the two constitute the source of multimodal real feedback information.

[0092] Difference calculation involves analyzing the deviation between the masked prediction sequence and the true feedback information. Specifically, this involves measuring the visual difference between the future video frame sequence and the behavioral deviation between the future action sequence. This is often done using pixel-level mean squared error, structural similarity metrics, or distance metrics in deep feature space. Differences can be calculated separately for video and motion deviations, or the two can be combined to form an overall performance evaluation metric.

[0093] The loss value, as the optimization objective function in the training process, reflects the comprehensive error level of the system in visual prediction and motion reasoning tasks, and directly determines the parameter adjustment direction and amplitude of the joint modeling module, visual diffusion decoder, and motion diffusion decoder. The smaller the loss value, the better the prediction performance and the higher the system stability.

[0094] In the field of robotic agents, the future video frame sequence in the masked prediction sequence can be the visual image generated by the robot's environmental perception module, while the real future video frame sequence comes from the real-time observation of the robot's camera system. The difference between the two can be determined through pixel-level comparison, structural similarity analysis, or semantic information alignment. The future action sequence in the masked prediction sequence is the operational trend output of the robot's motion control module, while the real future action sequence is the actual operational feedback of the robot's execution system. The difference between the two can be measured using pose change, path deviation, or behavioral matching metrics. These two types of differences are fused to generate a unified loss value, guiding the system's simultaneous optimization of the visual and action prediction processes.

[0095] In the healthcare sector, the future video frame sequence in the masked prediction sequence can include continuous medical imaging prediction data, while the actual future video frame sequence comes from the actual acquisition results of medical imaging equipment. The difference measurement method can be based on medical image reconstruction error, tissue structure consistency, or clinical feature recognition accuracy. The future action sequence in the masked prediction sequence can be the device operation prediction results of the medical assistance system, while the actual future action sequence is the actual operation record of medical staff or equipment. The difference can be based on operation trajectory, time series offset, or key step matching analysis. Ultimately, the comprehensive loss value reflects the overall error level of the system in the medical assistance decision-making and intelligent operation prediction process.

[0096] In FinTech business scenarios, the masked predicted future video frame sequence represents the predicted trading system interface data, while the real future video frame sequence represents the actual terminal interface feedback. Difference metrics can be based on interface element recurrence rate, visual layout consistency, or functional module matching. The masked predicted future action sequence represents the customer's operational behavior trend, while the real future action sequence represents the customer's actual operational path. Difference metrics involve behavioral trajectory comparison, operational logic matching, and anomaly deviation detection. The overall loss value comprehensively reflects the system's accuracy and stability in interface prediction, operational reasoning, and risk management tasks.

[0097] This embodiment forms a unified loss value by calculating the difference between the masked prediction sequence and the actual feedback information, which can accurately reflect the comprehensive error of the system in the visual prediction and motion reasoning process, promote the efficient collaborative optimization of the joint modeling module, visual diffusion decoder and motion diffusion decoder, improve the system's multimodal information fusion capability, environmental adaptability and task execution accuracy, and enhance the overall performance and stability in complex scenarios and uncertain information conditions.

[0098] S80, optimizing parameters of the joint modeling module, parameters of the visual diffusion decoder, and parameters of the motion diffusion decoder based on the loss value;

[0099] In this embodiment, the parameters of the joint modeling module, the parameters of the visual diffusion decoder, and the parameters of the action diffusion decoder are optimized based on the loss value, which belongs to a multi-component collaborative adaptive adjustment process driven by feedback results. The loss value is a comprehensive error metric calculated by the system after comparing the masked prediction sequence with the real feedback information, which reflects the accuracy level of the visual prediction and action reasoning results. The joint modeling module is used to fuse the latent visual token sequence and the action token sequence to generate a unified latent representation, and its parameters involve feature interaction structure, mapping weight matrix and position encoding parameters. The visual diffusion decoder is used to generate a predicted future video frame sequence based on the unified latent representation, and its parameters include diffusion step control parameters, denoising network structure parameters and visual feature restoration parameters. The action diffusion decoder is used to generate a predicted future action sequence, and its parameters cover time step setting, behavior trend extraction parameters and motion pattern generation parameters.

[0100] The optimization process uses the loss value as a reference. Through backpropagation and gradient update mechanisms, the partial derivatives of the loss value with respect to each parameter are calculated to form the parameter update amount. The above module parameters are dynamically adjusted to reduce the overall prediction error of the system and improve the multimodal information fusion capability, visual generation quality and motion reasoning accuracy.

[0101] In the robotic agent system, the loss-based optimization of the joint modeling module parameters, the visual diffusion decoder parameters, and the motion diffusion decoder parameters results from the deviation between the masked prediction sequence and the robot's actual environmental perception data and execution feedback. Joint modeling module parameter adjustment focuses on optimizing the visual and motion information fusion structure to enhance environmental understanding and decision-making relevance. Visual diffusion decoder parameter optimization includes the number of diffusion process steps, noise modeling structure, and visual restoration accuracy control to ensure that the generated video data has sufficient environmental detail and temporal coherence. Motion diffusion decoder parameter optimization focuses on adjusting the operation trend prediction structure and temporal resolution control to improve the accuracy and dynamic adaptability of the action sequence, thereby promoting the robot's navigation, obstacle avoidance, and operational stability in complex environments.

[0102] In healthcare, the loss is based on the difference between the masked predicted sequence and the actual medical images and device operation records. Joint modeling module parameter optimization improves the combined representation of medical image and operation behavior features. Visual diffusion decoder parameter adjustment optimizes the structural integrity and tissue detail preservation of generated medical images. Action diffusion decoder parameter optimization enhances the ability to accurately predict device operation trends, ensuring the accuracy of auxiliary diagnosis and the stability of operational processes during medical treatment.

[0103] In FinTech business scenarios, the difference between the masked prediction sequence and the actual system interface feedback and customer operation paths generates a loss value. Joint modeling module parameter optimization improves the ability to jointly express interface elements and customer behavior data. Visual diffusion decoder parameter optimization improves the clarity of interface change trend reasoning and the consistency of interaction logic. Action diffusion decoder parameter optimization enhances the accuracy of operation trend reasoning and the timeliness of response, ensuring the system's interactive stability and risk response capabilities in environments with high concurrency and data anomalies.

[0104] This embodiment dynamically optimizes the joint modeling module parameters, visual diffusion decoder parameters, and motion diffusion decoder parameters based on the loss value, thereby effectively reducing the multimodal information fusion deviation, improving the visual generation quality and motion reasoning accuracy, enhancing the system's adaptability and overall prediction performance in complex environments, and promoting the system to maintain efficient and stable operation in multiple tasks and scenarios.

[0105] S90, repeatedly performing the steps of applying a random mask to the historical sequence, encoding, processing to generate a unified latent representation, generating a prediction sequence, applying a random mask to the prediction sequence, determining a loss value and optimizing parameters until a preset training termination condition is met, thereby obtaining a trained joint modeling module, a visual diffusion decoder, and a motion diffusion decoder;

[0106] In this embodiment, the steps of applying random masking to the historical sequence, encoding, processing to generate a unified latent representation, generating a predicted sequence, applying random masking to the predicted sequence, determining the loss value, and optimizing parameters are repeated until the preset training termination conditions are met. This is a multi-stage training process built on a cyclic iterative structure, which aims to gradually optimize the performance of each component of the system through multiple error feedback and parameter adjustments to form a stable and reliable joint reasoning capability. The historical sequence includes historical image sequences and historical action sequences. The random masking operation is used to break the direct dependence of the data and enhance the model's adaptability to incomplete information. The encoding and joint modeling operations extract multimodal information expression, and the unified latent representation integrates visual and action features to improve the integrity of information expression and downstream prediction capabilities.

[0107] During the prediction sequence generation process, the visual diffusion decoder is responsible for restoring the future video frame sequence, while the motion diffusion decoder is responsible for restoring the future action sequence. The masked prediction sequence introduces random information loss, enhancing the model's robustness to future uncertainties. The loss value is calculated based on the deviation between the predicted sequence and the real data. The optimization parameters are adjusted through a gradient update mechanism to adjust the parameters of the joint modeling module, the visual diffusion decoder, and the motion diffusion decoder, improving the overall prediction accuracy of the system. The training process is terminated based on preset training termination conditions, which may include reaching a set number of iterations, the loss value falling below a preset threshold, or stable performance indicators.

[0108] Finally, the trained joint modeling module, visual diffusion decoder and motion diffusion decoder are obtained, which have the ability to stably fuse multimodal information, predict future trends and adapt to complex scenarios.

[0109] In the field of robotic agents, historical sequences are derived from the visual perception and action data of the robot's multimodal sensing system. During training, the aforementioned stages are executed cyclically, combining visual information with action feedback. Masking operations are used to simulate scenarios with information loss. Encoding and joint modeling enhance multimodal representation, and prediction sequences simulate future environmental changes and operational trends. The visual diffusion decoder generates high-fidelity video sequences of the environment, while the action diffusion decoder outputs action decision sequences. Through multiple iterations, system parameters are continuously optimized until the robot possesses high-precision visual understanding and action reasoning capabilities, suitable for path planning, object manipulation, and collaborative execution tasks in dynamic environments.

[0110] In healthcare scenarios, historical sequences can include medical imaging data and medical device operation records. The training process, through multiple masking and prediction operations, improves the system's ability to process incomplete medical information and predict future lesion development trends. The visual diffusion decoder generates continuous medical image sequences to assist doctors in understanding lesion evolution, while the action diffusion decoder generates action suggestion sequences to optimize diagnostic and treatment equipment configuration. Through multiple rounds of training, a stable and efficient medical decision-making support model is developed, enhancing the accuracy and safety of diagnosis and treatment.

[0111] In the FinTech sector, historical sequences represent interface interaction data and user operation records. During training, through repeated masking, encoding, and prediction operations, the system gradually strengthens its ability to comprehensively understand multi-source information, improving its ability to predict interface interaction trends and user behavior. The visual diffusion decoder generates a sequence of interface change trends, while the action diffusion decoder generates a sequence of customer operations. After multiple optimizations and adjustments, the system possesses efficient interactive reasoning capabilities, making it suitable for intelligent customer service, risk warning, and high-concurrency trading environments.

[0112] By repeatedly executing a loop structure of random masking, encoding, multimodal processing, prediction generation, mask interference, difference calculation, and parameter optimization, this embodiment enables the system to continuously obtain feedback information and dynamically adjust parameter configurations, gradually improving the multimodal information fusion capability and the accuracy of future trend predictions. Ultimately, a joint modeling module, visual diffusion decoder, and motion diffusion decoder with high reliability and environmental adaptability are formed, which are suitable for high-precision reasoning and decision-making needs in complex application scenarios in multiple fields.

[0113] S100, using the trained joint modeling module, visual diffusion decoder and action diffusion decoder to process the reasoning task and generate a target future action sequence and / or a target future video frame sequence.

[0114] In this embodiment, the trained joint modeling module, visual diffusion decoder and action diffusion decoder are used to process reasoning tasks to generate target future action sequences and / or target future video frame sequences, which belongs to a multimodal information reasoning process based on the completed training parameter configuration, and is mainly used for prediction and generation of unknown environments or future trends. The trained joint modeling module refers to a structure that has the ability to stably fuse potential visual token sequences and action token sequences after a cyclic optimization and error feedback training process, and can output a unified potential representation to fully express historical information and environmental characteristics. The visual diffusion decoder refers to an image sequence generation unit constructed based on the diffusion modeling principle, which receives a unified potential representation as input, and gradually generates a future video frame sequence that meets the prediction target, and has the function of restoring visual dynamic information and environmental evolution trends. The action diffusion decoder refers to an action sequence generation unit designed based on a diffusion structure, which receives a unified potential representation input, outputs a future action sequence through a multi-step generation process, and has the ability to simulate operation trends and decision outputs.

[0115] Reasoning tasks involve the system predicting future information and determining trends based on existing historical information and model parameters, even in the presence of unknown data or in real-world application environments. The generated target future action sequence represents the system's inference of possible future operations, actions, or decision sequences, while the target future video frame sequence represents a predicted expression of the evolving trends of the visual information in the environment. Both can be generated independently or simultaneously, depending on application requirements, enabling flexible reasoning capabilities across multiple scenarios.

[0116] In the field of robotic agents, reasoning tasks typically include predicting action paths, perceiving environmental dynamics, and inferring task execution strategies. Real-time visual images and action feedback are captured and fed into a trained joint modeling module to generate a unified latent representation. The visual diffusion decoder uses this representation to generate a target future video frame sequence, assisting the robot in proactively perceiving environmental trends. The action diffusion decoder generates a target future action sequence and outputs the corresponding action path, joint control, or grasping action sequence, enhancing the robot's autonomous decision-making and dynamic adaptability in complex environments.

[0117] In healthcare scenarios, reasoning tasks involve predicting lesion evolution, assisting with surgical procedures, and generating intelligent diagnosis and treatment recommendations. The system receives medical imaging data and medical procedure information. A joint modeling module fuses multimodal information and outputs a unified latent representation. The visual diffusion decoder generates a continuous sequence of future medical images to assist doctors in observing lesion trends. The action diffusion decoder generates a sequence of medical procedure recommendations, optimizing equipment configuration and operation paths, and improving diagnostic and treatment efficiency and safety.

[0118] In FinTech scenarios, reasoning tasks include predicting interface interactions, analyzing user action trends, and generating intelligent recommendation strategies. The system acquires user action data and interface state information to generate a unified latent representation. The visual diffusion decoder generates images of interface change trends, while the action diffusion decoder generates user action sequences. This aids in intelligent interface adjustments, predicting user behavior, and providing efficient personalized recommendation services.

[0119] This embodiment uses the trained joint modeling module, visual diffusion decoder and motion diffusion decoder to perform reasoning tasks. The system can fully utilize historical information and optimized multimodal fusion capabilities to accurately generate the target's future action sequence and target's future video frame sequence, improve environmental perception, trend prediction and operational decision-making capabilities, and is suitable for complex scenarios in multiple fields such as robotic intelligence, medical health and financial technology, and significantly enhance the system's intelligent reasoning level and practical application reliability.

[0120] The present invention relates to the field of artificial intelligence technology and can be applied to business scenarios such as robot intelligent body decision-making, financial technology, and medical health. A data sequence generation method, device, equipment, and medium based on joint modeling are disclosed, including: obtaining training samples including historical image sequences, historical action sequences, real future video frame sequences, and real future action sequences, applying a random mask to at least one of the historical image sequences or historical action sequences to obtain a masked historical sequence, encoding the historical image sequence in the masked historical sequence into a latent visual token sequence, encoding the historical action sequence in the masked historical sequence into an action token sequence, processing the latent visual token sequence and the action token sequence through a joint modeling module, generating a unified latent representation, and applying a random mask to at least one of the historical image sequences or historical action sequences to obtain a masked historical sequence. Indicates that a visual diffusion decoder is used to generate a predicted future video frame sequence, and a motion diffusion decoder is used to generate a predicted future action sequence. A random mask is applied to at least one of the predicted future video frame sequence or the predicted future action sequence to obtain a masked predicted sequence. A loss value is determined based on the difference between the masked predicted sequence and the real future video frame sequence and the real future action sequence. The parameters of the joint modeling module, the visual diffusion decoder, and the motion diffusion decoder are optimized based on the loss value. The above operation is repeated to obtain a trained joint modeling module, a visual diffusion decoder, and a motion diffusion decoder. The trained joint modeling module, the visual diffusion decoder, and the motion diffusion decoder are used to process the reasoning task to generate a target future action sequence and / or a target future video frame sequence. The present invention constructs a unified potential representation by fusing visual information and action information, and jointly optimizes the visual diffusion decoder and the motion diffusion decoder to avoid error transmission and information loss caused by separate predictions, thereby improving the prediction accuracy under multimodal data and the overall reasoning efficiency of the system.

[0121] In one embodiment, the above step S20 includes:

[0122] S201, determining a type of sequence to be masked, where the type of sequence to be masked is at least one of a historical image sequence or a historical action sequence;

[0123] S202, generating a random mask position indication corresponding to the type of the sequence to be masked;

[0124] S203, creating a mask matrix corresponding to the random mask position indication;

[0125] S204: Apply the mask matrix to the sequence type to be masked to obtain a masked historical sequence.

[0126] In this embodiment, a random mask is applied to at least one of the historical image sequence or the historical action sequence to obtain a masked historical sequence, which belongs to an input information interference processing mechanism based on a sequence data structure, and is used to introduce randomness and information shielding during the training process to enhance the robustness and generalization ability of the model. A historical image sequence refers to a collection of image information arranged in chronological order, reflecting environmental changes, visual scenes or appearance features of operation targets, and is derived from cameras, sensors or other visual data acquisition devices. A historical action sequence refers to an operation instruction, control signal or system execution status arranged in chronological order, reflecting the system's behavioral decisions and operation paths in historical stages, and is derived from an action control system, operation log or system feedback data.

[0127] Determining the type of sequence to be masked means selecting to perform a masking operation on at least one type of data in the historical image sequence or historical action sequence based on training requirements or model structure configuration. This supports masking only image information, only action information, or both types of information at the same time, and has flexible input interference configuration capabilities.

[0128] Generating random mask position indications corresponding to the type of sequence to be masked means determining some positions in the sequence as mask targets based on a pseudo-random number generation algorithm, a distribution control mechanism, or preset rules. The position indication information can be a sequence index, a timestamp, or a position code to ensure that the mask positions are random and dispersed, and to avoid data invalidation caused by excessive information masking.

[0129] Creating a mask matrix corresponding to the random mask position indication means constructing a mask matrix that matches the input sequence structure. By setting the corresponding position in the matrix as the occlusion mark value, the control information at the mask position is hidden, set to zero, replaced or disturbed. The mask matrix supports multi-dimensional structural expression and is adaptable to different types of data such as images and actions.

[0130] Applying a mask matrix to the sequence type to be masked to obtain a masked historical sequence means applying the mask matrix to the input data, blocking real information at selected locations and retaining complete data at other locations, forming a training input with missing information and complete structure. The masked historical sequence retains the overall data structure while artificially creating information gaps, thereby improving the environmental diversity and model adaptability during the training phase.

[0131] This embodiment introduces random masks into historical image sequences or historical action sequences to form masked historical sequences, which can effectively avoid the model's excessive dependence on complete input data, enhance the prediction ability in missing information or uncertain environments, and improve the system's robustness and generalization level to changes in visual and action information. It helps to maintain stable sequence modeling and reasoning effects in complex environments, multi-task requirements or incomplete data conditions.

[0132] In one embodiment, the above step S30 includes:

[0133] S301, extracting visual features of the historical image sequence in the masked historical sequence to obtain an image feature sequence;

[0134] S302, performing dimensionality reduction processing on the image feature sequence to generate a visual feature vector;

[0135] S303, inputting the visual feature vector into a pre-trained variational autoencoder, and outputting a potential visual token sequence through the pre-trained variational autoencoder;

[0136] S304, performing temporal resolution improvement processing on the historical action sequence in the masked historical sequence to obtain an action sequence with improved resolution;

[0137] S305, extracting motion features of the motion sequence after the resolution is improved to obtain a motion feature vector;

[0138] S306: Map the action feature vector into an action token sequence.

[0139] In this embodiment, the historical image sequence in the masked historical sequence is encoded as a potential visual token sequence, and the historical action sequence in the masked historical sequence is encoded as an action token sequence. This belongs to the process of structured expression of multimodal information, which aims to uniformly map visual and action information into recognizable low-dimensional symbolic expressions within the sequence modeling structure through feature extraction and conversion mechanisms, thereby improving the cross-modal fusion efficiency and the accuracy of upstream and downstream information transmission.

[0140] Extracting visual features from the masked historical image sequence to obtain an image feature sequence involves parsing the visual information content in the historical image sequence based on an image coding network, convolutional neural network, or visual Transformer architecture. This extracts multidimensional visual features, including image texture, edge structure, color distribution, and spatial positional relationships, to form a time-series image feature set. Visual features can include static image features, inter-frame dynamic information, or scene change trends, and are derived from the image processing module or video encoding unit.

[0141] Dimensionality reduction processing of image feature sequences to generate visual feature vectors refers to reducing the spatial dimension and data redundancy of image feature sequences through linear transformation, principal component analysis, feature compression network or other information dimensionality reduction technologies, while retaining the ability to express information while reducing the data size and computational complexity, and generating compact visual feature vectors that can be input into downstream sequence models.

[0142] Inputting the visual feature vector into a pre-trained variational autoencoder and outputting a latent visual token sequence through the pre-trained variational autoencoder means using a variational autoencoder structure that has been pre-trained on large-scale visual data. Based on probability distribution mapping and latent space reparameterization technology, the visual feature vector is encoded into a latent visual token sequence that conforms to the target distribution. The latent visual token sequence is a discrete tag or low-dimensional symbol that can be directly processed by the sequence model, and has the ability to condense visual information, compress input scale, and retain environmental structure.

[0143] The temporal resolution of the historical action sequence in the masked historical sequence is improved to obtain the action sequence with improved resolution. This means that the temporal sampling accuracy and expression continuity of the historical action sequence are improved through interpolation algorithms, sequence compensation mechanisms or reconstruction methods based on time series models, and the fine-grained temporal expression capability of action information is enhanced to adapt to the needs of high-resolution action prediction or dynamic process modeling.

[0144] Extracting motion features of action sequences with enhanced resolution to obtain motion feature vectors refers to parsing the dynamic information in action sequences based on sequence encoders, temporal feature extraction networks, or statistical feature analysis, extracting motion features including speed changes, directional trends, and temporal correlations, and generating compact motion feature vectors for structured expression of action information.

[0145] Mapping action feature vectors into action token sequences means converting action feature vectors into action token sequences required for sequence modeling through symbolic encoding, vector quantization, or dictionary-based mapping. Action token sequences are low-dimensional, structured, and parsable, making them convenient for simultaneous modeling with visual information in joint sequence models, thereby improving cross-modal information fusion and action prediction accuracy.

[0146] This embodiment encodes the masked historical image sequence into a potential visual token sequence, and the masked historical action sequence into an action token sequence. This can effectively compress the expression scale of multimodal information, improve the fusion efficiency and structural consistency of data in the sequence modeling process, avoid the increased computational complexity and prediction error diffusion problems caused by information dimension mismatch, further enhance the system's collaborative perception and efficient expression capabilities of visual and action information in complex environments, and help improve the accuracy and robustness of multimodal joint prediction.

[0147] In one embodiment, the above step S40 includes:

[0148] S401, concatenating the potential visual token sequence and the action token sequence to form a joint input sequence;

[0149] S402, performing a position coding operation on the combined input sequence to obtain a position coding sequence;

[0150] S403, inputting the position code sequence into the multi-head attention layer of the Transformer module to perform interaction relationship modeling;

[0151] S404: Generate the unified latent representation through the output layer of the Transformer module.

[0152] In this embodiment, the potential visual token sequence and the action token sequence are processed by a joint modeling module to generate a unified potential representation, which is a process of multimodal information fusion and temporal relationship modeling. The purpose is to structurally integrate visual information and action information under the same sequence modeling framework, generate a unified expression form that can be used for downstream generation and prediction, and improve information interaction efficiency and multimodal collaborative modeling capabilities.

[0153] Concatenating the latent visual token sequence and the action token sequence to form a joint input sequence involves combining the latent visual token sequence and the action token sequence based on sequence structure, according to chronological order, information priority, or other predefined concatenation rules, to form a joint input sequence containing both information sources. This joint input sequence preserves the original representation structure of the visual and action information, facilitating subsequent unified modeling. The sources include the latent visual token sequence and the action token sequence generated by the encoding module.

[0154] Performing positional encoding on the joint input sequence to generate a position-encoded sequence is a process that, based on sequence modeling's dependence on the temporal dimension or structural position, injects the temporal position, relative position, or other structural information of each element in the sequence into the sequence representation through a positional encoding mechanism, generating a position-aware encoded sequence. Positional encoding can employ sinusoidal positional encoding, learnable position vectors, or encoding methods based on relative position relationships. Positional encoding sequences enhance the structural differentiation within the joint input sequence, strengthening the model's temporal modeling capabilities.

[0155] Feeding position-encoded sequences into the Transformer module's multi-head attention layer for interactive relationship modeling involves feeding sequences containing positional information into a Transformer structure based on a multi-head self-attention mechanism. This multi-head attention layer simultaneously captures dependencies within sequences of varying dimensions and scopes, modeling the dynamic interactions, temporal associations, and potential coupling between visual and motion information. The multi-head attention mechanism enables efficient global information exchange, enhancing the depth of multimodal data fusion and predictive expressiveness.

[0156] Generating a unified latent representation through the output layer of the Transformer module involves structurally integrating and compressing the sequence representation after the multi-head attention layer completes interaction modeling, outputting a unified latent representation that integrates visual and motion information. This unified latent representation can be a low-dimensional latent space representation, a structured sequence vector, or other combined representations, which can be used for prediction, generation, and inference in downstream tasks.

[0157] This embodiment forms a joint input sequence by splicing the potential visual token sequence and the action token sequence, and models the interactive relationship based on position encoding and multi-head attention mechanism. It can enhance the collaborative expression and temporal association modeling effect between visual information and action information while maintaining the integrity of the information structure, avoid the problem of decreased prediction accuracy caused by information fragmentation, and improve the overall expression consistency and information utilization efficiency of the multimodal sequence, thereby improving the accuracy and robustness of subsequent prediction generation, and meeting the application requirements of joint reasoning and dynamic prediction in complex multimodal scenarios.

[0158] In one embodiment, the above step S50 includes:

[0159] S501, inputting the unified latent representation into a visual diffusion decoder, performing a multi-step noise addition operation in the visual diffusion decoder to generate a noise addition sequence;

[0160] S502, performing iterative denoising processing on the noise-added sequence to generate a predicted future video frame sequence;

[0161] S503, inputting the unified latent representation into an action diffusion decoder, performing a time-stepping noise injection operation in the action diffusion decoder, and generating a noise injection sequence;

[0162] S504: Perform iterative denoising processing on the noise injection sequence to generate a predicted future action sequence.

[0163] In this embodiment, based on the unified latent representation, a visual diffusion decoder is used to generate a predicted future video frame sequence, and an action diffusion decoder is used to generate a predicted future action sequence. This involves a decoding and generation process for the joint expression of multimodal information. The purpose is to obtain structured outputs related to the future visual state and future action trends respectively through a diffusion generation mechanism, thereby improving the coherence, authenticity and expression consistency of the prediction results.

[0164] Feeding the unified latent representation into the visual diffusion decoder involves feeding it with a unified latent representation that fuses visual and motion information, providing the complete context needed to predict future video frame sequences. The visual diffusion decoder is capable of generating visual sequences, typically using a structure based on a conditional diffusion model. This architecture can gradually generate high-quality video frame sequences through noise introduction and denoising.

[0165] Performing multi-step noise addition operations in the visual diffusion decoder to generate a noise-added sequence means that during the generation process, random noise is gradually injected into the visual sequence according to the settings of the diffusion model, forming an intermediate state sequence that gradually evolves from low quality to high quality. The noise addition operation enables the model to have the ability to express the inverse generation process from noise to data. The noise-added sequence reflects the dynamic evolution path of the generation process.

[0166] Iterative denoising is performed on the noise-added sequence to generate a predicted future video frame sequence. This means that after completing multi-step noise addition, an inverse process design based on the diffusion model is performed. Through multiple iterative denoising operations, the detail information and structural information in the visual sequence are gradually restored, and finally a predicted future video frame sequence with high resolution, continuity and authenticity is generated to express the trend of visual state changes in the future time period.

[0167] Inputting the unified latent representation into the action diffusion decoder means that in parallel with the visual generation process, the unified latent expression result is used as the input condition of the action diffusion decoder to provide the contextual information background required for action sequence generation. The action diffusion decoder has the ability to generate action trajectories, behavioral trends or control instruction sequences, and is suitable for complex dynamic prediction tasks.

[0168] A time-stepped noise injection operation is performed in the action diffusion decoder to generate a noise-injected sequence. This means that random noise is gradually injected into the action sequence in a time-stepping manner based on the timing characteristics of the action sequence, forming an intermediate state sequence dominated by noise. The time-stepped noise injection operation ensures that the action sequence has independent randomness control in each time segment, improving the diversity and temporal consistency of the generation process.

[0169] Performing iterative denoising on the noise-injected sequence to generate a predicted future action sequence means gradually restoring the details and dynamic features of the action sequence through multiple iterative denoising operations based on the diffusion decoding structure, generating a predicted future action sequence with coherence, logic, and environmental adaptability. This is used to express behavioral trends, control decisions, or action trajectories in the future time period, meeting the dynamic prediction and control needs in complex scenarios.

[0170] This embodiment inputs a unified latent representation into the visual diffusion decoder and the motion diffusion decoder respectively, combines multi-step noise addition and time-step noise injection processes, and cooperates with an iterative denoising mechanism. This can achieve efficient decoding of multimodal information and recovery of dynamic details in the process of generating future video frame sequences and future action sequences, avoiding the problem of incomplete expression caused by information fragmentation or a single data source, and effectively improving the accuracy, continuity and environmental adaptability of visual state and motion trend prediction, thereby enhancing the overall performance and application value of the multi-task dynamic prediction system in complex application environments in multiple fields such as robotic intelligence, medical health, and financial technology.

[0171] In one embodiment, the above step S70 includes:

[0172] S701, determining a first difference value between a future video frame sequence in a masked prediction sequence and a real future video frame sequence;

[0173] S702, determining a video loss component based on the first difference value;

[0174] S703, determining a second difference value between the future action sequence in the masked prediction sequence and the actual future action sequence;

[0175] S704, determining an action loss component based on the second difference value;

[0176] S705: Combine the video loss component and the motion loss component to obtain a loss value.

[0177] In this embodiment, the loss value is determined based on the difference between the future video frame sequence in the masked prediction sequence and the real future video frame sequence, as well as the difference between the future action sequence in the masked prediction sequence and the real future action sequence. This is a joint error evaluation process based on multi-source prediction results and real data, which aims to construct a comprehensive optimization feedback indicator by distinguishing the prediction deviations of visual information and action information, and provide quantitative guidance for subsequent model parameter updates.

[0178] Determining the first difference value between the future video frame sequence in the masked predicted sequence and the real future video frame sequence refers to comparing the predicted future video frame sequence after the mask operation with the real future video frame sequence in the corresponding time period frame by frame or as a whole based on the generation effect of the visual information. The specific calculation method of the difference may include pixel-level error, structural similarity, feature space distance, etc. The first difference value is used to quantify the degree of deviation between the visual prediction output and the real visual state.

[0179] Determining the video loss component based on the first difference value means converting the first difference value into a quantitative error index as a feedback parameter for the quality of visual information generation. The video loss component can adopt various forms such as weighted average error, structural loss function, perceptual loss function, etc. to ensure that the component comprehensively reflects the detail restoration, continuity and authenticity indicators in the process of video sequence generation.

[0180] Determining the second difference value between the future action sequence in the predicted sequence after masking and the actual future action sequence refers to comparing the predicted future action sequence after masking with the actual future action sequence in terms of timing, amplitude, dynamic trend and other dimensions based on the generation effect of the action information. The calculation of the second difference value can be combined with methods such as sequence distance, dynamic time warping, and trajectory matching error to comprehensively quantify the degree of difference between the action prediction results and the actual behavioral performance.

[0181] Determining the action loss component based on the second difference value means generating a quantitative error indicator reflecting the quality of action sequence generation based on the second difference value result. The action loss component can take various forms such as timing error accumulation, motion trend deviation, dynamic consistency deviation, etc., to ensure that this component comprehensively characterizes the prediction accuracy of the action sequence in terms of coherence, logic and environmental adaptability.

[0182] Combining the video loss component and the motion loss component to obtain the loss value means integrating the video loss component and the motion loss component into a single optimization target indicator through weighted fusion or other combination strategies. The loss value serves as the error feedback standard of the overall prediction system, taking into account both the visual expression effect and the motion prediction accuracy, providing a unified quantitative basis for subsequent multi-module collaborative optimization, and improving the comprehensive prediction ability of the overall system under complex dynamic tasks.

[0183] This embodiment determines the prediction differences between visual information and motion information respectively, independently calculates the video loss component and the motion loss component, and combines the two into a unified loss value. This can effectively avoid the optimization offset problem under the dominance of a single indicator, achieve a balanced evaluation of multimodal prediction outputs, improve the stability, accuracy and multi-task generalization ability of the joint modeling system in dynamic environments, and enhance the overall performance in complex prediction tasks in multiple fields such as robotic intelligence, medical health, and financial technology.

[0184] In one embodiment, the above step S80 includes:

[0185] S801, determining the gradient of the loss value with respect to the parameters of the joint modeling module, and adjusting the parameter update amount of the joint modeling module through an adaptive learning rate strategy based on the gradient;

[0186] S802, determining a gradient of the loss value with respect to a parameter of the visual diffusion decoder, and adjusting an update amount of the parameter of the visual diffusion decoder based on a preset gradient magnitude threshold constraint;

[0187] S803, determining a gradient of the loss value with respect to a parameter of the motion diffusion decoder, and adjusting a parameter update amount of the motion diffusion decoder according to time series correlation regularization based on the gradient;

[0188] S804: Update the parameters of the joint modeling module, the visual diffusion decoder, and the motion diffusion decoder respectively based on the parameter update amount.

[0189] In this embodiment, the parameters of the joint modeling module, the parameters of the visual diffusion decoder, and the parameters of the motion diffusion decoder are optimized based on the loss value, which is a dynamic parameter adjustment process for multi-module collaborative training. It aims to use unified error feedback information to implement differentiated and customized parameter optimization operations based on the structural differences and functional requirements of different functional modules in the system, thereby improving the learning efficiency and stability of the overall prediction system.

[0190] Determining the gradient of the loss value with respect to the parameters of the joint modeling module involves calculating the sensitivity of each joint modeling module parameter to the overall error, based on the relationship between the joint modeling module's output and the system's overall loss value. This gradient reflects the theoretical basis for the direction and magnitude of parameter adjustment and serves as the foundation for iteratively updating parameters such as the weight matrix, bias terms, and mapping layers within the joint modeling module structure. Joint modeling modules typically undertake the task of deeply integrating and associating multimodal information. Accurately calculating parameter gradients is directly related to the quality of multi-source data feature integration and potential information extraction.

[0191] Gradient-based adaptive learning rate strategies adjust the parameter updates of the joint modeling module. This involves combining dynamic feedback from the training process with factors such as loss trends, gradient fluctuations, and historical convergence results to dynamically adjust the actual update amplitudes of the joint modeling module's parameters, avoiding slow convergence or gradient oscillation issues associated with a single fixed learning rate. Adaptive learning rate strategies can include algorithms based on second-order gradient information, periodic attenuation mechanisms, and gradient statistical feature analysis to ensure stable and efficient training of the joint modeling module in the context of multi-task, high-dimensional, and complex data.

[0192] Determining the gradient of the loss value with respect to the parameters of the visual diffusion decoder involves using a backpropagation mechanism to calculate the gradient information of the internal parameters of the visual diffusion decoder based on the difference between the predicted future video frame sequence generated by the visual diffusion decoder and the actual future video frame sequence. The gradient reflects the contribution of each parameter to the video prediction error and serves as the basis for subsequent parameter adjustments. The visual diffusion decoder is typically based on a diffusion model structure and relies on a multi-step noise modeling and denoising process. Accurately obtaining the parameter gradient is crucial to the overall optimization of video generation quality, detail restoration, and visual coherence.

[0193] Adjusting the parameter updates of the visual diffusion decoder based on a preset gradient amplitude threshold constraint limits extreme fluctuations during parameter updates by setting a reasonable upper limit on the gradient amplitude, thus avoiding model instability, prediction anomalies, or training failures caused by gradient explosion or excessive parameter adjustments. The gradient amplitude threshold can be flexibly set based on specific task requirements and data distribution characteristics. Combined with other technical measures such as cropping, normalization, and stability regularization, this ensures the robustness and stability of the visual diffusion decoder in high-resolution video generation tasks.

[0194] Determining the gradient of the loss with respect to the parameters of the action diffusion decoder involves calculating the sensitivity of each action diffusion decoder parameter to the overall prediction error based on the difference between the predicted future action sequence generated by the action diffusion decoder and the actual future action sequence. This gradient information is used to guide the directional adjustment of the action diffusion decoder parameters, improving the accuracy and logical coherence of the generated action sequence. The action diffusion decoder is typically designed to model high-temporal-resolution, multi-dimensional action signals, accurately capturing complex dynamic features and temporal dependencies, and relying on an efficient gradient feedback mechanism.

[0195] Gradient-based regularization of time series correlation adjusts the parameter updates of the action diffusion decoder. This involves integrating the dynamic continuity and temporal stability requirements of action sequences, introducing time series correlation constraints, and regularizing the gradient information. This suppresses incoherent and abrupt parameter updates, thereby enhancing the dynamic consistency, temporal smoothness, and logical rationality of the action generation process. Time series correlation regularization can be implemented through methods such as sequence autocorrelation analysis, adjacent frame smoothing constraints, and dynamic trend preservation mechanisms, ensuring stable output and high-quality prediction capabilities of the action diffusion decoder for long sequences and complex behavioral patterns.

[0196] Updating the parameters of the joint modeling module, visual diffusion decoder and motion diffusion decoder respectively based on the parameter update amount means combining the parameter update information obtained by independent calculation of each module to synchronously update the internal parameters of the joint modeling module, visual diffusion decoder and motion diffusion decoder respectively, to ensure the overall consistency, structural stability and systematic improvement of prediction performance in the process of multi-module collaborative training, and to form a stable and efficient end-to-end joint optimization mechanism.

[0197] Example description: In an autonomous interactive service robot system, a human-machine collaborative intelligent agent is deployed for a multi-task environment. The system collects the robot's historical image sequences and historical action sequences in real time through a multi-sensor network. The historical image sequences include visual information captured by the robot's on-board camera, robotic arm joint camera, and external monitoring system. The historical action sequences include robotic arm operation instructions, mobile chassis trajectory control information, and interactive behavior feedback records. At the same time, training samples are constructed by combining real future video frame sequences and real future action sequences. The real future video frame sequences are obtained through visual monitoring during the actual task execution process, and the real future action sequences are obtained through action log records after the robot completes the task.

[0198] During the process of constructing training data, the system randomly selects at least one information channel in the historical image sequence or historical action sequence, applies a random mask, and forms a masked historical sequence. The position of the masked area is dynamically determined based on the system's adaptive strategy to ensure the diversity and comprehensiveness of information interference patterns at different training stages. The system then performs visual feature extraction on the historical image sequence in the masked historical sequence, analyzes the spatial structure information of the image sequence through a multi-scale convolutional neural network, outputs an image feature sequence, compresses the image feature sequence using a dimensionality reduction algorithm, and forms a compactly expressed visual feature vector. This visual feature vector is input into a pre-trained variational autoencoder to obtain a potential visual token sequence. At the same time, the system improves the temporal resolution of the historical action sequence in the masked historical sequence, refines the trend of action changes, extracts high-frequency dynamic features, obtains action feature vectors, and converts them into action token sequences through a feature mapping process.

[0199] Through the joint modeling module, the system splices the latent visual token sequence and the action token sequence to form a joint input sequence, performs position encoding operations, captures the internal structural information and temporal logical relationship of the sequence, and inputs the joint input sequence into the multi-head attention layer of the Transformer module to carry out cross-modal information interaction modeling, output a unified latent representation, and integrate visual and action information to form a complete environmental cognition and historical behavior modeling result.

[0200] Based on the unified latent representation, the system generates predicted future video frame sequences and predicted future action sequences through the visual diffusion decoder and the motion diffusion decoder respectively. The visual diffusion decoder performs multi-step noise addition and iterative denoising internally to gradually reconstruct high-resolution visual information. The motion diffusion decoder combines time-step noise injection and dynamic smoothing iterative denoising to generate a high-temporal-resolution action output sequence that conforms to the task logic.

[0201] To enhance the system's robustness and generalization capabilities, the system randomly selects at least one information sequence from the predicted future video frame sequence or predicted future action sequence and applies a random mask to it. This generates a masked prediction sequence, constructs prediction scenarios for situations where information is missing, and further examines the system's multimodal information completion capabilities. By calculating the first difference between the future video frame sequence and the true future video frame sequence, and the second difference between the future action sequence and the true future action sequence, the system determines the video loss component and the action loss component, respectively, and combines them to generate an overall loss value.

[0202] Based on the loss value back-propagation mechanism, the system optimizes the parameters of the joint modeling module, visual diffusion decoder and motion diffusion decoder respectively. The parameter update of the joint modeling module is combined with the adaptive learning rate strategy to dynamically adjust the amplitude. The parameter update of the visual diffusion decoder is subject to the gradient amplitude threshold constraint to prevent abnormal oscillations. The parameter update of the motion diffusion decoder is regularized according to the time series correlation to ensure the coherence and dynamic stability of the action sequence.

[0203] The above steps are repeated through a continuous cycle of historical sequence masking, encoding, modeling, prediction, masking, error calculation and parameter optimization until the system meets the preset training termination conditions, and obtains the trained joint modeling module, visual diffusion decoder and motion diffusion decoder. The system has the ability of autonomous perception, dynamic prediction and multimodal collaborative reasoning.

[0204] In the actual reasoning application stage, the system obtains the real-time environment input sequence, generates a unified potential representation for the reasoning stage based on the trained joint modeling module, and combines the visual diffusion decoder and the action diffusion decoder to output the target future video frame sequence and the target future action sequence on demand, driving the robot to achieve autonomous path planning, dynamic obstacle avoidance and continuous prediction of task actions in complex interactive scenarios, effectively improving the robot's decision-making intelligence and operational consistency in multi-tasks and multi-environment.

[0205] In the field of medical health, combining patient status monitoring and clinical decision support needs, the system obtains training samples including historical image sequences, historical action sequences, real future video frame sequences and real future action sequences. The historical image sequences are derived from the patient's body surface images, instrument operation perspective images and physiological monitoring screens collected by surgical robots, rehabilitation equipment or ward monitoring systems. The historical action sequences include operation action records performed by medical robots, posture adjustment instructions of rehabilitation assistive devices and patient autonomous or passive motion trajectory information. The real future video frame sequences and real future action sequences are constructed based on video records and motion capture data of the actual treatment process or rehabilitation process.

[0206] Based on an adaptive strategy, the system applies a random mask to at least one of the historical image sequences or historical action sequences, simulating scenarios with incomplete information or missing data. This generates a masked historical sequence, further improving the model's stability and ability to address abnormal data loss in medical monitoring. The system then extracts visual features from the masked historical image sequence, obtaining an image feature sequence. This is then processed through dimensionality reduction to generate a visual feature vector, which is then fed into a variational autoencoder pre-trained with medical imaging features to generate a latent visual token sequence. The system also improves the temporal resolution of the masked historical action sequence, extracting key motion features, converting them into action feature vectors, and mapping them into action token sequences.

[0207] Through the joint modeling module, the system splices the latent visual token sequence and the action token sequence to form a joint input sequence. The position encoding operation captures the sequence time information and structural characteristics. The Transformer multi-head attention mechanism fuses multimodal information and outputs a unified latent representation that comprehensively reflects the patient's historical images, action trends and equipment operation status.

[0208] Based on the unified latent representation, the system uses the visual diffusion decoder and the motion diffusion decoder in sequence to generate predicted future video frame sequences and predicted future action sequences. The visual diffusion decoder restores high-resolution medical images through multi-step noise addition and iterative denoising. The motion diffusion decoder generates action prediction sequences at high temporal resolution through time-step noise injection and denoising, thereby improving the system's ability to perceive patients' upcoming actions, posture adjustments, or medical equipment operation trends.

[0209] The system randomly applies a mask to the predicted sequence to generate a masked predicted sequence, simulating data-missing scenarios and enhancing system robustness. The system then calculates the difference between the masked predicted sequence and the true future video frame sequence and true future action sequence, determining the video loss component and the action loss component, respectively, and combining them to form a total loss value. Based on this loss value, the system optimizes the parameters of the joint modeling module, the visual diffusion decoder, and the action diffusion decoder. The joint modeling module is dynamically adjusted using an adaptive learning rate strategy. The visual diffusion decoder is optimized based on gradient amplitude threshold constraints, and the action diffusion decoder is adjusted using a time series correlation regularization strategy to ensure a coherent and stable output sequence.

[0210] The system continuously loops the above operations until the training termination conditions are met, and obtains the trained joint modeling module, visual diffusion decoder and motion diffusion decoder. When applied to reasoning tasks in the medical and health field, the system generates a unified potential representation based on real-time monitoring input, combines two types of diffusion decoders, and outputs the target future action sequence and target future video frame sequence on demand, assisting in the prediction of the patient's upcoming posture changes, movement trends or medical robot movements. It is widely used in rehabilitation training, surgical assistance, real-time monitoring of health status and remote medical equipment operation risk warning, effectively improving the safety of the medical process, patient care efficiency and remote operation accuracy.

[0211] In the field of financial technology, combining the needs of intelligent risk control, customer behavior modeling and financial transaction process prediction, the system obtains training samples including historical image sequences, historical action sequences, real future video frame sequences and real future action sequences. The historical image sequences can be derived from video surveillance of financial business outlets, screenshots of counter operation interfaces, images of intelligent customer service virtual interactive interfaces and other data. The historical action sequences include operation instructions executed by tellers, customers or financial service robots, user transaction behavior records or mouse, gesture and keyboard input trajectory information in the interactive system. The real future video frame sequences and real future action sequences are constructed based on the image data and system behavior logs of the actual business processing process.

[0212] The system simulates data incompleteness in financial transaction scenarios by randomly masking at least one of the historical image sequences or historical action sequences to generate a masked historical sequence. This enhances the model's robustness and predictive accuracy in the presence of missing abnormal information, video interference, or incomplete behavioral records. Visual features are further extracted from the masked historical image sequence to obtain an image feature sequence. This is then compressed through dimensionality reduction to obtain a visual feature vector, which is then fed into a pre-trained variational autoencoder to output a latent visual token sequence. The system also enhances the temporal resolution of the masked historical action sequence, extracting transaction behavior characteristics and operational trends, converting them into action feature vectors, and mapping them to generate an action token sequence.

[0213] The system adopts a joint modeling module to splice the latent visual token sequence and the action token sequence to form a joint input sequence, combines the position encoding operation to capture the time series structure and multimodal information in the financial business process, and models the complex interactive relationship between visual information and behavioral data through the Transformer multi-head attention layer. It outputs a unified latent representation, integrating the comprehensive feature expression reflecting the customer's historical behavior, operating environment and the status of the financial system.

[0214] Based on the unified latent representation, the system uses the visual diffusion decoder to perform multi-step noise addition and iterative denoising to generate a predicted future video frame sequence, simulating the image change trend of the counter scene, trading interface or risk warning system. Further, through the action diffusion decoder, combined with time-step noise injection and denoising processing, it generates a predicted future action sequence to reflect the customer's upcoming operation tendencies, system behavior changes or transaction path development trends.

[0215] The system applies a random mask to at least one of the generated predicted future video frame sequences or predicted future action sequences to generate a masked prediction sequence, enhancing the model's stability in the face of data anomalies, missing data, or system interference during the prediction process. The difference between the masked prediction sequence and the true future video frame sequence and true future action sequence is calculated, and the video loss component and action loss component are determined separately, which are combined to form a total loss value. The loss value is used to optimize the parameters of the joint modeling module, the visual diffusion decoder, and the action diffusion decoder. An adaptive learning rate strategy dynamically adjusts the joint modeling module. A gradient amplitude threshold constraint controls the changes in the visual diffusion decoder parameters. Time series correlation regularization improves the continuity and prediction consistency of the action diffusion decoder.

[0216] The system continues to execute the above loop until the preset training termination conditions are met, and obtains the trained joint modeling module, visual diffusion decoder and action diffusion decoder. When applied to reasoning tasks in the field of financial technology, it generates a unified potential representation based on real-time input data, and outputs the target future action sequence and target future video frame sequence through the diffusion decoder on demand, realizing high-precision prediction of financial business processing procedures, user behavior patterns and system operation trends. It is widely used in risk transaction behavior prediction, intelligent counter system anomaly detection, financial customer behavior analysis and real-time risk control auxiliary decision-making, helping to improve transaction security, business processing efficiency and the intelligence level of financial systems.

[0217] This embodiment implements customized parameter optimization processes based on adaptive learning rate strategy, gradient amplitude threshold constraint and time series correlation regularization for the joint modeling module, visual diffusion decoder and motion diffusion decoder respectively. It can effectively improve the overall training stability, parameter convergence speed and multimodal information expression ability of the multi-module joint system under complex prediction tasks, and enhance the system's adaptability, robustness and generalization effects in multi-source information joint modeling and high-accuracy prediction scenarios in fields such as robotic intelligence, medical health, and financial technology.

[0218] In one embodiment, a data sequence generation device based on joint modeling is provided, and the data sequence generation device based on joint modeling corresponds one-to-one to the data sequence generation method based on joint modeling in the above embodiment. Figure 3 , Figure 3This is a functional module diagram of a preferred embodiment of the data sequence generation device based on joint modeling of the present invention. It includes a training data preparation module 10, a history sequence masking module 20, a history sequence encoding module 30, a joint representation generation module 40, a prediction sequence generation module 50, a prediction sequence masking module 60, a loss calculation module 70, a parameter optimization module 80, a training control module 90, and an inference generation module 100. Each functional module is described in detail below:

[0219] A training data preparation module 10 is used to obtain training samples including historical image sequences, historical action sequences, real future video frame sequences, and real future action sequences;

[0220] a historical sequence masking module 20, configured to apply a random mask to at least one of the historical image sequence or the historical action sequence to obtain a masked historical sequence;

[0221] a history sequence encoding module 30 for encoding a history image sequence in the masked history sequence into a potential visual token sequence, and encoding a history action sequence in the masked history sequence into an action token sequence;

[0222] a joint representation generation module 40 for processing the latent visual token sequence and the action token sequence through a joint modeling module to generate a unified latent representation;

[0223] A prediction sequence generation module 50 is configured to generate a predicted future video frame sequence using a visual diffusion decoder and a predicted future action sequence using an action diffusion decoder based on the unified latent representation;

[0224] A prediction sequence masking module 60 is configured to apply a random mask to at least one of the predicted future video frame sequence or the predicted future action sequence to obtain a masked prediction sequence;

[0225] a loss calculation module 70 for determining a loss value based on a difference between a future video frame sequence in the masked predicted sequence and a true future video frame sequence, and a difference between a future action sequence in the masked predicted sequence and a true future action sequence;

[0226] a parameter optimization module 80, configured to optimize the parameters of the joint modeling module, the parameters of the visual diffusion decoder, and the parameters of the motion diffusion decoder based on the loss value;

[0227] A training control module 90 is configured to repeatedly execute the steps of applying random masking to the historical sequence, encoding, processing to generate a unified latent representation, generating a prediction sequence, applying random masking to the prediction sequence, determining a loss value, and optimizing parameters until a preset training termination condition is met, thereby obtaining a trained joint modeling module, a visual diffusion decoder, and a motion diffusion decoder;

[0228] The reasoning generation module 100 is used to process reasoning tasks using the trained joint modeling module, visual diffusion decoder and motion diffusion decoder to generate a target future action sequence and / or a target future video frame sequence.

[0229] In one embodiment, the historical sequence mask module 20 is specifically configured to:

[0230] Determining a type of sequence to be masked, where the type of sequence to be masked is at least one of a historical image sequence or a historical action sequence;

[0231] generating a random mask position indication corresponding to the type of sequence to be masked;

[0232] Creating a mask matrix corresponding to the random mask position indication;

[0233] Apply the mask matrix to the sequence type to be masked to obtain a masked history sequence.

[0234] In one embodiment, the historical sequence encoding module 30 is specifically configured to:

[0235] extracting visual features of the historical image sequence in the masked historical sequence to obtain an image feature sequence;

[0236] Performing dimensionality reduction processing on the image feature sequence to generate a visual feature vector;

[0237] Inputting the visual feature vector into a pre-trained variational autoencoder, and outputting a potential visual token sequence through the pre-trained variational autoencoder;

[0238] Performing temporal resolution improvement processing on the historical action sequence in the masked historical sequence to obtain an action sequence with improved resolution;

[0239] Extracting motion features of the motion sequence after the resolution is improved to obtain a motion feature vector;

[0240] The action feature vector is mapped into an action token sequence.

[0241] In one embodiment, the joint representation generation module 40 is specifically configured to:

[0242] splicing the potential visual token sequence and the action token sequence to form a joint input sequence;

[0243] Performing a position encoding operation on the combined input sequence to obtain a position encoding sequence;

[0244] Input the positional encoding sequence into the multi-head attention layer of the Transformer module to model the interaction relationship;

[0245] The unified latent representation is generated by the output layer of the Transformer module.

[0246] In one embodiment, the prediction sequence generation module 50 is specifically configured to:

[0247] inputting the unified latent representation into a visual diffusion decoder, performing a multi-step noise addition operation in the visual diffusion decoder to generate a noise addition sequence;

[0248] performing an iterative denoising process on the noise-added sequence to generate a predicted future video frame sequence;

[0249] Inputting the unified latent representation into an action diffusion decoder, performing a time-stepped noise injection operation in the action diffusion decoder to generate a noise injection sequence;

[0250] An iterative denoising process is performed on the noise injection sequence to generate a predicted future action sequence.

[0251] In one embodiment, the loss calculation module 70 is specifically configured to:

[0252] determining a first difference value between the future video frame sequence and the real future video frame sequence in the masked prediction sequence;

[0253] determining a video loss component based on the first difference value;

[0254] determining a second difference value between the future action sequence in the masked predicted sequence and the true future action sequence;

[0255] determining an action loss component based on the second difference value;

[0256] The video loss component and the motion loss component are combined to obtain a loss value.

[0257] In one embodiment, the parameter optimization module 80 is specifically configured to:

[0258] Determine the gradient of the loss value with respect to the parameters of the joint modeling module, and adjust the parameter update amount of the joint modeling module through an adaptive learning rate strategy based on the gradient;

[0259] Determining a gradient of the loss value with respect to a parameter of the visual diffusion decoder, and adjusting an update amount of the parameter of the visual diffusion decoder based on a preset gradient magnitude threshold constraint;

[0260] Determining a gradient of the loss value with respect to a parameter of the motion diffusion decoder, and adjusting an amount of parameter update of the motion diffusion decoder according to time series correlation regularization based on the gradient;

[0261] Parameters of the joint modeling module, the visual diffusion decoder, and the motion diffusion decoder are updated respectively based on the parameter update amount.

[0262] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide determination and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal via a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the server side of a data sequence generation method based on joint modeling.

[0263] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, memory, network interface, display screen and input device connected via a system bus. The processor of the computer device is used to provide determination and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the user side of a data sequence generation method based on joint modeling.

[0264] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0265] Obtain training samples including historical image sequences, historical action sequences, real future video frame sequences, and real future action sequences;

[0266] Applying a random mask to at least one of the historical image sequence or the historical action sequence to obtain a masked historical sequence;

[0267] encoding a historical image sequence in the masked history sequence into a latent visual token sequence, and encoding a historical action sequence in the masked history sequence into an action token sequence;

[0268] Processing the latent visual token sequence and the action token sequence through a joint modeling module to generate a unified latent representation;

[0269] Based on the unified latent representation, using a visual diffusion decoder to generate a predicted future video frame sequence, and using an action diffusion decoder to generate a predicted future action sequence;

[0270] applying a random mask to at least one of the predicted future video frame sequence or the predicted future action sequence to obtain a masked prediction sequence;

[0271] Determine a loss value based on the difference between the future video frame sequence in the masked prediction sequence and the true future video frame sequence, and the difference between the future action sequence in the masked prediction sequence and the true future action sequence;

[0272] Optimizing parameters of the joint modeling module, parameters of the visual diffusion decoder, and parameters of the motion diffusion decoder based on the loss value;

[0273] Repeat the steps of applying random masking to the history sequence, encoding, processing to generate a unified latent representation, generating a prediction sequence, applying random masking to the prediction sequence, determining the loss value and optimizing the parameters until the preset training termination condition is met, thereby obtaining the trained joint modeling module, visual diffusion decoder and action diffusion decoder;

[0274] The trained joint modeling module, visual diffusion decoder, and action diffusion decoder are used to process reasoning tasks and generate target future action sequences and / or target future video frame sequences.

[0275] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0276] Obtain training samples including historical image sequences, historical action sequences, real future video frame sequences, and real future action sequences;

[0277] Applying a random mask to at least one of the historical image sequence or the historical action sequence to obtain a masked historical sequence;

[0278] encoding a historical image sequence in the masked history sequence into a latent visual token sequence, and encoding a historical action sequence in the masked history sequence into an action token sequence;

[0279] Processing the latent visual token sequence and the action token sequence through a joint modeling module to generate a unified latent representation;

[0280] Based on the unified latent representation, using a visual diffusion decoder to generate a predicted future video frame sequence, and using an action diffusion decoder to generate a predicted future action sequence;

[0281] applying a random mask to at least one of the predicted future video frame sequence or the predicted future action sequence to obtain a masked prediction sequence;

[0282] Determine a loss value based on the difference between the future video frame sequence in the masked prediction sequence and the true future video frame sequence, and the difference between the future action sequence in the masked prediction sequence and the true future action sequence;

[0283] Optimizing parameters of the joint modeling module, parameters of the visual diffusion decoder, and parameters of the motion diffusion decoder based on the loss value;

[0284] Repeat the steps of applying random masking to the history sequence, encoding, processing to generate a unified latent representation, generating a prediction sequence, applying random masking to the prediction sequence, determining the loss value and optimizing the parameters until the preset training termination condition is met, thereby obtaining the trained joint modeling module, visual diffusion decoder and action diffusion decoder;

[0285] The trained joint modeling module, visual diffusion decoder, and action diffusion decoder are used to process reasoning tasks and generate target future action sequences and / or target future video frame sequences.

[0286] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can be found in the relevant descriptions of the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0287] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0288] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0289] It should be noted that if any software tools or components other than those of the Company appear in the embodiments of this application, they are merely for illustration and do not represent actual use. The above embodiments are intended only to illustrate the technical solutions of the present invention, not to limit them. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some of the technical features therein with equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A data sequence generation method based on joint modeling, characterized in that: The following steps are involved: Obtain training samples including historical image sequences, historical action sequences, real future video frame sequences, and real future action sequences; Applying a random mask to at least one of the historical image sequence or the historical action sequence to obtain a masked historical sequence; encoding a historical image sequence in the masked history sequence into a latent visual token sequence, and encoding a historical action sequence in the masked history sequence into an action token sequence; Processing the latent visual token sequence and the action token sequence through a joint modeling module to generate a unified latent representation; Based on the unified latent representation, using a visual diffusion decoder to generate a predicted future video frame sequence, and using an action diffusion decoder to generate a predicted future action sequence; applying a random mask to at least one of the predicted future video frame sequence or the predicted future action sequence to obtain a masked prediction sequence; Determine a loss value based on the difference between the future video frame sequence in the masked prediction sequence and the true future video frame sequence, and the difference between the future action sequence in the masked prediction sequence and the true future action sequence; Optimizing parameters of the joint modeling module, parameters of the visual diffusion decoder, and parameters of the motion diffusion decoder based on the loss value; Repeat the steps of applying random masking to the history sequence, encoding, processing to generate a unified latent representation, generating a prediction sequence, applying random masking to the prediction sequence, determining the loss value and optimizing the parameters until the preset training termination condition is met, thereby obtaining the trained joint modeling module, visual diffusion decoder and action diffusion decoder; The trained joint modeling module, visual diffusion decoder, and action diffusion decoder are used to process reasoning tasks and generate target future action sequences and / or target future video frame sequences.

2. The data sequence generation method based on joint modeling according to claim 1, characterized in that: Applying a random mask to at least one of the historical image sequence or the historical action sequence to obtain a masked historical sequence includes: Determining a type of sequence to be masked, where the type of sequence to be masked is at least one of a historical image sequence or a historical action sequence; generating a random mask position indication corresponding to the type of sequence to be masked; Creating a mask matrix corresponding to the random mask position indication; Apply the mask matrix to the sequence type to be masked to obtain a masked history sequence.

3. The data sequence generation method based on joint modeling according to claim 1, characterized in that: Encoding a historical image sequence in the masked history sequence into a potential visual token sequence, and encoding a historical action sequence in the masked history sequence into an action token sequence, including: extracting visual features of the historical image sequence in the masked historical sequence to obtain an image feature sequence; Performing dimensionality reduction processing on the image feature sequence to generate a visual feature vector; Inputting the visual feature vector into a pre-trained variational autoencoder, and outputting a potential visual token sequence through the pre-trained variational autoencoder; Performing temporal resolution improvement processing on the historical action sequence in the masked historical sequence to obtain an action sequence with improved resolution; Extracting motion features of the motion sequence after the resolution is improved to obtain a motion feature vector; The action feature vector is mapped into an action token sequence.

4. The data sequence generation method based on joint modeling according to claim 1, characterized in that: The latent visual token sequence and the action token sequence are processed by a joint modeling module to generate a unified latent representation, including: splicing the potential visual token sequence and the action token sequence to form a joint input sequence; Performing a position encoding operation on the combined input sequence to obtain a position encoding sequence; Input the positional encoding sequence into the multi-head attention layer of the Transformer module to model the interaction relationship; The unified latent representation is generated by the output layer of the Transformer module.

5. The data sequence generation method based on joint modeling according to claim 1, characterized in that: Based on the unified latent representation, a visual diffusion decoder is used to generate a predicted future video frame sequence, and an action diffusion decoder is used to generate a predicted future action sequence, including: inputting the unified latent representation into a visual diffusion decoder, performing a multi-step noise addition operation in the visual diffusion decoder to generate a noise addition sequence; performing an iterative denoising process on the noise-added sequence to generate a predicted future video frame sequence; Inputting the unified latent representation into an action diffusion decoder, performing a time-stepped noise injection operation in the action diffusion decoder to generate a noise injection sequence; An iterative denoising process is performed on the noise injection sequence to generate a predicted future action sequence.

6. The data sequence generation method based on joint modeling according to claim 1, characterized in that: The loss value is determined based on the difference between the future video frame sequence in the masked prediction sequence and the real future video frame sequence, and the difference between the future action sequence in the masked prediction sequence and the real future action sequence, including: determining a first difference value between the future video frame sequence and the real future video frame sequence in the masked prediction sequence; determining a video loss component based on the first difference value; determining a second difference value between the future action sequence in the masked predicted sequence and the true future action sequence; determining an action loss component based on the second difference value; The video loss component and the motion loss component are combined to obtain a loss value.

7. The data sequence generation method based on joint modeling according to claim 1, characterized in that: Optimizing parameters of the joint modeling module, parameters of the visual diffusion decoder, and parameters of the motion diffusion decoder based on the loss value, including: Determine the gradient of the loss value with respect to the parameters of the joint modeling module, and adjust the parameter update amount of the joint modeling module through an adaptive learning rate strategy based on the gradient; Determining a gradient of the loss value with respect to a parameter of the visual diffusion decoder, and adjusting an update amount of the parameter of the visual diffusion decoder based on a preset gradient magnitude threshold constraint; Determining a gradient of the loss value with respect to a parameter of the motion diffusion decoder, and adjusting an amount of parameter update of the motion diffusion decoder according to time series correlation regularization based on the gradient; Parameters of the joint modeling module, the visual diffusion decoder, and the motion diffusion decoder are updated respectively based on the parameter update amount.

8. A data sequence generation device based on joint modeling, characterized in that: The data sequence generation device based on joint modeling includes: A training data preparation module is used to obtain training samples including historical image sequences, historical action sequences, real future video frame sequences, and real future action sequences; a historical sequence masking module, configured to apply a random mask to at least one of the historical image sequence or the historical action sequence to obtain a masked historical sequence; a history sequence encoding module, configured to encode a history image sequence in the masked history sequence into a potential visual token sequence, and to encode a history action sequence in the masked history sequence into an action token sequence; a joint representation generation module, configured to process the latent visual token sequence and the action token sequence through a joint modeling module to generate a unified latent representation; a prediction sequence generation module, configured to generate a predicted future video frame sequence using a visual diffusion decoder and a predicted future action sequence using an action diffusion decoder based on the unified latent representation; A prediction sequence masking module, configured to apply a random mask to at least one of the predicted future video frame sequence or the predicted future action sequence to obtain a masked prediction sequence; a loss calculation module, configured to determine a loss value based on a difference between a future video frame sequence in the masked predicted sequence and a true future video frame sequence, and a difference between a future action sequence in the masked predicted sequence and a true future action sequence; a parameter optimization module, configured to optimize the parameters of the joint modeling module, the parameters of the visual diffusion decoder, and the parameters of the motion diffusion decoder based on the loss value; A training control module is configured to repeatedly execute the steps of applying random masking to the historical sequence, encoding, processing to generate a unified latent representation, generating a prediction sequence, applying random masking to the prediction sequence, determining a loss value, and optimizing parameters until a preset training termination condition is met, thereby obtaining a trained joint modeling module, a visual diffusion decoder, and an action diffusion decoder; The inference generation module is used to process the inference task using the trained joint modeling module, visual diffusion decoder and action diffusion decoder to generate target future action sequences and / or target future video frame sequences.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a data sequence generation program based on joint modeling that is stored in the memory and can be run on the processor. When the data sequence generation program based on joint modeling is executed by the processor, the steps of the data sequence generation method based on joint modeling as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The storage medium stores a data sequence generation program based on joint modeling, which, when executed by a processor, implements the steps of the data sequence generation method based on joint modeling as described in any one of claims 1 to 7.

Images