Vision sequence based planning method, apparatus, device and medium

Abstract

The present application relates to the technical field of artificial intelligence, which can be applied to business scenarios such as financial technology and medical health, and discloses a planning method and device based on visual sequence, equipment and medium, comprising: obtaining a training visual state sequence formed in an exploration trajectory, extracting a visual state pair composed of a training prefix visual state sequence and a training subsequent visual state, and performing initial training on a generative visual model; generating a plurality of candidate subsequent visual states based on the prefix visual state sequence of the target environment, obtaining discrete actions by analysis and determining the corresponding environmental feedback reward value, and further calculating the relative advantage; updating the generative visual model parameters based on the relative advantage, and generating a visual planning sequence. The present application avoids information loss caused by image-to-text conversion by planning and feedback updating in the visual state space, enhances the modeling capability of the model for spatial structure and dynamic process, realizes pure visual-driven dynamic planning, and improves the strategy generation effect and environmental adaptability.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a planning method, apparatus, device, and storage medium based on visual sequences. Background Technology

[0002] Existing planning methods based on Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) generally employ language inference mechanisms as the core decision-making basis. Even if the model input contains image information, the system typically needs to convert the image content into text form, such as visual descriptions, scene labels, or state summaries, before inputting it into the language model for planning inference. This "visual-to-text and then-inference" process has significant limitations at several technical levels: First, this modality conversion process inevitably weakens or loses the continuous spatial structure, physical state changes, and geometric features in the original visual information, especially in scenarios requiring high-precision understanding of image state evolution, such as dynamic object interactions or temporal evolution sequence modeling, where text representations are far from fully capturing the detailed expression of visual dynamics.

[0003] In the fintech sector, visual state planning capabilities are invaluable in tasks such as fraud prevention path identification, process decision review, and document image analysis. However, current methods largely rely on textual information extraction and static image descriptions for analysis and decision-making, lacking the ability to model the continuous behavioral paths or image state transition patterns implicit in image sequences. Furthermore, because existing models are primarily trained on pre-existing labeled data, they exhibit significant generalization limitations in unknown risk paths or cross-scenario financial tasks, making it difficult to adapt to dynamic changes in complex business environments.

[0004] In the healthcare field, image sequences are widely used for tasks such as surgical navigation, lesion evolution tracking, and multimodal clinical auxiliary diagnosis. In these scenarios, traditional language model-based solutions typically require textual descriptions of medical images or surgical videos before reasoning and decision-making, rather than directly performing dynamic planning based on image sequences. This textual intermediary not only reduces the accuracy and efficiency of treatment path reasoning but also fails to accurately capture the causal relationship between dynamic state changes in images and clinical action effects. Furthermore, standard supervised training methods rely excessively on static data annotation, lacking proactive exploration of the relationships between different operational paths and outcomes during diagnosis and treatment, making them ill-suited for decision-making needs in unknown cases or complex treatment processes.

[0005] In general, existing methods suffer from insufficient ability to model visual dynamics, loss of state information due to modal transitions, and weak generalization ability of planning strategies, especially limiting their performance in complex visual tasks requiring end-to-end autonomous planning. These problems are particularly prominent in applications in industries such as fintech and healthcare, necessitating the development of a pure visual planning path reasoning mechanism that does not require textual mediation while maintaining the integrity of image structural information. Summary of the Invention

[0006] The main objective of this invention is to provide a planning method, apparatus, device, and storage medium based on visual sequences, aiming to solve the technical problems of existing technologies being unable to achieve end-to-end, purely vision-driven dynamic planning processes, resulting in the loss of visual information and insufficient feedback on action effects in state transition modeling and task sequence generation.

[0007] To achieve the above objectives, the present invention provides a planning method based on visual sequences, comprising:

[0008] Obtain training visual state sequences corresponding to multiple exploration trajectories, and extract prefix and subsequent visual state pairs containing training prefix visual state sequences and training subsequent visual states based on the training visual state sequences.

[0009] The supervised generative visual model is trained by generating subsequent visual states that conform to the dynamics of the environment through the prefix and subsequent visual states, thus completing the initial training of the generative visual model.

[0010] Based on the prefix visual state sequence of the target environment, multiple candidate subsequent visual states are generated using the generative visual model that has completed initial training.

[0011] The visual state sequence is generated by combining the prefix visual state sequence of the target environment with each candidate subsequent visual state. The visual state sequence is parsed to obtain the corresponding discrete action, and the environmental feedback reward value of each discrete action is determined by combining the progress map.

[0012] Based on the environmental feedback reward value for each discrete action, the relative advantage of each candidate subsequent visual state is determined;

[0013] The parameters of the generative visual model are updated based on the relative advantages to obtain an optimized generative visual model.

[0014] The optimized generative visual model generates a visual planning sequence for the target task.

[0015] Furthermore, to achieve the above objectives, the present invention provides a planning device based on visual sequences, comprising:

[0016] The data acquisition and construction module is used to acquire training visual state sequences corresponding to multiple exploration trajectories, and extract prefix and subsequent visual state pairs containing training prefix visual state sequences and training subsequent visual states based on the training visual state sequences.

[0017] The initial training module is used to generate subsequent visual states that conform to the dynamic environment of the supervised generative visual model by using the prefix and subsequent visual states, thereby completing the initial training of the generative visual model.

[0018] The candidate generation module is used to generate multiple candidate subsequent visual states based on the prefix visual state sequence of the target environment using the generative visual model that has completed initial training.

[0019] The action evaluation module is used to combine the prefix visual state sequence of the target environment with each candidate subsequent visual state to generate a visual state sequence, parse the visual state sequence to obtain the corresponding discrete action, and combine the progress map to determine the environmental feedback reward value of each discrete action.

[0020] An advantage estimation module is used to determine the relative advantage of each candidate subsequent visual state based on the environmental feedback reward value for each discrete action.

[0021] The strategy optimization module is used to update the parameters of the generative visual model according to the relative advantage to obtain an optimized generative visual model.

[0022] The planning generation module is used to generate a visual planning sequence for the target task using the optimized generative visual model.

[0023] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and a visual sequence-based planning program stored in the memory and executable on the processor, wherein when the visual sequence-based planning program is executed by the processor, it implements the steps of the visual sequence-based planning method as described above.

[0024] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a planning program based on visual sequences, wherein the planning program based on visual sequences, when executed by a processor, implements the steps of the planning method based on visual sequences as described above.

[0025] Beneficial Effects: This invention relates to the field of artificial intelligence technology and can be applied to business scenarios such as fintech and healthcare. It discloses a planning method, apparatus, device, and medium based on visual sequences, comprising: acquiring training visual state sequences corresponding to multiple exploration trajectories; extracting prefix and subsequent visual state pairs formed by training prefix visual state sequences and training subsequent visual states; using these prefix and subsequent visual state pairs to initially train a generative visual model; generating multiple candidate subsequent visual states based on the prefix visual state sequence of the target environment using the initially trained generative visual model; parsing to obtain the corresponding discrete actions and determining the environmental feedback reward value in conjunction with a progress graph, thereby determining the relative advantage of each candidate subsequent visual state; updating the parameters of the generative visual model according to the relative advantage, and finally generating a visual planning sequence for the target task. This invention maintains the continuity of visual states and the closed loop of action feedback during the training and inference process of the generative visual model, enabling the model to perform dynamic planning directly in the visual state space. This avoids the information loss caused by image-to-text conversion in existing technologies, improves the modeling ability for spatial structure and physical evolution, and thus achieves pure vision-driven planning without the need for a language medium, improving the model's generalization ability and strategy effectiveness in complex environments. Attached Figure Description

[0026] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0027] Figure 1 This is a schematic diagram of an application environment for a planning method based on visual sequences according to an embodiment of the present invention;

[0028] Figure 2 This is a flowchart illustrating an embodiment of the planning method based on visual sequences of the present invention.

[0029] Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the planning device based on visual sequences of the present invention;

[0030] Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0031] Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0032] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0033] The visual sequence-based planning method provided in this invention can be applied to, for example... Figure 1In this application environment, the user terminal communicates with the server via a network. The server can obtain training visual state sequences corresponding to multiple exploration trajectories from the user terminal, extract prefix and subsequent visual state pairs formed by the training prefix visual state sequence and the training subsequent visual state, and use these prefix and subsequent visual state pairs to perform initial training on the generative visual model. Based on the prefix visual state sequence of the target environment, the generative visual model that has completed the initial training generates multiple candidate subsequent visual states, analyzes and obtains the corresponding discrete actions, and determines the environmental feedback reward value by combining the progress map, thereby determining the relative advantage of each candidate subsequent visual state. The parameters of the generative visual model are updated according to the relative advantage, and finally, a visual planning sequence for the target task is generated. This invention maintains the continuity of visual states and the closed loop of action feedback during the training and inference process of the generative visual model, enabling the model to perform dynamic planning directly in the visual state space. This avoids the information loss caused by image-to-text conversion in existing technologies, improves the modeling ability of spatial structure and physical evolution, and thus realizes pure vision-driven planning without language medium, improving the model's generalization ability and strategy effectiveness in complex environments. The user terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will now be described in detail through specific embodiments.

[0034] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the visual sequence-based planning method provided by the present invention. It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0035] like Figure 2 As shown, the planning method based on visual sequences proposed in this invention includes the following steps:

[0036] S10, obtain training visual state sequences corresponding to multiple exploration trajectories, and extract prefix and subsequent visual state pairs containing training prefix visual state sequences and training subsequent visual states based on the training visual state sequences.

[0037] In this embodiment, acquiring the training visual state sequence corresponding to multiple exploration trajectories refers to continuously recording a series of image frames or frame sequence information that change over time in the visual environment during multiple interaction processes. The source of the visual states can include image rendering results in a simulation environment, or raw images or preprocessed image data acquired by image sensors (such as RGB cameras or depth cameras) in real-world scenes. An exploration trajectory refers to a state-action-state sequence generated by an actor (e.g., a mobile robot, software agent, or automated component) continuously moving or operating in an environment according to a predetermined strategy. Here, the state is defined as a visual state represented by an image. Each exploration trajectory should include multiple visual states that are consecutive in time and satisfy a representative and comprehensive distribution in space or task objectives. The acquisition operation should support sequential reading based on timestamps and may optionally include additional task tags, path tags, or environmental metadata for subsequent filtering.

[0038] Based on the acquired training visual state sequence, it is necessary to extract combined units consisting of a prefix visual state sequence and subsequent visual states. The prefix visual state sequence refers to a subsequence selected chronologically from the visual state sequence. Its length can be a fixed value (e.g., 5 frames) or dynamically adjusted depending on the task scenario. Each prefix sequence should contain a continuous process of state evolution, thus reflecting the contextual semantics and state evolution trends. Subsequent visual states refer to the image frames immediately following the prefix visual state sequence, used to characterize the potential state extensions that may occur naturally in the current environment. The construction of prefix and subsequent visual state pairs is based on a sliding time window mechanism. For example, in a visual sequence containing 10 frames, a prefix window of length 5 can be used to slide and generate multiple prefix-corresponding subsequent state pairing instances.

[0039] In practice, trajectory generation can employ various strategies. An ε-greedy exploration strategy can be used to cover the uncertainty of the state space, a policy network can be used for probabilistic sampling, or a behavioral policy from reinforcement learning can be used for sampling. Visual state extraction can utilize a time-stepping sampling mechanism, selecting image frames at fixed frame intervals, or dynamically determining sampling points based on the rate of environmental change. When constructing the prefix visual state sequence, image data can be vectorized or kept in its original resolution for model training. The selection logic for subsequent visual states must ensure no frame overlap with the prefix sequence to maintain temporal causality. Furthermore, to enhance the diversity of combined data, multiple prefix length strategies can be employed for the same visual state sequence to generate diverse prefix-subsequent state pairs, thereby enhancing the predictive ability of the generative model under different trajectory length conditions.

[0040] In different scenarios, exploration trajectories can be collected and organized in various ways. For example, in a closed simulation platform, a large number of random actions can be executed by an automated control module to cover the spatial state, while recording the image output of each step to form an exploration trajectory. In dynamic real-world scenarios, such as intelligent monitoring systems or intelligent vehicle environments, image sequences are generated by continuously collecting visual sensor signals and combining them with timestamps, serving as training trajectories. The generation method of the prefix visual state sequence can adjust the length parameter according to the memory dependency of different tasks. In tasks with drastic fluctuations in spatial state, the window can be appropriately shortened to reduce input complexity, while in highly dependent tasks, the prefix can be lengthened to improve state prediction capabilities. Subsequent visual states can be the image immediately following the next frame, or multiple steps can be set to adapt to different planning step lengths.

[0041] To adapt to different training model structures, prefix and subsequent visual state pairs can be processed using different encoding methods. In convolutional neural network-based structures, the original image can be directly used as model input; in visual transformer-based structures, the image can be patched and mapped into vector embeddings; in embedding models that require state vectors as input, a precoding module can be introduced to convert the image into a unified-dimensional state feature representation. The combination can take the form of two-dimensional tensor pairs, three-dimensional tensor sequences, or be used as input-label pairs to supervise training modules, further enhancing the model's understanding and generalization ability regarding state sequences.

[0042] Example: In the healthcare field, this can be applied to predicting the operational path of surgical robots. By collecting image sequences from multiple actual operational processes, a prefix is ​​paired with subsequent images to train the model to learn state changes during continuous operations. For example, consecutive image frames from endoscopic surgery can be used to train a generative model to predict the next visual field movement, assisting the navigation system in planning the path in advance.

[0043] In the fintech sector, this technology can be applied to state prediction in counter visual recognition processes. For example, when a customer completes a transaction at a self-service terminal, a prefix sequence is constructed using preceding camera images to predict the state of subsequent images. This helps determine if there are any abnormal deviations or incomplete steps in the customer's operation sequence, enabling process behavior prediction and early intervention. In all these scenarios, modeling is entirely based on visual sequences without relying on text or action labels, thus enhancing the system's automated perception capabilities.

[0044] This embodiment achieves intuitive capture of state transition distribution by constructing prefix and subsequent visual state pairs from the exploration trajectory, improving the model's ability to model continuous state evolution under pure visual information, solving the problem of missing state representation under language mediation, and thus enhancing the model's accuracy in perceiving future states and its ability to express environmental dynamics.

[0045] S20, using the prefix and subsequent visual states, the supervised generative visual model generates subsequent visual states that conform to the dynamics of the environment, thus completing the initial training of the generative visual model.

[0046] In this embodiment, the process of generating a generative visual model through supervision using prefix and subsequent visual states relies on establishing a correspondence between input and output. In this structure, the prefix visual state sequence serves as the model input, and the subsequent visual states serve as the target output, guiding the generative visual model to learn the dynamic evolution of visual states. The prefix visual state sequence is a continuous visual segment over a period of time, representing the evolution trend of the environment over a certain historical length. The subsequent visual states represent the next possible state change under this trend. The pairing of prefix and subsequent visual states should satisfy temporal causality, that is, the subsequent state must not be earlier than the timestamp of any state in the prefix, thereby forming a predictive relationship that can be used for supervision.

[0047] Generative visual models are a class of model structures capable of generating complete or future states from partial states. Specific model structures may include encoder-decoder structures, recurrent neural network structures, visual transformer structures, or variations thereof. Temporal modeling modules or attention mechanisms can also be introduced to enhance the ability to model dependencies between states. The model input is typically a sequence of image frames of fixed or dynamic length. After feature extraction and representation compression, the sequence is fed into an intermediate temporal modeling module, which outputs the corresponding predicted representation of subsequent states. The model training phase employs supervised learning, using existing prefixes and subsequent visual state pairs to compare the deviation between the model's predictions and the actual state results, and adjusting parameters through gradient backpropagation.

[0048] Determining the measure of difference between the predicted and actual subsequent visual states is crucial for supervised training. This measure can be achieved using pixel-level L2 loss, structural similarity index, or feature space distance (e.g., comparing high-order semantic features extracted from the pre-trained model). For image variations at different spatiotemporal scales, multi-scale loss structures can be introduced to compare predictions with actual results at different resolutions, enhancing the model's generalization robustness. During model training, parameter optimization is performed based on the backpropagation algorithm for each update. Training can be batch-based, with each batch containing several prefix and subsequent visual state pairs; alternatively, incremental training based on single samples can improve convergence efficiency.

[0049] Whether training is complete requires determining whether the current model performance has reached a preset threshold. This can be done by judging the convergence of the loss on the validation set, or by introducing surrogate metrics (such as predicted frame clarity or task success rate) into the planning task. Initial training is considered complete once the model parameter variation falls below a set range for several consecutive rounds, or the validation metric reaches the platform's lower limit. The training termination condition design here should support an adaptive convergence mechanism to avoid overfitting or premature stopping due to static round settings.

[0050] In sequence modeling tasks, prefix and subsequent visual state pairs can be encoded as two-dimensional tensors and input into the model. For convolutional neural network-based architectures, the input images need to be normalized to a uniform size. Multiple prefix states can be stacked to form a single input tensor, or the temporal structure can be maintained by inputting them side-by-side through temporal channels. The prediction of subsequent states is then handled by the decoding module, which outputs the complete image tensor. During model training, gradient optimizers such as the Adam optimizer and RMSprop can be used to control the learning rate and improve training stability.

[0051] Model parameters can be initialized using random distribution, pre-trained model transfer parameters, or optimized parameters based on prior tasks. During training, data augmentation strategies, such as image blurring, rotation, and brightness perturbation, can be introduced to expand the diversity of prefix state combinations and improve the model's generalization ability under real-world interference conditions. To avoid blurry or degraded predicted images, a perceptual loss term, such as VGG perceptual loss, can be added to the loss function, or a discriminator in an adversarial training mechanism can be used to guide the generation module to output results that more closely resemble the distribution of real images.

[0052] The initial training completion judgment mechanism can judge the convergence trend based on the rate of model loss decrease within the training rounds. Alternatively, a certain number of prefix state combinations can be set, and the subsequent states can be predicted by the model. The continuity, rationality and consistency can be verified under manual annotation, thereby helping to determine whether the model meets the training convergence requirements under the current task.

[0053] Example: In the healthcare field, this can be used for model pre-training in smart ward scenarios for patient behavior monitoring. For instance, by using a bedside camera to collect image sequences of patients over a continuous period, and constructing prefix and subsequent state image pairs, the model can be trained to predict possible changes in patient position, trends in getting up, etc., which helps improve the ability to plan subsequent nursing actions or predict emergency responses.

[0054] In the fintech field, this technology can be applied to predictive learning of user behavior at bank self-service terminals. By collecting visual records of user operations, such as hand gestures and interface changes at ATMs, prefix and subsequent state image pairs are constructed for training. This allows the model to learn to predict the user's next possible interface action, which can then be used to determine whether the operation is compliant or requires a warning, thereby enhancing the device's intelligent response capabilities.

[0055] This embodiment can directly guide the generation module to grasp the mapping pattern of visual state evolution over time, completing the modeling of visual dynamics without relying on text or structural labels. Compared with indirect supervision through action labels or behavioral signals, it significantly reduces supervision costs and improves the model's adaptability to visual dynamic transfer, thus possessing better predictive coherence and state restoration capabilities in downstream planning tasks.

[0056] S30, based on the prefix visual state sequence of the target environment, generates multiple candidate subsequent visual states using the generative visual model that has completed initial training;

[0057] In this embodiment, the prefix visual state sequence of the target environment refers to an image sequence composed of multiple consecutive visual frames arranged in chronological order before the current decision moment, reflecting the visual change trajectory of the current task environment in a short period of history. This sequence typically originates from processes such as sensor acquisition, image stream reading, or cached state recovery. Its structure should maintain the chronological order and consistent dimensions to ensure the continuity and interpretability of the state sequence.

[0058] Generative vision models that have completed initial training refer to models that have undergone supervised training using a large number of prefix and subsequent visual state pairs in the early training phase. These models possess the ability to predict future states from observed states, with the prediction goal of approximating the image or feature representation of the true state as closely as possible. The internal structure of such models can employ neural network architectures that incorporate temporal modeling mechanisms, such as Transformers, temporal convolutional networks, and attention aggregation networks, to enhance the ability to characterize the temporal dependencies between historical states.

[0059] When using a generative visual model that has completed initial training for predictive inference based on a prefix visual state sequence of the target environment, multiple generation processes should be performed. Each time, the input is an identical prefix visual state sequence, but different model sampling paths, decoding initializations, noise perturbations, or sampling strategies are introduced to increase the diversity of generated subsequent visual states. This diverse generation mechanism aims to simulate the possible evolution paths of the environment under uncertain conditions and provide a rich candidate state space for subsequent action effect evaluation.

[0060] Multiple candidate subsequent visual states represent multiple possible future state images derived from the same input, and can be stored in the form of high-resolution images, feature maps, compressed embeddings, etc. The generation method can construct diverse candidate sets based on probability sampling, multi-path beam search, temperature-controlled sampling, or top-k / top-p truncation strategies, while retaining supplementary information such as prediction confidence scores for reference by subsequent evaluation mechanisms.

[0061] In practical applications, the prefix visual state sequence can be preprocessed by continuously acquiring image frames using a camera device, such as uniform image size, channel number normalization, and inter-frame interpolation alignment. This preprocessed sequence is then input into a trained generative vision model for multiple rounds of inference. Each round of inference can use different random initial seeds or introduce slight noise perturbations to cause the model output to fall into different prediction spaces, thus forming diverse candidate sets.

[0062] Each candidate subsequent visual state generated retains the model's decoding trajectory and confidence information, and can be accompanied by an inference path index for subsequent analysis. This candidate set can be sorted or filtered according to the model's self-assessment score, or all can be retained for subsequent evaluation. The number of generated states can be adaptively set according to the complexity of the target task; for example, 5-10 candidate states can be generated for navigation tasks, while more than 20 can be generated for mechanical prediction tasks to cover more state branches.

[0063] During the generation process, visual feature similarity constraints or spatial structure consistency discrimination mechanisms can be introduced to filter candidate states that are structurally broken or abnormally drifting, ensuring that the states in the candidate set are reasonable and continuous, which facilitates subsequent execution environment feedback calculation and action reasoning.

[0064] Example: In the healthcare business, inpatient rehabilitation assessment systems can use this mechanism to generate various candidate visual states that patients may exhibit, such as changes in body position, sitting up, and tendency to get out of bed. This helps the nursing system plan intervention behaviors in advance, such as raising and lowering bed rails and bedside reminders, thereby improving risk response efficiency.

[0065] In the fintech business, intelligent counter interaction systems can generate multiple candidate interface response behavior images based on the visual state sequence of the current customer operation, such as account switching, transfer preview, menu opening, etc., to identify whether there are non-standard operation paths or potential abnormal interaction behaviors, providing a basis for intelligent risk control or process verification.

[0066] This embodiment generates multiple possible subsequent visual states in the target environment. The model not only covers various potential state transition paths but also provides redundancy and diversity in the state space for subsequent action selection, improving the robustness of the overall prediction process. This approach avoids the overfitting problem caused by generating only a single optimal path in traditional methods. Furthermore, without introducing external action labels or text structures, it achieves multi-branch modeling of future visual dynamics, enhancing the model's responsiveness to uncertainties in real-world scenarios.

[0067] S40, combine the prefix visual state sequence of the target environment with each candidate subsequent visual state to generate a visual state sequence, parse the visual state sequence to obtain the corresponding discrete action, and combine the progress map to determine the environmental feedback reward value of each discrete action.

[0068] In this embodiment, the prefix visual state sequence of the target environment is a continuous image sequence that constitutes the basis for judging the current state. It usually comes from image data collected by the environmental perception system and has a clear temporal order. The candidate subsequent visual states are predicted images of future states generated by the generative visual model based on the prefix sequence. They have an uncertainty distribution and are used to simulate different potential state evolution paths.

[0069] Combining the prefix visual state sequence with candidate subsequent visual states aims to construct new state transition pairs that reflect the transition from a continuous state history to a future predicted state, possessing the ability to simulate the consequences of actions. This combination is not a simple image stitching, but rather establishes a contrastive relationship with input-output features for use in the action parsing process.

[0070] Parsing to obtain the corresponding discrete action refers to extracting the action label that can trigger the state transition from the state transition pair using a state-action mapping function or policy network. This action is a discrete action, representing a discrete set of executable operations, such as "forward," "turn left," or "grab," suitable for use in policy evaluation and optimization. This parsing process can be implemented based on convolutional coding structures, feature distance comparison, target pose estimation, etc., to map the association between state change trajectories and standard action labels.

[0071] A progress map is a structured infographic used to depict the stages of task execution and the expected evolution path of their states. It includes expected target scores or reward information for the effects of actions in different states. It does not rely on explicit action labels, but rather defines environmental feedback reward values ​​based on the rationality of the evolution between visual states and the distance to goal achievement.

[0072] The environmental feedback reward value reflects the positive or negative impact of each action (i.e., candidate subsequent visual states) on task progress or goal achievement. The calculation process combines state transition pairs with progress map information, visually determining whether the transition direction is close to or far from the expected goal, thus providing quantitative feedback. This mechanism avoids relying on the cost of real-world environmental interactions, directly evaluating the rationality of actions in visual space.

[0073] In the implementation process, each set of prefix visual state sequences and candidate subsequent visual states can first be image feature encoded to construct state transition pairs. For each state pair, the spatial feature differences between its initial state and predicted state are extracted, such as target position offset, object contour changes, and inter-frame structure flow changes, and then input into the action parsing model.

[0074] This model can take the state difference tensor after image registration as input, output the probability distribution in the standard action set, and select the action corresponding to the maximum response value as the discrete action identifier of the current state pair. Alternatively, it can use an analytical approach based on embedding spatial distance mapping, such as a K-nearest neighbor contrastive classifier or a template matching-based attention parsing network.

[0075] After obtaining the discrete actions for each state pair, the pre-defined progress graph for the current task is loaded. This progress graph is generated during the task initialization phase using expert examples or a rule definition system, and includes information such as state phase numbers, action preference paths, and phase weights. The parsed actions are mapped to the structural paths in the progress graph, and reward values ​​are obtained based on the progress effect of the corresponding task phase. For example, if an action causes the state to approach the target state, the reward value is positive; if it deviates from the target trajectory, the reward value is negative.

[0076] The reward value can be directly used in the subsequent policy gradient optimization process, or it can be used as a scoring basis to select the optimal candidate subsequent visual state.

[0077] Example description: In healthcare scenarios, rehabilitation training assistance systems can use the patient's current continuous motion video as a prefix visual state sequence, combined with several possible subsequent posture images predicted by the generative model, to parse discrete actions such as "getting up", "raising arms", and "turning over", and calculate the rehabilitation stage progress score through the training task map (progress map) to dynamically evaluate the rationality of the action and the timing of intervention.

[0078] In fintech business scenarios, based on the visual state sequence of changes in the user's current interactive interface and the predicted image of the next operation interface, the system can identify whether the user has performed operations such as "jump", "cancel", or "confirm". With the help of the business process diagram, it can determine whether the action has promoted the transaction process and then assign positive or negative environmental feedback scores to identify potential abnormal behavior or operational deviations.

[0079] This embodiment achieves effective evaluation of candidate subsequent visual states through this processing mechanism, avoiding reliance on real-world interaction data from the external environment. Furthermore, by injecting domain task knowledge through a progress graph, the reward calculation process gains structure and directionality. By mapping state transitions to discrete actions and performing visual spatial evaluation, not only is the efficiency of policy evaluation improved, but the generalization ability of action rationality judgment is also enhanced, maintaining visual consistency in the planning process while ensuring data closure.

[0080] S50, based on the environmental feedback reward value of each of the discrete actions, determine the relative advantage of each of the candidate subsequent visual states;

[0081] In this embodiment, the environmental feedback reward value for a discrete action is a numerical index obtained through prior calculation, used to evaluate the positive or negative contribution of the action in the current task phase. This value can be estimated based on the target proximity between visual states, or dynamically generated by fusing task structure diagrams, expert scores, or experience-based evolutionary reward functions. Reward values ​​may exhibit differences in scale, distribution, and interval; therefore, standardization and relativity analysis are required before they can be used in the model optimization process.

[0082] Relative advantage is an indicator that reflects the relative strength of a candidate's subsequent visual state within the current candidate set. It quantifies whether the current predicted action is superior to the average among similar candidates, and the significance of its superiority. The calculation of relative advantage relies not only on the individual score of each action but also on the overall statistical processing of the reward values ​​of all candidate samples, including calculating the mean and standard deviation, to measure the degree of deviation of individual reward values ​​relative to the statistical center.

[0083] The difference between the reward value of a discrete action and the average reward in the current candidate set is used to capture the degree of unexpected deviation of each candidate prediction action. The standard deviation serves as a normalization scaling factor for this deviation, further unifying the influence of reward value distribution under different task states. Finally, a normalized advantage value, also known as a z-score or standardized score, is generated by dividing the difference by the standard deviation, ensuring comparability and stability under different states and stages.

[0084] This process essentially constructs a relative scoring model, which assigns a relative quantitative index reflecting the quality of each predicted subsequent visual state. This index will directly serve as an input variable for optimizing the generative model's strategy objective; therefore, its calculation must maintain differentiability, numerical stability, and generalization ability.

[0085] In practical implementation, the environmental feedback reward values ​​of all discrete actions corresponding to the current candidate prediction results are first aggregated to form a numerical vector set. The mean of this set is calculated to obtain the expected reward value of the current candidate set; then the standard deviation is calculated for the same set to reflect the dispersion of the candidate reward distribution.

[0086] For each candidate subsequent visual state, its corresponding reward value is extracted and subtracted from the average reward value of the set to obtain its offset. Then, this offset is divided by the standard deviation to obtain the standardized advantage value of the candidate subsequent visual state, which serves as its relative advantage index. This calculation can be efficiently performed on parallel computing devices (such as GPUs) through vectorization.

[0087] In some implementations, if a small standard deviation leads to normalization amplification of errors, a numerical stabilization term can be introduced, such as adding an ε correction term to the standard deviation, to avoid division by zero or over-amplification of the effects of outliers. In scenarios with severely skewed distributions, nonlinear functions such as quantile ranking transformation and sigmoid mapping can be used to replace the z-score method to generate relative advantage values, thereby enhancing stability.

[0088] This relative advantage value will serve as input for policy optimization, influencing the intensity of model parameter updates or the sampling probability. It can also be used as a ranking criterion during the action selection phase to filter or reweight the importance of samples.

[0089] Example Explanation: In a healthcare setting, a rehabilitation robot system predicts a patient's next movement posture image and generates candidate images. After assigning a reward value to each candidate based on the training task map, the system performs a standardized analysis of the rehabilitation effectiveness of different movements. By calculating the relative advantage of each predicted movement, the system can identify which predictions effectively promote the patient to achieve the specified recovery goals, thus guiding the robot to select a better feedback path.

[0090] In fintech scenarios, investment advisory assistance systems predict multiple possible subsequent interface states based on user behavior sequences and calculate the profitability score of each operation using a conversion rate estimation model in the business process diagram. The system then uses relative advantage calculations to filter out the predicted actions with the highest monetization potential in the current market environment, guiding the recommendation logic to fine-tune its strategies and improve business conversion efficiency.

[0091] This embodiment addresses the lack of comparability between subsequent visual states of different candidates by leveraging relative advantages. It transforms absolute rewards into relative strengths using a unified statistical standard, thereby increasing the model's sensitivity to differences in sample quality during the update process. This mechanism further enhances the generative model's preference for high-quality candidates, helping to improve the convergence speed of the target state and the overall path rationality, and preventing the strategy from getting trapped in local optima with excessively high reward values.

[0092] S60, Update the parameters of the generative visual model according to the relative advantage to obtain the optimized generative visual model;

[0093] In this embodiment, relative advantage is a standardized metric calculated prior to the given information, used to quantify the performance of each candidate prediction result relative to the overall candidate set. Essentially, it reflects the degree to which the behavioral value of the action prediction deviates from the mean in the task environment, and can be used to dynamically adjust the model's learning intensity on the samples. A relative advantage greater than zero indicates that the prediction result is better than the average, while a relative advantage less than zero indicates that it is worse than the average.

[0094] Parameter updates in generative visual models require the construction of a set of optimization objectives that are compatible with relatively advantageous guiding features. To this end, the parameter update process references the probability distribution ratio between the current policy output and the original policy output. This probability ratio represents the degree of consistency between the old and new policies in generating candidate results, and is used to control the update magnitude, preventing the policy from deviating too quickly from its initial learning trajectory.

[0095] The optimization objective consists of three parts: the policy probability ratio, the relative advantage value, and a reference policy constraint term. The product of the policy probability ratio and the relative advantage term forms the weighted update direction, used to strengthen the response to high-advantage actions; the reference policy constraint term controls the range of model updates and prevents drastic policy changes. This structure originates from the concept of Proximal Policy Optimization (PPO) in reinforcement learning, but after adaptation, it can be used for updating predictive generative models without action supervision.

[0096] By performing parameter differentiation and backpropagation calculations on the above optimization objective, the optimal update direction of the current model under the existing training samples can be obtained, thereby completing the parameter update and forming the optimized model. This process can be executed periodically in each iteration, forming a training mechanism that gradually approaches the optimal prediction capability.

[0097] In the specific implementation process, it is first necessary to extract the policy output probability value corresponding to each candidate prediction result, that is, the probability of generating the subsequent visual state in the model output distribution. Simultaneously, the generation probability under the old policy parameters in the early training phase is recorded to calculate the probability ratio. For each candidate prediction result, the ratio of its current policy probability to the old policy probability is calculated. This ratio is then multiplied by the relative advantage obtained in the previous steps to form the weighted optimization gradient direction. If the ratio is too large or too small, stability issues will arise; therefore, a threshold range can be set or a clipping operation can be used to limit its value range. The optimization objective also includes a reference policy constraint term to limit the deviation between the parameter update and the old parameters. This constraint term can be defined using KL divergence, Euclidean distance, or other regularization methods. After forming the complete objective function, the gradient is calculated and the model parameters are updated using the backpropagation algorithm. In some implementations, to enhance training convergence, mini-batch samples can be used for phased updates, or an adaptive learning rate can be used to dynamically adjust the update speed. An entropy reward term can also be introduced to enhance the diversity of model output and improve the coverage of the generated distribution.

[0098] Example: In the healthcare field, a respiratory anomaly detection system can predict future suspicious states based on a patient's historical visual state sequence. After generating relative advantages for multiple predicted states, the system uses actual labeled data to calculate the strategy probability ratio, constructs an optimization objective function, and updates model parameters, thereby gradually strengthening the model's ability to capture abnormal breathing patterns and improving the accuracy of early warning.

[0099] In the fintech field, compliance risk prediction systems can predict the next user interface based on customer behavior sequences. After generating multiple predicted interfaces, the system constructs a reward function based on historical conversion rates. By guiding model parameter updates through relative advantage, the generated strategy gradually shifts towards interfaces with higher conversion potential and compliance levels, improving the timeliness of risk control intervention and the sensitivity of strategy adjustments.

[0100] This embodiment uses relative advantage as the update guide factor and combines it with the probability ratio of the policy output to construct the optimization objective, achieving explicit weighted updates of high-value candidate samples, thereby enhancing the model's responsiveness to key prediction patterns. This mechanism, while maintaining policy stability, effectively improves the model's adaptability and generalization ability to changes in environmental structure, avoiding getting trapped in local optima during training, and making the prediction results more practical and dynamically reasonable.

[0101] S70, the optimized generative visual model generates a visual planning sequence for the target task.

[0102] In this embodiment, the target task refers to a functionally oriented operation that needs to be completed through continuous changes in visual states, such as path arrival, state transition, and control execution. Its representation typically depends on the dynamic evolution of the visual scene. The visual planning sequence is an ordered set of consecutive visual states, representing a visual path that gradually transitions from an initial state to the target state. This sequence is not manually set externally but is generated step-by-step through autonomous reasoning, relying on a trained and optimized generative visual model, thus possessing end-to-end modeling characteristics.

[0103] Generative vision models, upon receiving an initial sequence of prefix visual states as input, can infer and generate subsequent visual states based on previously generated historical states. Throughout the generation process, the model continuously uses the currently generated state sequence as prefix input and recursively calls itself to generate the next state until a termination condition is met. The termination condition can be triggered by task-defined end markers, such as target region recognition, state convergence, or a maximum generation step limit; these are configurable external policies.

[0104] The generation of visual planning sequences follows an autoregressive inference structure. In each iteration, the current prefix visual state sequence is the set of all previously generated states, while the newly generated states are predictions of the next environmental dynamics. This structure not only preserves temporal order but also implicitly models the causal state transition patterns of the environment, forming a task-driven visual path.

[0105] Since the optimized generative vision model has already adjusted its parameters through the relative advantage mechanism during the previous training, it has a policy bias for specific task scenarios. Therefore, the sequences it generates have higher task achievement efficiency and reasonable state evolution, and can realize the deduction of dynamic planning paths without explicit action instructions.

[0106] The implementation process first requires acquiring the initial visual state corresponding to the task, which serves as the starting input for the current prefix state sequence. This state is then input into the optimized generative visual model, triggering the first subsequent state generation operation. The generated new state is concatenated with the original prefix sequence to form the updated prefix sequence. In each generation round, the model uses only the current prefix sequence as input, without providing explicit action instructions or language prompts, relying on the state transition patterns learned by the model itself to predict the next state. This generation process continues, updating the prefix sequence and calling the model for inference in each iteration. The task termination condition can be determined in the following ways: first, checking whether the current generated state meets structural constraints, such as the detection of target markers or the entry of key regions; second, monitoring whether the magnitude of the generated state change is lower than the convergence threshold, indicating that the sequence tends to stabilize; and third, presetting the maximum number of generation rounds to prevent the model from getting stuck in a loop. The generated visual state sequence will be collected into an ordered array for use in subsequent action execution, scene reconstruction, or downstream task-driven modules. To improve efficiency, this inference process can be accelerated using GPU parallel inference, cache optimization, and other methods.

[0107] Example: In the healthcare field, for automated equipment inspection tasks in intensive care units, the initial state image of the monitoring camera can be input as a prefix state sequence. The optimized model generates a visual state sequence of the automated inspection trajectory based on historical data, eliminating the need for doctors to specify a clear path and completing the entire process coverage without disturbing patients.

[0108] In the fintech field, for counter process review robot tasks, the system takes the camera data when the user first enters the window as the starting point input, and infers to generate a multi-stage service path image sequence from queuing to finally leaving the window. This is used to retrospectively identify abnormal behavior, evaluate service rhythm and optimize window resource allocation, thereby improving operational compliance and customer experience.

[0109] This embodiment uses an optimized generative visual model to autoregressively generate a visual planning sequence for the target task. It does not rely on language reasoning or action templates and directly realizes the dynamic evolution reasoning of the task path in the visual state space. This avoids the information loss and control delay problems caused by modality switching, while improving the model's adaptability in environments with clear geometric structures and complex state transitions. It significantly enhances the automatic planning capability and the accuracy of task completion.

[0110] This invention relates to the field of artificial intelligence technology and can be applied to business scenarios such as fintech and healthcare. It discloses a planning method, apparatus, device, and medium based on visual sequences, comprising: acquiring training visual state sequences corresponding to multiple exploration trajectories; extracting prefix and subsequent visual state pairs formed by training prefix visual state sequences and training subsequent visual states; using these prefix and subsequent visual state pairs to initially train a generative visual model; generating multiple candidate subsequent visual states based on the prefix visual state sequence of the target environment using the initially trained generative visual model; parsing to obtain the corresponding discrete actions and determining the environmental feedback reward value in conjunction with a progress graph, thereby determining the relative advantage of each candidate subsequent visual state; updating the parameters of the generative visual model according to the relative advantage, and finally generating a visual planning sequence for the target task. This invention maintains the continuity of visual states and the closed loop of action feedback during the training and inference process of the generative visual model, enabling the model to perform dynamic planning directly in the visual state space. This avoids the information loss caused by image-to-text conversion in existing technologies, improves the modeling ability for spatial structure and physical evolution, and thus achieves pure vision-driven planning without the need for a language medium, improving the model's generalization ability and strategy effectiveness in complex environments.

[0111] In one embodiment, step S10 above includes:

[0112] S101, performs exploration in the target environment through a random walk strategy, generating multiple exploration trajectories;

[0113] S102, Collect the visual states in a continuous time sequence in each exploration trajectory to form a training visual state sequence;

[0114] S103, extract multiple training prefix visual state sequences from each training visual state sequence;

[0115] S104, For each training prefix visual state sequence, extract the visual states in the training visual state sequence that are adjacent to the training prefix visual state sequence in time order as training subsequent visual states.

[0116] S105, each training prefix visual state sequence is combined with the corresponding training subsequent visual state to form the prefix and subsequent visual state pair.

[0117] In this embodiment, before constructing the vision-driven task planning, it is necessary to accumulate data samples that can characterize the state evolution of the target environment. This process is organized in units of exploration trajectories. Each exploration trajectory corresponds to a path that starts from an arbitrary initial state, continuously transitions through the target environment, and records the visual state at each moment. To obtain trajectory information with diverse and comprehensive state transitions, a random walk strategy can be used to initially traverse the target environment. This strategy does not rely on predefined action models but samples exploration paths in the state space in an unbiased manner, enhancing the breadth of trajectory samples.

[0118] During the exploration process, visual states arranged chronologically along each exploration trajectory need to be collected. Each frame of visual state represents the representation of the environment at the current moment in the perceptual modality, such as RGB images, depth maps, thermal imaging data, or other image modalities. The specific acquisition method depends on the perception capabilities of the actual deployed equipment. By arranging these continuous states in trajectory order to form a training visual state sequence, the temporal evolution logic and spatial continuity of the original environmental state can be preserved.

[0119] Building upon existing training visual state sequences, it is necessary to further construct supervised pairs that can be used for training generative visual models. Specifically, a sliding window extracts multiple training prefix visual state sequences from each training visual state sequence. Each prefix sequence is constructed from the initial state, following an increasing order of length. For example, a prefix of length 1 represents the first frame, a prefix of length 2 represents the first two frames, and so on. This approach helps the model learn the influence of historical states of different lengths on future states.

[0120] For each training prefix visual state sequence, the immediately following visual state needs to be extracted from its corresponding training visual state sequence as the subsequent training visual state. The adjacent relationship here follows temporal order, meaning the state immediately following the end time point of the current prefix sequence. This adjacent extraction method ensures the predictability and continuity of the state transitions between each prefix and its successor.

[0121] Finally, each training prefix visual state sequence is combined with its corresponding training subsequent visual state to form a prefix-subsequent visual state pair. Each pair constitutes a sample in the training dataset, used to supervise the model in predicting subsequent states given a historical visual state sequence. This data structure can be used to support autoregressive training mechanisms, enabling the model to learn to capture the spatial variation patterns and temporal evolution characteristics during state transitions.

[0122] This embodiment constructs prefix-subsequent supervision pairs based on continuous visual state sequences, enabling generative vision models to directly learn the dynamic evolution of the environment from historical visual states without relying on external action labels or language mediation. This enhances the model's ability to understand state transitions in visual space, effectively supporting prediction, reasoning, and path planning for unknown states in subsequent tasks. This approach preserves the temporal continuity and spatial consistency of the original environmental perception data, avoids semantic loss caused by modality switching, and provides data support for constructing purely vision-driven task planning.

[0123] In one embodiment, step S20 above includes:

[0124] S201, input the training prefix visual state sequence in the prefix and subsequent visual state pair into the generative visual model;

[0125] S202, Generative visual model generates predicted subsequent visual states based on the trained prefix visual state sequence;

[0126] S203, determine a state difference metric between the predicted subsequent visual state and the trained subsequent visual state in the prefix and subsequent visual state pair.

[0127] S204, Update the parameters of the generative visual model based on the state difference metric;

[0128] S205, Determine whether the updated generative visual model meets the training termination condition;

[0129] S206, when the training termination condition is not met, repeat the operations of inputting the training prefix visual state sequence into the generative visual model, generating the predicted subsequent visual state, determining the state difference metric, and updating the parameters of the generative visual model.

[0130] S207: When the training termination condition is met, the initial training of the generative visual model is completed.

[0131] In this embodiment, before building task planning capabilities, the generative visual model needs to be trained to predict future states based on historical visual states. This process uses prefix and subsequent visual state pairs as training data, continuously adjusting model parameters through supervised learning to fit the actual patterns of state transitions. First, the training prefix visual state sequence in each training sample is used as the model input. The training prefix visual state sequence is a set of image states with temporal order, typically representing perceived images of the environment at different points in time. The input method can employ an image encoder to extract features from each frame, forming a state representation matrix of uniform dimensions.

[0132] Generative vision models, after receiving a training sequence of prefix visual states, model the input states based on their internal structure (e.g., autoregressive structures, Transformer architectures, or temporal convolutional networks) to generate predicted subsequent visual states. The prediction process can employ conditional generation, generating the visual representation for the next time step based on the prefix sequence. The predicted subsequent visual states can take the form of images, feature maps, or multimodal representations, depending on the model design and task requirements.

[0133] To evaluate the difference between the generated result and the true subsequent state, a state difference metric function needs to be defined. This metric function can employ pixel-level mean squared error, structural similarity metrics, perceptual loss functions (such as LPIPS), or feature space-based similarity measures to quantify the difference between the predicted subsequent visual state and the trained subsequent visual state. The difference metric value reflects how well the model currently fits the state transition process.

[0134] The parameters of the generative vision model are updated via backpropagation based on the state difference metric. This update process can be based on gradient descent optimization methods, such as Adam and RMSProp, to iteratively optimize the model weights to reduce prediction error. After each parameter update, the system needs to determine whether the current model has reached the training termination condition. The termination condition can be convergence of the validation set error, reaching the upper limit of the training epochs, or the performance metric exceeding a preset threshold. If the termination condition has not been met, the process of inputting the prefix sequence, generating the predicted state, calculating the difference metric, and updating the parameters is repeated, forming a loop training mechanism.

[0135] When the judgment condition is met, the loop terminates, indicating that the generative visual model has a strong state prediction ability, the initial training phase is completed, and the model can be used to infer the state evolution in the target environment.

[0136] This embodiment constructs a generative visual model with state prediction capabilities by using prefix and subsequent visual state pairs as supervised training data. This model can effectively learn the distribution characteristics of state evolution over time in the environment. The training process employs a parameter update mechanism driven by difference metrics, which does not rely on external language information or action labels. This fundamentally solves the problem of traditional models' dependence on language mediation and explicit action sequences, improving the model's direct understanding and prediction capabilities of visual information. This lays the foundation for generating planning sequences under purely visual conditions.

[0137] In one embodiment, step S30 above includes:

[0138] S301, Receive the state observation input of the target environment, and construct a continuous visual state representation based on the state observation input;

[0139] S302, Based on the continuous visual state representation, generate a prefix visual state sequence of the target environment that includes the current visual state and the previous visual state;

[0140] S303, input the prefix visual state sequence of the target environment into the generative visual model that has completed the initial training;

[0141] S304, the generative visual model that has completed the initial training performs multiple inference operations on the prefix visual state sequence of the target environment, and each inference operation generates a candidate subsequent visual state.

[0142] S305, gather all generated candidate subsequent visual states to form multiple candidate subsequent visual states.

[0143] In this embodiment, after the initial training of the generative visual model is completed, inference operations based on the model need to be performed in the target environment to obtain feasible future visual state candidates. The first step in inference in the target environment is to receive state observation input from the environment. This input may include visual image frames at the current moment, and the source may be a robot camera, a smart terminal perception module, an image sensor in an automated device, etc. In order to express the temporal continuity of the environmental state, state observations at multiple consecutive time points need to be fused to construct a continuous visual state representation. This representation not only includes the current state image but also implies historical evolution trends.

[0144] Based on the aforementioned continuous visual state representation, a prefix visual state sequence of the target environment is constructed. This sequence should cover the image states at multiple time points prior to the current moment, forming a temporally ordered state stream used to express the local environmental dynamics. The length and sampling interval of the prefix visual state sequence can be set according to the temporal characteristics of the task; for example, a shorter interval can be selected in a rapidly changing environment to preserve dynamic details.

[0145] After the prefix visual state sequence is constructed, it is fed as input into the initially trained generative vision model. This model already possesses the state transition distribution learned during training and can perform conditional inference on the input sequence to generate predictive future state images. To generate multiple optional future states, the model needs to support multiple inference operations under the same input conditions. These multiple inferences can be based on introducing random noise, using different seeds, and implementing sampling strategies (such as Top-k, temperature sampling, etc.) to generate multiple samples. The output of each inference operation is a candidate subsequent visual state, representing a possible future scenario that the model believes may occur under the current prefix conditions.

[0146] Finally, all candidate subsequent visual states from multiple inference results are aggregated to obtain a candidate set consisting of multiple images, which serves as input for action evaluation and selection in the next stage. This candidate set enriches the model's understanding of the diversity of future environmental evolution and provides a possible basis for subsequent policy optimization.

[0147] This embodiment efficiently generates multiple candidate visual states that conform to the evolutionary laws of the environment by inputting continuous visual states from the target environment into a generative visual model and performing multiple inference operations on the model. This mechanism avoids the language-mediated state transition process, directly models the state evolution path in the image space, and enhances the ability to model physical laws and spatial changes. The candidate states generated by multiple inferences are distributed and diverse, providing greater space for subsequent policy selection based on feedback evaluation, and significantly improving the robustness and accuracy of visual planning in uncertain environments.

[0148] In one embodiment, step S40 above includes:

[0149] S401, the prefix visual state sequence of the target environment is combined with each candidate subsequent visual state to form multiple visual state pairs;

[0150] S402 maps each visual state pair to a corresponding discrete action type through a state-action parsing function;

[0151] S403, Obtain the pre-generated progress chart;

[0152] S404, based on the progress map and discrete action type, determine the environmental feedback reward value for each discrete action.

[0153] In this embodiment, after acquiring multiple candidate subsequent visual states, a series of visual state pairs for action reasoning are constructed by combining them with the prefix visual state sequence of the target environment. Each visual state pair consists of a historical visual state sequence and a predicted future state image, forming a state transition segment from the current known state to the potential next state. This combination operation can be implemented based on image frame sequence stitching, state vector docking, or tensor dimension expansion, ensuring that the model can identify and retain the information features of state changes during subsequent processing.

[0154] The aforementioned visual state pairs are input into a state-action parsing function, which derives the corresponding discrete action type based on the state change characteristics. This state-action parsing function can be trained on labeled data or constructed through physics engine simulation, spatial displacement estimation, or image similarity mapping. The core objective of this function is to decode visual state changes into action identifiers with clear semantics, such as "move forward," "turn left," or "grab an object," ensuring that subsequent action quality assessments have practical execution value.

[0155] Subsequently, a pre-generated progress map is obtained, which represents the phased goals, state transition paths, and completion measurement methods during the task's progress. The progress map can be generated through domain knowledge construction, expert strategy definition, or learning algorithm derivation. Essentially, it is a reference trajectory or node graph structure for task execution, which can clearly indicate the relationship between the current state and the target state.

[0156] Finally, the discrete action types mapped by the visual state are combined with the progress map to evaluate the value of each action in advancing the current task. This evaluation process can incorporate methods such as state node matching, path distance estimation, and stage transition rewards to quantify the positive contribution of each action to achieving the goal. Ultimately, an environmental feedback reward value is assigned to each discrete action for subsequent model updates and policy evaluation.

[0157] This embodiment constructs visual state pairs and determines the action type based on a state-action parsing function. It then combines this with a progress graph to calculate the environmental feedback reward value. This eliminates the reliance on external text annotations or rule engines for action evaluation, allowing it to be completed entirely based on the changing trends of image states and task structure reasoning. This mechanism establishes a clear mapping between state prediction and action selection, effectively improving the model's adaptability and feedback sensitivity in unknown environments, and enhancing the training efficiency and task completion quality of the generative model in complex planning tasks.

[0158] In one embodiment, step S50 includes:

[0159] S501, obtain the environmental feedback reward value of all discrete actions in the current group;

[0160] S502, determine the average and standard deviation of the current group's environmental feedback reward values;

[0161] S503, for each candidate subsequent visual state, determine the difference between the environmental feedback reward value of the discrete action corresponding to the candidate subsequent visual state and the average value;

[0162] S504, divide the difference by the standard deviation to generate the relative advantage value of the candidate subsequent visual state.

[0163] In this embodiment, to address the performance differences between discrete actions corresponding to candidate subsequent visual states, a metric for relative advantage needs to be introduced for model optimization feedback. First, the environmental feedback reward value corresponding to each discrete action is obtained within the candidate action set. This reward value is generated after constructing visual state pairs and analyzing them in conjunction with the progress graph structure, exhibiting clear task relevance and state transition responsiveness. The candidate set is typically generated by inference from a generative visual model that has completed initial training, and its size can be flexibly adjusted according to model computational resources and the task's temporal span.

[0164] After obtaining the environmental feedback reward values ​​for all discrete actions in the set, the statistical mean and standard deviation of these values ​​are calculated as two key parameters characterizing the overall performance distribution of the current candidate set. The mean is used to determine the central tendency of the reward values, and the standard deviation is used to describe the dispersion of the rewards and assist in the subsequent standardization process, avoiding evaluation bias caused by uneven distribution of reward values.

[0165] Based on the obtained statistics, for each candidate subsequent visual state, its corresponding discrete action is extracted, and its environmental feedback reward value is found. Then, the difference between this reward value and the mean is calculated, which reflects the degree of deviation of the candidate from the average level in the action space.

[0166] Furthermore, dividing this difference by the standard deviation yields a standardized score, which represents the relative advantage of the discrete action corresponding to the candidate's subsequent visual state within the current candidate set. This standardization process suppresses the influence of anomalous reward values, improving the stability and comparability of the evaluation indicators. Its mathematical expression conforms to the z-score mechanism in statistical inference, achieving normalized comparisons across sample sets through dimensionless processing.

[0167] Throughout the process, the relative advantage value is not only used to rank and select candidate subsequent visual states, but also serves as a key weighting factor in the next stage of generative visual model parameter updates, providing directional guidance for policy learning. This process ensures the consistency and convergence of the evaluation criteria for candidate states, preventing the model from being misled by local high rewards and falling into invalid paths. The entire evaluation chain ensures that each candidate state can be quantitatively analyzed using statistically normalized indicators, forming a set of candidate states with actual discriminative power.

[0168] This embodiment achieves a normalized evaluation of diverse action feedback by calculating the relative advantage value of actions corresponding to candidate subsequent visual states. This effectively overcomes the problems of inconsistent reward scales or outlier interference, enabling the generative visual model to stably optimize parameters and converge to a better policy path during multiple training rounds. This method enhances the discriminativeness and interpretability of candidate state evaluation, providing a reliable foundation for policy guidance and action sampling in subsequent generative models.

[0169] In one embodiment, step S60 above includes:

[0170] S601, Determine the probability ratio of the new parameter strategy to the old parameter strategy in the generative visual model;

[0171] S602, construct a strategy optimization objective based on the probability ratio and the relative advantage;

[0172] S603, add a reference strategy constraint term to the strategy optimization objective;

[0173] S604, optimize the strategy optimization objective to update the parameters of the generative visual model, and obtain the optimized generative visual model.

[0174] In this embodiment, to effectively inject the feedback of the relative advantage obtained from the previous evaluation into the structure of the generative vision model, the model parameters need to be updated based on policy optimization. First, while keeping the original structure of the generative vision model unchanged, a probability ratio calculation operation is performed on the new parameter policy composed of the current parameters and the old parameter policy before the update in the candidate state space. This probability ratio reflects the change in preference of the two sets of parameter policies for predicting subsequent candidate visual states under the same input distribution, and can serve as a basis for dynamic adjustment of the gradient direction.

[0175] Next, the probability ratio is combined with the previously calculated relative advantage value to construct a policy optimization objective function. This objective function measures the improvement of the new policy's performance on the current candidate state set and numerically guides the parameter update direction towards candidate states with higher reward potential. The construction method typically uses the probability ratio multiplied by the relative advantage to ensure that gradient backpropagation focuses more on the actual effective reward differences, thus improving learning efficiency.

[0176] To prevent policy divergence or drastic shifts during optimization, a reference policy constraint term is introduced into the policy optimization objective to limit the degree of deviation of the new policy relative to the old policy. This constraint term is typically implemented by embedding a regularization term based on KL divergence into the objective function, ensuring stability during policy updates, and especially preventing policy jumps when the distribution of the candidate state set changes drastically.

[0177] After constructing the policy optimization objective function, a gradient descent-based method is used to optimize and solve the objective function, obtaining a new parameter set for the generative visual model. This parameter update process can be performed through iterative solving, batch training, or parallel gradient aggregation to improve convergence efficiency and generalization ability. The resulting optimized generative visual model retains the original model structure while incorporating relatively advantageous information and fine-tuning its predictive capabilities, enabling it to generate subsequent visual state sequences with higher task completion rates under similar prefix visual state sequences. This update process achieves embedded feedback of reinforcement learning signals without introducing additional modules, effectively improving the model's responsiveness to the task objective.

[0178] Example: In intelligent risk control in the financial sector, it is often necessary to model customer behavior sequences and predict risk evolution trends to assist automated decision-making systems in completing tasks such as credit approval, transaction monitoring, and compliance checks.

[0179] In a certain intelligent lending system, to predict compliance risks in customers' future transactions, the system uses historical transaction images as input and drives the planning and reasoning process by constructing a sequence of visual states for financial transactions. First, the system randomly simulates customer transaction trajectories under different market conditions, generating multiple exploration trajectories including account balance graphs, transfer structure graphs, and fund flow heatmaps. Each trajectory consists of visual states organized chronologically, forming a sequence of visual states for training. Within these sequences, the system extracts a sequence of transaction prefix graphs with increasing order of magnitude and pairs it with the immediately following transaction image to form training prefix-subsequent visual state pairs.

[0180] Next, the system inputs these prefixes and subsequent visual state pairs into the initial generative visual model, generating subsequent visual state maps that conform to the flow of funds through supervised training. During training, the system continuously compares the predicted subsequent visual states with the actual sampled images using image difference metrics such as the structural similarity index (SSIM) or feature embedding difference, and adjusts the model parameters in reverse based on this difference until the convergence condition is met.

[0181] After initial training, when a user initiates a new transaction, the system receives observations of the target environment state, consisting of current account information, historical operation charts, and behavioral feature maps, and encodes them into a prefix visual state sequence. This prefix sequence is then input into a generative visual model to perform multiple inferences, generating a possible subsequent transaction behavior map each time, forming a candidate set of subsequent visual states. These candidate states may include risky behavioral patterns such as large transfers, frequent small transactions, and cross-account fund transfers.

[0182] Subsequently, the system combines the prefix visual state sequence with each candidate subsequent visual state to construct the current possible transaction evolution pair, and parses it into discrete actions using a preset transaction-risk mapping model (state-action parsing function), such as "triggering manual review," "entering the risk pool," and "marking abnormal transactions." Based on a progress chart constructed from the customer's historical performance, the system then determines the expected reward value of each of the above actions in the current risk assessment process, such as whether it helps reduce the probability of default or optimizes credit resource allocation.

[0183] For each candidate visual state, the system calculates the mean and standard deviation of all current candidate actions based on the environmental feedback reward value of its corresponding action, and calculates a relative advantage value to reflect the superiority of the candidate state in the current scene. Using these relative advantages, the system constructs a new policy optimization objective function, taking the probability ratio of the current model's predicted policy to the old policy as the objective term. After incorporating reference policy constraints such as KL divergence, the system performs policy optimization to update the parameters of the generative visual model.

[0184] Finally, using the optimized generative visual model, the system takes a new sequence of user transaction prefix graphs as input and outputs a visual planning sequence composed of a series of predicted financial behavior graphs, providing support for risk monitoring. The system can output planning results such as high-risk transaction trigger nodes, optimized path suggestions, and potential intervention points, which can be used as a reference for subsequent decision-making models.

[0185] Through the above process, the financial intelligence system can not only perform planning and prediction based on a purely visual transaction state diagram without the need for text conversion, but also realize a closed loop from learning from historical behavior, generative reasoning, risk response analysis to strategy optimization feedback, significantly improving the autonomous planning and risk response capabilities in the financial field based on multimodal data-driven approaches.

[0186] In the healthcare field, for tasks such as predicting patient state evolution and planning intervention pathways, an intelligent diagnostic and treatment assistance system based on visual state sequences can be constructed. In the intensive care unit (ICU) management system, the clinical monitoring platform continuously collects images of patients' physiological states, including trend maps of vital signs (such as heart rate, blood pressure, and blood oxygen saturation), visualizations of bedside monitoring equipment, historical medication heatmaps, and body position images. The system uses random simulation to extract multiple state evolution trajectories from the real monitoring history of multiple critically ill patients, thereby forming visual state sequences for training. Each visual state sequence is organized chronologically, representing the progression of the patient's condition from initial to critical. The system extracts multiple training prefix visual state sequences and pairs them with their temporally adjacent subsequent visual images to construct prefix-subsequent visual state pairs for model learning.

[0187] These training prefixes are input into a generative visual model along with subsequent visual states. The model learns to predict future disease progression in images based on the prefix sequences, such as whether phenomena like sudden respiratory drop, arrhythmia, or a surge in inflammation indices will occur. The training process iteratively optimizes the model by comparing the differences between the predicted subsequent visual states and the actual evolutionary graphs. Indicators such as image embedding difference and graph convolution matching score can be used to adjust model parameters until convergence.

[0188] After initial training, when a patient is in a certain state, the system collects their latest continuous physiological images and environmental illustrations, encodes them into a prefix visual state sequence of the target environment, and inputs it into a generative visual model for multiple forward inference operations. Each time, a possible subsequent evolutionary image is generated, forming multiple candidate subsequent visual states. These candidate states may correspond to different clinical pathways, such as stable condition, mild deterioration, severe deterioration, or impending emergency intervention.

[0189] The system combines these candidate images with prefix images to form visual state pairs. Through a state-action parsing function, it identifies possible diagnostic and treatment response actions, such as "intensify monitoring," "adjust ventilator parameters," "postpone transfer," and "immediately initiate emergency procedures." Next, the system retrieves a disease progression map constructed from patient characteristics and similar past medical records. Combining this with the current candidate action, it determines the environmental feedback reward value, such as increasing the probability of survival, reducing ICU stay, or improving intervention effectiveness.

[0190] After obtaining the environmental feedback reward values ​​for all candidate actions, the system calculates the mean and standard deviation of these values. Then, it calculates and standardizes the difference between the action corresponding to each candidate state and the mean, generating relative advantage values. Based on these relative advantages, the system constructs an optimization objective function, weighting the probability ratio of the current policy to the old policy, and incorporating the model policy stability constraint to perform an update operation and optimize the generative visual model.

[0191] Ultimately, the optimized generative visual model can rapidly generate a series of predictive evolutionary path sequences based on the new patient physiological state map, forming a visual planning sequence tailored to the current patient's condition. This planning sequence will be used to assist clinicians in identifying potential risks in advance, developing personalized treatment intervention pathways, and improving the success rate of rescue and treatment efficiency.

[0192] Through this planning process, the system has achieved the ability to autonomously complete state prediction and intervention planning based directly on visual image sequences without converting complex medical images into text or labels. This solves the problem that existing language-based models are difficult to accurately model spatial physiological evolution and physical response mechanisms, and provides decision support tools with reasoning capabilities for critical care management.

[0193] This embodiment guides the parameter updates of the generative vision model through relative advantage values ​​and integrates probability ratios and reference policy constraints during policy optimization, achieving fine-tuning of the generation policy. This process not only preserves the predictive ability obtained from initial training but also enables the model to generate subsequent visual states with higher task completion rates through coupled optimization with environmental feedback. Compared to traditional training methods that rely on static supervision signals, this update mechanism improves the model's adaptability to action effects under different combinations of visual states, effectively enhancing its robustness and generalization ability in performing visual planning tasks in complex dynamic scenes.

[0194] In one embodiment, a planning device based on visual sequences is provided, which corresponds one-to-one with the planning method based on visual sequences in the above embodiments. (Refer to...) Figure 3 , Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the visual sequence-based planning device of the present invention. The modules include a data acquisition and construction module 10, an initial training module 20, a candidate generation module 30, an action evaluation module 40, an advantage estimation module 50, a strategy optimization module 60, and a planning generation module 70. Detailed descriptions of each functional module are as follows:

[0195] The data acquisition and construction module 10 is used to acquire training visual state sequences corresponding to multiple exploration trajectories, and extract prefix and subsequent visual state pairs containing training prefix visual state sequences and training subsequent visual states based on the training visual state sequences.

[0196] The initial training module 20 is used to generate subsequent visual states that conform to the dynamic environment of the supervised generative visual model by using the prefix and subsequent visual states, thereby completing the initial training of the generative visual model.

[0197] The candidate generation module 30 is used to generate multiple candidate subsequent visual states based on the prefix visual state sequence of the target environment using the generative visual model that has completed initial training.

[0198] Action evaluation module 40 is used to combine the prefix visual state sequence of the target environment with each candidate subsequent visual state to generate a visual state sequence, parse the visual state sequence to obtain the corresponding discrete action, and combine the progress map to determine the environmental feedback reward value of each discrete action.

[0199] Advantage estimation module 50 is used to determine the relative advantage of each candidate subsequent visual state based on the environmental feedback reward value of each discrete action;

[0200] The strategy optimization module 60 is used to update the parameters of the generative visual model according to the relative advantage to obtain an optimized generative visual model.

[0201] The planning generation module 70 is used to generate a visual planning sequence for the target task through the optimized generative visual model.

[0202] In one embodiment, the data acquisition and construction module 10 is specifically used for:

[0203] Exploration is performed in the target environment using a random walk strategy, generating multiple exploration trajectories;

[0204] Collect the visual states in a continuous time sequence for each exploration trajectory to form a training visual state sequence;

[0205] Extract multiple training prefix visual state sequences from each training visual state sequence;

[0206] For each training prefix visual state sequence, extract the visual states that are temporally adjacent to the training prefix visual state sequence in the training visual state sequence as training subsequent visual states.

[0207] Each training prefix visual state sequence is combined with its corresponding training subsequent visual state to form the prefix and subsequent visual state pair.

[0208] In one embodiment, the initial training module 20 is specifically used for:

[0209] The training prefix visual state sequence in the prefix and subsequent visual state pair is input into the generative visual model;

[0210] The generative visual model generates a predicted subsequent visual state based on the training prefix visual state sequence;

[0211] Determine a state difference metric between the predicted subsequent visual state and the trained subsequent visual state in the prefix and subsequent visual state pair.

[0212] The parameters of the generative visual model are updated based on the state difference metric.

[0213] Determine whether the updated generative visual model meets the training termination condition;

[0214] When the training termination condition is not met, repeat the operations of inputting the training prefix visual state sequence into the generative visual model, generating the predicted subsequent visual state, determining the state difference metric, and updating the parameters of the generative visual model.

[0215] When the training termination condition is met, the initial training of the generative visual model is completed.

[0216] In one embodiment, the candidate generation module 30 is specifically used for:

[0217] Receive state observation input from the target environment and construct a continuous visual state representation based on the state observation input;

[0218] Based on the continuous visual state representation, a prefix visual state sequence of the target environment containing the current visual state and the previous visual state is generated.

[0219] The prefix visual state sequence of the target environment is input into the generative visual model that has completed initial training;

[0220] The generative visual model that has completed initial training performs multiple inference operations on the prefix visual state sequence of the target environment, and each inference operation generates a candidate subsequent visual state.

[0221] All generated candidate subsequent visual states are aggregated to form multiple candidate subsequent visual states.

[0222] In one embodiment, the motion evaluation module 40 is specifically used for:

[0223] The prefix visual state sequence of the target environment is combined with each candidate subsequent visual state to form multiple visual state pairs;

[0224] Each visual state pair is mapped to a corresponding discrete action type using a state-action parsing function.

[0225] Get the pre-generated progress chart;

[0226] Based on the progress chart and discrete action types, determine the environmental feedback reward value for each discrete action.

[0227] In one embodiment, the advantage estimation module 50 is specifically used for:

[0228] Obtain the environmental feedback reward values ​​for all discrete actions within the current group;

[0229] Determine the average and standard deviation of the current group's environmental feedback reward values;

[0230] For each candidate subsequent visual state, determine the difference between the environmental feedback reward value of the discrete action corresponding to the candidate subsequent visual state and the average value;

[0231] Divide the difference by the standard deviation to generate the relative advantage value of the candidate subsequent visual state.

[0232] In one embodiment, the strategy optimization module 60 is specifically used for:

[0233] Determine the probability ratio of the new parameter strategy to the old parameter strategy in the generative visual model;

[0234] A strategy optimization objective is constructed based on the probability ratio and the relative advantage;

[0235] Add a reference policy constraint term to the policy optimization objective;

[0236] The optimization objective of the strategy is optimized to update the parameters of the generative visual model, thereby obtaining an optimized generative visual model.

[0237] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used for communication with external user terminals via a network connection. When executed by the processor, the computer program implements the functions or steps of a visual sequence-based planning method on the server side.

[0238] In one embodiment, a computer device is provided, which may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements a user-side function or step of a visual sequence-based planning method.

[0239] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0240] Obtain training visual state sequences corresponding to multiple exploration trajectories, and extract prefix and subsequent visual state pairs containing training prefix visual state sequences and training subsequent visual states based on the training visual state sequences.

[0241] The supervised generative visual model is trained by generating subsequent visual states that conform to the dynamics of the environment through the prefix and subsequent visual states, thus completing the initial training of the generative visual model.

[0242] Based on the prefix visual state sequence of the target environment, multiple candidate subsequent visual states are generated using the generative visual model that has completed initial training.

[0243] The visual state sequence is generated by combining the prefix visual state sequence of the target environment with each candidate subsequent visual state. The visual state sequence is parsed to obtain the corresponding discrete action, and the environmental feedback reward value of each discrete action is determined by combining the progress map.

[0244] Based on the environmental feedback reward value for each discrete action, the relative advantage of each candidate subsequent visual state is determined;

[0245] The parameters of the generative visual model are updated based on the relative advantages to obtain an optimized generative visual model.

[0246] The optimized generative visual model generates a visual planning sequence for the target task.

[0247] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0248] Obtain training visual state sequences corresponding to multiple exploration trajectories, and extract prefix and subsequent visual state pairs containing training prefix visual state sequences and training subsequent visual states based on the training visual state sequences.

[0249] The supervised generative visual model is trained by generating subsequent visual states that conform to the dynamics of the environment through the prefix and subsequent visual states, thus completing the initial training of the generative visual model.

[0250] Based on the prefix visual state sequence of the target environment, multiple candidate subsequent visual states are generated using the generative visual model that has completed initial training.

[0251] The visual state sequence is generated by combining the prefix visual state sequence of the target environment with each candidate subsequent visual state. The visual state sequence is parsed to obtain the corresponding discrete action, and the environmental feedback reward value of each discrete action is determined by combining the progress map.

[0252] Based on the environmental feedback reward value for each discrete action, the relative advantage of each candidate subsequent visual state is determined;

[0253] The parameters of the generative visual model are updated based on the relative advantages to obtain an optimized generative visual model.

[0254] The optimized generative visual model generates a visual planning sequence for the target task.

[0255] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and user side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0256] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0257] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0258] It should be noted that if any software tools or components not belonging to this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use. The embodiments described above are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A planning method based on visual sequences, characterized in that, Includes the following steps: The training visual state sequence corresponding to multiple exploration trajectories is obtained, and a prefix and subsequent visual state pair containing a training prefix visual state sequence and a training subsequent visual state is extracted based on the training visual state sequence. The visual state sequence refers to a series of image frames or frame sequence information that continuously record the changes in the visual environment over time during multiple interaction processes. The prefix visual state sequence refers to a sub-sequence selected from the visual state sequence in chronological order. The subsequent visual state refers to the image frame immediately following the prefix visual state sequence. The supervised generative visual model is trained by generating subsequent visual states that conform to the dynamics of the environment through the prefix and subsequent visual states, thus completing the initial training of the generative visual model. Based on the prefix visual state sequence of the target environment, multiple candidate subsequent visual states are generated using the generative visual model that has completed initial training. The process involves combining the prefix visual state sequence of the target environment with each candidate subsequent visual state to generate a visual state sequence, parsing the visual state sequence to obtain the corresponding discrete actions, and determining the environmental feedback reward value for each discrete action in conjunction with a progress map. This includes: combining the prefix visual state sequence of the target environment with each candidate subsequent visual state to form multiple visual state pairs; mapping each visual state pair to a corresponding discrete action type using a state-action parsing function; obtaining a pre-generated progress map, which is a structured infographic used to depict the task execution stages and their expected state evolution paths; and determining the environmental feedback reward value for each discrete action based on the progress map and the discrete action type. Determining the relative advantage of each candidate subsequent visual state based on the environmental feedback reward value of each discrete action includes: obtaining the environmental feedback reward values ​​of all discrete actions in the current group; determining the average value and standard deviation of the environmental feedback reward values ​​in the current group; for each candidate subsequent visual state, determining the difference between the environmental feedback reward value of the discrete action corresponding to the candidate subsequent visual state and the average value; and dividing the difference by the standard deviation to generate the relative advantage value of the candidate subsequent visual state. The parameters of the generative visual model are updated based on the relative advantages to obtain an optimized generative visual model. The optimized generative visual model generates a visual planning sequence for the target task.

2. The planning method based on visual sequences as described in claim 1, characterized in that, Obtain training visual state sequences corresponding to multiple exploration trajectories, and extract prefix and subsequent visual state pairs containing training prefix visual state sequences and training subsequent visual states based on the training visual state sequences, including: Exploration is performed in the target environment using a random walk strategy, generating multiple exploration trajectories; Collect the visual states in a continuous time sequence for each exploration trajectory to form a training visual state sequence; Extract multiple training prefix visual state sequences from each training visual state sequence; For each training prefix visual state sequence, extract the visual states that are temporally adjacent to the training prefix visual state sequence in the training visual state sequence as training subsequent visual states. Each training prefix visual state sequence is combined with its corresponding training subsequent visual state to form the prefix and subsequent visual state pair.

3. The planning method based on visual sequences as described in claim 1, characterized in that, The supervised generative visual model is trained by generating subsequent visual states that conform to the dynamics of the environment using the prefix and subsequent visual states, thus completing the initial training of the generative visual model, including: The training prefix visual state sequence in the prefix and subsequent visual state pair is input into the generative visual model; The generative visual model generates a predicted subsequent visual state based on the training prefix visual state sequence; Determine a state difference metric between the predicted subsequent visual state and the trained subsequent visual state in the prefix and subsequent visual state pair. The parameters of the generative visual model are updated based on the state difference metric. Determine whether the updated generative visual model meets the training termination condition; When the training termination condition is not met, repeat the operations of inputting the training prefix visual state sequence into the generative visual model, generating the predicted subsequent visual state, determining the state difference metric, and updating the parameters of the generative visual model. When the training termination condition is met, the initial training of the generative visual model is completed.

4. The planning method based on visual sequences as described in claim 1, characterized in that, Based on the prefix visual state sequence of the target environment, multiple candidate subsequent visual states are generated using a generative visual model that has completed initial training, including: Receive state observation input from the target environment and construct a continuous visual state representation based on the state observation input; Based on the continuous visual state representation, a prefix visual state sequence of the target environment containing the current visual state and the previous visual state is generated. The prefix visual state sequence of the target environment is input into the generative visual model that has completed initial training; The generative visual model that has completed initial training performs multiple inference operations on the prefix visual state sequence of the target environment, and each inference operation generates a candidate subsequent visual state. All generated candidate subsequent visual states are aggregated to form multiple candidate subsequent visual states.

5. The planning method based on visual sequences as described in claim 1, characterized in that, The parameters of the generative visual model are updated based on the relative advantage to obtain an optimized generative visual model, including: Determine the probability ratio of the new parameter strategy to the old parameter strategy in the generative visual model; A strategy optimization objective is constructed based on the probability ratio and the relative advantage; Add a reference policy constraint term to the policy optimization objective; The optimization objective of the strategy is optimized to update the parameters of the generative visual model, thereby obtaining an optimized generative visual model.

6. A planning device based on visual sequences, characterized in that, The visual sequence-based planning device includes: The data acquisition and construction module is used to acquire training visual state sequences corresponding to multiple exploration trajectories, and extract prefix and subsequent visual state pairs containing training prefix visual state sequences and training subsequent visual states based on the training visual state sequences. The visual state sequence refers to a series of image frames or frame sequence information that continuously record changes in the visual environment over time during multiple interaction processes. The prefix visual state sequence refers to a subsequence selected from the visual state sequence in chronological order. The subsequent visual state refers to the image frame immediately following the prefix visual state sequence. The initial training module is used to generate subsequent visual states that conform to the dynamic environment of the supervised generative visual model by using the prefix and subsequent visual states, thereby completing the initial training of the generative visual model. The candidate generation module is used to generate multiple candidate subsequent visual states based on the prefix visual state sequence of the target environment using the generative visual model that has completed initial training. The action evaluation module is used to combine the prefix visual state sequence of the target environment with each candidate subsequent visual state to generate a visual state sequence, parse the visual state sequence to obtain the corresponding discrete action, and determine the environmental feedback reward value of each discrete action in conjunction with a progress map. This includes: combining the prefix visual state sequence of the target environment with each candidate subsequent visual state to form multiple visual state pairs; mapping each visual state pair to a corresponding discrete action type using a state-action parsing function; obtaining a pre-generated progress map, which is a structured infographic used to depict the task execution stages and their expected state evolution paths; and determining the environmental feedback reward value of each discrete action based on the progress map and the discrete action type. The advantage estimation module is used to determine the relative advantage of each candidate subsequent visual state based on the environmental feedback reward value of each discrete action, including: obtaining the environmental feedback reward values ​​of all discrete actions in the current group; determining the average value and standard deviation of the environmental feedback reward values ​​in the current group; for each candidate subsequent visual state, determining the difference between the environmental feedback reward value of the discrete action corresponding to the candidate subsequent visual state and the average value; dividing the difference by the standard deviation to generate the relative advantage value of the candidate subsequent visual state. The strategy optimization module is used to update the parameters of the generative visual model according to the relative advantage to obtain an optimized generative visual model. The planning generation module is used to generate a visual planning sequence for the target task using the optimized generative visual model.

7. A computer device, characterized in that, The computer device includes a memory, a processor, and a visual sequence-based planning program stored in the memory and executable on the processor, wherein the visual sequence-based planning program, when executed by the processor, implements the steps of the visual sequence-based planning method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The storage medium stores a planning program based on visual sequences, which, when executed by a processor, implements the steps of the planning method based on visual sequences as described in any one of claims 1-5.

Images