Digital image action dynamic adjustment method based on reinforcement learning

Abstract

The invention discloses a digital image action dynamic adjustment method based on reinforcement learning, which comprises the following steps: collecting multi-source interaction data between a user and a digital image, performing semantic analysis, and generating semantic sub-target vectors; constructing an improved hierarchical reinforcement learning structure according to the score value, generating a sub-target intention embedding vector, inputting the sub-target intention embedding vector into a style regulator and a low-layer action strategy network, and outputting an action trajectory reference path; dynamically adjusting a graph structure action skill library by utilizing a graph attention mechanism, and generating an action combination candidate graph; constructing a Koopman linearization subspace control model, executing subspace trajectory optimization, and outputting a continuous action execution instruction; updating the control model and the strategy network according to the environment feedback data to realize strategy iterative optimization; and finally packaging the key module into a digital image behavior packet. According to the method, the real-time performance, the coherence and the semantic consistency of digital image action adjustment are improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence, and particularly relates to a digital image action dynamic adjustment method based on reinforcement learning. BACKGROUND

[0002] With the wide application of virtual humans and digital images in emerging scenarios such as social platforms, online education, virtual customer service, digital marketing and metaverse, the behavior intelligence and natural action expression of digital images gradually become core factors affecting user interaction experience. Traditional digital image action control mainly relies on preset action libraries and fixed behavior scheduling logic, usually adopting a rule-driven or template matching-based approach, which is difficult to effectively adapt to semantic differences of user instructions, individual preferences and changing interaction contexts, resulting in strong action repetition, rigid performance and lack of flexibility, and is difficult to meet the needs of humanized and emotionalized interaction.

[0003] In recent years, reinforcement learning has made a series of breakthroughs in the field of robot control and intelligent strategy generation, providing a new idea for the action generation and dynamic adjustment of digital images. By constructing a policy network based on interaction rewards, the reinforcement learning method can enable the system to continuously optimize the behavior strategy through environmental feedback, forming strong adaptability and generalization ability. However, directly applying the traditional reinforcement learning structure to the action adjustment of digital images still faces multiple challenges. First, the action adjustment task of digital images usually has the characteristics of target decomposition and hierarchical decision-making, and it is difficult to model complex task structures by relying only on a single-layer policy network, which can easily lead to slow policy convergence or local optimum. Second, user input is often expressed in natural language, emotional state, behavior response and other multi-modal data, and there are granularity differences and semantic ambiguities between different modal data, making it difficult for traditional policy modeling methods to achieve semantic consistency analysis and action mapping. In addition, the action execution of digital images itself has continuous control characteristics and physical consistency requirements, and reinforcement learning strategies often lack controllable state evolution mechanisms, resulting in generated actions that are not coherent, stable or distorted.

[0004] Under this background, some research attempts to apply hierarchical reinforcement learning to digital human action control scenarios, by introducing a hierarchical structure of high-level task scheduling and low-level action execution, to improve the policy expression ability and task adaptability. The high-level strategy is responsible for generating sub-goals at the medium or semantic level, and the low-level strategy executes specific action sequences according to the sub-goals. Although this method alleviates the problem of task complexity to some extent, in the existing structure, the generation of sub-goals often relies on artificial setting or fixed decomposition logic, lacking dynamic modeling ability for user intent, and being unable to effectively cope with the characteristics of semantic ambiguity and strong polysemy in natural language expression. In addition, the collaborative strategy updating mechanism between layers is not mature, and the training process is easily disturbed by high-dimensional action space, resulting in unstable training process, and the final control effect still difficult to meet the requirements of real and natural expression.

[0005] On the other hand, the trajectory control problem of action is also one of the core links that restricts the natural expression of digital image. In the process of digital image action execution, the system not only needs to consider the mapping between semantic instructions and action intention, but also needs to ensure that the actual execution trajectory meets the human cognition in terms of time continuity, posture smoothness and physical accessibility. Traditional reinforcement learning has modeling difficulties in nonlinear system control, and it is difficult to effectively handle the potential structural constraints and dynamic coupling relationship in the action state space. Therefore, how to combine the control method with strong explainability and robust modeling ability to make up for the instability in the traditional reinforcement learning control stage has become an important direction of current research.

[0006] Koopman operator theory provides strong support for solving such nonlinear system control problems. By constructing a state mapping function to lift the original nonlinear system to a high-dimensional subspace, its linear modeling and control are realized, and the Koopman method can provide predictable and optimized control structure while preserving the dynamic characteristics of the original system. Existing researches have used Koopman models for robot control, traffic system optimization and other fields, and have achieved certain results, but there is still a lack of systematic modeling framework in the dynamic action control of digital image. Especially in the interactive system driven by multiple goals, semantic instructions and user behavior, how to construct a goal-oriented Koopman control model to make it deeply integrated with reinforcement learning structure and form a closed-loop feedback mechanism between action intention analysis, trajectory optimization and strategy update is still a challenging key issue.

[0007] In summary, the existing technology has the problems of insufficient semantic sub-goal modeling capability, difficulty in expressing user differentiated intention with hierarchical strategy structure, lack of controllability and explainability in the control stage, loose strategy update mechanism, etc. when realizing the dynamic adjustment of digital image action, which cannot meet the high-quality, personalized and natural smooth action control demand in human-computer interaction. Therefore, it is urgent to propose a new method that integrates improved hierarchical reinforcement learning and Koopman linear subspace control, establish a complete technical path integrating semantic goal driving, individual style regulation, action atlas generation and trajectory optimization control, and improve the dynamic action expression ability and adaptability of digital image in multi-scene, multi-user and multi-task interaction. SUMMARY

[0008] To solve the problems in the prior art, the application provides a digital image action dynamic adjustment method based on reinforcement learning, which combines an improved hierarchical reinforcement learning structure and a Koopman linearization subspace control model, and aims at the problems of lack of semantic understanding, lack of style regulation and poor trajectory continuity of existing digital image actions, constructs a semantic sub-target driven high-low layer strategy network, a style regulation mechanism, a graph structure action skill library and a controllable action trajectory optimization control model, and systematically realizes personalized action generation and dynamic adjustment for user intentions. The method has the advantages of strong semantic understanding ability, high style adaptability, natural action connection, high trajectory control precision and strong strategy adaptive updating capability, and is suitable for various interactive application scenarios such as virtual human interaction, virtual customer service and immersive digital scene.

[0009] The technical scheme of the application is as follows:

[0010] A digital image action dynamic adjustment method based on reinforcement learning comprises the following steps:

[0011] Multi-source interaction data between a user and a digital image is collected, semantic analysis is performed on the multi-source interaction data, a semantic sub-target vector is generated, and a target granularity score value is calculated according to the semantic sub-target vector;

[0012] An improved hierarchical reinforcement learning structure is constructed according to the target granularity score value, the improved hierarchical reinforcement learning structure comprises a high layer strategy network, a style regulation controller and a low layer action strategy network, a sub-target intention embedding vector is generated through the high layer strategy network, a style regulation vector is generated through the style regulation controller, the sub-target intention embedding vector and the style regulation vector are jointly input into the low layer action strategy network, and an action trajectory reference path is output;

[0013] A graph structure action skill library is constructed, and the adjacency structure of the graph structure action skill library is dynamically adjusted through a graph attention mechanism according to the semantic sub-target vector and the action trajectory reference path, and an action combination candidate graph is generated;

[0014] A Koopman linearization subspace control model is constructed, an observable mapping function is established by using training data, and a linear system model in a Koopman subspace is obtained by training based on a subspace state vector and a control input;

[0015] Trajectory optimization is performed in the Koopman subspace according to the action trajectory reference path and the action combination candidate graph, an optimal subspace action control sequence is generated, the optimal subspace action control sequence is input into a decoder reflector and reflected to an original action state space, and continuous action execution instructions of the digital image are output;

[0016] Collect the environment feedback data after the continuous action execution instruction is executed, update the parameters according to the environment feedback data, and update the graph structure action skill library and the linear system model at the same time, and realize the policy iteration optimization of the improved hierarchical reinforcement learning structure;

[0017] Save the updated high-level policy network, style regulator, low-level action policy network, graph structure action skill library and Koopman linearization subspace control model as a digital image behavior package.

[0018] Further, the multi-source interaction data between the user and the digital image is collected, the multi-source interaction data is semantically analyzed to generate a semantic sub-target vector, and a target granularity score value is calculated according to the semantic sub-target vector, specifically:

[0019] Collecting multi-source interaction data between the user and the digital image, the multi-source interaction data including natural language instructions, scene labels, user behavior data and historical response records;

[0020] Uniformly format the multi-source interaction data to construct a semantic analysis input vector;

[0021] Input the semantic analysis input vector into the semantic analysis network, extract the context semantic features, and generate the semantic sub-target vector;

[0022] Calculate the target granularity score value based on the semantic complexity, instruction abstraction degree and behavior expectation explicitness in the semantic sub-target vector.

[0023] Further, the high-level policy network in the improved hierarchical reinforcement learning structure receives the semantic sub-target vector and the target granularity score value as input, generates a sub-target intention embedding vector using a dynamic hierarchical mechanism, and the dynamic hierarchical mechanism adaptively adjusts the number of sub-targets and the abstraction level of the high-level policy output according to the target granularity score value.

[0024] The style regulator receives the sub-target intention embedding vector, the user personality parameter and the context emotion label, and generates a style regulation vector using a fusion mapping network, the fusion mapping network combines the personality and emotion factors by attention mechanism, realizes the personalized adjustment of the action style and behavior trend;

[0025] The low-level action policy network receives the splicing features of the sub-target intention embedding vector and the style regulation vector as input, and outputs a continuous action trajectory reference path based on the conditional policy generation structure, the action trajectory reference path is composed of multiple time step target state points, and the action combination is generated.

[0026] Further, the improved hierarchical reinforcement learning structure is constructed according to the target granularity score value, the improved hierarchical reinforcement learning structure includes a high-level policy network, a style regulator and a low-level action policy network, a sub-target intention embedding vector is generated through the high-level policy network, a style regulation vector is generated through the style regulator, the sub-target intention embedding vector and the style regulation vector are jointly input into the low-level action policy network, and an action trajectory reference path is output, and specifically:

[0027] The semantic sub-target vector is input into the high-level policy network, and a sub-target intention embedding vector is output;

[0028] User individuality parameters are generated based on the user behavior data and historical response records, the user individuality parameters include action style preference, response rhythm tendency and risk tolerance feature;

[0029] Context emotion labels are constructed in combination with natural language instructions and user behavior feedback in the current interaction round, the context emotion labels reflect emotional orientation and emotional intensity under the current interaction state;

[0030] The sub-target intention embedding vector, the user individuality parameters and the context emotion labels are input into the style regulator to generate a style regulation vector;

[0031] The sub-target intention embedding vector and the style regulation vector are spliced to form a fusion feature representation, which is input into the low-level action policy network to output an action trajectory reference path.

[0032] Further, the graph structure action skill library is constructed, the adjacency structure of the graph structure action skill library is dynamically adjusted based on the semantic sub-target vector and the action trajectory reference path through a graph attention mechanism, and an action combination candidate graph is generated, and specifically:

[0033] The graph structure action skill library is constructed based on a predefined set of atomic action skills, each atomic action skill is represented as a node of a graph, and a set of edges between nodes is constructed according to historical behavior data to calculate state transition probabilities between atomic actions;

[0034] Initialization is performed on the graph structure action skill library to form an adjacency matrix with associated weights, describing the transition strength and reachable relationship between action skills;

[0035] The semantic sub-target vector and the action trajectory reference path are input into a graph attention mechanism module as joint regulation information, the adjacency matrix is dynamically updated, and the edge weights are adjusted to highlight the action transition paths related to the current semantic target;

[0036] Based on the updated graph structure action skill library, combination search is performed in the candidate action nodes to generate an action combination candidate graph, and a selectable skill sequence is provided.

[0037] Further, the Koopman linearization subspace control model is constructed, an observable mapping function is established using training data, a linear system model in the Koopman subspace is obtained based on a subspace state vector and a control input, and specifically,

[0038] A target-oriented Koopman linearization subspace control model is constructed, which includes an observable mapping function module and a target-enhanced subspace linear system modeling module.

[0039] An original action state vector sequence, a control input sequence and a corresponding semantic sub-target vector sequence are collected, the original action state vector is input into the observable mapping function module composed of a multi-layer neural network to generate a subspace state vector.

[0040] The subspace state vector, the control input sequence and the semantic sub-target vector are input into the target-enhanced subspace linear modeling module to fit and construct a linear system including a state transition matrix, a control input mapping matrix and a target guide mapping matrix.

[0041] The model is trained and evaluated based on trajectory reconstruction error, semantic response consistency index and system stability constraint, and when each performance index meets a preset threshold, the final target-oriented Koopman linearization subspace control model parameters are determined and solidified as a subspace prediction model in the action trajectory optimization stage.

[0042] Further, the trajectory optimization is performed in the Koopman subspace according to the action trajectory reference path and the action combination candidate atlas to generate an optimal subspace action control sequence, which is input into a decoder to be mapped to the original action state space to output continuous action execution instructions of the digital image.

[0043] Based on the Koopman linearization subspace control model, the action trajectory reference path and the action combination candidate atlas are mapped to corresponding subspace state vector sequences and control input sequences.

[0044] In the Koopman linearization subspace control model, optimal control trajectory solving is performed to minimize trajectory deviation loss and control cost function, and an optimal subspace action control sequence is generated.

[0045] The optimal subspace control sequence and the corresponding sub-target vector are input into the Koopman linearization subspace control model to predict an optimal subspace state sequence.

[0046] The optimal subspace state sequence is mapped back to the action state sequence in the original action state space using the trained decoder function, and continuous action execution instructions of the digital image are generated.

[0047] Further, the environment feedback data collected after the execution of the continuous action execution instruction is used to update the parameters, and the graph structure action skill library and the linear system model are updated at the same time, so that the policy iteration optimization of the improved hierarchical reinforcement learning structure is realized, specifically as follows:

[0048] The environment feedback data collected after the execution of the continuous action execution instruction includes user response characteristics, behavior result states and action naturalness indicators.

[0049] The environment feedback data and the multi-source interaction data are jointly used to construct an experience replay dataset, and the network parameters of the high-level policy network, the style regulator and the low-level action policy network are updated respectively, and the reinforcement learning policy iteration training is performed through the policy gradient optimization method.

[0050] Based on the observed action deviation and trajectory response error in the environment feedback data, the node transition weight and the adjacent structure in the graph structure action skill library are adaptively adjusted by using the minimum reconstruction error criterion.

[0051] The state-control input-target vector triple sample in the latest environment feedback sample is used to incrementally update the state transition matrix, control mapping matrix and target guiding matrix of the Koopman linearization subspace control model, improve the adaptability of the control model to the change of user's intention, and realize the dynamic optimization of the improved hierarchical reinforcement learning structure.

[0052] Further, the updated high-level policy network, style regulator, low-level action policy network, graph structure action skill library and Koopman linearization subspace control model are saved as a digital figure behavior package, specifically as follows:

[0053] The updated high-level policy network, style regulator, low-level action policy network, graph structure action skill library and Koopman linearization subspace control model are uniformly encapsulated to construct a digital figure behavior package.

[0054] The parameters, training weights, structure topologies and input / output interfaces of each module in the digital figure behavior package are archived and encoded to form a standardized model file that can be loaded and deployed.

[0055] The digital figure behavior package is loaded in a new interaction scene, new multi-source interaction data is received, action strategy reasoning and control execution are completed through the improved hierarchical reinforcement learning structure, and dynamic action adjustment tasks of digital figures in cross-scene are supported.

[0056] Compared with the prior art, the present application has the following beneficial effects:

[0057] The application breaks through the multiple bottlenecks of the traditional digital interactive system in the aspects of action generation continuity, individual expression accuracy, semantic intention adaptability and execution control stability, realizes dynamic action planning, optimization and adaptive execution facing user personalized needs and context scene changes, and has high intelligence and wide adaptability.

[0058] Firstly, the application makes key innovations in the aspects of semantic understanding and action generation driving mechanism. By collecting multi-source interactive data such as natural language instructions between users and digital images, scene labels, user behavior data and historical response records, and fusing semantic analysis and target granularity scoring mechanism, the complex and multi-modal user input is effectively converted into a structured semantic sub-target vector. The semantic sub-target vector not only clearly defines the target intention of digital image action planning, but also provides fine and continuous target driving basis for subsequent hierarchical strategy modeling, solving the problems of rough target analysis and unclear intention transmission in traditional methods.

[0059] Secondly, the application proposes an improved hierarchical reinforcement learning method with an innovative structure. The high-level strategy network generates a sub-target intention embedding representation based on the semantic sub-target vector, combines a style regulator constructed by user historical behavior, individual parameters and context emotion labels, and realizes individualized style control. The low-level action strategy network fuses the intention embedding and the style regulation vector, and outputs a continuous action trajectory reference path, realizing the linkage and adaptation of strategy execution level and expression style. The hierarchical structure not only realizes the decoupling of target levels, but also enhances the controllability and interpretability of strategy generation, effectively dealing with the problems of style mutation and strategy confusion in multi-scene dynamic interaction.

[0060] In addition, the application adopts the design of a graph structure action skill library in the action skill modeling and combination mechanism, models atomic action skills as graph structure nodes, constructs the transition probability between edges through historical behavior data, and adjusts the adjacency structure in real time based on the graph attention mechanism, thereby constructing a dynamic action combination candidate graph. This structure fully utilizes the experience accumulation of historical execution data, realizes the structurization, modularization and combination flexibility of action generation, so that the digital image can quickly combine a semantic consistent and natural connected action sequence under the driving of different semantic sub-targets.

[0061] Further, the present application introduces a Koopman linearization subspace control model to solve the problem of high computational complexity and low trajectory control accuracy of traditional action control models in nonlinear dynamic process modeling. By constructing an observable mapping function to embed the original action state into the Koopman subspace, a linear dynamic system is used to complete the action control strategy modeling in the subspace, and trajectory optimization and control input reasoning are performed, and finally the original action space is reflected back through the decoder, realizing the accurate mapping of control input to physical action. This method greatly improves the operation efficiency and execution stability of the system in complex nonlinear control scenarios, and has good engineering deployability.

[0062] Finally, in terms of strategy optimization, the present application collects environmental feedback data after continuous action execution instructions, including user response characteristics, behavior result state and action naturalness indicators, dynamically updates high-level strategy network, style regulator, low-level action strategy network and Koopman control model parameters, and forms a self-adaptive closed-loop strategy updating mechanism. This mechanism realizes the co-evolution of strategy and control model, not only can continuously improve the interaction quality according to user feedback, but also has the ability of continuous learning and migration, and the ability to adapt to new scenarios and new users is significantly enhanced. BRIEF DESCRIPTION OF DRAWINGS

[0063] Figure 1 Fig. 1 is a flowchart of a digital figure action dynamic adjustment method based on reinforcement learning;

[0064] Figure 2 Fig. 3 is a schematic diagram of an improved hierarchical reinforcement learning structure. DETAILED DESCRIPTION

[0065] The present application will be further illustrated below in conjunction with the drawings and specific embodiments, and it should be understood that these embodiments are only used to illustrate the present application and not to limit the scope of the present application. After reading the present application, those skilled in the art can make various modifications to the present application, and all such modifications fall within the scope defined by the appended claims.

[0066] Embodiment one:

[0067] A digital figure action dynamic adjustment method based on reinforcement learning of the present application, as shown in Figure 1 and Figure 2 , includes:

[0068] Step 1) Collecting multi-source interaction data between the user and the digital figure, performing semantic analysis on the multi-source interaction data, generating a semantic sub-target vector, and calculating a target granularity score value according to the semantic sub-target vector;

[0069] Step 2) Construct an improved hierarchical reinforcement learning structure according to the target granularity score value, the improved hierarchical reinforcement learning structure including a high-level policy network, a style regulator, and a low-level action policy network, generate a sub-target intention embedding vector through the high-level policy network and a style regulation vector through the style regulator, input the sub-target intention embedding vector and the style regulation vector into the low-level action policy network together, and output an action trajectory reference path;

[0070] Step 3) Construct a graph structure action skill library, dynamically adjust the adjacency structure of the graph structure action skill library according to the semantic sub-target vector and the action trajectory reference path through a graph attention mechanism, and generate an action combination candidate graph;

[0071] Step 4) Construct a Koopman linearization subspace control model, establish an observable mapping function using training data, and train a linear system model in the Koopman subspace based on a subspace state vector and a control input;

[0072] Step 5) Perform trajectory optimization in the Koopman subspace according to the action trajectory reference path and the action combination candidate graph, generate an optimal subspace action control sequence, input the optimal subspace action control sequence into a decoder to reflect the original action state space, and output continuous action execution instructions for a digital figure;

[0073] Step 6) Collect environment feedback data after the continuous action execution instructions are executed, update parameters according to the environment feedback data, and simultaneously update the graph structure action skill library and the linear system model, and realize policy iterative optimization of the improved hierarchical reinforcement learning structure;

[0074] Step 7) Save the updated high-level policy network, style regulator, low-level action policy network, graph structure action skill library, and Koopman linearization subspace control model as a digital figure behavior package.

[0075] The application can accurately extract semantic sub-target vectors by collecting multi-source interaction data between the user and the digital image and performing semantic analysis, and dynamically adjust the target abstraction level according to the target granularity score value, thereby realizing fine modeling of user intent and context awareness. The improved hierarchical reinforcement learning structure models the user's personality and emotional state in combination with the style regulator, realizes personalized adjustment and natural expression of action style, and improves the coordination and emotional fit of the interaction response. Through the graph attention mechanism, the adjacency structure of the graph structure action skill library is dynamically updated, the action combination path can be flexibly adjusted according to the current semantic target and the trajectory reference path, and the diversity and adaptability of complex action planning are enhanced. The Koopman linearization subspace control model introduces an observable mapping and linear modeling mechanism, effectively improving the computational efficiency and trajectory generation accuracy in nonlinear behavior control tasks, and outputting high continuity and consistency of digital image action instructions through trajectory optimization and inverse mapping. The strategy network and control model are iteratively optimized in combination with environmental feedback data, and a reinforcement learning closed-loop system with adaptive ability is constructed, realizing dynamic adjustment and generalization enhancement of the strategy in long-term interaction. Finally, the trained core model module is encapsulated as a digital image behavior package, supporting cross-scene deployment and strategy migration, significantly improving the practicality, expandability and intelligent level of the digital image system.

[0076] In one embodiment, multi-source interaction data between a user and a digital image is collected, the multi-source interaction data is semantically analyzed to generate a semantic sub-target vector, and a target granularity score value is calculated based on the semantic sub-target vector, specifically:

[0077] Multi-source interaction data between a user and a digital image is collected, and the multi-source interaction data includes natural language instructions, scene labels, user behavior data, and historical response records.

[0078] The multi-source interaction data is uniformly formatted to construct a semantic analysis input vector.

[0079] The semantic analysis input vector is input into a semantic analysis network to extract context semantic features and generate a semantic sub-target vector.

[0080] A target granularity score value is calculated based on the semantic complexity, instruction abstraction degree, and behavior expectation explicitness in the semantic sub-target vector.

[0081] By collecting multi-source interaction data including natural language instructions, scene labels, user behavior data and historical response records, the interaction context between the user and the digital avatar can be fully captured, and the breadth and depth of semantic understanding can be enhanced. The multi-modal data is uniformly formatted and the semantic analysis input vector is constructed, so that the semantic analysis network can fully integrate multiple data types, improve the consistency and accuracy of semantic modeling. The context semantic features extracted by the semantic analysis network have stronger instruction understanding ability and scene association ability, which helps to generate semantic sub-target vector that is more in line with the actual intention. By introducing multi-dimensional indicators such as semantic complexity, instruction abstraction degree and behavior expectation clarity to evaluate the semantic sub-target vector, the target granularity can be quantitatively modeled, and the system can adaptively adjust the target splitting strategy, providing more accurate target guidance and abstract level support for the hierarchical strategy generation and action control of the subsequent reinforcement learning structure, thereby significantly improving the response flexibility and task completion degree of the digital avatar in complex and variable interaction scenarios.

[0082] In one embodiment, the high-level policy network in the improved hierarchical reinforcement learning structure receives the semantic sub-target vector and the target granularity score value as input, generates a sub-target intention embedding vector using a dynamic hierarchical mechanism, and dynamically adjusts the number of sub-targets and the abstraction level of the high-level policy output according to the target granularity score value.

[0083] The style regulator receives the sub-target intention embedding vector, the user personality parameter and the context emotion label, generates a style regulation vector using a fusion mapping network, and realizes personalized adjustment of action style and behavior trend by weighting and combining personality and emotion factors through attention mechanism.

[0084] The low-level action policy network receives the spliced features of the sub-target intention embedding vector and the style regulation vector as input, outputs a continuous action trajectory reference path based on the conditional policy generation structure, and the action trajectory reference path is composed of multiple time step target state points to guide action combination generation.

[0085] The improved hierarchical reinforcement learning structure plays a key role in enhancing the dynamic adjustment process of digital figure actions. First, the high-level policy network introduces a dynamic hierarchical mechanism by combining the semantic sub-goal vector and the target granularity score value, enabling the policy output to adaptively adjust the abstraction level and number of sub-goals, thereby enhancing the target understanding ability for different semantic complexity, significantly improving the response accuracy and understanding precision of the system in processing ambiguous expressions or multi-target nested instructions. Second, the style regulator embeds the sub-goal intention vector into the user personality parameters and context emotion labels through the fusion mapping network, introducing personalized and emotional regulation mechanisms before action generation, enabling the system to generate actions that are more natural, lively, and in line with user preferences, enhancing the personalization of digital figures and the interactive affinity. In addition, the low-level action policy network generates a structured action trajectory reference path based on the conditional policy structure after receiving the fused feature input, ensuring the logical continuity and dynamic adaptability of the action generation process, effectively guiding the subsequent combination generation operation of the graph structure action skill library. In summary, the improved structure enhances the flexibility, personalization, and context adaptability of digital figure action planning through high and low level policy coordination, sub-goal level adaptive adjustment, and emotional style personalized fusion control, improving the accuracy, coherence, and user satisfaction of overall action expression.

[0086] In one embodiment, an improved hierarchical reinforcement learning structure is constructed according to the target granularity score value, the improved hierarchical reinforcement learning structure including a high-level policy network, a style regulator, and a low-level action policy network, the sub-goal intention embedding vector is generated by the high-level policy network and the style regulation vector is generated by the style regulator, the sub-goal intention embedding vector and the style regulation vector are input into the low-level action policy network together, and the action trajectory reference path is output, specifically:

[0087] The semantic sub-goal vector is input into the high-level policy network, and the sub-goal intention embedding vector is output.

[0088] User personality parameters are generated based on user behavior data and historical response records, including action style preference, response rhythm tendency, and risk tolerance characteristics.

[0089] Context emotion labels are constructed by combining natural language instructions and user behavior feedback in the current interaction round, reflecting the emotional orientation and emotional intensity under the current interaction state.

[0090] The sub-goal intention embedding vector, user personality parameters, and context emotion labels are input into the style regulator to generate a style regulation vector.

[0091] The sub-target intention embedding vector is spliced with the style regulation vector to form a fusion feature representation, which is input into a low-level action policy network to output an action trajectory reference path.

[0092] The application realizes a closed-loop control mechanism from high-level semantic understanding to personalized action generation by introducing an improved hierarchical reinforcement learning structure guided by target granularity score values. First, the semantic sub-target vector is input into a high-level policy network to output a sub-target intention embedding vector, effectively realizing the abstraction and structuring of semantic targets, enabling the system to accurately extract the core of the intention when facing complex natural language expressions from users, and enhancing the recognition ability of task-driven. Second, by statistical analysis of user behavior data and historical response records, action style preferences, response rhythm tendencies and risk tolerance characteristics are extracted to form user personality parameters, enabling the system to identify and remember the interaction characteristics of users in different scenarios and establish a long-term user portrait. Further, context emotion labels are constructed by combining the natural language instructions in the current round and user behavior feedback, thereby capturing the emotional fluctuations and emotional intensity of the user in the current interaction process, realizing emotion perception and real-time adaptation. Subsequently, the sub-target intention embedding vector, user personality parameters and context emotion labels are input into a style regulator, and the generated style regulation vector adjusts the weighted fusion of multiple source features, making the action style more consistent with the individual characteristics and current emotional state of the user in emotional expression and rhythm control. Finally, the fusion of the sub-target intention embedding vector and the style regulation vector forms the low-level policy input, realizing the accurate output of the action trajectory reference path and improving the continuity, flexibility and personalized expression effect of action generation. Overall, this embodiment strengthens the coupling ability of semantic understanding and personality modeling, enabling the action decision-making process to maintain semantic accuracy while considering individual preferences and emotional adaptation, thereby significantly improving the naturalness of digital images in multi-round human-computer interaction, user satisfaction and interactive immersion.

[0093] In one embodiment, a graph structure action skill library is constructed, and the adjacency structure of the graph structure action skill library is dynamically adjusted according to the semantic sub-target vector and the action trajectory reference path through a graph attention mechanism to generate an action combination candidate graph, specifically:

[0094] Based on a pre-defined set of atomic action skills, each atomic action skill is represented as a node of a graph, and the state transition probabilities between atomic actions are calculated based on historical behavior data to construct a set of edges between nodes;

[0095] Initialization is performed on the graph structure action skill library to form an adjacency matrix with associated weights, describing the transition strength and reachable relationship between action skills;

[0096] The semantic sub-target vector and the action trajectory reference path are input into a graph attention mechanism module as joint regulation information, the adjacency matrix is dynamically updated, and the edge weight is adjusted to highlight the action transition path related to the current semantic target.

[0097] Based on the updated graph structure action skill library, combination search is performed in the candidate action node to generate an action combination candidate graph, and a selectable skill sequence is provided.

[0098] The semantic sub-target vector and the action trajectory reference path are input into a graph attention mechanism module as joint regulation information, the adjacency matrix is dynamically updated, and the edge weight is adjusted to highlight the action transition path related to the current semantic target.

[0099] In one embodiment, a Koopman linearization subspace control model is constructed, an observable mapping function is established using training data, a linear system model in the Koopman subspace is obtained based on the subspace state vector and the control input, and specifically:

[0100] A target-oriented Koopman linearization subspace control model is constructed, which includes an observable mapping function module and a target-enhanced subspace linear system modeling module.

[0101] The original action state vector sequence, the control input sequence and the corresponding semantic sub-target vector sequence are collected, the original action state vector is input into the observable mapping function module composed of a multi-layer neural network to generate a subspace state vector;

[0102] The subspace state vector, the control input sequence and the semantic sub-target vector are input into the target enhanced subspace linear modeling module to fit and construct a linear system including a state transition matrix, a control input mapping matrix and a target guide mapping matrix;

[0103] Based on the trajectory reconstruction error, the semantic response consistency index and the system stability constraint, the model is trained and evaluated, and when the performance indicators meet the preset threshold, the final target-oriented Koopman linearization subspace control model parameters are determined and solidified as the subspace prediction model in the action trajectory optimization stage.

[0104] The present application realizes accurate modeling and efficient prediction of the action state of a digital image by introducing a target-oriented Koopman linearization subspace control model. Based on the original action state vector, the control input sequence and the semantic sub-target vector, the nonlinear original state is converted into a subspace state with linear characteristics by constructing an observable mapping function. This linear mapping not only improves the stability of the model in the state expression process, but also provides clear structural support for the design and optimization of the subsequent controller. During modeling, the system integrates semantic sub-target information and constructs a target enhanced linear subspace system, so that the control model can not only express the relationship between the state and the control input, but also capture the guiding effect of user intention on action evolution. This fusion method enables the digital image to exhibit higher semantic matching degree and action response flexibility when executing complex instructions. At the same time, the method introduces multi-dimensional performance indicators such as trajectory reconstruction error, semantic response consistency and system stability to train and evaluate the model, thereby ensuring the generalization ability and control accuracy of the model under different semantic targets. The Koopman linearization subspace control model finally trained can be embedded in the action trajectory optimization module for use, so that the system can stably generate a control sequence with high adaptability and continuity to the target during action execution, avoiding the control drift and action break phenomenon caused by high difficulty in nonlinear modeling in traditional methods, and significantly improving the intelligent performance of virtual characters in dynamic interaction scenarios and the user interaction experience.

[0105] In one embodiment, trajectory optimization is performed in the Koopman subspace according to the action trajectory reference path and the action combination candidate atlas to generate an optimal subspace action control sequence, which is input into the decoder to reflect the original action state space, and the continuous action execution instruction of the digital image is output, specifically:

[0106] Based on the Koopman linearization subspace control model, the action trajectory reference path and the action combination candidate atlas are mapped to the corresponding subspace state vector sequence and control input sequence;

[0107] In the Koopman linearization subspace control model, the optimal control trajectory solving is performed to minimize the trajectory deviation loss and the control cost function, and the optimal subspace action control sequence is generated.

[0108] The optimal subspace control sequence and the corresponding subspace target vector are input into the Koopman linearization subspace control model to predict the optimal subspace state sequence.

[0109] The trained decoder function is used to perform inverse mapping on the optimal subspace state sequence to restore it to the action state sequence in the original action state space, and the continuous action execution instruction of the digital image is generated.

[0110] The present application jointly optimizes the action trajectory reference path and the action combination candidate atlas by means of the Koopman linearization subspace control model, effectively improving the coherence and response accuracy of the digital image in the dynamic scene. By mapping the action trajectory reference path and the action combination candidate atlas into state vector sequences and control input sequences in the Koopman subspace, the system can perform optimal control solving in a subspace with linear dynamic characteristics. Compared with traditional nonlinear control methods, the solving complexity and computational overhead are greatly reduced, and the interpretability and convergence stability of the optimization results are enhanced. In the trajectory optimization process, the optimization strategy of minimizing the trajectory deviation loss and the control cost as the core target makes the generated action control sequence accurately fit the original intention path, while avoiding redundant control and enhancing system execution efficiency. Combined with the auxiliary input of the subspace target vector, the response matching ability of the optimization result to the semantic target is further enhanced, ensuring that the digital image presents a high consistency of semantic behavior trend during task execution. After completing the trajectory optimization, the system performs inverse mapping on the optimal subspace state sequence through the trained decoder to restore it to the continuous action sequence in the original action space, and finally outputs the continuous action execution instruction of the digital image. This method not only ensures the physical executability and semantic coherence of the action sequence, but also significantly improves the naturalness and coordination of the action generation, avoiding the problems of action breakage, response delay and semantic deviation in traditional methods, and showing stronger action intelligence and user adaptability in multiple rounds of complex interaction.

[0111] In one embodiment, the environment feedback data after collecting the continuous action execution instruction is executed, the parameters are updated according to the environment feedback data, and the graph structure action skill library and the linear system model are updated at the same time, realizing the policy iteration optimization of the improved hierarchical reinforcement learning structure, specifically:

[0112] Collecting environment feedback data after the execution of continuous action execution instructions, the environment feedback data including user response features, behavior result states and action naturalness indicators;

[0113] Jointly constructing an experience replay dataset from the environment feedback data and the multi-source interaction data, updating the network parameters of the high-level policy network, the style regulator and the low-level action policy network respectively, and performing reinforcement learning policy iteration training through a policy gradient optimization method;

[0114] Based on the observed action bias and trajectory response error in the environment feedback data, the node transition weights and adjacency structure in the graph structure action skill library are adaptively adjusted using the minimum reconstruction error criterion;

[0115] Synchronously using the triple sample of state-control input-target vector in the latest environment feedback sample, the state transition matrix, control mapping matrix and target guide matrix of the Koopman linearization subspace control model are incrementally updated to improve the adaptability of the control model to user intent changes, and dynamic optimization of the improved hierarchical reinforcement learning structure is realized.

[0116] By collecting environment feedback data after the execution of continuous action execution instructions, the system realizes dynamic updating of the policy network and the control model, significantly improving the adaptability and robustness of the digital image in complex interaction processes. The collected environment feedback data covers user response features, behavior result states and action naturalness indicators, which can fully reflect the matching effect of action execution on user intent and the behavior deviation in the interaction process. These data not only provide real and usable supervision signals for subsequent policy optimization, but also constitute a high-quality experience replay dataset, which helps the reinforcement learning model to converge stably in multiple rounds of interaction.

[0117] The application constructs an experience replay set by combining the feedback data with the original multi-source interaction data, and the system can update the parameters of the high-level policy network, the style regulator and the low-level action policy network at the same time, thereby enhancing the individual expression ability and target matching ability of the action policy at different abstraction levels. The iterative training is performed by using the policy gradient optimization method, so that the system can continuously adjust the action generation strategy to maximize the user response score and the interaction satisfaction, effectively avoiding the problem that the traditional fixed strategy is difficult to generalize. In addition, by analyzing the action deviation and trajectory response error in the environmental feedback, the adjacency structure of the graph structure action skill library is adaptively adjusted, and the connection weight between nodes can dynamically reflect the current interaction intention and action transfer trend, so that the generated action combination is more relevant and stable in execution, significantly improving the adaptability of the skill library to new targets and new scenes. At the same time, the system also uses the state-control input-target vector triplets in the feedback samples to incrementally update the key matrix of the linearized subspace control model, effectively enhancing the prediction ability and sensitivity to user behavior changes of the control model. The strategy optimization mechanism not only enables the improved hierarchical reinforcement learning structure to maintain accuracy and consistency under changing user intentions and environmental conditions, but also provides technical support for the long-term interaction learning ability of the digital avatar, making it perform excellently in multiple dimensions such as personalization, coherence and real-time performance.

[0118] In one embodiment, the updated high-level policy network, style regulator, low-level action policy network, graph structure action skill library and Koopman linearized subspace control model are saved as a digital avatar behavior package, specifically:

[0119] The updated high-level policy network, style regulator, low-level action policy network, graph structure action skill library and Koopman linearized subspace control model are uniformly encapsulated to build a digital avatar behavior package;

[0120] The parameters, training weights, structure topology and input / output interfaces of each module in the digital avatar behavior package are archived and encoded to form a standardized model file that can be loaded and deployed;

[0121] The digital avatar behavior package is loaded in a new interaction scenario, new multi-source interaction data is received, and action policy reasoning and control execution are completed through the improved hierarchical reinforcement learning structure, supporting dynamic action adjustment tasks of digital avatars across scenarios.

[0122] The updated high-level policy network, the style regulator, the low-level action policy network, the graph structure action skill library and the Koopman linearization subspace control model are uniformly packaged as a digital image behavior package, which not only realizes the modular management and efficient calling of each module, but also significantly improves the migration and deployment efficiency of the system in different interactive scenarios. By uniformly archiving and coding the configuration parameters, training weights, structure topology and input and output interfaces of each component module in the behavior package, the system can generate standardized and portable model files, which are convenient for quick loading and version control, and provide basic support for synchronous deployment between multiple devices and large-scale promotion. As an extensible agent capability carrier, the digital image behavior package enables the system to perform action strategy reasoning and control output without retraining when facing new users, new semantic targets or new interactive situations, thereby significantly shortening the response delay and computational overhead and improving the real-time performance and stability of user experience. In addition, the structure also has good sustainable iteration capability, and through continuous collection of feedback data in the interactive process and completion of policy updating, periodic reconstruction and performance optimization of the behavior package can be realized, ensuring the continuous intelligent evolution of the digital image in the long-term use process. More importantly, the behavior package saving mechanism enables the digital image to have the ability of "memory" and "habit", and in future scenarios, it can perform differential transfer learning based on historical behavior packages to generate action responses with individual style, target sensitivity and dynamic adaptation capability, greatly improving the individuality and generalization capability of the system. Therefore, the behavior package mechanism provides key basic support for the dynamic adjustment of digital image actions in multi-scene, multi-user and multi-task environments, and has significant engineering promotion value and application prospect.

[0123] In order to verify the feasibility of the application in implementation, the application is applied to a virtual character driving platform in a child voice interactive education scene. The platform faces children aged 5 to 10, and provides a function module of multi-round voice interactive teaching with a "digital image teacher", including English oral practice, picture book explanation, behavior guidance and interactive game content. In actual application, the traditional preset animation template and fixed action script have problems such as rigid response action, lack of natural emotion expression and difficulty in adapting to different semantic targets. Especially when facing complex or emotional instructions, the response of the digital image is significantly mismatched, affecting the immersion and interaction enthusiasm of children. The application realizes adaptive action generation for semantic sub-targets in this scene through the action dynamic adjustment mechanism driven by reinforcement learning, which significantly improves the naturalness and accuracy of interaction.

[0124] In the test phase, 8 primary schools in a certain area were selected, and a total of 320 students were distributed in two groups for the experiment. Among them, 160 people in the control group used the old version system without the technology of the application, and 160 people in the experiment group used the improved platform system integrating the application. The interactive content includes three types of tasks: "English dialogue question and answer", "rhythm game interaction" and "safety education simulation". The system collects various data such as user voice input, response semantics, interaction rounds, action execution feedback, child expression emotion labeling and teacher scores in the process of multiple interactions.

[0125] For example, in the "English dialogue question and answer" task, when the child inputs "Can you show me how to say thank you in English?", the traditional system only plays the recording and makes a mechanical nodding action, while the system using the application analyzes the emotional intent in the instruction, identifies it as "request demonstration + emotional expression", dynamically generates a composite behavior chain containing voice imitation and semantic emphasis actions, and generates soft tone and smile expression through the style regulation mechanism according to the user's past preferences, making the overall performance more affable. In "rhythm game interaction", after the child inputs "Let's dance faster!", the experimental group system can dynamically identify the "speed enhancement" intent based on the target granularity score value, and adjust the edge weight between nodes in the graph structure action skill library, so that the generated action is more rhythmic and synchronous. After using the Koopman subspace control model, the continuity score of the action transition is improved.

[0126] The specific experimental results are shown in the following table:

[0127] Table 1 Performance comparison table of experimental group and control group system in different task scenarios

[0128] Item Control group performance Experimental group performance Promotion range Average motion response time (seconds) 2.1 1.4 -33.3% Action semantic adaptation rate 61.2% 87.6% +43.2% Motion trajectory continuity score 6.7 (full score 10) 8.2 (full score 10) +22.4% Average trajectory offset 1.15 (unit: motion coding space distance) 0.75 (unit: motion coding space distance) -35.2% Environmental feedback convergence rounds 17.8 rounds 10.5 rounds -41.0% Interaction satisfaction survey (children) Satisfaction ratio 65.3% Satisfaction ratio 93.1% +42.5% Teacher assessment recognition 68.6% 89.4% +30.3%

[0129] As can be seen from Table 1, the experimental group using the method of the application is significantly better than the control group in multiple key performance indicators. First, in terms of action response time, the experimental group is 1.4 seconds on average, which is about 33.3% shorter than the control group, indicating that the improved hierarchical reinforcement learning structure combined with sub-target analysis and style regulation mechanism can quickly generate personalized action response paths, improving the real-time performance of the system. Second, the action semantic adaptation rate increased from 61.2% to 87.6%, indicating that the action generation method driven by semantic sub-target vector can more accurately correspond to the deep intent of user input, significantly enhancing the naturalness of interaction.

[0130] In addition, the continuity score of the action trajectory is improved from 6.7 to 8.2, and the average trajectory offset is reduced by 35.2%, verifying the effectiveness of the Koopman linearization subspace control model in the smoothness and stability of action generation. At the same time, the convergence round of the policy network is also shortened from 17.8 rounds to 10.5 rounds, with a decline of 41%, reflecting that the reinforcement learning structure of the present application has faster policy optimization capability under the driving of environmental feedback. The synchronous improvement of user subjective satisfaction and teacher recognition (respectively +42.5% and +30.3%) further verifies the practicability and innovation of the present application from the actual experience level.

[0131] The present application not only systematically enhances the traditional digital image control system in the theoretical structure, but also shows excellent dynamic response ability, personalized matching ability and system robustness in practical application, fully meeting the intelligent control needs of digital image with high naturalness and strong interaction. The method has wide popularization prospect, especially suitable for multi-type semantic driven interactive scenes such as education accompanying, virtual assistant, rehabilitation training and the like.

[0132] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A method for dynamically adjusting the actions of a digital avatar based on reinforcement learning, characterized in that, include: Collect multi-source interaction data between users and digital avatars, perform semantic parsing on the multi-source interaction data, generate semantic sub-target vectors, and calculate target granularity scores based on the semantic sub-target vectors; An improved hierarchical reinforcement learning structure is constructed based on the target granularity score. The improved hierarchical reinforcement learning structure includes a high-level policy network, a style regulator, and a low-level action policy network. The high-level policy network generates a sub-target intent embedding vector, and the style regulator generates a style regulation vector. The sub-target intent embedding vector and the style regulation vector are input into the low-level action policy network to output an action trajectory reference path. A graph-structured action skill library is constructed. Based on the semantic sub-target vector and the action trajectory reference path, the adjacency structure of the graph-structured action skill library is dynamically adjusted through a graph attention mechanism to generate a candidate graph of action combinations. Construct a Koopman linearized subspace control model, establish an observable mapping function using training data, and obtain a linear system model in the Koopman subspace based on the subspace state vector and control input. Based on the motion trajectory reference path and motion combination candidate map, trajectory optimization is performed in the Koopman subspace to generate the optimal subspace motion control sequence. The sequence is then input into the decoder and mapped back to the original motion state space, outputting a digital image of continuous motion execution instructions. Collect environmental feedback data after the execution of continuous action execution instructions, update parameters based on environmental feedback data, and simultaneously update the graph structure action skill library and linear system model to achieve iterative optimization of the strategy of the improved hierarchical reinforcement learning structure; Save the updated high-level strategy network, style regulator, low-level action strategy network, graph-structured action skill library, and Koopman linearized subspace control model as a digital character behavior package.

2. The method for dynamic adjustment of digital image actions based on reinforcement learning according to claim 1, characterized in that, The process involves collecting multi-source interaction data between users and digital avatars, performing semantic parsing on the multi-source interaction data to generate semantic sub-target vectors, and calculating target granularity scores based on the semantic sub-target vectors. Specifically: Collect multi-source interaction data between users and digital avatars, including natural language commands, scene tags, user behavior data, and historical response records; Perform unified formatting on multi-source interactive data to construct semantic parsing input vectors; The semantic parsing input vector is fed into the semantic parsing network to extract contextual semantic features and generate semantic sub-target vectors; The target granularity score is calculated based on the semantic complexity, instruction abstraction, and behavioral expectation clarity in the semantic sub-target vector.

3. The method for dynamic adjustment of digital image actions based on reinforcement learning according to claim 2, characterized in that, The high-level policy network in the improved hierarchical reinforcement learning structure receives semantic sub-target vectors and target granularity scores as inputs, and generates sub-target intent embedding vectors using a dynamic hierarchical mechanism. The dynamic hierarchical mechanism adaptively adjusts the number of sub-targets and the abstraction level of the high-level policy output according to the target granularity scores. The style modifier receives the sub-target intent embedding vector, user personality parameters, and contextual emotion labels. It uses a fusion mapping network to generate a style control vector. The fusion mapping network uses an attention mechanism to weight and combine personality and emotion factors to achieve personalized adjustment of action style and behavioral trends. The low-level action policy network receives the concatenated features of the sub-target intent embedding vector and style control vector as input, and outputs a continuous action trajectory reference path based on the conditional policy generation structure. The action trajectory reference path is composed of target state points at multiple time steps, which guides the generation of action combinations.

4. The method for dynamic adjustment of digital image actions based on reinforcement learning according to claim 3, characterized in that, The improved hierarchical reinforcement learning structure is constructed based on the target granularity score. This structure includes a high-level policy network, a style regulator, and a low-level action policy network. The high-level policy network generates a sub-target intent embedding vector, and the style regulator generates a style regulation vector. The sub-target intent embedding vector and the style regulation vector are input together into the low-level action policy network, which outputs a motion trajectory reference path. Specifically: Input the semantic sub-target vector into the high-level policy network, and output the sub-target intent embedding vector; User personality parameters are generated based on the user behavior data and historical response records. These user personality parameters include action style preferences, response rhythm tendencies, and risk tolerance characteristics. By combining natural language instructions and user behavior feedback in the current interaction round, a contextual emotion label is constructed, which reflects the emotional orientation and intensity of the current interaction state. The style modifier is input by embedding sub-target intent into a vector, user personality parameters, and upper and lower emotion tags to generate a style modifier vector. The sub-target intent embedding vector and style control vector are concatenated to form a fused feature representation, which is then input into the low-level action policy network and outputs an action trajectory reference path.

5. The method for dynamic adjustment of digital image actions based on reinforcement learning according to claim 4, characterized in that, The construction of the graph-structured action skill library involves dynamically adjusting the adjacency structure of the graph-structured action skill library based on semantic sub-target vectors and action trajectory reference paths using a graph attention mechanism to generate a candidate graph of action combinations. Specifically: A graph-structured action skill library is constructed based on a predefined set of atomic action skills. Each atomic action skill is represented as a node in the graph, and the state transition probability between atomic actions is calculated based on historical behavior data to construct a set of edges between nodes. Initialize the graph-structured action skill library to form an adjacency matrix with associated weights, which describes the transition strength and reachability relationship between action skills; The semantic sub-target vector and the action trajectory reference path are used as joint control information and input into the graph attention mechanism module to dynamically update the adjacency matrix and adjust the edge weights to highlight the action transfer path related to the current semantic target. Based on the updated graph-structured action skill library, a combination search is performed on candidate action nodes to generate an action combination candidate graph, providing optional skill sequences.

6. The method for dynamic adjustment of digital image actions based on reinforcement learning according to claim 5, characterized in that, The construction of the Koopman linearized subspace control model involves establishing an observable mapping function using training data, and training a linear system model in the Koopman subspace based on the subspace state vector and control input. Specifically: A goal-oriented Koopman linearized subspace control model is constructed, which includes an observable mapping function module and a goal-enhanced subspace linear system modeling module. The original action state vector sequence, control input sequence and corresponding semantic sub-target vector sequence are collected. The original action state vector is input into the observable mapping function module composed of a multi-layer neural network to generate subspace state vectors. The subspace state vector, control input sequence, and semantic subtarget vector are input into the target-enhanced subspace linear modeling module to fit and construct a linear system including a state transition matrix, a control input mapping matrix, and a target guidance mapping matrix; The model is trained and evaluated based on trajectory reconstruction error, semantic response consistency index and system stability constraints. When each performance index meets the preset threshold, the parameters of the final goal-oriented Koopman linearized subspace control model are determined and solidified into the subspace prediction model for the motion trajectory optimization stage.

7. The method for dynamic adjustment of digital image actions based on reinforcement learning according to claim 6, characterized in that, The process involves performing trajectory optimization in the Koopman subspace based on the action trajectory reference path and action combination candidate map, generating the optimal subspace action control sequence, inputting it into the decoder for inverse mapping to the original action state space, and outputting a digitally visualized continuous action execution instruction. Specifically: Based on the Koopman linearized subspace control model, the motion trajectory reference path and motion combination candidate map are mapped to the corresponding subspace state vector sequence and control input sequence. In the Koopman linearized subspace control model, the optimal control trajectory is solved to generate the optimal subspace motion control sequence with the goal of minimizing the trajectory offset loss and control cost function. The optimal subspace control sequence and the corresponding sub-objective vector are input into the Koopman linearized subspace control model to predict the optimal subspace state sequence. The trained decoder function is used to reverse map the optimal subspace state sequence to restore the action state sequence in the original action state space, and generate a digital image of continuous action execution instructions.

8. The method for dynamic adjustment of digital image actions based on reinforcement learning according to claim 7, characterized in that, The process involves collecting environmental feedback data after the execution of continuous action commands, updating parameters based on this data, and simultaneously updating the graph-structured action skill base and the linear system model. This enables iterative optimization of the strategy within the improved hierarchical reinforcement learning structure. Specifically: Collect environmental feedback data after the execution of the continuous action execution instructions. The environmental feedback data includes user response characteristics, behavior result status and action naturalness index. An experience playback dataset is jointly constructed by environmental feedback data and multi-source interaction data. The network parameters of the high-level policy network, style regulator and low-level action policy network are updated respectively. Reinforcement learning policy iterative training is performed by policy gradient optimization method. Based on the motion deviation and trajectory response error observed in the environmental feedback data, the node transfer weights and adjacency structure in the graph-structured motion skill library are adaptively adjusted by adopting the minimum reconstruction error criterion. By simultaneously utilizing the triplet samples of state-control input-target vector in the latest environmental feedback samples, the state transition matrix, control mapping matrix, and target guidance matrix of the Koopman linearized subspace control model are incrementally updated, thereby improving the control model's adaptability to changes in user intent and achieving dynamic optimization of the improved hierarchical reinforcement learning structure.

9. The method for dynamic adjustment of digital image actions based on reinforcement learning according to claim 8, characterized in that, The process of saving the updated high-level policy network, style regulator, low-level action policy network, graph-structured action skill library, and Koopman linearized subspace control model as a digital avatar behavior package specifically involves: The updated high-level strategy network, style regulator, low-level action strategy network, graph structure action skill library and Koopman linearized subspace control model are uniformly encapsulated to construct a digital image behavior package. The configuration parameters, training weights, structural topology, and input / output interfaces of each module in the digital image behavior package are archived and encoded to form a standardized model file that can be loaded and deployed. The digital avatar behavior package is loaded into the new interactive scenario, and new multi-source interactive data input is received. The action strategy reasoning and control execution are completed through an improved hierarchical reinforcement learning structure, supporting the dynamic action adjustment task of the digital avatar in cross-scenario scenarios.

Images