Method and apparatus for policy optimization of multi-modal action model, and medium

Abstract

The application relates to the technical field of artificial intelligence, can be applied to a financial technology, medical health and other business system platform, and discloses a multi-modal action model strategy optimization method, device, equipment and medium, the method comprises the following steps: relationship analysis is carried out among acquired image data, language instructions and the behavior action sequence of a target user, relationship dependency is obtained, an initial multi-modal action model is constructed in combination with initial training parameters, the initial multi-modal action model is fine-tuned by using acquired task-specific data, a fine-tuned multi-modal action model is obtained, environment interaction data sets of a target environment are acquired, the environment interaction data sets are sampled one by one by using the fine-tuned multi-modal action model, a plurality of target interaction trajectories are generated, and the selection strategy in the fine-tuned multi-modal action model is optimized, and a target selection strategy is obtained. The application improves the selection strategy accuracy of the multi-modal action model when the model faces new situations or insufficient data.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a strategy optimization method, apparatus, device, and medium for a multimodal action model. Background Technology

[0002] Current multimodal action models, namely vision-language-action (VLA) models, while achieving some success in specific tasks, rely on pre-collected offline data for training. This makes the models unable to adapt to new environments and tasks. When faced with constantly changing real-world environments, traditional training methods often cannot handle the gradual accumulation of model errors, thus affecting long-term task performance. Furthermore, obtaining large amounts of high-quality human demonstration data remains a significant challenge, especially when data is insufficient, which can lead to a substantial drop in the performance of multimodal action models.

[0003] In the healthcare field, multimodal action models hold promise for assisting physicians in diagnosis and surgery. However, their reliance on vast amounts of historical data for training makes them less effective in handling new diseases or unseen medical situations. Furthermore, the need for high-quality data during training limits their applicability, particularly for rare diseases or when data is scarce. More importantly, the models' inability to adapt in real-time to complex and changing patient physiological states and surgical environments can lead to a gradual accumulation of decision-making errors, ultimately impacting treatment outcomes.

[0004] In the fintech sector, multimodal action models can provide personalized recommendations based on clients' financial data. However, relying on vast amounts of historical transaction data and predefined risk analysis makes them ineffective in responding to rapid market changes and unforeseen events. The high data requirements significantly increase model training costs, and in the absence of sufficient data, models may fail to accurately assess risk or predict market trends, leading to flawed investment decisions. As the market environment constantly evolves, models struggle to adapt in real time, particularly in the face of new economic policies or global financial events, making them prone to misjudgments and erroneous decisions.

[0005] Therefore, in current technology, multimodal action models rely on a large amount of historical data and cannot adapt to environmental changes in real time. This leads to problems such as low accuracy in selecting optimization strategies when facing new situations or insufficient data, resulting in low generalization ability of the models. Summary of the Invention

[0006] This invention provides a strategy optimization method, apparatus, device, and medium for multimodal action models, with the main purpose of solving the problem of low generalization ability of multimodal action models when facing new situations or insufficient data.

[0007] Firstly, to achieve the above objectives, the present invention provides a strategy optimization method for a multimodal action model, comprising:

[0008] Acquire image data, language instructions, and the target user's action sequence, and construct an initial training parameter set based on the image data, language instructions, and action sequence;

[0009] Analyze the dependencies between parameters in the initial training parameter set, and construct an initial multimodal action model based on the initial training parameter set and the dependencies.

[0010] Acquire task-specific data, and use the task-specific data to fine-tune the initial multimodal action model to obtain a fine-tuned multimodal action model;

[0011] Obtain the environmental interaction dataset of the target environment, and use the fine-tuned multimodal action model to sample each interaction in the environmental interaction dataset to generate several target interaction trajectories;

[0012] The selection strategy in the fine-tuned multimodal action model is optimized based on the target interaction trajectory to obtain the target selection strategy.

[0013] Secondly, the present invention also provides a strategy optimization device for a multimodal action model, comprising:

[0014] The parameter set construction module is used to acquire image data, language instructions, and the target user's action sequence, and to construct an initial training parameter set based on the image data, language instructions, and action sequence.

[0015] An initial model building module is used to analyze the relationship dependencies between parameters in the initial training parameter set, and to build an initial multimodal action model based on the initial training parameter set and the relationship dependencies.

[0016] The initial model fine-tuning module is used to acquire task-specific data and fine-tune the initial multimodal action model using the task-specific data to obtain a fine-tuned multimodal action model.

[0017] The data interaction sampling module is used to acquire the environmental interaction dataset of the target environment, and to perform interaction sampling on the environmental interaction dataset one by one using the fine-tuned multimodal action model to generate several target interaction trajectories.

[0018] The model strategy optimization module is used to optimize the selection strategy in the fine-tuned multimodal action model based on the target interaction trajectory to obtain the target selection strategy.

[0019] Thirdly, the present invention also provides an electronic device, the electronic device comprising:

[0020] At least one processor; and,

[0021] A memory communicatively connected to the at least one processor; wherein,

[0022] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the strategy optimization method for the multimodal action model described above.

[0023] Fourthly, the present invention also provides a computer-readable storage medium storing at least one computer program, which is executed by a processor in an electronic device to implement the strategy optimization method for the multimodal action model described above.

[0024] This invention acquires image data, language commands, and action sequences of the target user. Based on these data, an initial training parameter set is constructed, enabling systematic processing and integration of multimodal information. This provides a rich and unified input foundation, helping multimodal models better understand and predict complex tasks. The invention analyzes the dependencies between parameters in the initial training parameter set and constructs an initial multimodal action model based on these dependencies. By combining these dependencies with a standard parameter set and optimizing using an objective function, the model can efficiently select and process actions across different modalities, improving both accuracy and the ability to handle complex tasks and multimodal data. Finally, task-specific data is acquired and used to fine-tune the initial multimodal action model, resulting in a fine-tuned multimodal action model. This optimized model can execute tasks more accurately and adapt to different tasks. To meet the requirements of the task and improve the efficiency and accuracy of the model in practical applications, an environmental interaction dataset of the target environment is obtained. The fine-tuned multimodal action model is used to sample each interaction in the dataset, generating several target interaction trajectories. By calculating the advantage value of each trajectory, the target interaction trajectory that best meets the preset conditions can be accurately selected, ensuring that the model can continuously optimize the decision path and provide the optimal strategy for task execution. Based on the target interaction trajectory, the selection strategy in the fine-tuned multimodal action model is optimized to obtain the target selection strategy. The optimized selection strategy is then used to sample the interaction dataset again and select actions, enabling the decision path to be adjusted in real time under different environments and continuously iterated and optimized. This not only improves the model's adaptability but also ensures the continuous effectiveness of the strategy in long-term execution, enabling it to maintain efficient task execution capabilities in constantly changing environments and improving the accuracy of the model's selection strategy when facing new situations or insufficient data. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of an application environment for a strategy optimization method for a multimodal action model according to an embodiment of the present invention;

[0026] Figure 2 This is a flowchart illustrating a strategy optimization method for a multimodal action model according to an embodiment of the present invention.

[0027] Figure 3 This is a flowchart illustrating the initial multimodal action model fine-tuning process in a strategy optimization method for a multimodal action model according to an embodiment of the present invention.

[0028] Figure 4 A schematic diagram of a strategy optimization device for a multimodal action model provided in an embodiment of the present invention;

[0029] Figure 5 A schematic diagram of the structure of an electronic device that implements a strategy optimization method for a multimodal action model according to an embodiment of the present invention;

[0030] Figure 6 This is another structural schematic diagram of an electronic device that implements a strategy optimization method for a multimodal action model, according to an embodiment of the present invention.

[0031] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0032] The principles and spirit of the present invention will now be described with reference to several specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0033] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0034] This application provides a strategy optimization method for a multimodal action model. The execution subject of this method includes, but is not limited to, at least one electronic device that can be configured to execute the device provided in this application, such as a server or a terminal. In other words, the strategy optimization method for a multimodal action model can be executed by software or hardware installed on a terminal device or a server device. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0035] This invention provides a strategy optimization method for a multimodal action model, applicable to applications such as... Figure 1 In this application environment, the client communicates with the server via a network. The server can obtain image data, language commands, and the target user's action sequences from the client. Based on the image data, language commands, and action sequences, it constructs an initial training parameter set, which can systematically process and integrate multimodal information, providing a rich and unified input foundation to help the multimodal model better understand and predict complex tasks. It analyzes the dependencies between parameters in the initial training parameter set and constructs an initial multimodal action model based on the initial training parameter set and the dependencies. By combining the dependencies with the standard parameter set and optimizing using an objective function, it ensures that the model can efficiently select and process actions between different modalities, improving not only the model's accuracy but also its ability to handle complex tasks and multimodal data. Finally, it acquires task-specific data and uses this data to fine-tune the initial multimodal action model, resulting in a fine-tuned multimodal action model. The optimized fine-tuned multimodal action model can execute tasks more accurately, adapt to the needs of different tasks, and improve performance. The invention improves the efficiency and accuracy of the model in practical applications by acquiring an environmental interaction dataset of the target environment. The fine-tuned multimodal action model samples each interaction in the dataset to generate several target interaction trajectories. By calculating the advantage value of each trajectory, the model can accurately select the target interaction trajectory that best meets the preset conditions, ensuring that the model can continuously optimize the decision path and provide the optimal strategy for task execution. Based on the target interaction trajectory, the selection strategy in the fine-tuned multimodal action model is optimized to obtain a target selection strategy. The optimized selection strategy is then used to sample and select actions again from the interaction dataset, enabling real-time adjustment of the decision path in different environments and continuous iterative optimization. This not only improves the model's adaptability but also ensures the continuous effectiveness of the strategy in long-term execution, maintaining high-efficiency task execution capabilities in constantly changing environments. It also improves the accuracy of the model's selection strategy when facing new situations or insufficient data. Finally, the target selection strategy is output and fed back to the user client. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The following detailed description of specific embodiments further illustrates the invention.

[0036] The following is an explanation of this invention, which optimizes a multimodal action model to better adapt it to specific tasks and environments. An initial multimodal action model is constructed by acquiring image data, language commands, and behavioral action sequences. This model is then fine-tuned to better suit the needs of specific tasks. The fine-tuned model is used to sample interactions from a dataset of interactions in the target environment, generating multiple target interaction trajectories for further evaluation. By analyzing these interaction trajectories, the selection strategy within the model is optimized to obtain a target selection strategy. This improves the model's performance in real-world tasks, optimizes the decision-making ability of the multimodal model, and ensures more accurate task execution in complex environments.

[0037] Reference Figure 2 The diagram shown is a flowchart illustrating a strategy optimization method for a multimodal action model according to an embodiment of the present invention. In this embodiment, the strategy optimization method for a multimodal action model includes:

[0038] S1. Acquire image data, language instructions, and the target user's action sequence, and construct an initial training parameter set based on the image data, language instructions, and action sequence.

[0039] In this embodiment of the invention, image data refers to the visual information of the environment in which the target user is located when performing a task, usually existing in the form of video frames or image sequences; language instructions refer to the natural language description of the task objective by the target user, including specific operational intentions; and behavioral action sequences refer to the action trajectories generated by the target user during the execution of the task, reflecting the actual operation process of the task.

[0040] Keyframes are extracted from image data and encoded into visual feature vectors using a convolutional neural network. Language instructions are segmented and embedded, and semantic vector representations are extracted. Behavioral action sequences are structured and encoded, mapped into discrete or continuous action representations. Image features, language instruction vectors, and action sequences at corresponding time steps are aligned and combined to construct an initial training parameter set, providing an input basis for the multimodal model.

[0041] In specific healthcare scenarios, this technology can be used for patient monitoring and health intervention. By extracting key patient imaging data (such as CT scans or X-ray images) and encoding keyframe images into visual feature vectors using convolutional neural networks, lesion areas can be accurately identified. Combined with doctors' verbal instructions (such as diagnostic instructions or treatment plans), word segmentation and embedding techniques are used to transform these instructions into semantic vectors, helping the model understand the doctor's commands. Sequences of behavioral actions (such as operations performed by a surgical robot or steps in drug dispensing) are structured and encoded into discrete or continuous action representations, aligned with image features and instruction vectors to construct an initial set of training parameters. This multimodal data serves as the input basis for the model, enabling applications such as automated diagnosis, precision treatment planning, and real-time monitoring.

[0042] In specific fintech scenarios, it can be used for tasks such as intelligent investment advisory, risk assessment, and financial planning. By extracting users' transaction records and market chart data, convolutional neural networks are used to encode key image frames (such as stock market charts) into visual feature vectors. Simultaneously, combined with users' financial needs expressed in language (e.g., "Buy stock X" or "View financial status"), semantic vectors are generated through word segmentation and embedding to help the system understand user needs. User behavioral data (such as historical investment behavior and risk preferences) is structured and encoded, mapped into discrete or continuous action representations. Image features, language commands, and behavioral data are aligned and combined to construct an initial set of training parameters, providing the input foundation for a multimodal model, enabling functions such as accurate investment decisions, risk prediction, and personalized financial advice.

[0043] In this embodiment of the invention, constructing an initial training parameter set based on the image data, the language instructions, and the action sequence includes:

[0044] Extract keyframes from the image data;

[0045] Obtain the sampling point positions of the preset convolution kernel in the key frame, and generate the offset of each sampling point position according to the key frame;

[0046] The sampling point position is adaptively offset using the offset, and the preset convolution kernel is updated using the offset sampling point position to obtain a deformable convolution kernel;

[0047] The keyframes are used to extract features using the deformable convolutional kernel to obtain visual encoding;

[0048] The language instructions are segmented to obtain several instruction words;

[0049] The instruction word is dynamically fine-tuned and embedded based on the preset action sequence feedback to obtain the word embedding corresponding to the instruction word;

[0050] Dynamic convolutional fusion is performed on all the word embeddings to obtain the language semantic vector;

[0051] The sequence of actions is normalized to obtain normalized action data.

[0052] The behavior normalization data is encoded to obtain the behavior action code;

[0053] The visual encoding, the language semantic vector, and the behavior action encoding are aligned and combined at time steps to obtain the initial training parameter set.

[0054] In detail, the image data is analyzed to extract representative keyframes. A set of convolutional kernel sampling points for feature extraction is determined based on each keyframe. Based on information such as texture changes, edge distribution, or semantic features of the image region in the keyframe, the position of each convolutional kernel sampling point is adaptively modeled using an offset prediction module to generate a corresponding two-dimensional offset.

[0055] The fixed position coordinates of each sampling point of the convolution kernel are added to the corresponding offset to obtain the updated sampling position. At non-integer positions, bilinear interpolation is used to obtain image feature values. Based on the updated set of sampling positions, deformable convolution kernels are constructed, enabling the convolution operation to have the ability to perceive local deformation of the input image and adaptive regional structure.

[0056] Convolutional operations are performed on regions with spatial deformation features in the image based on deformable sampling locations. Local details and structural change information in keyframes are captured by dynamically adjusted sampling points, and image features with higher representation capabilities are extracted. The extracted multi-scale or multi-channel image features are encoded and integrated to generate visual codes for subsequent fusion.

[0057] Language instructions are processed using word segmentation tools (such as BERT, WordPiece, or SentencePiece). This typically involves splitting the sentence within the instruction into several words or subwords according to rules such as spaces and punctuation marks. When dealing with complex language structures, the word segmenter is trained using a dictionary or corpus to handle compound words, technical terms, or other words that are difficult to recognize directly. In this way, language instructions are transformed into a series of independent instruction words, each containing semantic and syntactic information.

[0058] Choose a pre-trained word embedding model, such as Word2Vec, GloVe, or BERT. Use the execution matching degree between the action sequence and the corresponding instruction word as a feedback signal. Fine-tune the pre-trained word embedding model to make the word embedding in the semantic space more consistent with its behavioral semantics in the specific task scenario. Combine the fine-tuned model to perform contextual embedding on the instruction word to generate a word embedding representation that reflects the intention of the action execution.

[0059] Based on the current context information and instruction semantic features, multiple convolution kernel parameters are adaptively generated. The responses of different convolution kernels at different positions are weighted and calculated. Local semantic structure is extracted through a sliding window method. At the same time, global semantic information is fused by a dynamic weight adjustment mechanism to obtain a language semantic vector that can reflect the overall intent and context of the instruction.

[0060] Action data is collected at each time step, which may contain different action parameters (such as position, velocity, acceleration, etc.). Each action feature is processed using normalization methods. For example, min-max normalization maps the action data to the interval [0, 1]; Z-score normalization transforms the data into a standard normal distribution with a mean of 0 and a standard deviation of 1, ensuring that all action data are on the same scale.

[0061] Choose the appropriate encoding method based on the task requirements. For example, if the behavioral data consists of discrete action categories, one-hot encoding can be used, mapping each action to an independent binary vector where only the dimension representing the action category is 1, and the rest are 0. If the behavioral data is continuous, embedding representation can be used, transforming each continuous action into a low-dimensional dense vector through a specific mapping method. The encoded behavioral action data can retain the semantic information of the actions and be input into the model in a fixed format, helping the model understand the relationships and patterns between different actions.

[0062] The visual encoding of images, the semantic vectors of language instructions, and the encoding of actions are aligned according to time steps to ensure that their corresponding timestamps are consistent. For example, in a video sequence, the visual encoding of each frame needs to be synchronized with the language instructions and actions at the corresponding time points. These aligned modal data are combined at each time step. Through concatenation, weighted averaging, or other fusion methods, the visual, language, and action data at each time step are integrated into a comprehensive feature vector, forming the initial training parameter set and providing a comprehensive and consistent input for the multimodal model.

[0063] The process of constructing an initial training parameter set based on image data, language instructions, and action sequences systematically processes and integrates multimodal information, providing a rich and unified input foundation to help multimodal models better understand and predict complex tasks. Visual encoding extracts key visual features from images, language semantic vectors capture the deeper meaning of instructions through word segmentation and embedding, and action encoding standardizes and structures action sequences to ensure the comparability and consistency of action information. Through time step alignment and combination, data from all modalities can be aligned in chronological order, enhancing the model's ability to understand temporal data. This multimodal training input allows the model to simultaneously learn the relationships between different modalities, improving the accuracy and efficiency of task execution, especially when dealing with complex scenarios involving the coordination of vision, language, and behavior.

[0064] S2. Analyze the relationship dependencies between the parameters in the initial training parameter set, and construct an initial multimodal action model based on the initial training parameter set and the relationship dependencies.

[0065] In this embodiment of the invention, the parameters of different modalities are dimensionally aligned and normalized. Two sets of standard parameters are randomly selected as modal pair analysis groups. The Pearson correlation coefficient between the dimensions of each set of parameters is calculated and compared with a preset correlation threshold to generate the relationship dependency between modal pairs. Finally, the strategy action selection model is optimized through the objective function and parameter relationship group to finally construct the initial multimodal action model.

[0066] In specific healthcare scenarios, this technology can be used for intelligent diagnosis and personalized treatment recommendations. By performing multimodal processing on medical imaging data (such as X-rays and CT scans), patient health records, and doctor instructions, and dimensional alignment and normalization of parameters from different modalities (such as image features, text instructions, and treatment behavior data), the Pearson correlation coefficient between them is calculated to analyze the correlation between different modalities, determine the dependency between modal pairs, and ultimately construct an initial multimodal action model capable of accurate diagnosis, generating treatment suggestions, and monitoring patient health status, thereby improving the accuracy and efficiency of medical decision-making.

[0067] In specific fintech scenarios, it can be used in intelligent risk assessment and investment decision support systems. Different modalities of this data (such as image features, text commands, and customer behavior data) are dimensionally aligned and normalized. Pearson correlation coefficients between each dimension are calculated, further analyzing the correlation between market data and user behavior. This generates the dependency relationships between each modality pair, helping to understand the interaction between customer investment preferences and market dynamics. Ultimately, an initial multimodal action model is constructed to accurately predict market trends, assess investment risks, and provide personalized investment recommendations.

[0068] In this embodiment of the invention, the step of analyzing the relationship dependencies between parameters in the initial training parameter set, and constructing an initial multimodal action model based on the initial training parameter set and the relationship dependencies, includes:

[0069] Align the parameters of the initial training parameter group by dimension to obtain the aligned parameter group;

[0070] Normalize each parameter in the alignment parameter group to obtain a standard parameter group;

[0071] Two parameters are randomly selected from the standard parameter set to form a mode pair analysis group;

[0072] Determine the Pearson correlation coefficient for each dimension among the modal pairs in the analysis group;

[0073] The Pearson correlation coefficients for all dimensions are summed and averaged to obtain the overall correlation of the mode pairs in the analysis group.

[0074] The overall correlation is compared with a preset correlation threshold, and the relationship dependency between each group of modes and the analysis group is generated based on the comparison results.

[0075] Associating the relationship dependency with the standard parameter set yields a parameter relationship set;

[0076] Obtain the objective function, and use the parameter relationship group and the objective function to optimize the preset strategy action selection model to obtain the initial multimodal action model.

[0077] In detail, the parameter dimensions of each modality are analyzed to identify potential dimensional differences between different modalities. For example, the visual encoding of image data may be a high-dimensional vector, while language semantic vectors and behavioral action encodings may be lower-dimensional vectors. Through techniques such as interpolation, dimensionality reduction, or dimensionality increase, these parameters with different dimensions are adjusted to the same or similar dimensions to ensure they have a consistent representation. This transforms the data from each modality into the same scale and dimension, resulting in an aligned parameter set.

[0078] Based on the distribution of each parameter, an appropriate normalization method is selected to transform the parameters into a standard normal distribution with a mean of 0 and a standard deviation of 1. Normalization ensures that the parameters of all modalities are on the same scale, preventing certain features from dominating the model's learning process due to their large values. The parameters after normalization form a standard parameter set.

[0079] Two sets of parameters are randomly selected from the standard parameter set as modality pair analysis groups. Each set of parameters represents data from different modalities, such as visual features, language semantics, or behavioral action encoding. For the selected modality pair analysis groups, the Pearson correlation coefficient between each dimension is calculated to measure the degree of linear correlation between the two variables. The formula for calculating the Pearson correlation coefficient is as follows:

[0080]

[0081] in, and These represent the two parameters in the modal pair analysis group at the [number]th [time]. Dimension value, and This is represented by the mean of the two parameters in the modality pair analysis group. By calculating the Pearson correlation coefficient for each dimension, the correlation between modality pairs can be quantified, and the interdependence between different modes can be understood.

[0082] First, calculate the Pearson correlation coefficient between each dimension of each modality pair. Then, sum the correlation coefficients of all dimensions and divide by the total number of dimensions to obtain an average value. This overall correlation represents the overall linear correlation between the two modalities, reflecting their similarity and dependence across multiple dimensions, and helps determine which modalities have a strong dependency on the task.

[0083] In this embodiment of the invention, comparing the overall correlation with a preset correlation threshold and generating the relationship dependency between each group of modes and the analysis group based on the comparison result includes:

[0084] When the comparison result is that the overall correlation is less than or equal to a preset correlation threshold, there is no dependency between the modality pairs analysis groups corresponding to the overall correlation.

[0085] When the comparison result is that the overall correlation is greater than the preset correlation threshold, the overall correlation is compared with the preset first dependency strength and the preset second dependency strength respectively;

[0086] If the overall correlation is less than or equal to the preset first dependency strength, then the weak dependency is taken as the relational dependency between the modal pairs analysis groups corresponding to the overall correlation.

[0087] If the overall correlation is greater than a preset first dependency strength and less than a preset second dependency strength, then medium dependency is taken as the relational dependency between the modal pairs analysis groups corresponding to the overall correlation.

[0088] If the overall correlation is greater than or equal to the preset second dependency strength, then strong dependency is taken as the relational dependency between the modal pairs analysis groups corresponding to the overall correlation.

[0089] Specifically, if the overall correlation is less than or equal to a preset correlation threshold, it indicates that there is no significant dependency between the modality and the analyzed groups. For cases where the overall correlation is greater than the preset correlation threshold, it is necessary to further compare the overall correlation with the first dependency strength and the second dependency strength: if the overall correlation is less than or equal to the first dependency strength, it indicates that the modality has a weak dependency on the analyzed groups, and is marked as "weak dependency"; if the overall correlation is greater than the first dependency strength but less than the second dependency strength, it indicates that the dependency between the modality pairs is moderate, and is marked as "moderate dependency"; if the overall correlation is greater than or equal to the second dependency strength, it indicates that the dependency is strong, and is marked as "strong dependency", which can clearly distinguish the dependency strength between different modality pairs.

[0090] In detail, this dependency information is associated with each parameter in the standard parameter set, marking which parameters have strong dependencies in the model. For example, if the overall correlation of a set of mode pairs is high, then this set of parameters will be marked as highly dependent and added to the parameter relationship set. The parameter relationship set not only includes all parameters in the standard parameter set, but also annotates the dependency information between each parameter, allowing subsequent optimization processes to focus on parameter combinations with strong dependencies.

[0091] The objective function is typically set according to task requirements, such as maximizing the probability of an action sequence to improve the overall performance of the model. Using the previously obtained parameter relationships as input helps the model understand which parameters have a greater impact on the final decision. By optimizing the model using the objective function, the model will automatically adjust its policy selection process, gradually increasing its attention to and weighting of highly relevant parameters. Through optimization algorithms such as gradient descent, the model's parameters are continuously updated, enabling more accurate action selection in real-world tasks and generating an initial multimodal action model. The formula for calculating the objective function is shown below:

[0092]

[0093] in, Represents image data, Represents a sequence of actions. Represents language instructions, Indicates data distribution, It is the strategy of the initial multimodal action model. Indicates the action at the current time step. This indicates the action of the previous time step in the current time step. Indicates the current time step. Indicates the total time steps. This represents the logarithm of the probability that the model will choose the action at the current time step, given the image data, language instructions, and actions from the previous time step and past time steps. This represents the objective function.

[0094] By dimensional alignment and normalization, parameters from different modalities can be compared on the same scale, ensuring data uniformity and comparability. Randomly selecting and calculating the Pearson correlation coefficient of the modality pair analysis group quantifies the correlation between different modalities, providing crucial information for the model to understand intermodal dependencies. Comparing the overall correlation with a preset threshold clarifies the dependence strength between each modality pair, providing a more precise direction for model optimization. By combining relational dependencies with a standard parameter set and optimizing using an objective function, the model is ensured to efficiently select and process actions across different modalities, generating an initial multimodal action model. This not only improves model accuracy but also enhances its ability to handle complex tasks and multimodal data.

[0095] S3. Obtain task-specific data, and use the task-specific data to fine-tune the initial multimodal action model to obtain a fine-tuned multimodal action model.

[0096] In this embodiment of the invention, visual-specific data and language features related to a specific task are extracted from task-specific data. An initial multimodal action model is used to process the visual-specific data and language features, and a suitable action is selected. The initially selected action is compared with the real action-specific data in the task-specific data. The initial selection loss value is calculated based on the difference. The initial selection loss value is minimized, and the initial learning rate is dynamically adjusted to finally obtain a fine-tuned multimodal action model.

[0097] In specific healthcare scenarios, this technology can be used to improve the accuracy of personalized treatment and diagnostic support systems. It extracts medical imaging data (such as CT scans and MRI images) and linguistic features from medical records (such as doctors' diagnoses and medical text). An initial model processes this visual and linguistic data, selects appropriate treatment or diagnostic plans, compares the model's initial selection with the patient's actual treatment outcomes, calculates the initial selection loss value, and dynamically adjusts the learning rate to improve the model's adaptability to different patient data. Ultimately, the model can generate more accurate treatment suggestions, helping doctors develop personalized treatment plans and improving diagnostic efficiency and treatment effectiveness.

[0098] In specific fintech scenarios, this can be used for intelligent investment decision-making and risk assessment. It extracts linguistic features from clients' historical transaction data, dynamic market images (such as stock charts and financial reports), and relevant market analysis reports. An initial multimodal model generates preliminary investment decisions or risk assessments based on this visual data and linguistic features. By comparing these preliminary decisions with actual market performance, calculating and selecting loss values, and dynamically adjusting the learning rate, the fine-tuned model can more accurately analyze clients' risk preferences, providing customized investment advice and risk warnings, thus improving the intelligence and accuracy of financial services.

[0099] Figure 3 This is a flowchart illustrating the initial multimodal action model fine-tuning process in a strategy optimization method for a multimodal action model provided in an embodiment of the present invention.

[0100] In this embodiment of the invention, the step of fine-tuning the initial multimodal action model using the task-specific data to obtain a fine-tuned multimodal action model includes:

[0101] Multiple batches of the task-specific data are sampled to obtain different batches of visual-specific data, language features, and action-specific data;

[0102] The initial multimodal action model is used to select actions based on the visual-specific data and the language features to obtain the initial selected actions;

[0103] The initial selection action is compared with the action-specific data, and the initial selection loss value is determined based on the comparison result;

[0104] The initial selection loss value is minimized to obtain the minimum selection loss value;

[0105] An initial learning rate and a periodic hot restart mechanism are obtained, and the initial learning rate is dynamically adjusted using the periodic hot restart mechanism to obtain an updated learning rate.

[0106] The initial multimodal action model is updated using the minimum selection loss value and the update learning rate to obtain a fine-tuned multimodal action model.

[0107] In detail, the task dataset is divided into time, scene category, or instruction target dimensions. Representative visual-specific data, language features, and action-specific data are extracted from each data subset to ensure the differences in sample distribution, semantic coverage, and behavioral diversity among batches of data. The diversity of training samples is enhanced by multiple batch sampling, providing rich multimodal input support for subsequent model fine-tuning and generalization improvement.

[0108] Regarding visually specific data, keyframes or image regions reflecting the task objective are extracted from image or video sequences. This visually specific data typically contains details helpful for decision-making. Regarding linguistic features, the focus is primarily on task-related textual information, usually including natural language processing techniques (such as word segmentation and named entity recognition) to extract important words, semantic information, or contextual key information from the text. Regarding action-specific data, the focus is primarily on actions that actually occur within the task. The extracted visually specific data and linguistic features provide multimodal input to the model, supporting effective reasoning and decision-making in specific tasks.

[0109] The extracted visual-specific data and language features are used as input to the model. The visual-specific data and language features are fused together, and the relationships between different modalities are combined. Based on these fused features, possible actions are evaluated according to the task objectives (such as diagnosis, prediction, decision-making, etc.), and a preliminary action is selected as the output to obtain the initial selected action, which represents the model's preliminary decision under task-specific data.

[0110] The initial action selected by the model is compared with the actual action to be performed in the task (i.e., action-specific data). This can be achieved by calculating the difference between the two, typically measured using a loss function such as mean squared error (MSE) or cross-entropy loss. The deviations of the model-selected action from the actual action in the task are quantified across various dimensions, generating an initial loss value that reflects the action selection error under the current task data. The smaller the loss value, the closer the model's initial action selection is to the actual task-specific action; the larger the loss value, the greater the gap between the model's selection and the actual action.

[0111] After calculating the initial selection loss, the gradient information in the loss function is used to analyze how to adjust the parameters to effectively reduce the loss. Through backpropagation, the loss value is passed to each parameter layer of the initial multimodal action model, guiding the direction and magnitude of parameter updates. Through multiple iterations, the weights and biases of the initial multimodal action model are gradually adjusted, making the model's output action selection increasingly closer to the actual action in the task-specific data. Ultimately, through continuous optimization, the initial multimodal action model can minimize the selection loss, indicating that the difference between the model's predicted action and the actual action has been minimized.

[0112] The initial learning rate and corresponding periodic hot restart mechanism parameters are obtained. Based on the hot restart mechanism, the initial learning rate is non-linearly decayed according to a sine or cosine law in each preset training period, and the learning rate is reset to the initial value or scaled version at the end of the period, thus forming a dynamic learning rate change curve with periodic increases and decreases. The local optimum is found during the training process through the hot restart mechanism, which improves the model's convergence stability and global search capability, and finally obtains the updated learning rate for model optimization.

[0113] Based on the calculated minimum selection loss value, the deviation between the current predicted action and the actual action is identified. This loss value is then propagated to each layer of the model via backpropagation. Combined with the updated learning rate, parameters are adjusted based on the current gradient information to reduce loss and improve the accuracy of action selection. The updated learning rate determines the step size of each parameter update, ensuring gradual convergence to a better solution while avoiding over-adjustment. Through multiple iterations, the model continuously optimizes its parameters, making action selection increasingly accurate under task-specific data. After a series of updates, the initial multimodal action model is fine-tuned into a more tailored multimodal action model that better meets the specific task requirements.

[0114] Fine-tuning the initial multimodal action model using task-specific data can significantly improve its performance on specific tasks. Task-relevant visual and linguistic features are extracted to ensure the initial multimodal action model receives highly task-related information. Action selection is performed using this data, and the results are compared with real actions in actual task data to calculate the initial selection loss. The model is optimized by minimizing the loss value, gradually reducing bias and improving accuracy. Dynamically adjusting the learning rate adaptively adjusts the update pace based on training progress, avoiding overly fast or slow learning and ensuring efficient and stable convergence. The optimized, fine-tuned multimodal action model can perform tasks more accurately, adapting to the needs of different tasks and improving the model's efficiency and accuracy in practical applications, thereby achieving better decision support and task execution.

[0115] S4. Obtain the environmental interaction dataset of the target environment, and use the fine-tuned multimodal action model to sample each interaction in the environmental interaction dataset to generate several target interaction trajectories.

[0116] In this embodiment of the invention, a task starting point is randomly sampled from the environmental interaction dataset to determine the initial interaction context. Environmental interaction data and corresponding interactive language commands at the task starting point are extracted. This data is then processed using a fine-tuned multimodal action model to select the most suitable interactive action. The selected interactive action is downsampled to form a continuous sequence of interactive actions. The corresponding actions are executed in the target environment, and execution results are generated based on changes in the environmental state. By analyzing the execution results, target interaction trajectories that meet preset conditions are selected.

[0117] In specific healthcare scenarios, it can be used for personalized treatment and diagnostic assistance. Task-specific data is extracted from patients' medical imaging data (such as CT and MRI scans) and clinical records. Combined with doctors' diagnostic instructions, this visual and verbal information is analyzed to select appropriate treatment plans, generate preliminary diagnostic actions, and generate interactive trajectories based on patients' actual feedback (such as changes in their condition). The effectiveness of different treatment strategies is evaluated, and the optimal solution is selected. Ultimately, a customized treatment plan is provided for each patient, optimizing the medical decision-making process.

[0118] In specific fintech scenarios, it can be used in intelligent investment and risk assessment systems. It extracts historical transaction data, market trend charts, and linguistic features from relevant financial news or analytical reports, and uses preliminary models to select investment actions or risk control strategies. The initially selected investment plans are compared with actual market performance, and interactive trajectories are generated based on real-time market data feedback. The performance of different investment strategies is evaluated, and the most advantageous investment plan is selected based on indicators such as investment return and risk, ultimately helping financial institutions and investors make more accurate decisions.

[0119] In this embodiment of the invention, the step of using the fine-tuned multimodal action model to perform interaction sampling on the environmental interaction dataset one by one to generate several target interaction trajectories includes:

[0120] A task starting point is randomly sampled from the environmental interaction dataset;

[0121] Extract the environmental interaction data and interaction language commands at the starting point of the task;

[0122] The fine-tuned multimodal action model is used to select actions from the environmental interaction data and the interaction language instructions to obtain the interactive execution action at the starting point of the task.

[0123] The interactive actions are downsampled to obtain an interactive action sequence;

[0124] The interactive action sequence is executed on the target environment, and an execution result is generated based on the state changes of the target environment;

[0125] An initial interaction trajectory is generated based on the execution result, and an advantage value is determined for each initial interaction trajectory. Target interaction trajectories that meet preset conditions are then selected based on the advantage value.

[0126] Specifically, a starting point for the current task is randomly selected from multiple interaction records in the dataset. The starting point typically includes the current environmental state information (such as images, sensor data, user behavior, etc.) and the corresponding interactive language instructions (such as user commands or system responses). By analyzing the environmental interaction data at the task starting point, the specific state of the environment at the current moment can be extracted, along with the language instructions associated with that state. This information provides the necessary context for subsequent action selection and model inference.

[0127] Environmental interaction data (such as images and sensor information) extracted from the task's starting point and interactive language commands (such as user-inputted commands or questions) are fed into the fine-tuned multimodal model. Through learned cross-modal correlations, visual and linguistic information are combined to generate a comprehensive feature representation. The model's internal policy selection mechanism then evaluates and selects the most appropriate action to respond to the current environmental state and language command. For example, in a smart home scenario, the model might choose to perform actions such as turning the air conditioner on or off or adjusting the lights based on data such as room temperature, lighting, and user commands. The resulting interactive action represents the model's optimal response at the given task starting point.

[0128] The original interactive actions may involve multiple detailed operations or frequent changes in actions. To obtain a more concise sequence of interactive actions, the original actions are sampled at time step intervals or according to specific rules. Key action steps are selected, representing the main turning points of environmental changes and user intent, thus eliminating redundant and repetitive operations. Through this downsampling method, the final sequence of interactive actions will contain a more concise and representative set of actions, effectively reflecting the core process of the task, reducing the burden of subsequent calculations and analysis, and ensuring the achievement of task objectives.

[0129] The downsampled sequence of interactive actions is applied one by one to the target environment. Each action represents an interaction between the system and the environment, such as changing environmental parameters, performing a task, or responding to user input. When these actions are executed, the state of the target environment changes accordingly based on the nature of the action. For example, in a smart home scenario, executing the action "turn on the air conditioner" may cause a change in the ambient temperature. The changes in the environment after each interactive action are continuously tracked and recorded, generating corresponding execution results. These results reflect the actual impact of the interactive actions on the target environment, helping the system evaluate the effectiveness of the actions and provide feedback for subsequent decisions.

[0130] Each execution result is combined with its corresponding action sequence to form a complete initial interaction trajectory. This trajectory records all interaction steps, state changes, and action execution results from the task's start to its end. An advantage value is calculated for each trajectory to measure its effectiveness during task execution, such as task completion efficiency, success rate, and user satisfaction. The advantage value is typically determined based on the achievement of the task objective and environmental feedback; a higher advantage value indicates better performance during execution. Target interaction trajectories that meet preset criteria or thresholds are selected. These target trajectories have high advantage values, maximize the achievement of task objectives, and provide the model with the optimal strategy execution path.

[0131] By fine-tuning a multimodal action model and sampling interactions one by one from the environmental interaction dataset, the model's adaptability and execution efficiency can be effectively improved during task execution. Randomly sampling the task starting point and extracting relevant environmental data and interactive language commands enables the model to make accurate decisions based on real-time environmental information. By generating and downsampling interactive actions, the action sequence is simplified, reducing computational and processing complexity. During the execution of these actions, execution results are generated through real-time feedback, further evaluating the effectiveness of the interactive actions. Based on the initial interactive trajectories generated from the execution results, the advantage value of each trajectory can be calculated to accurately select the target interactive trajectory that best meets the preset conditions. This ensures that the model can continuously optimize the decision path, providing the optimal strategy for task execution and improving the overall system's intelligence level and the accuracy and efficiency of task completion.

[0132] S5. Optimize the selection strategy in the fine-tuned multimodal action model based on the target interaction trajectory to obtain the target selection strategy.

[0133] In this embodiment of the invention, the target interaction trajectory is divided into successful interaction trajectories and failed interaction trajectories. The probability ratio of each type of trajectory is calculated, and a strategy optimization loss value is calculated based on this ratio. A target reward function is generated according to a preset reward rule to further optimize the initial selection strategy. The optimized strategy is then used to resample the environmental interaction dataset, and the corresponding action is selected based on the new strategy to generate an optimized interaction trajectory. The optimized interaction trajectory returns to the trajectory classification step for statistical analysis of successful and failed trajectories. When the number of returns exceeds a preset threshold, the finally generated optimized selection strategy is determined as the target selection strategy.

[0134] In specific healthcare scenarios, this technology can be used for intelligent treatment recommendation and disease prediction systems. By interactively sampling patient medical data (such as medical images and medical records) and physician diagnostic instructions, treatment strategies can be progressively optimized to determine which treatment options have a higher success rate and which have a higher failure rate in historical cases. By analyzing successful and unsuccessful treatment trajectories, calculating probability ratios, and generating reward functions based on reward rules, treatment decision-making strategies can be optimized. This helps physicians select the most appropriate treatment method based on the patient's specific situation, thereby improving treatment outcomes and patient satisfaction.

[0135] In specific fintech scenarios, this can be applied to intelligent investment and risk management systems. By analyzing data from different investment strategies and market behaviors, it can learn and optimize investment decision-making paths. For example, in stock investment decisions, the model classifies historical investment decision trajectories to identify which decisions resulted in profits and which resulted in losses. Investment strategies are adjusted based on the probability ratio of successful to unsuccessful strategies, and a reward function is used to encourage high-return, low-risk investment decisions. As the strategy is optimized, it can provide investors with more accurate and personalized investment advice, while continuously adjusting and optimizing the strategy based on real-time market changes, ultimately improving investment returns and risk control effectiveness.

[0136] In this embodiment of the invention, the step of optimizing the selection strategy in the fine-tuned multimodal action model based on the target interaction trajectory to obtain a target selection strategy includes:

[0137] The target interaction trajectory is classified to obtain successful interaction trajectories and failed interaction trajectories;

[0138] Determine the probability ratio of the successful interaction trajectory and the failed interaction trajectory respectively, and determine the strategy optimization loss value based on the probability ratio;

[0139] Obtain the reward rules, adaptively modify the reward rules, and generate a dynamically adjustable target reward function based on the modified reward rules;

[0140] Obtain the initial selection strategy of the fine-tuned multimodal action model, and optimize the initial selection strategy using the strategy optimization loss value and the target reward function to obtain the optimized selection strategy;

[0141] The optimized interaction trajectory is obtained by resampling and selecting actions on the environmental interaction dataset using the optimization selection strategy.

[0142] The optimized interaction trajectory is returned to the step of classifying the target interaction trajectory to obtain successful interaction trajectories and failed interaction trajectories, and the number of returns is counted;

[0143] When the number of returns exceeds a preset return threshold, the final optimized selection strategy is used as the target selection strategy.

[0144] In detail, a successful interaction trajectory refers to a series of interactions during execution that ultimately achieves the predetermined task objective or meets specific performance metrics. A failed interaction trajectory refers to those that fail to achieve the task objective or meet the expected results, possibly due to unsuitable decisions, incorrect actions, or environmental changes. This trajectory classification helps identify which strategies perform well in task execution and which have room for improvement, providing a basis for subsequent strategy optimization.

[0145] By statistically analyzing the number of successful and failed interaction trajectories across all interaction trajectories, and calculating their ratios, a success probability ratio and a failure probability ratio are derived. These ratios reflect the probability of successful and failed trajectories occurring in a given task environment. Based on these probability ratios, a policy optimization loss value is calculated, which depends on the gap between successful and failed interaction trajectories. The goal is to maximize the proportion of successful trajectories through policy optimization. A lower proportion of successful trajectories results in a higher loss value, indicating that the policy needs further optimization to improve the task success rate. By minimizing this loss value, the chosen policy can be optimized, gradually improving the model's accuracy and efficiency in task execution.

[0146] Define the key objectives and metrics of the task, and define corresponding reward rules based on these objectives. For example, provide positive rewards for successfully achieving the objectives and penalties for failing to achieve them. Adjust the distribution or weighting of reward values ​​based on dynamic factors such as task completion, action execution error, risk control indicators, or environmental changes, so that the reward rules can more accurately reflect the current strategy performance. On this basis, construct a target reward function that supports real-time adjustment. Map the task execution results to the defined reward rules to provide feedback signals during model optimization. The design of the target reward function ensures that the model can adjust its behavior according to preset standards during training, ultimately optimizing the selected strategy towards improving task success rate and performance.

[0147] The initial policy selection strategy is a preliminary decision rule made by the model during the early stages of training based on existing knowledge and experience. It may contain some errors or biases. The calculated policy optimization loss value reflects the performance gap of the strategy when performing the task; the smaller the loss value, the closer the strategy is to the optimal one. Combined with the objective reward function, a reward or penalty signal is provided for each decision step according to the task objective. By optimizing the loss value and reward function, the model adjusts its decision strategy, gradually correcting the shortcomings of the initial policy selection strategy. The optimization goal is to maximize the success rate and efficiency of the task, ultimately obtaining an optimized policy selection strategy. This strategy is more adaptable to the task requirements than the initial strategy, providing more accurate action selection and helping the model complete the task more efficiently.

[0148] The optimized selection strategy is applied to the environmental interaction dataset to generate new interaction trajectories. These optimized interaction trajectories, through the actions of selecting new strategies, reflect how the model interacts with the environment under the improved decision rules. These optimized interaction trajectories then return to the trajectory classification step of the target interaction trajectory. In this step, the model classifies the new trajectories, identifying which are successful and which are unsuccessful interactions. By statistically analyzing the ratio of successful to unsuccessful trajectories and the number of returns, it can be determined whether the optimized selection strategy can consistently produce positive results. When the number of returns for the optimized strategy exceeds a preset threshold, it indicates that the strategy has become stable and effective after multiple iterations. At this point, the optimized selection strategy is determined as the target selection strategy, becoming the final decision-making basis for the system in future task execution.

[0149] By classifying and analyzing the target interaction trajectories, successful and unsuccessful trajectories are clearly distinguished, providing clear feedback for subsequent optimization. Calculating the probability ratio of successful to unsuccessful trajectories and determining the optimization loss value based on this ratio helps quantify the merits of the strategy, allowing for targeted adjustments to the selection strategy. The target reward function generated by combining reward rules guides the model towards more efficient and successful optimization, thereby maximizing the probability of task success. The optimized selection strategy is then used to sample and select actions on the interaction dataset again, enabling real-time adjustments to the decision path and continuous iterative optimization in different environments. By statistically analyzing the number of returns and judging the stability of the optimization strategy, when the number of returns exceeds a preset threshold, the target selection strategy can be determined. This not only improves the model's adaptability but also ensures the strategy's continued effectiveness in long-term execution, enabling the system to maintain high-efficiency task execution capabilities in constantly changing environments.

[0150] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0151] like Figure 4The diagram shown is a functional block diagram of a strategy optimization device for a multimodal action model provided in an embodiment of the present invention.

[0152] In this embodiment of the disclosure, a strategy optimization device for a multimodal action model is provided, which corresponds one-to-one with the strategy optimization method for a multimodal action model described in the above embodiments. For example... Figure 4 As shown, the strategy optimization device 100 for a multimodal action model can be installed in an electronic device. According to its functions, the strategy optimization device 100 for a multimodal action model includes a parameter group construction module 101, an initial model construction module 102, an initial model fine-tuning module 103, a data interaction sampling module 104, and a model strategy optimization module 105. Detailed descriptions of each functional module are as follows:

[0153] The parameter group construction module 101 is used to acquire image data, language instructions and the target user's action sequence, and construct an initial training parameter group based on the image data, the language instructions and the action sequence;

[0154] The initial model building module 102 is used to analyze the relationship dependencies between parameters in the initial training parameter set, and to build an initial multimodal action model based on the initial training parameter set and the relationship dependencies.

[0155] The initial model fine-tuning module 103 is used to acquire task-specific data and fine-tune the initial multimodal action model using the task-specific data to obtain a fine-tuned multimodal action model.

[0156] The data interaction sampling module 104 is used to acquire the environmental interaction dataset of the target environment, and to perform interaction sampling on the environmental interaction dataset one by one using the fine-tuned multimodal action model to generate several target interaction trajectories.

[0157] The model strategy optimization module 105 is used to optimize the selection strategy in the fine-tuned multimodal action model according to the target interaction trajectory to obtain the target selection strategy.

[0158] In one embodiment, the parameter set construction module 101 constructs an initial training parameter set based on the image data, the language instructions, and the action sequence, including:

[0159] Extract keyframes from the image data;

[0160] Obtain the sampling point positions of the preset convolution kernel in the key frame, and generate the offset of each sampling point position according to the key frame;

[0161] The sampling point position is adaptively offset using the offset, and the preset convolution kernel is updated using the offset sampling point position to obtain a deformable convolution kernel;

[0162] The keyframes are used to extract features using the deformable convolutional kernel to obtain visual encoding;

[0163] The language instructions are segmented to obtain several instruction words;

[0164] The instruction word is dynamically fine-tuned and embedded based on the preset action sequence feedback to obtain the word embedding corresponding to the instruction word;

[0165] Dynamic convolutional fusion is performed on all the word embeddings to obtain the language semantic vector;

[0166] The sequence of actions is normalized to obtain normalized action data.

[0167] The behavior normalization data is encoded to obtain the behavior action code;

[0168] The visual encoding, the language semantic vector, and the behavior action encoding are aligned and combined at time steps to obtain the initial training parameter set.

[0169] In one embodiment, the initial model construction module 102 performs analysis of the dependencies between parameters in the initial training parameter set, and constructs an initial multimodal action model based on the initial training parameter set and the dependencies, including:

[0170] Align the parameters of the initial training parameter group by dimension to obtain the aligned parameter group;

[0171] Normalize each parameter in the alignment parameter group to obtain a standard parameter group;

[0172] Two parameters are randomly selected from the standard parameter set to form a mode pair analysis group;

[0173] Determine the Pearson correlation coefficient for each dimension among the modal pairs in the analysis group;

[0174] The Pearson correlation coefficients for all dimensions are summed and averaged to obtain the overall correlation of the mode pairs in the analysis group.

[0175] The overall correlation is compared with a preset correlation threshold, and the relationship dependency between each group of modes and the analysis group is generated based on the comparison results.

[0176] Associating the relationship dependency with the standard parameter set yields a parameter relationship set;

[0177] Obtain the objective function, and use the parameter relationship group and the objective function to optimize the preset strategy action selection model to obtain the initial multimodal action model.

[0178] In one embodiment, the initial model building module 102 compares the overall relevance with a preset relevance threshold and generates the relationship dependency between each group of modalities and the analysis group based on the comparison result, including:

[0179] When the comparison result is that the overall correlation is less than or equal to a preset correlation threshold, there is no dependency between the modality pairs analysis groups corresponding to the overall correlation.

[0180] When the comparison result is that the overall correlation is greater than the preset correlation threshold, the overall correlation is compared with the preset first dependency strength and the preset second dependency strength respectively;

[0181] If the overall correlation is less than or equal to the preset first dependency strength, then the weak dependency is taken as the relational dependency between the modal pairs analysis groups corresponding to the overall correlation.

[0182] If the overall correlation is greater than a preset first dependency strength and less than a preset second dependency strength, then medium dependency is taken as the relational dependency between the modal pairs analysis groups corresponding to the overall correlation.

[0183] If the overall correlation is greater than or equal to the preset second dependency strength, then strong dependency is taken as the relational dependency between the modal pairs analysis groups corresponding to the overall correlation.

[0184] In one embodiment, the initial model fine-tuning module 103 performs fine-tuning of the initial multimodal action model using the task-specific data to obtain a fine-tuned multimodal action model, including:

[0185] Multiple batches of the task-specific data are sampled to obtain different batches of visual-specific data, language features, and action-specific data;

[0186] The initial multimodal action model is used to select actions based on the visual-specific data and the language features to obtain the initial selected actions;

[0187] The initial selection action is compared with the action-specific data, and the initial selection loss value is determined based on the comparison result;

[0188] The initial selection loss value is minimized to obtain the minimum selection loss value;

[0189] An initial learning rate and a periodic hot restart mechanism are obtained, and the initial learning rate is dynamically adjusted using the periodic hot restart mechanism to obtain an updated learning rate.

[0190] The initial multimodal action model is updated using the minimum selection loss value and the update learning rate to obtain a fine-tuned multimodal action model.

[0191] In one embodiment, the data interaction sampling module 104 performs interaction sampling on the environmental interaction dataset using the fine-tuned multimodal action model to generate several target interaction trajectories, including:

[0192] A task starting point is randomly sampled from the environmental interaction dataset;

[0193] Extract the environmental interaction data and interaction language commands at the starting point of the task;

[0194] The fine-tuned multimodal action model is used to select actions from the environmental interaction data and the interaction language instructions to obtain the interactive execution action at the starting point of the task.

[0195] The interactive actions are downsampled to obtain an interactive action sequence;

[0196] The interactive action sequence is executed on the target environment, and an execution result is generated based on the state changes of the target environment;

[0197] An initial interaction trajectory is generated based on the execution result, and an advantage value is determined for each initial interaction trajectory. Target interaction trajectories that meet preset conditions are then selected based on the advantage value.

[0198] In one embodiment, the model strategy optimization module 105 performs strategy optimization on the selection strategy in the fine-tuned multimodal action model based on the target interaction trajectory to obtain a target selection strategy, including:

[0199] The target interaction trajectory is classified to obtain successful interaction trajectories and failed interaction trajectories;

[0200] Determine the probability ratio of the successful interaction trajectory and the failed interaction trajectory respectively, and determine the strategy optimization loss value based on the probability ratio;

[0201] Obtain the reward rules, adaptively modify the reward rules, and generate a dynamically adjustable target reward function based on the modified reward rules;

[0202] Obtain the initial selection strategy of the fine-tuned multimodal action model, and optimize the initial selection strategy using the strategy optimization loss value and the target reward function to obtain the optimized selection strategy;

[0203] The optimized interaction trajectory is obtained by resampling and selecting actions on the environmental interaction dataset using the optimization selection strategy.

[0204] The optimized interaction trajectory is returned to the step of classifying the target interaction trajectory to obtain successful interaction trajectories and failed interaction trajectories, and the number of returns is counted;

[0205] When the number of returns exceeds a preset return threshold, the final optimized selection strategy is used as the target selection strategy.

[0206] In this invention, a strategy optimization device for a multimodal action model is described. First, the invention acquires image data, language commands, and action sequences of a target user. Based on these data, an initial training parameter set is constructed, enabling systematic processing and integration of multimodal information. This provides a rich and unified input foundation, helping the multimodal model better understand and predict complex tasks. The device analyzes the dependencies between parameters in the initial training parameter set and constructs an initial multimodal action model based on these dependencies. By combining the dependencies with a standard parameter set and optimizing using an objective function, the model can efficiently select and process actions across different modalities. This not only improves the model's accuracy but also enhances its ability to handle complex tasks and multimodal data. Finally, task-specific data is acquired and used to fine-tune the initial multimodal action model, resulting in a fine-tuned multimodal action model. This optimized model is more precise. To accurately execute tasks and adapt to the needs of different tasks, the model's efficiency and accuracy in practical applications are improved. Then, an environmental interaction dataset of the target environment is acquired. The fine-tuned multimodal action model is used to sample each interaction in the dataset, generating several target interaction trajectories. By calculating the advantage value of each trajectory, the target interaction trajectory that best meets the preset conditions can be accurately selected, ensuring that the model can continuously optimize the decision path and provide the optimal strategy for task execution. Finally, the selection strategy in the fine-tuned multimodal action model is optimized based on the target interaction trajectory to obtain the target selection strategy. The optimized selection strategy is then used to sample and select actions again from the interaction dataset, enabling real-time adjustment of the decision path in different environments and continuous iterative optimization. This not only improves the model's adaptability but also ensures the continuous effectiveness of the strategy in long-term execution, maintaining high-efficiency task execution capabilities in constantly changing environments and improving the accuracy of the model's selection strategy when facing new situations or insufficient data. Specific limitations of the strategy optimization device for a multimodal action model can be found in the limitations of the strategy optimization method for a multimodal action model described above, and will not be repeated here. The modules in the aforementioned multimodal action model strategy optimization device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0207] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the server-side functions or steps of a policy optimization method for a multimodal action model.

[0208] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements client-side functions or steps of a multimodal action model strategy optimization method.

[0209] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0210] Acquire image data, language instructions, and the target user's action sequence, and construct an initial training parameter set based on the image data, language instructions, and action sequence;

[0211] Analyze the dependencies between parameters in the initial training parameter set, and construct an initial multimodal action model based on the initial training parameter set and the dependencies.

[0212] Acquire task-specific data, and use the task-specific data to fine-tune the initial multimodal action model to obtain a fine-tuned multimodal action model;

[0213] Obtain the environmental interaction dataset of the target environment, and use the fine-tuned multimodal action model to sample each interaction in the environmental interaction dataset to generate several target interaction trajectories;

[0214] The selection strategy in the fine-tuned multimodal action model is optimized based on the target interaction trajectory to obtain the target selection strategy.

[0215] In the several embodiments provided by this invention, it should be understood that the disclosed devices and apparatuses can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0216] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0217] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0218] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0219] In some embodiments of this example, a computer-readable storage medium is provided, on which a computer program is stored, characterized in that the computer program, when executed by a processor, implements the steps of the method described in the above embodiments.

[0220] The readable storage medium of the present invention stores a computer program, which, when executed by a processor of an electronic device, can perform the following:

[0221] Acquire image data, language instructions, and the target user's action sequence, and construct an initial training parameter set based on the image data, language instructions, and action sequence;

[0222] Analyze the dependencies between parameters in the initial training parameter set, and construct an initial multimodal action model based on the initial training parameter set and the dependencies.

[0223] Acquire task-specific data, and use the task-specific data to fine-tune the initial multimodal action model to obtain a fine-tuned multimodal action model;

[0224] Obtain the environmental interaction dataset of the target environment, and use the fine-tuned multimodal action model to sample each interaction in the environmental interaction dataset to generate several target interaction trajectories;

[0225] The selection strategy in the fine-tuned multimodal action model is optimized based on the target interaction trajectory to obtain the target selection strategy.

[0226] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0227] Computer-readable storage media may also store at least one computer-executable program / instruction, such as computer-readable instructions. Computer-readable storage media include, but are not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Computer-readable storage media may include, for example, read-only memory (ROM), hard disk, flash memory, etc. For example, a non-transitory computer-readable storage medium may be connected to a computing device such as a computer, and then, when the computing device executes the computer-readable instructions stored on the computer-readable storage medium, the various methods described above can be performed.

[0228] In addition, the computer device may include (but is not limited to) a data bus, an input / output (I / O) bus, a display, and input / output devices (e.g., keyboard, mouse, speakers, etc.).

[0229] The processor can communicate with external devices via the I / O bus through wired or wireless networks.

[0230] In one embodiment, the at least one computer-executable instruction may also be compiled into or comprise a software product / computer program product, wherein one or more computer-executable instructions are executed by a processor to perform the steps of the various functions and / or methods in the embodiments described herein.

[0231] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc.

[0232] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0233] In the embodiments provided in this disclosure, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0234] It should be noted that, in this disclosure, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element limited by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0235] It should be noted that if any software tools or components not belonging to this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use. The above are merely preferred embodiments of the present invention and do not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A strategy optimization method for a multimodal action model, characterized in that, The method includes: Acquire image data, language instructions, and the target user's action sequence, and construct an initial training parameter set based on the image data, language instructions, and action sequence; Analyze the dependencies between parameters in the initial training parameter set, and construct an initial multimodal action model based on the initial training parameter set and the dependencies. Acquire task-specific data, and use the task-specific data to fine-tune the initial multimodal action model to obtain a fine-tuned multimodal action model; Obtain the environmental interaction dataset of the target environment, and use the fine-tuned multimodal action model to sample each interaction in the environmental interaction dataset to generate several target interaction trajectories; The selection strategy in the fine-tuned multimodal action model is optimized based on the target interaction trajectory to obtain the target selection strategy; The step of analyzing the dependencies between parameters in the initial training parameter set and constructing an initial multimodal action model based on the initial training parameter set and the dependencies includes: Align the parameters between the initial training parameter groups to obtain an aligned parameter group, and normalize each parameter in the aligned parameter group to obtain a standard parameter group; Two parameters are randomly selected from the standard parameter set to form a mode pair analysis group, and the Pearson correlation coefficient of each dimension between the mode pair analysis groups is determined. The Pearson correlation coefficients for all dimensions are summed and averaged to obtain the overall correlation of the mode pairs in the analysis group. The overall correlation is compared with a preset correlation threshold, and the relationship dependency between each group of modes and the analysis group is generated based on the comparison results. The parameter relationship group is obtained by associating the relationship dependency with the standard parameter group; Obtain the objective function, and use the parameter relationship group and the objective function to optimize the preset strategy action selection model to obtain the initial multimodal action model.

2. The strategy optimization method for a multimodal action model as described in claim 1, characterized in that, The step of constructing an initial training parameter set based on the image data, the language instructions, and the action sequence includes: Extract keyframes from the image data; Obtain the sampling point positions of the preset convolution kernel in the key frame, and generate the offset of each sampling point position according to the key frame; The sampling point position is adaptively offset using the offset, and the preset convolution kernel is updated using the offset sampling point position to obtain a deformable convolution kernel; The keyframes are used to extract features using the deformable convolutional kernel to obtain visual encoding; The language instructions are segmented to obtain several instruction words; The instruction word is dynamically fine-tuned and embedded based on the preset action sequence feedback to obtain the word embedding corresponding to the instruction word; Dynamic convolutional fusion is performed on all the word embeddings to obtain the language semantic vector; The sequence of actions is normalized to obtain normalized action data. The behavior normalization data is encoded to obtain the behavior action code; The visual encoding, the language semantic vector, and the behavior action encoding are aligned and combined at time steps to obtain the initial training parameter set.

3. The strategy optimization method for a multimodal action model as described in claim 1, characterized in that, The step of comparing the overall correlation with a preset correlation threshold and generating the relationship dependency between each modality and the analysis group based on the comparison result includes: When the comparison result is that the overall correlation is less than or equal to a preset correlation threshold, there is no dependency between the modality pairs analysis groups corresponding to the overall correlation. When the comparison result is that the overall correlation is greater than the preset correlation threshold, the overall correlation is compared with the preset first dependency strength and the preset second dependency strength respectively; If the overall correlation is less than or equal to the preset first dependency strength, then the weak dependency is taken as the relational dependency between the modal pairs analysis groups corresponding to the overall correlation. If the overall correlation is greater than a preset first dependency strength and less than a preset second dependency strength, then medium dependency is taken as the relational dependency between the modal pairs analysis groups corresponding to the overall correlation. If the overall correlation is greater than or equal to the preset second dependency strength, then strong dependency is taken as the relational dependency between the modal pairs analysis groups corresponding to the overall correlation.

4. The strategy optimization method for a multimodal action model as described in claim 1, characterized in that, The step of fine-tuning the initial multimodal action model using the task-specific data to obtain a fine-tuned multimodal action model includes: Multiple batches of the task-specific data are sampled to obtain different batches of visual-specific data, language features, and action-specific data; The initial multimodal action model is used to select actions based on the visual-specific data and the language features to obtain the initial selected actions; The initial selection action is compared with the action-specific data, and the initial selection loss value is determined based on the comparison result; The initial selection loss value is minimized to obtain the minimum selection loss value; An initial learning rate and a periodic hot restart mechanism are obtained, and the initial learning rate is dynamically adjusted using the periodic hot restart mechanism to obtain an updated learning rate. The initial multimodal action model is updated using the minimum selection loss value and the update learning rate to obtain a fine-tuned multimodal action model.

5. The strategy optimization method for a multimodal action model as described in claim 1, characterized in that, The step of sampling each interaction in the environmental interaction dataset using the fine-tuned multimodal action model to generate several target interaction trajectories includes: A task starting point is randomly sampled from the environmental interaction dataset; Extract the environmental interaction data and interaction language commands at the starting point of the task; The fine-tuned multimodal action model is used to select actions from the environmental interaction data and the interaction language instructions to obtain the interactive execution action at the starting point of the task. The interactive actions are downsampled to obtain an interactive action sequence; The interactive action sequence is executed on the target environment, and an execution result is generated based on the state changes of the target environment; An initial interaction trajectory is generated based on the execution result, and an advantage value is determined for each initial interaction trajectory. Target interaction trajectories that meet preset conditions are then selected based on the advantage value.

6. The strategy optimization method for a multimodal action model as described in claim 1, characterized in that, The step of optimizing the selection strategy in the fine-tuned multimodal action model based on the target interaction trajectory to obtain the target selection strategy includes: The target interaction trajectory is classified to obtain successful interaction trajectories and failed interaction trajectories; Determine the probability ratio of the successful interaction trajectory and the failed interaction trajectory respectively, and determine the strategy optimization loss value based on the probability ratio; Obtain the reward rules, adaptively modify the reward rules, and generate a dynamically adjustable target reward function based on the modified reward rules; Obtain the initial selection strategy of the fine-tuned multimodal action model, and optimize the initial selection strategy using the strategy optimization loss value and the target reward function to obtain the optimized selection strategy; The optimized interaction trajectory is obtained by resampling and selecting actions on the environmental interaction dataset using the optimization selection strategy. The optimized interaction trajectory is returned to the step of classifying the target interaction trajectory to obtain successful interaction trajectories and failed interaction trajectories, and the number of returns is counted; When the number of returns exceeds a preset return threshold, the final optimized selection strategy is used as the target selection strategy.

7. A policy optimization apparatus for a multimodal action model, used to implement the policy optimization method for a multimodal action model as described in any one of claims 1 to 6, characterized in that, The device includes: The parameter set construction module is used to acquire image data, language instructions, and the target user's action sequence, and to construct an initial training parameter set based on the image data, language instructions, and action sequence. An initial model building module is used to analyze the relationship dependencies between parameters in the initial training parameter set, and to build an initial multimodal action model based on the initial training parameter set and the relationship dependencies. The initial model fine-tuning module is used to acquire task-specific data and fine-tune the initial multimodal action model using the task-specific data to obtain a fine-tuned multimodal action model. The data interaction sampling module is used to acquire the environmental interaction dataset of the target environment, and to perform interaction sampling on the environmental interaction dataset one by one using the fine-tuned multimodal action model to generate several target interaction trajectories. The model strategy optimization module is used to optimize the selection strategy in the fine-tuned multimodal action model based on the target interaction trajectory to obtain the target selection strategy.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the strategy optimization method for the multimodal action model as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the strategy optimization method for the multimodal action model as described in any one of claims 1 to 6.

Images