Multi-modal task decision method and device based on feature enhancement

Abstract

The application provides a feature enhancement-based multimodal task decision method and device, and relates to the technical field of computers.The method comprises the following steps: calculating frame difference information between adjacent image frames in a plurality of image frames; inputting the plurality of image frames into a semantic encoder to obtain semantic features, and inputting the frame difference information into a motion encoder to obtain motion features; inputting the semantic features and the motion features into a cross-attention network to obtain enhanced semantic features and enhanced motion features; and inputting the enhanced semantic features and the enhanced motion features into a task decision network related to a target task to generate corresponding decision information.The feature enhancement-based multimodal task decision method and device provided by the application greatly improve the feature extraction capability of a visual encoder by guiding the visual encoder to focus on objects related to a task by using knowledge perception features with prior knowledge provided by a visual language model in a training stage.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a multimodal task decision-making method and apparatus based on feature enhancement. Background Technology

[0002] Visual encoders, as a core component of computer vision tasks, are widely used in fields such as image classification, object detection, image generation, and video understanding.

[0003] Visual encoders extract global or local features from images using convolutional neural networks (CNNs) or Transformer architectures, providing semantic information for downstream tasks. For example, in autonomous driving scenarios, visual encoders can extract information about objects of interest to the autonomous driving system.

[0004] However, despite the excellent performance of visual encoders in most scenarios, their feature extraction capabilities still have significant limitations, especially in complex scenes, cross-modal alignment, and fine-grained tasks. Summary of the Invention

[0005] The purpose of this application is to provide a multimodal task decision-making method and apparatus based on feature enhancement. By utilizing the knowledge-aware features with prior knowledge provided by the visual language model during the training phase, the visual encoder is guided to focus on task-related objects, which greatly improves the feature extraction capability of the visual encoder.

[0006] This application provides a feature-enhanced multimodal task decision-making method, including:

[0007] Multiple image frames acquired by a sensor are obtained, and the frame difference information between any two adjacent image frames is calculated. The frame difference information is used to characterize the difference between two adjacent image frames. The multiple image frames are input into a semantic encoder to obtain the semantic features output by the semantic encoder, and the frame difference information is input into a motion encoder to obtain the motion features output by the motion encoder. The semantic features and the motion features are input into a cross-attention network for feature enhancement to obtain enhanced semantic features and enhanced motion features. The enhanced semantic features and the enhanced motion features are input into a task decision network related to the target task to generate corresponding decision information. During training, the semantic encoder aligns the semantic features output by the semantic encoder with the knowledge-aware features provided by the visual language model, thereby guiding the semantic encoder to focus on objects related to the target task.

[0008] Optionally, calculating the frame difference information between any two adjacent image frames in the plurality of image frames includes: obtaining the color value of each pixel in the first image frame and the second image frame, and calculating the difference between the color value of each pixel in the first image frame and the color value of the corresponding pixel in the second image frame to obtain the pixel color difference between the first image frame and the second image frame; generating frame difference information between the first image frame and the second image frame based on the pixel color difference; wherein the first image frame and the second image frame are two adjacent image frames in the plurality of image frames.

[0009] Optionally, the step of inputting the semantic features and the motion features into a cross-attention network for feature enhancement to obtain enhanced semantic features and enhanced motion features includes: inputting the semantic features as query values ​​and the motion features as key values ​​into the cross-attention network to obtain a fused feature representation of the semantic features and the motion features; and using the fused feature representation to perform feature enhancement on the semantic features and the motion features respectively to obtain the enhanced semantic features and the enhanced motion features.

[0010] Optionally, the semantic encoder is trained based on the following steps: acquiring multiple sample image frames; inputting the multiple sample image frames into the semantic encoder to obtain a first semantic feature, and inputting the multiple sample image frames and the target prompt word into a visual language model to obtain a knowledge-aware feature with prior knowledge; calculating the similarity loss between the first semantic feature and the knowledge-aware feature, and adjusting the parameters of the semantic encoder based on the similarity loss value.

[0011] Optionally, the step of inputting the plurality of sample image frames and the target prompt word into a visual language model to obtain knowledge-aware features with prior knowledge includes: inputting the plurality of sample image frames and the target prompt word into a visual language model to obtain a feature mask; inputting the feature mask into the semantic encoding model to obtain knowledge-aware features with prior knowledge; wherein the knowledge-aware features and the first semantic features output by the semantic encoding model have the same dimension.

[0012] Optionally, after inputting the plurality of sample image frames into a semantic encoder to obtain a first semantic feature, and inputting the plurality of sample image frames and the target prompt word into a visual language model to obtain knowledge-aware features with prior knowledge, the method further includes: inputting the knowledge-aware features as query values ​​and the motion features as key values ​​into the cross-attention network to obtain a fused feature representation of the first motion feature and the first semantic feature output by the action encoder; using the fused feature representation to perform feature enhancement on the first semantic feature and the first motion feature respectively to obtain enhanced second semantic features and enhanced second motion features; and training a task decision network related to the target task based on the second semantic feature and the second motion feature.

[0013] This application also provides a feature-enhanced multimodal task decision-making device, comprising:

[0014] An information acquisition module is used to acquire multiple image frames collected by a sensor and calculate the frame difference information between any two adjacent image frames; the frame difference information is used to characterize the difference between two adjacent image frames. A feature extraction module is used to input the multiple image frames into a semantic encoder to obtain the semantic features output by the semantic encoder, and to input the frame difference information into a motion encoder to obtain the motion features output by the motion encoder. A feature enhancement module is used to input the semantic features and the motion features into a cross-attention network for feature enhancement to obtain enhanced semantic features and enhanced motion features. A task decision module is used to input the enhanced semantic features and the enhanced motion features into a task decision network related to the target task to generate corresponding decision information. During training, the semantic encoder aligns the semantic features output by the semantic encoder with the knowledge-aware features provided by the visual language model, thereby guiding the semantic encoder to focus on objects related to the target task.

[0015] Optionally, the information acquisition module is specifically used to acquire the color value of each pixel in the first image frame and the second image frame, and calculate the difference between the color value of each pixel in the first image frame and the color value of the corresponding pixel in the second image frame to obtain the pixel color difference between the first image frame and the second image frame; the information acquisition module is further used to generate frame difference information between the first image frame and the second image frame based on the pixel color difference; wherein, the first image frame and the second image frame are two adjacent image frames among the plurality of image frames.

[0016] Optionally, the feature enhancement module is specifically used to input the semantic feature as a query value and the motion feature as a key value into the cross-attention network to obtain a fused feature representation of the semantic feature and the motion feature; the feature enhancement module is also specifically used to perform feature enhancement on the semantic feature and the motion feature using the fused feature representation to obtain the enhanced semantic feature and the enhanced motion feature.

[0017] Optionally, the device further includes: a training module; the information acquisition module is further configured to acquire multiple sample image frames; the training module is configured to input the multiple sample image frames into a semantic encoder to obtain a first semantic feature, and input the multiple sample image frames and the target prompt word into a visual language model to obtain a knowledge-aware feature with prior knowledge; the training module is configured to calculate the similarity loss between the first semantic feature and the knowledge-aware feature, and adjust the parameters of the semantic encoder based on the similarity loss value.

[0018] Optionally, the feature enhancement module is further configured to input the plurality of sample image frames and target prompt words into a visual language model to obtain a feature mask; the feature enhancement module is further configured to input the feature mask into the semantic encoding model to obtain knowledge-aware features with prior knowledge; wherein the knowledge-aware features have the same dimension as the first semantic features output by the semantic encoding model.

[0019] Optionally, the feature enhancement module is further configured to input the knowledge-aware feature as a query value and the motion feature as a key value into the cross-attention network to obtain a fused feature representation of the first motion feature and the first semantic feature output by the action encoder; the feature enhancement module is further configured to use the fused feature representation to enhance the first semantic feature and the first motion feature respectively to obtain the enhanced second semantic feature and the enhanced second motion feature; the training module is further configured to train a task decision network related to the target task based on the second semantic feature and the second motion feature.

[0020] This application also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the feature-enhanced multimodal task decision-making method as described above.

[0021] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the feature-enhanced multimodal task decision methods described above.

[0022] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the feature-enhanced multimodal task decision method described above.

[0023] The multimodal task decision-making method and apparatus based on feature enhancement provided in this application first acquires multiple image frames collected by a sensor and calculates the frame difference information between any two adjacent image frames; the frame difference information is used to characterize the difference between two adjacent image frames. Then, the multiple image frames are input into a semantic encoder to obtain semantic features output by the semantic encoder, and the frame difference information is input into a motion encoder to obtain motion features output by the motion encoder. The semantic features and motion features are then input into a cross-attention network for feature enhancement to obtain enhanced semantic features and enhanced motion features. Finally, the enhanced semantic features and enhanced motion features are input into a task decision network related to the target task to generate corresponding decision information. During training, the semantic encoder aligns its output semantic features with the knowledge-aware features provided by a visual language model, thereby guiding the semantic encoder to focus on objects related to the target task. Thus, by using the knowledge-aware features provided by the visual language model during the training phase to guide the visual encoder to focus on task-related objects, the feature extraction capability of the visual encoder is greatly improved. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is one of the flowcharts illustrating the feature-enhanced multimodal task decision-making method provided in this application;

[0026] Figure 2 This is the second flowchart of the feature-enhanced multimodal task decision-making method provided in this application;

[0027] Figure 3 This is the third flowchart of the feature-enhanced multimodal task decision-making method provided in this application;

[0028] Figure 4 This is a schematic diagram of the semantic encoder training process provided in this application;

[0029] Figure 5 This is a schematic diagram of the representation interaction process of Cross-attention provided in this application;

[0030] Figure 6 This is a schematic diagram of the structure of the feature-enhanced multimodal task decision-making device provided in this application;

[0031] Figure 7 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0033] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0034] To address the issue of insufficient feature extraction capabilities in visual encoders in related technologies, this application provides a feature-enhanced multimodal task decision-making method. In this method, the visual encoder utilizes knowledge-aware features with prior knowledge provided by the visual language model during the training phase to guide the visual encoder to focus on task-related objects, thereby greatly improving the feature extraction capabilities of the visual encoder.

[0035] like Figure 1 The diagram shown is a flowchart of a multimodal task decision-making method based on feature enhancement provided in this application embodiment. The semantic features extracted by the semantic encoder (i.e., the aforementioned visual encoder) and the motion features extracted by the motion encoder are input into a cross-attention network to achieve complementary feature information through interaction, and finally, the fused features are obtained. The fused features are then input into a decision network corresponding to the target task to generate decision information.

[0036] It should be noted that the method proposed in this solution uses pure visual input, eliminating the need for expensive sensors such as LiDAR and time-lapse cameras. Supervised training is performed using Visual Learning Model (VLM) during the training phase, achieving representation enhancement without compromising algorithm efficiency. This approach is suitable for scenarios using mobile computing with relatively limited computing power, such as end-to-end autonomous driving and vision-based robot control. In other words, the target task in this embodiment can be either an autonomous driving task or a robot control task.

[0037] The following description, in conjunction with the accompanying drawings, details the feature-enhanced multimodal task decision-making method provided in this application through specific embodiments and application scenarios.

[0038] like Figure 2 As shown in the embodiment of this application, a multimodal task decision-making method based on feature enhancement is provided. This method may include the following steps 201 to 204:

[0039] Step 201: Acquire multiple image frames collected by the sensor, and calculate the frame difference information between any two adjacent image frames in the multiple image frames.

[0040] The frame difference information is used to characterize the difference between two adjacent image frames.

[0041] For example, the sensor can be an image sensor (e.g., a camera) or a radar sensor (e.g., a millimeter-wave radar). After acquiring multiple image frames through the sensor, the frame difference information between any two adjacent image frames is calculated, and the multiple image frames and the frame difference information are respectively input into the semantic encoder and the motion encoder for feature extraction.

[0042] Specifically, step 201 above, which calculates the frame difference information between any two adjacent image frames in the plurality of image frames, may further include the following steps 201a1 and 201a2:

[0043] Step 201a1: Obtain the color value of each pixel in the first image frame and the second image frame, and calculate the difference between the color value of each pixel in the first image frame and the color value of the corresponding pixel in the second image frame to obtain the pixel color difference between the first image frame and the second image frame.

[0044] Step 201a2: Based on the pixel color difference, generate frame difference information between the first image frame and the second image frame.

[0045] Wherein, the first image frame and the second image frame are two adjacent image frames among the plurality of image frames.

[0046] For example, such as Figure 3As shown, taking the aforementioned target task as an autonomous driving task as an example, three raw image frames observed are acquired from the camera sensor of the intelligent agent. First, the frame difference between adjacent frames needs to be calculated to obtain the frame difference information between any two adjacent image frames. Then, the image frames are input into the semantic encoder, and the frame difference information is input into the motion encoder to obtain semantic features. (Semantic features) and motion features (Motion features).

[0047] Step 202: Input the multiple image frames into the semantic encoder to obtain the semantic features output by the semantic encoder, and input the frame difference information into the motion encoder to obtain the motion features output by the motion encoder.

[0048] For example, after obtaining the semantic features output by the semantic encoder and the motion features output by the motion encoder, it is necessary to interact the two features through a cross-attention network and perform feature enhancement.

[0049] Step 203: Input the semantic features and motion features into a cross-attention network for feature enhancement to obtain enhanced semantic features and enhanced motion features.

[0050] Specifically, step 203 above may also include the following steps 203a and 203b:

[0051] Step 203a: Input the semantic features as query values ​​and the motion features as key values ​​into the cross-attention network to obtain the fused feature representation of the semantic features and the motion features.

[0052] Step 203b: Use the fused feature representation to perform feature enhancement on the semantic features and the motion features respectively, to obtain the enhanced semantic features and the enhanced motion features.

[0053] For example, a cross-attention network is used to compute the cross-attention between semantic features and motion features, resulting in a fused feature representation of the two features. Then, the fused feature representation is used to enhance both the semantic and motion features, resulting in enhanced semantic features and enhanced motion features.

[0054] Step 204: Input the enhanced semantic features and the enhanced motion features into the task decision network related to the target task to generate corresponding decision information.

[0055] During training, the semantic encoder aligns its output semantic features with the knowledge-aware features provided by the visual language model, thereby guiding the semantic encoder to focus on objects related to the target task.

[0056] For example, by inputting the enhanced semantic features and motion features into the decision network corresponding to the target task, the decision network can generate more accurate and reasonable decision information using the enhanced features.

[0057] Optionally, in the embodiments of this application, a representation supervision method based on VLM can be introduced to train the semantic encoder during the entire training process of the model network, so as to improve the feature extraction capability of the semantic encoder.

[0058] For example, such as Figure 4 The diagram shown is a schematic flowchart of the semantic encoder training process provided in an embodiment of this application. The training process includes the following steps 401 to 403:

[0059] Step 401: Obtain multiple sample image frames.

[0060] Step 402: Input the multiple sample image frames into the semantic encoder to obtain the first semantic feature, and input the multiple sample image frames and the target prompt word into the visual language model to obtain the knowledge perception feature with prior knowledge.

[0061] Step 403: Calculate the similarity loss between the first semantic feature and the knowledge-aware feature, and adjust the parameters of the semantic encoder based on the similarity loss value.

[0062] For example, in this embodiment, a Visual Language Model (VLM) is introduced to provide prior common sense for representation learning. The encoder is supervised at the feature layer, thereby enhancing the representation information and overcoming the limitations of traditional methods. Figure 3 The visual reinforcement learning shown lacks interpretability and struggles to align with real representations. Furthermore, semantic-motion representation decoupling is employed. For the semantic branch, feature similarity is used to align the extracted representation with the representation generated by the visual language model. For the motion branch, cross-attention is used for interaction, enabling effective extraction of motion information from semantic objects.

[0063] For example, such as Figure 3As shown, during the training process, in order to utilize the common sense information provided by the Visual Language Model (a type of Large Language Model, LLM), we input the observed images into the Visual Language Model and use text prompts (prompts) to ask questions, allowing the Visual Language Model (VLM) to perform inference, thereby obtaining object information related to the agent's decision-making in the observations. In order to effectively map this information to the feature space, thereby enabling supervised representation extraction, we can use an image segmentation algorithm based on CLIP (a type of Visual Language Model) to generate a feature mask, thereby aligning these key objects in the image.

[0064] For example, this application embodiment takes into account the use of a dual-branch network to extract different representations. Therefore, this application embodiment adopts two methods for supervision. First, for the semantic branch, explicit supervision is performed using feature similarity loss, which aligns the extracted semantic features with the features with prior knowledge provided by the visual language model (i.e., the aforementioned knowledge-aware features), thereby guiding the semantic encoder's area of ​​interest to some key objects (i.e., objects related to the target task).

[0065] Specifically, step 402 above may also include steps 402a1 and 402a2:

[0066] Step 402a1: Input the multiple sample image frames and target prompt words into the visual language model to obtain the feature mask.

[0067] Step 402a2: Input the feature mask into the semantic coding model to obtain knowledge-aware features with prior knowledge.

[0068] The knowledge-aware feature has the same dimension as the first semantic feature output by the semantic encoding model.

[0069] For example, in the embodiments of this application, the similarity loss between knowledge-aware features and semantic features can be calculated using the following formula:

[0070] (Formula 1)

[0071] in, The output of the visual language model, It is by The knowledge-aware features obtained by inputting into the semantic encoder are guaranteed to have the same dimension as the semantic representation.

[0072] Specifically, after step 402 above, the training method provided in this application embodiment may further include steps 404 to 406:

[0073] Step 404: Input the knowledge-aware feature as the query value and the motion feature as the key value into the cross-attention network to obtain the fused feature expression of the first motion feature and the first semantic feature output by the action encoder.

[0074] Step 405: Use the fused feature representation to enhance the first semantic feature and the first motion feature respectively, to obtain the enhanced second semantic feature and the enhanced second motion feature.

[0075] Step 406: Train the task decision network related to the target task based on the second semantic feature and the second motion feature.

[0076] For example, during the training phase, the motion representation output by the motion encoder and the semantic representation generated by the visual language model calculate cross-attention, while during the testing phase, the motion representation output by the motion encoder interacts with the semantic representation generated by the semantic encoder, enabling both branches of the network to effectively select key regions.

[0077] like Figure 5 The diagram illustrates the cross-attention representation interaction process provided in this embodiment. This cross-attention network can be specifically represented by the following formulas two and three:

[0078] (Formula 2)

[0079] (Formula 3)

[0080] Where X represents the fusion feature representation.

[0081] For example, after obtaining the enhanced features, the representation can also be compressed using a fully connected layer. and Features after fusion The input is fed into the decision network, which outputs an executable decision.

[0082] For example, to reduce noise throughout the training process, the motion state prediction loss can be introduced using Equation 4, and the reward loss function can be introduced using Equation 5:

[0083] (Formula 4)

[0084] (Formula 5)

[0085] For example, the objective optimization function of reinforcement learning can be represented by the following Equation 6:

[0086] (Formula 6)

[0087] For example, action evaluation can be represented by the following formula seven, and policy loss can be represented by the following formula eight:

[0088] (Formula 7)

[0089] (Formula 8)

[0090] in, Update the Bellman equation. The value function represents the Critic network's ability to accurately evaluate the value of decisions. It is the policy loss that enables the policy distribution generated by the policy network to align with the Q-value.

[0091] The feature-enhanced multimodal task decision-making method provided in this application introduces a visual language model into the visual reinforcement learning task, providing prior common sense for representation learning. It supervises the encoder at the feature layer, thereby enhancing the representation information and overcoming the problems of lack of interpretability and difficulty in aligning with real representations in visual reinforcement learning. Secondly, it adopts semantic-motion representation decoupling. For the semantic branch, feature similarity is used to align the extracted representation with the representation generated by the visual language model. For the motion branch, cross-attention is used for interaction, which enables it to effectively extract the motion information of semantic objects.

[0092] For example, taking an experiment designed based on the CARLA simulation platform, the vehicle acts as a reinforcement learning agent, aiming to travel the longest possible distance along the road in the diagram while avoiding collisions. The experiment shows that this approach can improve the agent's control decision-making performance. Table 1 presents the average results of the CARLA simulation, demonstrating that the autonomous vehicle based on this approach can increase the average travel distance by 30% and reduce the average collision intensity by 8%.

[0093]

[0094] Table 1

[0095] This application's embodiments include ablation experiments designed based on the proposed method to verify its effectiveness. By decoupling the various designed modules, models M1-M4 were obtained:

[0096] M1: Employs a single-branch encoder mechanism, with three frames of images as input, and does not use multimodal LLM or VLM for supervision.

[0097] M2: It adopts a dual-branch structure, with inputs being a single-frame image and the frame difference between two adjacent frames, and does not use multimodal LLM and VLM for supervision.

[0098] M3: The structure is the same as M2, but multimodal LLM and VLM are used to align semantic branches based on feature similarity.

[0099] M4, based on M3, utilizes the representations generated by VLM to supervise motion representations and semantic-motion interactions through cross-attention.

[0100] Table 2 presents the experimental results. M1 had the lowest performance because the pixel displacement was very small for adjacent frames, making it difficult to capture motion representations. Although M1's input was a multi-frame image, the encoder tended to focus more on semantic information and ignore motion information, resulting in a high collision rate. M2 adopted a dual-branch network structure to decouple motion representations, which significantly improved performance. M3 introduced prior semantic knowledge on the basis of M2, which improved performance, but due to the lack of supervision for the motion branch, it was still prone to collisions with some fast-moving objects in the scene, such as other vehicles. M4 introduced prior representation guidance in the semantic-motion representation interaction, enabling the model to pay more attention to some objects and predict their movement, further improving obstacle avoidance capabilities.

[0101]

[0102] Table 2

[0103] The multimodal task decision-making method based on feature enhancement provided in this application first acquires multiple image frames collected by a sensor and calculates the frame difference information between any two adjacent image frames. This frame difference information characterizes the difference between two adjacent image frames. Then, the multiple image frames are input into a semantic encoder to obtain semantic features output by the semantic encoder, and the frame difference information is input into a motion encoder to obtain motion features output by the motion encoder. The semantic features and motion features are then input into a cross-attention network for feature enhancement to obtain enhanced semantic features and enhanced motion features. Finally, the enhanced semantic features and enhanced motion features are input into a task decision network related to the target task to generate corresponding decision information. During training, the semantic encoder aligns its output semantic features with the knowledge-aware features provided by a visual language model, thereby guiding the semantic encoder to focus on objects related to the target task. Thus, by using the knowledge-aware features provided by the visual language model during the training phase to guide the visual encoder to focus on task-related objects, the feature extraction capability of the visual encoder is greatly improved.

[0104] It should be noted that the multimodal task decision-making method based on feature enhancement provided in this application can be executed by a multimodal task decision-making device based on feature enhancement, or by a control module within that device for executing the method. This application uses the execution of the multimodal task decision-making method based on feature enhancement by a multimodal task decision-making device as an example to illustrate the multimodal task decision-making device based on feature enhancement provided in this application.

[0105] It should be noted that, in the embodiments of this application, the multimodal task decision-making methods based on feature enhancement are illustrated using one accompanying figure from one of the embodiments of this application as an example. In specific implementation, the multimodal task decision-making methods based on feature enhancement illustrated in the accompanying figures of the above methods can also be implemented in conjunction with any other figures shown in the above embodiments, which will not be elaborated here.

[0106] The following describes the feature-enhanced multimodal task decision-making device provided in this application. The description below corresponds to the feature-enhanced multimodal task decision-making method described above.

[0107] Figure 6 A schematic diagram of the structure of the feature-enhanced multimodal task decision-making device provided in the embodiments of this application is shown below. Figure 6 As shown, it specifically includes:

[0108] The information acquisition module 601 is used to acquire multiple image frames collected by the sensor and calculate the frame difference information between any two adjacent image frames in the multiple image frames; the frame difference information is used to characterize the difference between two adjacent image frames; the feature extraction module 602 is used to input the multiple image frames into a semantic encoder to obtain the semantic features output by the semantic encoder, and to input the frame difference information into a motion encoder to obtain the motion features output by the motion encoder; the feature enhancement module 603 is used to input the semantic features and the motion features into a cross-attention network for feature enhancement to obtain enhanced semantic features and enhanced motion features; the task decision module 604 is used to input the enhanced semantic features and the enhanced motion features into a task decision network related to the target task to generate corresponding decision information; wherein, during the training process, the semantic encoder aligns the semantic features output by the semantic encoder with the knowledge-aware features with prior knowledge provided by the visual language model, thereby guiding the semantic encoder to focus on objects related to the target task.

[0109] Optionally, the information acquisition module 601 is specifically used to acquire the color value of each pixel in the first image frame and the second image frame, and calculate the difference between the color value of each pixel in the first image frame and the color value of the corresponding pixel in the second image frame to obtain the pixel color difference between the first image frame and the second image frame; the information acquisition module 601 is further used to generate frame difference information between the first image frame and the second image frame based on the pixel color difference; wherein, the first image frame and the second image frame are two adjacent image frames among the plurality of image frames.

[0110] Optionally, the feature enhancement module 603 is specifically used to input the semantic feature as a query value and the motion feature as a key value into the cross-attention network to obtain a fused feature representation of the semantic feature and the motion feature; the feature enhancement module 603 is also specifically used to perform feature enhancement on the semantic feature and the motion feature using the fused feature representation to obtain the enhanced semantic feature and the enhanced motion feature.

[0111] Optionally, the device further includes: a training module; the information acquisition module 601 is further configured to acquire multiple sample image frames; the training module is configured to input the multiple sample image frames into a semantic encoder to obtain a first semantic feature, and input the multiple sample image frames and the target prompt word into a visual language model to obtain a knowledge-aware feature with prior knowledge; the training module is configured to calculate the similarity loss between the first semantic feature and the knowledge-aware feature, and adjust the parameters of the semantic encoder based on the similarity loss value.

[0112] Optionally, the feature enhancement module 603 is further configured to input the plurality of sample image frames and target prompt words into a visual language model to obtain a feature mask; the feature enhancement module 603 is further configured to input the feature mask into the semantic encoding model to obtain knowledge-aware features with prior knowledge; wherein the knowledge-aware features have the same dimension as the first semantic features output by the semantic encoding model.

[0113] Optionally, the feature enhancement module 603 is further configured to input the knowledge-aware feature as a query value and the motion feature as a key value into the cross-attention network to obtain a fused feature representation of the first motion feature and the first semantic feature output by the action encoder; the feature enhancement module 603 is further configured to use the fused feature representation to enhance the first semantic feature and the first motion feature respectively to obtain the enhanced second semantic feature and the enhanced second motion feature; the training module is further configured to train a task decision network related to the target task based on the second semantic feature and the second motion feature.

[0114] The multimodal task decision-making device based on feature enhancement provided in this application first acquires multiple image frames collected by a sensor and calculates the frame difference information between any two adjacent image frames. This frame difference information characterizes the difference between two adjacent image frames. Then, the multiple image frames are input into a semantic encoder to obtain semantic features output by the semantic encoder, and the frame difference information is input into a motion encoder to obtain motion features output by the motion encoder. The semantic features and motion features are then input into a cross-attention network for feature enhancement to obtain enhanced semantic features and enhanced motion features. Finally, the enhanced semantic features and enhanced motion features are input into a task decision network related to the target task to generate corresponding decision information. During training, the semantic encoder aligns its output semantic features with the knowledge-aware features provided by a visual language model, thereby guiding the semantic encoder to focus on objects related to the target task. Thus, by using the knowledge-aware features provided by the visual language model during the training phase to guide the visual encoder to focus on task-related objects, the feature extraction capability of the visual encoder is greatly improved.

[0115] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7As shown, the electronic device may include: a processor 710, a communications interface 720, a memory 730, and a communications bus 740, wherein the processor 710, the communications interface 720, and the memory 730 communicate with each other through the communications bus 740. The processor 710 can call logic instructions in the memory 730 to execute a feature-enhanced multimodal task decision-making method. This method includes: first, acquiring multiple image frames collected by a sensor and calculating frame difference information between any two adjacent image frames; the frame difference information is used to characterize the difference between two adjacent image frames; then, inputting the multiple image frames into a semantic encoder to obtain semantic features output by the semantic encoder, and inputting the frame difference information into a motion encoder to obtain motion features output by the motion encoder; then, inputting the semantic features and the motion features into a cross-attention network for feature enhancement to obtain enhanced semantic features and enhanced motion features; finally, inputting the enhanced semantic features and the enhanced motion features into a task decision network related to the target task to generate corresponding decision information; wherein, during training, the semantic encoder aligns the semantic features output by the semantic encoder with the knowledge-aware features provided by the visual language model, thereby guiding the semantic encoder to focus on objects related to the target task. Thus, by using the knowledge-aware features provided by the visual language model during the training phase to guide the visual encoder to focus on task-related objects, the feature extraction capability of the visual encoder is greatly improved.

[0116] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0117] On the other hand, this application also provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer can execute the feature-enhanced multimodal task decision-making method provided by the methods described above. This method includes: first, acquiring multiple image frames collected by a sensor, and calculating frame difference information between any two adjacent image frames; the frame difference information is used to characterize the difference between two adjacent image frames; then, inputting the multiple image frames into a semantic encoder to obtain the semantic features output by the semantic encoder, and... The frame difference information is input into a motion encoder to obtain the motion features output by the motion encoder. The semantic features and motion features are then input into a cross-attention network for feature enhancement, resulting in enhanced semantic features and enhanced motion features. Finally, the enhanced semantic features and enhanced motion features are input into a task decision network related to the target task to generate corresponding decision information. During training, the semantic encoder aligns its output semantic features with the knowledge-aware features provided by the visual language model, thereby guiding the semantic encoder to focus on objects relevant to the target task. Thus, by utilizing the knowledge-aware features provided by the visual language model during the training phase to guide the visual encoder to focus on task-related objects, the feature extraction capability of the visual encoder is greatly improved.

[0118] Furthermore, this application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the aforementioned feature-enhanced multimodal task decision-making methods. The method includes: first, acquiring multiple image frames collected by a sensor and calculating frame difference information between any two adjacent image frames; the frame difference information is used to characterize the difference between two adjacent image frames; then, inputting the multiple image frames into a semantic encoder to obtain semantic features output by the semantic encoder, and inputting the frame difference information into a motion encoder to obtain motion features output by the motion encoder; and inputting the semantic features and the motion features into a cross-attention network for feature enhancement to obtain enhanced semantic features and enhanced motion features; finally, inputting the enhanced semantic features and the enhanced motion features into a task decision network related to the target task to generate corresponding decision information; wherein, during training, the semantic encoder aligns the semantic features output by the semantic encoder with the knowledge-aware features provided by a visual language model, thereby guiding the semantic encoder to focus on objects related to the target task. Thus, by using the knowledge-aware features provided by the visual language model during the training phase to guide the visual encoder to focus on task-related objects, the feature extraction capability of the visual encoder is greatly improved.

[0119] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0120] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A multimodal task decision-making method based on feature enhancement, characterized in that, include: Acquire multiple image frames captured by the sensor, and calculate the frame difference information between any two adjacent image frames in the multiple image frames; The frame difference information is used to characterize the difference between two adjacent image frames; The multiple image frames are input into a semantic encoder to obtain the semantic features output by the semantic encoder, and the frame difference information is input into a motion encoder to obtain the motion features output by the motion encoder. The semantic features and motion features are input into a cross-attention network for feature enhancement, resulting in enhanced semantic features and enhanced motion features. The enhanced semantic features and the enhanced motion features are input into a task decision network related to the target task to generate corresponding decision information; During training, the semantic encoder aligns its output semantic features with the knowledge-aware features provided by the visual language model, thereby guiding the semantic encoder to focus on objects related to the target task.

2. The multimodal task decision-making method based on feature enhancement according to claim 1, characterized in that, The calculation of the frame difference information between any two adjacent image frames in the plurality of image frames includes: Obtain the color value of each pixel in the first image frame and the second image frame, and calculate the difference between the color value of each pixel in the first image frame and the color value of the corresponding pixel in the second image frame to obtain the pixel color difference between the first image frame and the second image frame. Based on the pixel color difference, frame difference information between the first image frame and the second image frame is generated; Wherein, the first image frame and the second image frame are two adjacent image frames among the plurality of image frames.

3. The multimodal task decision-making method based on feature enhancement according to claim 1, characterized in that, The step of inputting the semantic features and the motion features into a cross-attention network for feature enhancement to obtain enhanced semantic features and enhanced motion features includes: The semantic features are used as query values, and the motion features are used as key values, which are then input into the cross-attention network to obtain a fused feature representation of the semantic features and the motion features. The semantic features and motion features are enhanced using the fused feature representation to obtain the enhanced semantic features and the enhanced motion features.

4. The multimodal task decision-making method based on feature enhancement according to claim 1, characterized in that, The semantic encoder is trained based on the following steps: Acquire multiple sample image frames; The multiple sample image frames are input into the semantic encoder to obtain the first semantic feature, and the multiple sample image frames and the target prompt word are input into the visual language model to obtain the knowledge perception feature with prior knowledge. Calculate the similarity loss between the first semantic feature and the knowledge-aware feature, and adjust the parameters of the semantic encoder based on the similarity loss value.

5. The feature-enhanced multimodal task decision-making method according to claim 4, characterized in that, The step of inputting the multiple sample image frames and target prompt words into a visual language model to obtain knowledge-aware features with prior knowledge includes: The multiple sample image frames and target prompt words are input into a visual language model to obtain a feature mask; The feature mask is input into the semantic coding model to obtain knowledge-aware features with prior knowledge; The knowledge-aware feature has the same dimension as the first semantic feature output by the semantic encoding model.

6. The feature-enhanced multimodal task decision-making method according to claim 4 or 5, characterized in that, After inputting the plurality of sample image frames into a semantic encoder to obtain first semantic features, and inputting the plurality of sample image frames and target prompt words into a visual language model to obtain knowledge-aware features with prior knowledge, the method further includes: The knowledge-aware features are used as query values, and the motion features are used as key values, which are then input into the cross-attention network to obtain a fused feature representation of the first motion feature and the first semantic feature output by the motion encoder. The first semantic feature and the first motion feature are enhanced using the fused feature representation to obtain the enhanced second semantic feature and the enhanced second motion feature; The task decision network related to the target task is trained based on the second semantic feature and the second motion feature.

7. A multimodal task decision-making device based on feature enhancement, characterized in that, The device includes: The information acquisition module is used to acquire multiple image frames collected by the sensor and calculate the frame difference information between any two adjacent image frames; the frame difference information is used to characterize the difference between two adjacent image frames. The feature extraction module is used to input the multiple image frames into the semantic encoder to obtain the semantic features output by the semantic encoder, and to input the frame difference information into the motion encoder to obtain the motion features output by the motion encoder. The feature enhancement module is used to input the semantic features and the motion features into the cross-attention network for feature enhancement, so as to obtain enhanced semantic features and enhanced motion features; The task decision module is used to input the enhanced semantic features and the enhanced motion features into a task decision network related to the target task to generate corresponding decision information; During training, the semantic encoder aligns its output semantic features with the knowledge-aware features provided by the visual language model, thereby guiding the semantic encoder to focus on objects related to the target task.

8. The feature-enhanced multimodal task decision-making device according to claim 7, characterized in that, The device further includes: a training module; The information acquisition module is also used to acquire multiple sample image frames; The training module is used to input the multiple sample image frames into the semantic encoder to obtain the first semantic features, and input the multiple sample image frames and the target prompt words into the visual language model to obtain knowledge perception features with prior knowledge. The training module is used to calculate the similarity loss between the first semantic feature and the knowledge-aware feature, and to adjust the parameters of the semantic encoder based on the similarity loss value.

9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the feature-enhanced multimodal task decision method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the feature-enhanced multimodal task decision method as described in any one of claims 1 to 6.

Images