Multi-modal task decision-making method and device based on feature enhancement

Abstract

The invention provides a multi-modal task decision-making method and device based on feature enhancement, and relates to the technical field of computers, and the method comprises the steps: calculating frame difference information between adjacent image frames in a plurality of image frames; inputting the plurality of image frames into a semantic encoder to obtain semantic features, and inputting the frame difference information into a motion encoder to obtain motion features; inputting the semantic features and the motion features into a cross attention network to obtain enhanced semantic features and enhanced motion features; and inputting the enhanced semantic features and motion features into a task decision network related to the target task to generate corresponding decision information. According to the multi-modal task decision-making method and device based on feature enhancement provided by the invention, the visual encoder is guided to pay attention to the object related to the task by utilizing the knowledge perception features with priori knowledge provided by the visual language model in the training stage, so that the feature extraction capability of the visual encoder is greatly improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a multimodal task decision-making method and device based on feature enhancement. Background Art

[0002] As a core component of computer vision tasks, visual encoders are widely used in image classification, target detection, image generation, video understanding and other fields.

[0003] Visual encoders use convolutional neural networks (CNNs) or Transformer architectures to extract global or local features of images, providing semantic information for downstream tasks. For example, in autonomous driving scenarios, visual encoders can extract object information that autonomous driving is interested in.

[0004] However, despite the excellent performance of visual encoders in most scenarios, their feature extraction capabilities still have significant limitations, especially in complex scenes, cross-modal alignment and fine-grained tasks. Summary of the Invention

[0005] The purpose of this application is to provide a multimodal task decision method and device based on feature enhancement, which greatly improves the feature extraction capability of the visual encoder by using the knowledge perception features with prior knowledge provided by the visual language model in the training stage to guide the visual encoder to focus on objects related to the task.

[0006] This application provides a multimodal task decision-making method based on feature enhancement, including: Acquire multiple image frames captured by the sensor, and calculate the frame difference information between any two adjacent image frames in the multiple image frames; the frame difference information is used to characterize the difference between two adjacent image frames; input the multiple image frames into a semantic encoder to obtain the semantic features output by the semantic encoder, and input the frame difference information into a motion encoder to obtain the motion features output by the motion encoder; input the semantic features and the motion features into a cross-attention network for feature enhancement to obtain enhanced semantic features and enhanced motion features; input the enhanced semantic features and the enhanced motion features into a task decision network related to the target task to generate corresponding decision information; wherein, during the training process, the semantic encoder aligns the semantic features output by the semantic encoder with the knowledge perception features with prior knowledge provided by the visual language model, thereby guiding the semantic encoder to focus on objects related to the target task.

[0007] Optionally, the calculating of the frame difference information between any two adjacent image frames among the multiple image frames includes: obtaining the color value of each pixel in the first image frame and the second image frame, and calculating the difference between the color value of each pixel in the first image frame and the color value of the corresponding pixel in the second image frame to obtain the pixel color difference between the first image frame and the second image frame; based on the pixel color difference, generating the frame difference information between the first image frame and the second image frame; wherein, the first image frame and the second image frame are two adjacent image frames among the multiple image frames.

[0008] Optionally, the inputting the semantic features and the motion features into the cross-attention network for feature enhancement to obtain enhanced semantic features and enhanced motion features includes: inputting the semantic features as query values ​​and the motion features as key values ​​into the cross-attention network to obtain a fused feature expression of the semantic features and the motion features; and using the fused feature expression to perform feature enhancement on the semantic features and the motion features respectively to obtain the enhanced semantic features and the enhanced motion features.

[0009] Optionally, the semantic encoder is trained based on the following steps: obtaining multiple sample image frames; inputting the multiple sample image frames into the semantic encoder to obtain a first semantic feature, and inputting the multiple sample image frames and the target prompt word into a visual language model to obtain a knowledge perception feature with prior knowledge; calculating the similarity loss between the first semantic feature and the knowledge perception feature, and adjusting the parameters of the semantic encoder based on the similarity loss value.

[0010] Optionally, the step of inputting the multiple sample image frames and the target prompt word into the visual language model to obtain a knowledge perception feature with prior knowledge includes: inputting the multiple sample image frames and the target prompt word into the visual language model to obtain a feature mask; inputting the feature mask into the semantic coding model to obtain a knowledge perception feature with prior knowledge; wherein the knowledge perception feature has the same dimension as the first semantic feature output by the semantic coding model.

[0011] Optionally, after inputting the multiple sample image frames into the semantic encoder to obtain the first semantic feature, and inputting the multiple sample image frames and the target prompt word into the visual language model to obtain the knowledge perception feature with prior knowledge, the method also includes: inputting the knowledge perception feature as the query value and the motion feature as the key value into the cross-attention network to obtain the fused feature expression of the first motion feature output by the action encoder and the first semantic feature; using the fused feature expression to perform feature enhancement on the first semantic feature and the first motion feature respectively to obtain the enhanced second semantic feature and the enhanced second motion feature; and training the task decision network related to the target task based on the second semantic feature and the second motion feature.

[0012] The present application also provides a multimodal task decision-making device based on feature enhancement, comprising: An information acquisition module is used to acquire multiple image frames captured by a sensor and calculate the frame difference information between any two adjacent image frames in the multiple image frames; the frame difference information is used to characterize the difference between two adjacent image frames; a feature extraction module is used to input the multiple image frames into a semantic encoder to obtain semantic features output by the semantic encoder, and input the frame difference information into a motion encoder to obtain motion features output by the motion encoder; a feature enhancement module is used to input the semantic features and the motion features into a cross-attention network for feature enhancement to obtain enhanced semantic features and enhanced motion features; a task decision module is used to input the enhanced semantic features and the enhanced motion features into a task decision network related to a target task to generate corresponding decision information; wherein, during the training process, the semantic encoder aligns the semantic features output by the semantic encoder with the knowledge perception features with prior knowledge provided by the visual language model, thereby guiding the semantic encoder to focus on objects related to the target task.

[0013] Optionally, the information acquisition module is specifically used to obtain the color value of each pixel in the first image frame and the second image frame, and calculate the difference between the color value of each pixel in the first image frame and the color value of the corresponding pixel in the second image frame to obtain the pixel color difference between the first image frame and the second image frame; the information acquisition module is also specifically used to generate frame difference information between the first image frame and the second image frame based on the pixel color difference; wherein, the first image frame and the second image frame are two adjacent image frames among the multiple image frames.

[0014] Optionally, the feature enhancement module is specifically used to input the semantic feature as a query value and the motion feature as a key value into the cross-attention network to obtain a fused feature expression of the semantic feature and the motion feature; the feature enhancement module is also specifically used to use the fused feature expression to perform feature enhancement on the semantic feature and the motion feature respectively to obtain the enhanced semantic feature and the enhanced motion feature.

[0015] Optionally, the device also includes: a training module; the information acquisition module is also used to acquire multiple sample image frames; the training module is used to input the multiple sample image frames into a semantic encoder to obtain a first semantic feature, and input the multiple sample image frames and the target prompt word into a visual language model to obtain a knowledge perception feature with prior knowledge; the training module is used to calculate the similarity loss between the first semantic feature and the knowledge perception feature, and adjust the parameters of the semantic encoder based on the similarity loss value.

[0016] Optionally, the feature enhancement module is further used to input the multiple sample image frames and the target prompt word into the visual language model to obtain a feature mask; the feature enhancement module is further used to input the feature mask into the semantic coding model to obtain a knowledge-aware feature with prior knowledge; wherein the knowledge-aware feature has the same dimension as the first semantic feature output by the semantic coding model.

[0017] Optionally, the feature enhancement module is also used to input the knowledge perception feature as a query value and the motion feature as a key value into the cross-attention network to obtain a fused feature expression of the first motion feature output by the action encoder and the first semantic feature; the feature enhancement module is also used to use the fused feature expression to perform feature enhancement on the first semantic feature and the first motion feature respectively to obtain an enhanced second semantic feature and an enhanced second motion feature; the training module is also used to train a task decision network related to the target task based on the second semantic feature and the second motion feature.

[0018] The present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of any of the above-mentioned multimodal task decision methods based on feature enhancement.

[0019] The present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-described multimodal task decision methods based on feature enhancement are implemented.

[0020] The present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described multimodal task decision methods based on feature enhancement.

[0021] The present application provides a multimodal task decision method and device based on feature enhancement. First, a plurality of image frames collected by a sensor are obtained, and the frame difference information between any two adjacent image frames in the plurality of image frames is calculated; the frame difference information is used to characterize the difference between the two adjacent image frames; then, the plurality of image frames are input into a semantic encoder to obtain the semantic features output by the semantic encoder, and the frame difference information is input into a motion encoder to obtain the motion features output by the motion encoder; and the semantic features and the motion features are input into a cross-attention network for feature enhancement to obtain enhanced semantic features and enhanced motion features; finally, the enhanced semantic features and the enhanced motion features are input into a task decision network related to the target task to generate corresponding decision information; wherein, during the training process, the semantic encoder aligns the semantic features output by the semantic encoder with the knowledge perception features with prior knowledge provided by the visual language model, thereby guiding the semantic encoder to focus on objects related to the target task. In this way, by using the knowledge perception features with prior knowledge provided by the visual language model to guide the visual encoder to focus on objects related to the task during the training phase, the feature extraction capability of the visual encoder is greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the present application or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0023] Figure 1 This is one of the flow charts of the multimodal task decision method based on feature enhancement provided by this application; Figure 2 This is the second flowchart of the multimodal task decision-making method based on feature enhancement provided by this application; Figure 3 This is the third flowchart of the multimodal task decision method based on feature enhancement provided by this application; Figure 4 This is a flowchart of the semantic encoder training process provided by this application; Figure 5 This is a schematic diagram of the cross-attention representation interaction process provided by this application; Figure 6 This is a schematic diagram of the structure of a multimodal task decision-making device based on feature enhancement provided by this application; Figure 7 It is a structural diagram of the electronic device provided in this application. DETAILED DESCRIPTION

[0024] To make the objectives, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0025] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0026] In response to the problem of insufficient feature extraction capability of visual encoders in related technologies, an embodiment of the present application provides a multimodal task decision method based on feature enhancement. In this method, the visual encoder uses the knowledge perception features with prior knowledge provided by the visual language model during the training phase to guide the visual encoder to focus on objects related to the task, greatly improving the feature extraction capability of the visual encoder.

[0027] like Figure 1 As shown, it is a flow chart of the multimodal task decision method based on feature enhancement provided in an embodiment of the present application, in which the semantic features extracted by the semantic encoder (i.e., the above-mentioned visual encoder) and the motion features extracted by the motion encoder are input into the cross-attention network for interaction to realize the complementarity of feature information, and finally the fused features are obtained, and the fused features are input into the decision network corresponding to the target task to generate decision information.

[0028] It should be noted that the proposed method uses purely visual input, eliminating the need for expensive sensors such as lidar and time-of-day cameras. Using a VLM for supervised training during the training phase allows for representation enhancement without sacrificing algorithm efficiency. This approach is suitable for scenarios where mobile computing and relatively limited computing power are required, such as end-to-end autonomous driving and vision-based robotic control. In other words, the target task in the embodiments of this application can be either an autonomous driving task or a robotic control task.

[0029] The following describes in detail the feature-enhanced multimodal task decision method provided by the embodiment of the present application through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0030] like Figure 2 As shown, an embodiment of the present application provides a multimodal task decision method based on feature enhancement, which may include the following steps 201 to 204: Step 201: Acquire multiple image frames captured by a sensor, and calculate frame difference information between any two adjacent image frames in the multiple image frames.

[0031] The frame difference information is used to represent the difference between two adjacent image frames.

[0032] Exemplarily, the above-mentioned sensor can be an image sensor (for example, a camera) or a radar sensor (for example, a millimeter-wave radar). After collecting multiple image frames through the above-mentioned sensor, the frame difference information between any two adjacent image frames is calculated, and the multiple image frames and the frame difference information are respectively input into the semantic encoder and the motion encoder for feature extraction.

[0033] Specifically, in the above step 201, the step of calculating the frame difference information between any two adjacent image frames in the plurality of image frames may further include the following steps 201a1 and 201a2: Step 201a1, obtain the color value of each pixel in the first image frame and the second image frame, and calculate the difference between the color value of each pixel in the first image frame and the color value of the corresponding pixel in the second image frame to obtain the pixel color difference between the first image frame and the second image frame.

[0034] Step 201a2: Generate frame difference information between the first image frame and the second image frame based on the pixel color difference value.

[0035] The first image frame and the second image frame are two adjacent image frames among the multiple image frames.

[0036] For example, Figure 3As shown in the figure, taking the above target task as an example of autonomous driving, the three original image frames observed are obtained from the camera sensor of the intelligent agent. First, we need to calculate the frame difference between adjacent frames to obtain the frame difference information of any two adjacent image frames. Then, the image frames are input into the semantic encoder (SemanticEncoder), and the frame difference information is input into the motion encoder (Motion Encoder) to obtain semantic features respectively. (Semantic features) and motion features (Motion features).

[0037] Step 202: Input the multiple image frames into a semantic encoder to obtain semantic features output by the semantic encoder, and input the frame difference information into a motion encoder to obtain motion features output by the motion encoder.

[0038] For example, after obtaining the semantic features output by the semantic encoder and the motion features output by the motion encoder, it is necessary to interact the two features through a cross-attention network and perform feature enhancement.

[0039] Step 203: Input the semantic features and the motion features into a cross attention network for feature enhancement to obtain enhanced semantic features and enhanced motion features.

[0040] Specifically, the above step 203 may further include the following steps 203a and 203b: Step 203a: input the semantic feature as a query value and the motion feature as a key value into the cross attention network to obtain a fusion feature expression of the semantic feature and the motion feature.

[0041] Step 203b: perform feature enhancement on the semantic feature and the motion feature respectively using the fused feature expression to obtain the enhanced semantic feature and the enhanced motion feature.

[0042] For example, a cross-attention network is used to calculate the cross attention of semantic features and motion features to obtain a fused feature representation of the two features. Subsequently, the fused feature representation is used to enhance the semantic features and motion features respectively, obtaining enhanced semantic features and enhanced motion features.

[0043] Step 204: Input the enhanced semantic features and the enhanced motion features into a task decision network related to the target task to generate corresponding decision information.

[0044] During the training process, the semantic encoder aligns the semantic features output by the semantic encoder with the knowledge perception features with prior knowledge provided by the visual language model, thereby guiding the semantic encoder to focus on objects related to the target task.

[0045] For example, by inputting the enhanced semantic features and motion features into the decision network corresponding to the target task, the decision network can use the enhanced features to generate more accurate and reasonable decision information.

[0046] Optionally, in an embodiment of the present application, during the training process of the entire model network, a VLM-based representation supervision method can be introduced to train the semantic encoder to improve the feature extraction capability of the semantic encoder.

[0047] For example, Figure 4 FIG. 4 is a flow chart of a semantic encoder training process according to an embodiment of the present application. The training process includes the following steps 401 to 403: Step 401: Acquire multiple sample image frames.

[0048] Step 402: Input the multiple sample image frames into a semantic encoder to obtain a first semantic feature, and input the multiple sample image frames and the target prompt word into a visual language model to obtain a knowledge perception feature with prior knowledge.

[0049] Step 403: Calculate the similarity loss between the first semantic feature and the knowledge-aware feature, and adjust the parameters of the semantic encoder based on the similarity loss value.

[0050] For example, in the embodiment of the present application, a visual language model (VLM) is introduced to provide prior common sense for representation learning, supervise the encoder at the feature layer, enhance the representation information, and overcome the Figure 3 The problem of visual reinforcement learning shown in this paper is that it lacks interpretability and has difficulty aligning with real representations. In addition, the semantic-motion representation is decoupled. For the semantic branch, feature similarity is used to align the extracted representation with the representation generated by the visual language model. For the motion branch, cross-attention is used for interaction, enabling it to effectively extract the motion information of semantic objects.

[0051] For example, Figure 3As shown in the figure, during the training process, in order to utilize the common sense information provided by the visual language model (a type of large language model LLM), we input the observed image into the visual language model and use text prompts to inquire, allowing the visual language model VLM to reason, thereby obtaining object information related to the agent's decision in the observation; in order to effectively map this information to the feature space, so as to be able to supervise the extraction of representations, we can use the image segmentation algorithm based on CLIP (a type of visual language model) to generate a feature mask, so as to align these key objects in the image.

[0052] For example, the embodiment of the present application takes into account the use of a dual-branch network to extract different representations. Therefore, the embodiment of the present application adopts two methods for supervision. First, for the semantic branch, feature similarity loss is used for explicit supervision to align the extracted semantic features with the features with prior knowledge provided by the visual language model (i.e., the knowledge-aware features mentioned above), thereby directing the semantic encoder's focus area to some key objects (i.e., objects related to the target task).

[0053] Specifically, the above step 402 may further include the following steps 402a1 and 402a2: Step 402a1: Input the plurality of sample image frames and the target prompt word into a visual language model to obtain a feature mask.

[0054] Step 402a2: Input the feature mask into the semantic coding model to obtain knowledge-aware features with prior knowledge.

[0055] The knowledge perception feature and the first semantic feature output by the semantic encoding model have the same dimension.

[0056] For example, in the embodiment of the present application, the similarity loss between the knowledge perception feature and the semantic feature can be calculated using the following formula 1: (Formula 1) in, is the output of the visual language model, is The knowledge-aware features obtained by inputting into the semantic encoder are guaranteed to have the same dimension as the semantic representation.

[0057] Specifically, after step 402, the training method provided in the embodiment of the present application may further include the following steps 404 to 406: Step 404: Input the knowledge perception feature as a query value and the motion feature as a key value into the cross attention network to obtain a fusion feature expression of the first motion feature output by the action encoder and the first semantic feature.

[0058] Step 405: Use the fused feature expression to perform feature enhancement on the first semantic feature and the first motion feature respectively to obtain an enhanced second semantic feature and an enhanced second motion feature.

[0059] Step 406: Train a task decision network related to the target task based on the second semantic feature and the second motion feature.

[0060] For example, during the training phase, the motion representation output by the motion encoder and the semantic representation generated by the visual language model calculate cross-attention, while during the test, the motion representation output by the motion encoder interacts with the semantic representation generated by the semantic encoder, so that both branches of the network can effectively select key areas.

[0061] like Figure 5 As shown in FIG, the representation interaction process of the cross-attention provided by the embodiment of the present application is shown. The cross-attention network can be specifically represented by the following formula 2 and formula 3: (Formula 2) (Formula 3) Among them, X is the fusion feature expression.

[0062] For example, after obtaining the enhanced features, the fully connected layer can be used to compress the representation into and , the fused features It is input into the decision network and outputs an executable decision.

[0063] For example, during the entire training process, in order to reduce noise, the motion state prediction loss can be introduced by the following formula 4, and the reward loss function can be introduced by the following formula 5: (Formula 4) (Formula 5) For example, the objective optimization function of reinforcement learning can be expressed by the following formula 6: (Formula 6) For example, the action evaluation can be expressed by the following formula 7, and the policy loss can be expressed by the following formula 8: (Formula 7) (Formula 8) in, is the Bellman update equation, Represents the value function, which enables the Critic network to correctly evaluate the value of the decision, It is the policy loss that enables the policy distribution generated by the policy network to be aligned with the Q value.

[0064] The multimodal task decision method based on feature enhancement provided in the embodiment of the present application introduces a visual language model into the visual reinforcement learning task, provides prior common sense for representation learning, supervises the encoder at the feature layer, realizes the enhancement of representation information, and overcomes the problems of lack of interpretability and difficulty in aligning with real representations in visual reinforcement learning; secondly, semantic-motion representation decoupling is adopted. For the semantic branch, feature similarity is used to align the extracted representation to the representation generated by the visual language model. For the motion branch, cross-attention is used for interaction, so that it can effectively extract the motion information of semantic objects.

[0065] For example, in an experiment designed based on the CARLA simulation platform, the vehicle acts as a reinforcement learning agent, with the goal of driving as far as possible along the road shown in the figure while avoiding collisions. The experiment demonstrates that this solution improves the agent's control decision-making performance. Table 1 shows the average results from the CARLA simulation, showing that autonomous vehicles based on this solution can increase average driving distance by 30% and reduce average collision intensity by 8%.

[0066] Table 1 The present invention designs an ablation experiment based on the designed method to verify the effectiveness of the method. By decoupling the designed modules, models M1-M4 are obtained: M1: A single-branch encoder structure is used, its input is three frames of images, and multimodal LLM and VLM are not used for supervision.

[0067] M2: A dual-branch structure is adopted, with the input being a single-frame image and the frame difference between two adjacent frames, and no multimodal LLM and VLM are used for supervision.

[0068] M3: The structure is the same as M2, and multimodal LLM and VLM are used to align the feature similarity of semantic branches.

[0069] M4, based on M3, uses the representation generated by VLM to supervise motion representation and semantic-motion interaction through cross-attention.

[0070] Table 2 shows the experimental results, among which M1 has the lowest performance. This is because the pixel displacement between adjacent frames is very small, making it difficult to capture motion representation. Although the input of M1 is multi-frame images, the encoder tends to focus more on semantic information and ignore motion information, resulting in a high collision rate; M2 adopts a dual-branch network structure to decouple motion representation, which significantly improves performance; M3 introduces prior semantic knowledge on the basis of M2, which improves performance, but due to the lack of supervision of the motion branch, it is still prone to collision with some fast-moving objects in the scene, such as other vehicles; M4 introduces the guidance of prior representation in the semantic-motion representation interaction, which enables the model to pay more attention to some objects and predict their movement, further improving the obstacle avoidance ability.

[0071] Table 2 The multimodal task decision method based on feature enhancement provided in the embodiment of the present application is as follows: first, a plurality of image frames collected by a sensor are obtained, and the frame difference information between any two adjacent image frames in the plurality of image frames is calculated; the frame difference information is used to characterize the difference between the two adjacent image frames; then, the plurality of image frames are input into a semantic encoder to obtain the semantic features output by the semantic encoder, and the frame difference information is input into a motion encoder to obtain the motion features output by the motion encoder; and the semantic features and the motion features are input into a cross-attention network for feature enhancement to obtain enhanced semantic features and enhanced motion features; finally, the enhanced semantic features and the enhanced motion features are input into a task decision network related to the target task to generate corresponding decision information; wherein, during the training process, the semantic encoder aligns the semantic features output by the semantic encoder with the knowledge perception features with prior knowledge provided by the visual language model, thereby guiding the semantic encoder to focus on objects related to the target task. In this way, by using the knowledge perception features with prior knowledge provided by the visual language model to guide the visual encoder to focus on objects related to the task during the training phase, the feature extraction capability of the visual encoder is greatly improved.

[0072] It should be noted that the execution subject of the feature enhancement-based multimodal task decision method provided in the embodiment of the present application can be a feature enhancement-based multimodal task decision device, or a control module in the feature enhancement-based multimodal task decision device for executing the feature enhancement-based multimodal task decision method. In the embodiment of the present application, the feature enhancement-based multimodal task decision device executing the feature enhancement-based multimodal task decision method is taken as an example to illustrate the feature enhancement-based multimodal task decision device provided in the embodiment of the present application.

[0073] It should be noted that in the embodiments of the present application, the multimodal task decision method based on feature enhancement shown in the drawings of the above-mentioned methods is illustrated as an example in combination with one of the drawings in the embodiments of the present application. In specific implementation, the multimodal task decision method based on feature enhancement shown in the drawings of the above-mentioned methods can also be implemented in combination with any other drawings that can be combined as shown in the above-mentioned embodiments, and will not be repeated here.

[0074] The following describes the multimodal task decision device based on feature enhancement provided by the present application. The multimodal task decision method based on feature enhancement described below and described above can refer to each other.

[0075] Figure 6 A schematic diagram of the structure of a multimodal task decision-making device based on feature enhancement provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, specifically including: An information acquisition module 601 is used to acquire multiple image frames captured by a sensor and calculate the frame difference information between any two adjacent image frames in the multiple image frames; the frame difference information is used to characterize the difference between two adjacent image frames; a feature extraction module 602 is used to input the multiple image frames into a semantic encoder to obtain semantic features output by the semantic encoder, and to input the frame difference information into a motion encoder to obtain motion features output by the motion encoder; a feature enhancement module 603 is used to input the semantic features and the motion features into a cross-attention network for feature enhancement to obtain enhanced semantic features and enhanced motion features; a task decision module 604 is used to input the enhanced semantic features and the enhanced motion features into a task decision network related to a target task to generate corresponding decision information; wherein, during the training process, the semantic encoder aligns the semantic features output by the semantic encoder with the knowledge perception features with prior knowledge provided by the visual language model, thereby guiding the semantic encoder to focus on objects related to the target task.

[0076] Optionally, the information acquisition module 601 is specifically used to obtain the color value of each pixel in the first image frame and the second image frame, and calculate the difference between the color value of each pixel in the first image frame and the color value of the corresponding pixel in the second image frame to obtain the pixel color difference between the first image frame and the second image frame; the information acquisition module 601 is also specifically used to generate frame difference information between the first image frame and the second image frame based on the pixel color difference; wherein, the first image frame and the second image frame are two adjacent image frames among the multiple image frames.

[0077] Optionally, the feature enhancement module 603 is specifically used to input the semantic feature as a query value and the motion feature as a key value into the cross-attention network to obtain a fused feature expression of the semantic feature and the motion feature; the feature enhancement module 603 is also specifically used to use the fused feature expression to perform feature enhancement on the semantic feature and the motion feature respectively to obtain the enhanced semantic feature and the enhanced motion feature.

[0078] Optionally, the device also includes: a training module; the information acquisition module 601 is also used to obtain multiple sample image frames; the training module is used to input the multiple sample image frames into the semantic encoder to obtain a first semantic feature, and input the multiple sample image frames and the target prompt word into the visual language model to obtain a knowledge perception feature with prior knowledge; the training module is used to calculate the similarity loss between the first semantic feature and the knowledge perception feature, and adjust the parameters of the semantic encoder based on the similarity loss value.

[0079] Optionally, the feature enhancement module 603 is also used to input the multiple sample image frames and the target prompt word into the visual language model to obtain a feature mask; the feature enhancement module 603 is also used to input the feature mask into the semantic coding model to obtain a knowledge perception feature with prior knowledge; wherein the knowledge perception feature has the same dimension as the first semantic feature output by the semantic coding model.

[0080] Optionally, the feature enhancement module 603 is also used to input the knowledge perception feature as a query value and the motion feature as a key value into the cross-attention network to obtain a fused feature expression of the first motion feature output by the action encoder and the first semantic feature; the feature enhancement module 603 is also used to use the fused feature expression to perform feature enhancement on the first semantic feature and the first motion feature respectively to obtain an enhanced second semantic feature and an enhanced second motion feature; the training module is also used to train a task decision network related to the target task based on the second semantic feature and the second motion feature.

[0081] The multimodal task decision device based on feature enhancement provided by the present application first obtains multiple image frames collected by the sensor and calculates the frame difference information between any two adjacent image frames in the multiple image frames; the frame difference information is used to characterize the difference between the two adjacent image frames; then, the multiple image frames are input into the semantic encoder to obtain the semantic features output by the semantic encoder, and the frame difference information is input into the motion encoder to obtain the motion features output by the motion encoder; and the semantic features and the motion features are input into the cross-attention network for feature enhancement to obtain enhanced semantic features and enhanced motion features; finally, the enhanced semantic features and the enhanced motion features are input into the task decision network related to the target task to generate corresponding decision information; wherein, during the training process, the semantic encoder aligns the semantic features output by the semantic encoder with the knowledge perception features with prior knowledge provided by the visual language model, thereby guiding the semantic encoder to focus on objects related to the target task. In this way, by using the knowledge perception features with prior knowledge provided by the visual language model to guide the visual encoder to focus on objects related to the task during the training phase, the feature extraction capability of the visual encoder is greatly improved.

[0082] Figure 7 An example of a physical structure diagram of an electronic device is shown below. Figure 7As shown, the electronic device may include: a processor (processor) 710 , a communication interface (Communications Interface) 720 , a memory (memory) 730 and a communication bus 740 , wherein the processor 710 , the communication interface 720 and the memory 730 communicate with each other via the communication bus 740 . The processor 710 can call the logic instructions in the memory 730 to execute a multimodal task decision method based on feature enhancement, which includes: first, obtaining multiple image frames captured by the sensor, and calculating the frame difference information between any two adjacent image frames in the multiple image frames; the frame difference information is used to characterize the difference between two adjacent image frames; then, inputting the multiple image frames into the semantic encoder to obtain the semantic features output by the semantic encoder, and inputting the frame difference information into the motion encoder to obtain the motion features output by the motion encoder; and inputting the semantic features and the motion features into the cross-attention network for feature enhancement to obtain enhanced semantic features and enhanced motion features; finally, inputting the enhanced semantic features and the enhanced motion features into the task decision network related to the target task to generate corresponding decision information; wherein, during the training process, the semantic encoder aligns the semantic features output by the semantic encoder with the knowledge perception features with prior knowledge provided by the visual language model, thereby guiding the semantic encoder to focus on objects related to the target task. In this way, by using the knowledge perception features with prior knowledge provided by the visual language model during the training phase to guide the visual encoder to focus on objects related to the task, the feature extraction ability of the visual encoder is greatly improved.

[0083] In addition, the logical instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the existing technology, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program code.

[0084] On the other hand, the present application also provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the multimodal task decision method based on feature enhancement provided by the above methods, the method comprising: first, obtaining multiple image frames captured by a sensor, and calculating frame difference information between any two adjacent image frames in the multiple image frames; the frame difference information is used to characterize the difference between two adjacent image frames; thereafter, the multiple image frames are input into a semantic encoder to obtain semantic features output by the semantic encoder, and The frame difference information is input into the motion encoder to obtain the motion features output by the motion encoder; and the semantic features and the motion features are input into the cross-attention network for feature enhancement to obtain enhanced semantic features and enhanced motion features; finally, the enhanced semantic features and the enhanced motion features are input into the task decision network related to the target task to generate corresponding decision information; wherein, during the training process, the semantic encoder aligns the semantic features output by the semantic encoder with the knowledge perception features with prior knowledge provided by the visual language model, thereby guiding the semantic encoder to focus on objects related to the target task. In this way, by using the knowledge perception features with prior knowledge provided by the visual language model to guide the visual encoder to focus on objects related to the task during the training phase, the feature extraction capability of the visual encoder is greatly improved.

[0085] On the other hand, the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the above-mentioned multimodal task decision method based on feature enhancement, the method comprising: first, obtaining multiple image frames captured by a sensor, and calculating the frame difference information between any two adjacent image frames in the multiple image frames; the frame difference information is used to characterize the difference between two adjacent image frames; thereafter, inputting the multiple image frames into a semantic encoder to obtain the semantic features output by the semantic encoder, and inputting the frame difference information into a motion encoder to obtain the motion features output by the motion encoder; and inputting the semantic features and the motion features into a cross-attention network for feature enhancement to obtain enhanced semantic features and enhanced motion features; finally, inputting the enhanced semantic features and the enhanced motion features into a task decision network related to the target task to generate corresponding decision information; wherein, during the training process, the semantic encoder aligns the semantic features output by the semantic encoder with the knowledge perception features with prior knowledge provided by the visual language model, thereby guiding the semantic encoder to focus on objects related to the target task. In this way, by using the knowledge perception features with prior knowledge provided by the visual language model during the training phase to guide the visual encoder to focus on objects related to the task, the feature extraction ability of the visual encoder is greatly improved.

[0086] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0087] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A multimodal task decision method based on feature enhancement, characterized in that: include: Acquire multiple image frames captured by the sensor, and calculate frame difference information between any two adjacent image frames in the multiple image frames; The frame difference information is used to represent the difference between two adjacent image frames; Inputting the plurality of image frames into a semantic encoder to obtain semantic features output by the semantic encoder, and inputting the frame difference information into a motion encoder to obtain motion features output by the motion encoder; Inputting the semantic features and the motion features into a cross attention network for feature enhancement to obtain enhanced semantic features and enhanced motion features; Inputting the enhanced semantic features and the enhanced motion features into a task decision network related to the target task to generate corresponding decision information; During the training process, the semantic encoder aligns the semantic features output by the semantic encoder with the knowledge perception features with prior knowledge provided by the visual language model, thereby guiding the semantic encoder to focus on objects related to the target task.

2. The multimodal task decision method based on feature enhancement according to claim 1, characterized in that: The calculating frame difference information between any two adjacent image frames in the plurality of image frames includes: Obtaining a color value of each pixel in a first image frame and a second image frame, and calculating a difference between the color value of each pixel in the first image frame and the color value of the corresponding pixel in the second image frame, to obtain a pixel color difference between the first image frame and the second image frame; generating frame difference information between the first image frame and the second image frame based on the pixel color difference value; The first image frame and the second image frame are two adjacent image frames among the multiple image frames.

3. The multimodal task decision method based on feature enhancement according to claim 1, characterized in that: Inputting the semantic features and the motion features into a cross attention network for feature enhancement to obtain enhanced semantic features and enhanced motion features includes: Inputting the semantic feature as a query value and the motion feature as a key value into the cross attention network to obtain a fusion feature expression of the semantic feature and the motion feature; The semantic feature and the motion feature are respectively enhanced using the fused feature expression to obtain the enhanced semantic feature and the enhanced motion feature.

4. The multimodal task decision method based on feature enhancement according to claim 1, characterized in that: The semantic encoder is trained based on the following steps: Acquire multiple sample image frames; Inputting the plurality of sample image frames into a semantic encoder to obtain a first semantic feature, and inputting the plurality of sample image frames and a target prompt word into a visual language model to obtain a knowledge perception feature with prior knowledge; A similarity loss between the first semantic feature and the knowledge-aware feature is calculated, and parameters of the semantic encoder are adjusted based on the similarity loss value.

5. The multimodal task decision method based on feature enhancement according to claim 4, characterized in that: The step of inputting the plurality of sample image frames and the target prompt word into a visual language model to obtain knowledge perception features with prior knowledge includes: Inputting the plurality of sample image frames and the target prompt word into a visual language model to obtain a feature mask; Inputting the feature mask into the semantic encoding model to obtain knowledge-aware features with prior knowledge; The knowledge perception feature and the first semantic feature output by the semantic encoding model have the same dimension.

6. The multimodal task decision method based on feature enhancement according to claim 4 or 5, characterized in that: After inputting the plurality of sample image frames into a semantic encoder to obtain a first semantic feature, and inputting the plurality of sample image frames and a target prompt word into a visual language model to obtain a knowledge perception feature with prior knowledge, the method further includes: Inputting the knowledge perception feature as a query value and the motion feature as a key value into the cross attention network to obtain a fusion feature expression of the first motion feature output by the action encoder and the first semantic feature; performing feature enhancement on the first semantic feature and the first motion feature respectively by using the fused feature expression to obtain an enhanced second semantic feature and an enhanced second motion feature; A task decision network related to the target task is trained based on the second semantic feature and the second motion feature.

7. A multimodal task decision-making device based on feature enhancement, characterized in that: The device comprises: An information acquisition module is used to acquire multiple image frames collected by the sensor and calculate frame difference information between any two adjacent image frames in the multiple image frames; the frame difference information is used to represent the difference between the two adjacent image frames; a feature extraction module, configured to input the plurality of image frames into a semantic encoder to obtain semantic features output by the semantic encoder, and input the frame difference information into a motion encoder to obtain motion features output by the motion encoder; A feature enhancement module, configured to input the semantic features and the motion features into a cross attention network for feature enhancement, thereby obtaining enhanced semantic features and enhanced motion features; A task decision module, configured to input the enhanced semantic features and the enhanced motion features into a task decision network related to a target task to generate corresponding decision information; During the training process, the semantic encoder aligns the semantic features output by the semantic encoder with the knowledge perception features with prior knowledge provided by the visual language model, thereby guiding the semantic encoder to focus on objects related to the target task.

8. The multimodal task decision-making device based on feature enhancement according to claim 7, characterized in that: The device further comprises: a training module; The information acquisition module is further used to acquire multiple sample image frames; The training module is configured to input the plurality of sample image frames into a semantic encoder to obtain a first semantic feature, and input the plurality of sample image frames and a target prompt word into a visual language model to obtain a knowledge perception feature with prior knowledge; The training module is used to calculate the similarity loss between the first semantic feature and the knowledge perception feature, and adjust the parameters of the semantic encoder based on the similarity loss value.

9. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the multimodal task decision method based on feature enhancement according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the computer program is executed by a processor, the steps of the multimodal task decision method based on feature enhancement as claimed in any one of claims 1 to 6 are implemented.

Images