Multi-modal multi-dimensional feature collaborative representation learning method and device for open environment

Abstract

The application provides a multi-modal multi-dimensional feature collaborative representation learning method and device for an open environment, effectively solving the problem that the existing multi-modal multi-dimensional feature collaborative representation learning method for an open environment cannot solve the environment and modal two types of shift while maintaining task discrimination ability. The method comprises the following steps: mapping a multi-dimensional feature collaborative representation learning task of the open environment to a task model in advance, and training the task model; receiving multi-modal perception data based on the trained task model, dividing the perception dimensions of the multi-modal perception data, and extracting the dimensional features in each perception dimension; processing the dimensional features through a preset three-level adversarial constraint module to obtain collaborative features, and inputting the collaborative features into a global prediction module; processing the collaborative features to obtain a target task prediction result in the open environment.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a method and apparatus for learning multimodal and multidimensional feature collaborative representations for open environments. Background Technology

[0002] Multimodal, multidimensional feature learning, by fusing heterogeneous data from different perceptual sources such as vision and audio, fully leverages the complementarity between multidimensional features to alleviate performance bottlenecks caused by the lack of single-dimensional information, and has made significant progress in fields such as video understanding and human-computer interaction. However, in real-world applications in open environments, the environmental distribution of training and testing data is offset, which has become a core factor restricting the model's generalization ability.

[0003] To alleviate this problem, robust generalization techniques for open environments have been gradually developed. The core of these techniques is to train models using limited-source environmental data so that they can maintain excellent performance in unknown target environments. The key lies in extracting environment-independent features that focus on the core semantics of the task and filter out environmental interference. These methods are highly effective in single-dimensional scenarios.

[0004] However, when the generalization problem is extended to multimodal and multidimensional scenarios, the complexity increases significantly: the system needs to simultaneously cope with environmental distribution shifts and heterogeneous modal shifts, and the two types of shifts are coupled with each other. The perception mechanisms and statistical characteristics of different perception dimensions differ significantly, forming modal shifts; while the asymmetric impact of environmental disturbances on different modalities further exacerbates the perception difficulty.

[0005] Existing technologies have obvious limitations: traditional environment generalization methods are designed for single dimensions and assume data homogeneity, making it difficult to adapt to multimodal heterogeneous features. Independent learning of environment-invariant features in each dimension can lead to semantic inconsistencies between dimensions. Existing multimodal fusion methods do not explicitly model modality invariance and do not jointly constrain the two types of biases, which can easily lead to problems such as dominant modality in decision-making and fragmented feature distribution, thus weakening the reliability of the model in unknown environments.

[0006] In summary, multimodal and multidimensional feature collaborative representation learning for open environments needs to simultaneously address both environment and modality shift problems, achieving collaborative learning with both environment and modality invariance. Balancing these two aspects and overcoming the offset coupling problem while maintaining task discriminative capability remains a critical technical challenge. Summary of the Invention

[0007] In view of this, the purpose of this application is to provide a multimodal and multidimensional feature collaborative representation learning method and apparatus for open environments, which effectively solves the problem that existing multimodal and multidimensional feature collaborative representation learning methods for open environments cannot solve the two-class shift of environment and mode while maintaining task discrimination ability.

[0008] In a first aspect, embodiments of this application provide a multimodal, multidimensional feature collaborative representation learning method for open environments, the method comprising: The multidimensional feature collaborative representation learning task of the open environment is pre-mapped as a task model, and the task model is trained based on a comprehensive loss function; the task model includes a feature extraction module, a three-level adversarial constraint module, and a global prediction module; Based on the trained task model, multimodal perception data in an open environment is received, the multimodal perception data is divided into perception dimensions, and the dimensional features in each perception dimension are extracted through the feature extraction module. The dimensional features are processed collaboratively by a pre-set three-level adversarial constraint module to obtain globally consistent collaborative features, and these collaborative features are then input into the global prediction module. Based on the processing of the collaborative features by the global prediction module, the target task prediction result in the open environment is obtained, so as to complete the multimodal and multidimensional feature collaborative representation learning task for the open environment.

[0009] In conjunction with the first aspect, this application provides a first possible implementation of the first aspect, wherein the step of obtaining globally consistent collaborative features by collaboratively processing the dimensional features through a preset three-level adversarial constraint module includes: The three-level adversarial constraint module is pre-configured so that it has multiple units; different units have different processing methods. The processing method is executed so that the corresponding unit performs corresponding collaborative processing on the received dimensional features.

[0010] In conjunction with the first aspect, this application provides a second possible implementation of the first aspect, wherein the unit includes at least a distributed environment sensitivity discrimination unit; The processing method described above is executed so that the corresponding unit performs corresponding collaborative processing on the received dimensional features, including: The distributed environment sensitivity discrimination unit identifies environment-specific patterns with single-dimensional features. Suppress the environment-specific patterns of the single-dimensional features to output environment-robust single-dimensional features.

[0011] In conjunction with the first aspect, this application provides a third possible implementation of the first aspect, wherein the unit includes at least a heterogeneous dimension constancy discrimination unit; The processing method described above is executed so that the corresponding unit performs corresponding collaborative processing on the received dimensional features, including: The heterogeneous dimension constancy discrimination unit distinguishes the feature types of different perception dimensions. Suppress the specific modalities of each perceptual dimension to output dimension-constant single-dimensional features.

[0012] In conjunction with the first aspect, this application provides a fourth possible implementation of the first aspect, wherein the unit includes at least a global scene collaborative discrimination unit; The processing method described above is executed so that the corresponding unit performs corresponding collaborative processing on the received dimensional features, including: The fused feature is obtained by integrating all dimension-constant single-dimensional features and environment-robust single-dimensional features; Suppress environmentally sensitive information in the fused features and output globally consistent collaborative features.

[0013] In conjunction with the first aspect, this application provides a fifth possible implementation of the first aspect, wherein processing the collaborative features based on the global prediction module to obtain the target task prediction result in an open environment includes: The collaborative features are mapped layer by layer using a multilayer perceptron structure, and the predicted probability distribution of each target task category is output. Based on the predicted probability distribution, the task category with the highest probability is selected as the target task prediction result.

[0014] In conjunction with the first aspect, this application provides a sixth possible implementation of the first aspect, wherein training the task model based on the comprehensive loss function includes: The task model performs feature extraction, three-level adversarial constraint processing, and global task prediction on the training data to obtain the process output results. A comprehensive loss function is constructed that integrates task prediction loss and level 3 adversarial constraint loss to calculate the comprehensive loss value from the output results of the process. Based on the comprehensive loss value, the parameters of each module of the task model are iteratively updated according to the parameter grouping rules until the task model converges.

[0015] Secondly, embodiments of this application provide a multimodal, multidimensional feature collaborative representation learning device for open environments, the device comprising: The training module is used to pre-map the multi-dimensional feature collaborative representation learning task of the open environment into a task model, and train the task model based on a comprehensive loss function; the task model includes a feature extraction module, a three-level adversarial constraint module, and a global prediction module; The receiving module is used to receive multimodal perception data in an open environment based on the task model after training, divide the multimodal perception data into perception dimensions, and extract the dimensional features in each perception dimension through the feature extraction module. The input module is used to collaboratively process the dimensional features through a pre-set three-level adversarial constraint module to obtain globally consistent collaborative features, and input the collaborative features into the global prediction module; The processing module is used to process the collaborative features based on the global prediction module to obtain the target task prediction result in the open environment, so as to complete the multimodal and multidimensional feature collaborative representation learning task for the open environment.

[0016] Thirdly, embodiments of this application provide an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of any of the methods described in the "Multimodal Multidimensional Feature Collaborative Representation Learning Method for Open Environments".

[0017] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of any of the methods for learning multimodal and multidimensional feature collaborative representations for open environments.

[0018] This application provides a multimodal, multidimensional feature collaborative representation learning method for open environments. The method pre-maps the multimodal feature collaborative representation learning task for the open environment as a task model and trains the task model based on a comprehensive loss function. The task model includes a feature extraction module, a three-level adversarial constraint module, and a global prediction module. Next, based on the trained task model, it receives multimodal perception data from the open environment, divides the multimodal perception data into perception dimensions, and extracts dimensional features from each perception dimension through the feature extraction module. Then, it collaboratively processes the dimensional features through a pre-set three-level adversarial constraint module to obtain globally consistent collaborative features, and inputs these collaborative features into the global prediction module. Finally, it processes the collaborative features based on the global prediction module to obtain the target task prediction result in the open environment, thus completing the multimodal, multidimensional feature collaborative representation learning task for open environments. Based on the above methods, this application precisely solves the intertwined problem of environmental distribution shift and heterogeneous modality shift through the synergistic effect of a pre-built three-level adversarial constraint module. It simultaneously achieves collaborative learning of environmental invariance and modality invariance, successfully constructing a unified and robust multimodal multidimensional collaborative representation space, breaking the limitation of existing methods that isolate and process single shifts. It also effectively solves the problems of multidimensional feature semantic misalignment and dominant modality-driven decision-making in traditional methods. While fully preserving the complementarity of each perception dimension, it avoids feature distribution fragmentation and ensures semantic consistency between dimensions. Furthermore, it significantly improves the model's generalization ability and prediction reliability in unknown open environments, effectively alleviates perception degradation caused by environmental interference, and enables the model to maintain excellent task discrimination performance in cross-environment and multimodal complex scenarios, perfectly adapting to the practical application needs of real open scenarios such as video understanding, human-computer interaction, and autonomous driving. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 The diagram illustrates a flowchart of a multimodal, multidimensional feature collaborative representation learning method for open environments provided in an embodiment of this application. Figure 2 A schematic diagram comparing the input architectures of single-dimensional feature learning and multi-dimensional feature collaborative learning provided in the embodiments of this application is shown. Figure 3 This illustration shows a schematic diagram of the method architecture for a multimodal, multidimensional feature collaborative representation learning method for open environments, provided by an embodiment of this application. Figure 4 This paper shows a structural block diagram of a multimodal, multidimensional feature collaborative representation learning device for open environments, provided in an embodiment of this application. Figure 5 A structural block diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0022] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0023] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0024] Currently, existing technologies cannot effectively address the dual offset coupling problem in open environments, making it difficult to construct a collaborative representation space that combines environment and modality invariance. This results in insufficient generalization ability and reliability of multimodal models in unknown open scenarios, necessitating a novel learning method to overcome these shortcomings.

[0025] Based on this, the embodiments of this application provide a multimodal and multidimensional feature collaborative representation learning method and apparatus for open environments, which are described below through embodiments.

[0026] Example 1 To facilitate understanding of this embodiment, a detailed description of the multimodal, multidimensional feature collaborative representation learning method for open environments disclosed in this application embodiment will be provided first. For example... Figure 1 The diagram illustrates a flowchart of a multimodal, multidimensional feature collaborative representation learning method for open environments. This application provides a multimodal, multidimensional feature collaborative representation learning method for open environments, the method comprising: S101. The multi-dimensional feature collaborative representation learning task of the open environment is pre-mapped into a task model, and the task model is trained based on a comprehensive loss function; the task model includes a feature extraction module, a three-level adversarial constraint module, and a global prediction module; S102. Based on the task model after training, receive multimodal perception data in an open environment, divide the multimodal perception data into perception dimensions, and extract the dimensional features in each perception dimension through the feature extraction module. S103. The dimensional features are processed collaboratively by a pre-set three-level adversarial constraint module to obtain globally consistent collaborative features, and the collaborative features are input into the global prediction module. S104. Based on the global prediction module, the collaborative features are processed to obtain the target task prediction result in the open environment, so as to complete the multimodal and multidimensional feature collaborative representation learning task for the open environment.

[0027] Due to the uncertainty of open environments, there is an environmental distribution shift between different source environments, specifically manifested as the joint distribution of various environments. They are all different. The goal of this application is to train a collaborative representation model based on known environmental data, enabling it to generalize to open and unknown environments not encountered during the training phase. And the joint distribution of this unknown environment It differs from the joint distribution of all known environments (i.e., for any...) ,satisfy ).

[0028] In step S101, this application pre-maps the multi-dimensional feature collaborative representation learning task of the open environment into a task model, thereby achieving the effect of task visualization. The core requirements of extracting robust collaborative features and achieving generalization in the open environment are transformed into modular division of labor within the model. This clarifies that the model needs to possess the full-process capability of feature extraction, bias constraint, and prediction completion, avoiding training without a clear structural direction. The task model includes an input space and an output space, wherein the input space is defined as... ,Include The perception dimension corresponds to multimodal data in an open environment, specifically heterogeneous modalities such as vision and audio; the output space is denoted as... The multi-dimensional feature collaborative representation learning task is set to include A known source environment dataset Each environment dataset consists of It consists of 10 labeled samples, i.e. In the sample, the input data The label data is in the form of multi-dimensional feature splicing. To correspond with the annotation, the task model is trained using the known source environment dataset as training data. Based on the comprehensive loss function, the task model is continuously trained on the known source environment dataset to adjust its internal parameters until the model reaches a convergent state, thus obtaining the trained task model. The obtained task model includes a feature extraction module, a three-level adversarial constraint module, and a global prediction module. These three modules are the core components of the task model, with clear division of labor and collaborative work, ultimately achieving the core goal of multimodal and multidimensional feature collaborative representation and accurate prediction in an open environment.

[0029] In a specific implementation of step S101, one embodiment is as follows: training the task model based on the comprehensive loss function includes: S1011. The task model performs feature extraction, three-level adversarial constraint processing, and global task prediction on the training data to obtain the process output result. S1012. Construct a comprehensive loss function that integrates task prediction loss and level 3 adversarial constraint loss to calculate the comprehensive loss value from the output results of the process. S1013. Based on the comprehensive loss value, iteratively update the parameters of each module of the task model according to the parameter grouping rules until the task model converges.

[0030] In steps S1011-S1013, during multiple training iterations of the task model, this application fully executes the entire process of feature extraction, three-level adversarial constraint processing, and global task prediction for each training round, thereby obtaining the output result of each training round. The parameter update for each round is guided by a comprehensive loss function. This comprehensive loss function is constructed based on the task prediction loss and the three-level adversarial constraint loss, simultaneously reflecting two core issues: the accuracy of task prediction and whether the features have eliminated environmental / dimensional biases. The comprehensive loss function calculates the comprehensive loss value from the process output result and is expressed as follows: (1); in For the comprehensive loss function, The cross-entropy loss function is used for task prediction loss. , , These are the loss functions in the three-level adversarial constraint module, used to suppress dimension-specific bias, environmental bias, and global environmental bias, respectively. Based on the comprehensive loss value, the module parameters of the task model, including the feature extraction module, the three-level adversarial constraint module, and the global prediction module, are iteratively updated according to parameter grouping rules. Specifically, the module parameters of each module are divided into three groups: dimensional feature extraction module parameters... (correspond ), global prediction module parameters (correspond ), and hierarchical adversarial discrimination unit parameters These correspond to the distributed environment sensitivity discrimination unit, respectively. Global Scene Collaborative Discrimination Unit and heterogeneous dimensional constancy discriminant unit The learning rate is denoted as .

[0031] Global prediction module parameter update rules:

[0032] The parameter update rules for each unit in the Level 3 adversarial constraint module are as follows:

[0033]

[0034] ; Parameter update rules for the dimensional feature extraction module: Parameter updates are achieved by fusing the gradient predicted from the perceptual task with the adversarial bias gradient propagated via the gradient inversion layer.

[0035] in and The gradient reversal strength coefficients for the environment-sensitive discrimination unit and the dimensionality-constant discrimination unit in the three-level adversarial constraint module are used. After multiple iterations, the comprehensive loss value is calculated to be minimized, indicating that the task model has converged. At this point, the environment / dimensionality bias has been eliminated, and the task prediction is most accurate. After training convergence, the task model enters the inference phase. At this stage, no further loss calculation or parameter updates are performed; only the trained task model is used to complete feature extraction and task prediction. The parameter grouping update strategy divides the model parameters into three groups: parameters of the feature extraction module, parameters of each discrimination unit in the three-level adversarial constraint module, and parameters of the global prediction module. Independent gradient calculations and updates are performed on each group, and the update magnitude of each parameter is adjusted in conjunction with the preset gradient reversal strength coefficients. This effectively avoids interference between parameters of different modules and prevents problems such as gradient vanishing and gradient exploding. This collaborative optimization approach not only improves the training efficiency of the task model (reducing redundant steps in parameter adjustment and shortening the training cycle) but also ensures the convergence stability of the task model (ensuring that the model can converge stably to the optimal state). Ultimately, it reduces the difficulty and deployment cost of engineering implementation of the task model, enabling the technical solution of this application to quickly adapt to the deployment needs of actual application scenarios.

[0036] The loss function corresponding to the distributed environment sensitivity discrimination unit in the third-level adversarial constraint module is the cross-entropy loss function, which is defined as follows:

[0037] in For environment labels, projection layer Used to map features of different dimensions to the same dimensional space; the loss function of the heterogeneous dimensional constancy discrimination unit in the three-level adversarial constraint module is expressed by formula (8):

[0038] in The dimension (modal) label; the loss function corresponding to the global scene collaborative discrimination unit in the three-level adversarial constraint module is expressed by formula (9):

[0039] The cross-entropy loss function for task prediction loss is expressed by formula (10):

[0040] in This is the global task prediction module.

[0041] In step S102, after the task model has reached convergence through iterative training, this application receives multimodal perception data in an open environment. The multimodal perception data includes at least two types of perception information, such as visual, audio, and optical flow. Figure 2 As shown, the multimodal perception data is divided into perception dimensions, that is, the perception data corresponding to vision, audio, and sensors are identified respectively, and the dimensional features of each perception dimension are extracted by the feature extraction module. Specifically, the feature extraction module... The first Perceptual data input from each perceptual dimension Mapping to dimensional features This allows for the extraction of spatial features from the video dimension and temporal features from the audio dimension. It preserves the unique information of each dimension and lays the foundation for subsequent collaborative optimization. It avoids the difficulty of processing different types of data mixed together, solves the problem of heterogeneity and disorder in multimodal perception data, and makes subsequent constraints and optimizations more targeted, such as suppressing illumination interference separately for the video dimension and suppressing noise interference separately for the audio dimension.

[0042] In S103, this application pre-sets a three-level adversarial constraint module. This module comprises three units: a distributed environment sensitivity discrimination unit, a heterogeneous dimensional constancy discrimination unit, and a global scene collaborative discrimination unit. These three units do not work independently but follow a hierarchical logic of single-dimensional constraints, cross-dimensional constraints, and global fusion constraints. They work collaboratively and progressively optimize the features of each dimension. Through the pre-set three-level adversarial constraint module, the dimensional features are processed in a layered and collaborative manner, gradually eliminating the interference of environmental distribution shifts and heterogeneous modal shifts, resulting in globally consistent collaborative features, such as... Figure 3 As shown, the collaborative features are input into the global prediction module to provide robust and unified feature support for subsequent target task prediction in open environments. Based on the hierarchical logic between the three units, the problem of the intertwining of environmental distribution shift and heterogeneous modality shift is accurately solved. The collaborative learning of environmental invariance and modality invariance is realized simultaneously, and a unified and robust multimodal multidimensional collaborative representation space is successfully constructed. This breaks the limitation of existing methods in handling single shifts in isolation and effectively solves the problems of multidimensional feature semantic misalignment and dominant modality decision-making in traditional methods. While fully preserving the complementarity of each perception dimension, feature distribution fragmentation is avoided, and semantic consistency between dimensions is ensured. This significantly improves the model's generalization ability and prediction reliability in unknown open environments, effectively alleviates perception degradation caused by environmental interference, and enables the model to maintain excellent task discrimination performance in cross-environment and multimodal complex scenarios. It perfectly adapts to the actual application needs of real open scenarios such as video understanding, human-computer interaction, and autonomous driving.

[0043] In a specific implementation of step S103, one embodiment is as follows: the method of obtaining globally consistent collaborative features by collaboratively processing the dimensional features through a pre-set three-level adversarial constraint module includes: S1031. The three-level adversarial constraint module is pre-configured so that the three-level adversarial constraint module has multiple units; different units have different processing methods. S1032. Execute the processing method so that the corresponding unit performs corresponding collaborative processing on the received dimensional features.

[0044] In this application, a three-level adversarial constraint module is set up based on the hierarchical logic of single-dimensional constraints, cross-dimensional constraints, and global fusion constraints. The three-level adversarial constraint module is pre-configured to have multiple units, including a distributed environment sensitivity discrimination unit, a heterogeneous dimension constancy discrimination unit, and a global scene collaborative discrimination unit. Different units have different loss functions to suppress corresponding environment / dimensional biases and execute the processing methods of each unit to enable the corresponding units to complete collaborative processing of the dimensional features. Through the collaborative efforts of the three-level adversarial constraint module, the distributed environment... The environment-sensitive discrimination unit specifically suppresses environment distribution shifts, the heterogeneous dimension constancy discrimination unit specifically suppresses heterogeneous modality shifts, and the global scene collaborative discrimination unit further strengthens the global-level shift suppression and feature collaboration. The three units work in a progressive and collaborative manner to achieve simultaneous learning of environment invariance and modality invariance. This not only frees features from the interference of specific environments but also enables unified alignment of features from different modalities, ultimately constructing a unified and robust collaborative representation space. This addresses the technical pain point of the intertwining of two types of shifts at the core level and completely breaks the limitation of existing methods that can only handle single shifts in isolation, providing core technical support for multimodal learning in open environments.

[0045] In a specific implementation of step S1032, one embodiment is as follows: the unit includes at least a distributed environment sensitivity discrimination unit; The processing method described above is executed so that the corresponding unit performs corresponding collaborative processing on the received dimensional features, including: A1. The distributed environment sensitivity discrimination unit identifies environment-specific patterns with single-dimensional features. A2. Suppress the environment-specific patterns of the single-dimensional features to output environment-robust single-dimensional features.

[0046] In steps A1-A2, this application processes single-dimensional features in a distributed environment sensitivity discrimination unit, that is, processes the feature level to identify environment-specific patterns of single-dimensional features. The distributed environment sensitivity discrimination unit adopts a multilayer perceptron (MLP) structure, and the input is the features of each dimension. (in The output module predicts the environment to which the features belong, and extracts features from each dimension. By receiving the inverse gradient signal through the gradient inversion layer, the suppression is achieved. The environment-specific patterns in the data, such as the interference information in the visual dimension affected by changes in lighting and background texture, and the interference information in the audio dimension affected by device noise and environmental noise, enable the task model to learn a more robust cross-environment representation. That is, each dimension of the feature retains only the core semantic information related to the target task and removes the interference components related to the collection environment, ensuring that the features of a single dimension have a stable representation in different environments, laying the foundation for subsequent cross-dimensional constraints.

[0047] In the specific implementation of step S1032, another embodiment exists in which the unit includes at least a heterogeneous dimension constancy discrimination unit. The processing method described above is executed so that the corresponding unit performs corresponding collaborative processing on the received dimensional features, including: B1. The heterogeneous dimension constancy discrimination unit distinguishes the feature types of different perception dimensions; B2. Suppress the specific modalities of each perceptual dimension to output dimension-constant single-dimensional features.

[0048] In steps B1-B2, after completing the single-dimensional environment offset suppression, this application uniformly inputs all single-dimensional features processed by distributed environment sensitivity constraints into a heterogeneous dimension constancy discrimination unit. This heterogeneous dimension constancy discrimination unit employs a multilayer perceptron, utilizing the high discriminative power of different dimension-specific patterns to distinguish feature types of different perceptual dimensions. Furthermore, the loss function in this heterogeneous dimension constancy discrimination unit eliminates dimensional identity information that hinders collaborative alignment. (The last sentence appears to be incomplete and possibly refers to a different application.) Similarly, the gradient inversion layer receives the inverted gradient to suppress... Modality-specific patterns, such as the strong spatial structure of visual features and the strong temporal dynamics of audio features, mitigate modality-specific interference, breaking down the heterogeneity barriers between different perceptual dimensions and achieving constancy learning of dimensional features. In other words, through this constraint processing, features from different perceptual dimensions are mapped to the same feature space, eliminating inherent differences in perceptual mechanisms, statistical properties, and semantic representations across modalities. This prevents features from a particular dominant perceptual dimension from dominating subsequent feature fusion and decision-making processes, ensuring that features across all dimensions have a unified semantic scale and achieving cross-dimensional feature alignment.

[0049] In the specific implementation of step S1032, there is another embodiment in which the unit includes at least a global scene collaborative discrimination unit; The processing method described above is executed so that the corresponding unit performs corresponding collaborative processing on the received dimensional features, including: C1. The fused feature is obtained by fusing all dimension-constant single-dimensional features and environment-robust single-dimensional features; C2. Suppress environmentally sensitive information in the fusion features and output globally consistent collaborative features.

[0050] In steps C1-C2, after the first two constraint processing steps, each single-dimensional feature has environmental robustness and dimensional constancy. At this point, all optimized single-dimensional features are fused to obtain the initial fused features. The initial fused features are then input into a global scene collaborative discrimination unit. This discrimination unit employs a multilayer perceptron structure, taking the initial fused features as input and outputting the global environment prediction result to which the fused features belong. This involves feature extraction modules across all dimensions. At the global level, the gradient inversion layer receives the reverse gradient, thereby suppressing the environmentally sensitive information in the fused feature $f_{fuse}$ and enhancing the environmental consistency of the global collaborative space. That is, the gradient inversion layer built into the global scene collaborative discrimination unit further suppresses the residual environmentally sensitive information in the initial fused feature, strengthens the environmental consistency of the global fused feature, and integrates the complementary information of features in each dimension to eliminate feature redundancy and semantic misalignment problems that may occur during the fusion process. Finally, it outputs a globally consistent collaborative feature. This collaborative feature retains the core complementary information of each perception dimension and completely gets rid of the dual interference of environmental distribution shift and heterogeneous modality shift. It presents a unified and robust distribution state in the entire feature space and realizes the collaborative representation of multimodal and multidimensional features.

[0051] In step S104, this application directly inputs the globally consistent collaborative features obtained after collaborative processing by the three-level adversarial constraint module into the global prediction module of the task model. Based on the global prediction module, the collaborative features are processed, specifically including feature parsing, task mapping, etc., to obtain the target task prediction result in the open environment. This includes tasks such as multimodal scene recognition, target classification, and behavior analysis, to complete multimodal and multidimensional feature collaborative representation learning tasks for open environments. The global prediction module enhances the discriminative ability of the task model in target prediction tasks by fusing feature information from all perceptual dimensions to generate the final prediction result. Before processing the collaborative features, the global prediction module performs adaptive preprocessing on the collaborative features. The preprocessing process is concise, efficient, and does not destroy the core semantic information of the collaborative features. Specifically, it includes two key steps: First, feature normalization, which maps all values ​​of the collaborative features to a preset unified range (such as [0,1]) to eliminate interference caused by differences in the scale of the original features in different perceptual dimensions and ensure the fairness of subsequent feature parsing and task mapping. Second, redundant feature screening, which uses a lightweight feature screening mechanism to remove small redundant components that may remain in the collaborative features (such redundant components are generated during the fusion of multidimensional features and do not affect the core semantics related to the task, but increase the computational load), and retains feature components that are highly related to the target task. This not only improves the computational efficiency of subsequent processing but also further strengthens the correlation between features and the target task.

[0052] The optimization objective of this application is to adjust the feature extraction modules for each dimension. With the global task prediction module The parameters are chosen to minimize the expected perception error of the system in an open and unknown environment. The optimization objective can be expressed as:

[0053] in, Represents the mathematical expectation. This represents the loss function.

[0054] Looking at the entire process of the task model, from the initial dimensional division of multimodal perception data in the open environment and the extraction of features from each dimension by the feature extraction module, to the collaborative processing of dimensional features by the three-level adversarial constraint module and the generation of globally consistent collaborative features, and then to the global prediction module processing collaborative features and generating accurate target task prediction results, the entire process forms a complete flow of data input, feature processing, feature optimization, and task prediction. Each link precisely addresses the core requirements and solves technical problems: it not only solves the coupling problem of environmental distribution shift and heterogeneous modality shift in the open environment, but also realizes the collaborative representation and unified semantic alignment of multimodal and multidimensional features, giving full play to the complementarity of multimodal features, effectively alleviating the performance bottleneck caused by the lack or degradation of single-dimensional information, and ensuring the generalization ability and prediction reliability of the model in the unknown open environment.

[0055] In a specific implementation of step S104, one embodiment is as follows: based on the global prediction module processing the collaborative features, the target task prediction result in the open environment is obtained, including: S1041. A multilayer perceptron structure is used to map the collaborative features layer by layer, and the predicted probability distribution of each target task category is output. S1042. Based on the predicted probability distribution, select the task category with the highest probability as the target task prediction result.

[0056] In steps S1041-S1042, the collaborative features are input to the core computing unit of the global prediction module. This unit adopts a multilayer perceptron (MLP) structure, which is adapted to the structure of each discrimination unit in the three-level adversarial constraint module. At the same time, it is specifically optimized for the target task prediction requirements in an open environment to avoid the problem of features being disconnected from tasks. The module contains multiple fully connected layers and activation functions (such as ReLU activation function). Each layer has a clear division of labor and works collaboratively: The first fully connected layer is responsible for mapping the preprocessed collaborative features from high-dimensional to mid-dimensional dimensions, transforming the high-dimensional collaborative features into intermediate feature representations adapted to task prediction. The intermediate features retain the core complementary information of each perceptual dimension, such as the spatial structure information of visual features and the temporal dynamic information of audio features, and are transformed into feature forms that are easy for the model to calculate and recognize, breaking down the barrier of high-dimensional features being difficult to analyze; The middle multi-layer fully connected layers are responsible for deep analysis of the intermediate features, mining the semantic information contained within the features layer by layer, realizing the non-linear mapping of features through activation functions, gradually stripping away weak features that are irrelevant to the target task, strengthening the feature components related to the target task category, such as scene recognition, object classification, behavior analysis, etc., and continuously improving the correlation between features and the target task; The last fully connected layer is responsible for mapping the deeply analyzed intermediate features to the category space of the target task, outputting the predicted probability distribution of each target task category. For example, if the target task is multimodal scene recognition in an open environment, the output results are the probability values ​​of different scene categories such as "kitchen," "city road," and "office area." If the target task is behavior analysis, the output results are the probability values ​​of different behavior categories such as "walking," "driving," and "operating equipment." If the target task is target classification, the output results are the probability values ​​of various targets (such as "vehicles," "pedestrians," and "obstacles"). The probability distribution can intuitively reflect the model's prediction confidence for each task category, providing a quantitative basis for determining the final prediction result. Based on the target task category probability distribution output by the core computing unit of the global prediction module, the maximum likelihood estimation method is used to select the optimal prediction category, that is, to select the task category with the highest probability value as the final target task prediction result in the open environment. This selection method can maximize the reliability of the prediction results, meeting the core requirements of "accurate prediction and high confidence" in an open environment.

[0057] To verify the effectiveness of the method provided in this application, the performance of the provided method is compared with that of state-of-the-art methods on the EPIC-Kitchens dataset and the Human-Animal-Cartoon (HAC) dataset. Both datasets contain data in three perceptual dimensions: video, optical flow, and audio. EPIC-Kitchens includes three different physical environments (D1, D2, D3), and HAC includes three heterogeneous scene environments (H, A, C). Multidimensional feature fusion experiments are implemented using the MMAction2 toolkit. Specific implementation parameters are as follows: Feature extraction module: Video feature extraction uses a SlowFast network with an output dimension of 2304; Audio feature extraction uses a ResNet-18 network with an output dimension of 512; Optical flow feature extraction uses a SlowFast slow path branch with an output dimension of 2048; Hierarchical adversarial constraint module: Distributed environment sensitivity discrimination unit with a hidden layer dimension of 256; Global scene collaborative discrimination unit with a hidden layer dimension of 512; Heterogeneous dimension constancy discrimination unit with a hidden layer dimension of 256; Training parameters: AdamW... The optimizer has a learning rate of 0.0001 and a batch size of 16.

[0058] Table 1. Performance Comparison of the Method Provided in This Application with Current Advanced Methods

[0059] Based on the experimental data in Table 1, the proposed method exhibits the following significant characteristics in its perception performance in open environments: Leading performance in all-scenario perception: Our method achieves state-of-the-art average perception accuracy on both the authoritative EPIC-Kitchens and HAC benchmark datasets across all perception dimension combinations (including two-dimensional combinations: video + audio, video + optical flow, audio + optical flow, and three-dimensional full fusion). For example, in the three-dimensional full fusion scenario, our method achieves an average accuracy of 68.35% on the HAC dataset, significantly outperforming all existing state-of-the-art methods.

[0060] Significant synergistic gains from multidimensional features: Experimental results show that the perceptual performance of the method steadily improves with the increase of perceptual dimensions. On the EPIC-Kitchens dataset, the method improves from 61.88% for two dimensions (audio + video) to 63.59% for three-dimensional full fusion. This demonstrates that the hierarchical adversarial constraint module proposed in this invention can effectively extract and align complementary information between heterogeneous dimensions, maximizing the synergistic representation efficiency of multi-source data in an open environment.

[0061] High robustness in open environments: In migration experiments involving different physical environments and heterogeneous scenarios, this method demonstrated strong environmental adaptability. In particular, in test domains with drastic environmental distribution shifts and complex interference factors, this method maintained stable performance, verifying the necessity of using hierarchical discriminant units to collaboratively model environmental invariance and dimensional constancy for open environment perception.

[0062] This application also conducted ablation experiments on the three-level adversarial constraint module based on the EPIC-Kitchens dataset and the Human-Animal-Cartoon (HAC) dataset, and obtained the ablation experiment results shown in Table 2.

[0063] Table 2 Ablation Experiment Results of Level 3 Counterbalance Constraint Module

[0064] Based on the progressive ablation experimental results in Table 2, the technical contributions of each component of this method can be demonstrated from the following three dimensions: Heterogeneous Dimensional Constancy Constraint Effectiveness: By introducing a heterogeneous dimensionality constancy discriminant unit on top of the baseline model, the method improved the perceptual accuracy on the EPIC-Kitchens and HAC datasets by 0.63% and 0.46%, respectively. This indicates that by erasing dimensional (modal) identity information in features through adversarial games, the model's dependence on a single perceptual dimension can be effectively reduced, a cross-dimensional collaborative representation space is initially constructed, and the fundamental role of modality invariance learning in improving generalization performance is demonstrated.

[0065] Global environment consistency constraints Key support: In the introduction Based on this, a global scene collaborative discrimination unit is further added. Subsequently, the method's performance saw a significant leap, improving by 2.54% and 2.50% on the two datasets, respectively. This substantial improvement validates the necessity of imposing environment alignment constraints at the global fusion level: it effectively suppresses the cumulative environment bias introduced during multidimensional feature fusion, ensuring the macroscopic robustness of collaborative representation features under different open scenarios.

[0066] Distributed environmental sensitivity constraints Refined gain: Finally, a distributed environment sensitivity discrimination unit is introduced to act on the feature layers of each dimension. The method achieved optimal performance, improving upon the previous stage by 1.83% and 2.36% respectively. This indicates that global alignment alone is insufficient; it is necessary to suppress specific environmental sensitivity biases in each perceptual dimension (video, audio, optical flow) during the feature extraction stage to prevent environmental interference signals from entering the fusion space at the source.

[0067] The hierarchical adversarial constraint module proposed in this method exhibits significant component synergy. By organically combining a distributed environment-sensitivity discrimination unit, a heterogeneous dimension constancy discrimination unit, and a global scene collaborative discrimination unit, a full-path constraint system is constructed from the local dimensional hierarchy to the global fusion space. In this system, the distributed environment-sensitivity discrimination unit pre-filters environment-specific noise in each perception dimension during the feature extraction stage; the heterogeneous dimension constancy discrimination unit achieves cross-dimensional semantic alignment by stripping dimensional identity labels; and the global scene collaborative discrimination unit further locks the robustness of the overall representation at the fusion level. The synergistic effect of these three components effectively alleviates the deep coupling challenge of environment distribution shift and heterogeneous modality shift in open scenes. Experimental results demonstrate that this three-in-one collaborative mechanism is not simply a performance accumulation, but rather achieves a significant leap in perception accuracy (from 5.00% to 5.32%) while ensuring the discriminativeness of the perception task. This fully validates the core creativity and technical necessity of this method in handling asymmetric environment degradation and constructing a multi-dimensional collaborative representation space.

[0068] Example 2 This application also provides a multimodal, multidimensional feature collaborative representation learning device for open environments, such as... Figure 4 The diagram shows a block diagram of a multimodal, multidimensional feature collaborative representation learning device for open environments. The function implemented by this device corresponds to the steps of the agile development method for executing vehicle air conditioning control software on a terminal device, as described above. This device can be understood as a server component including a processor. The multimodal, multidimensional feature collaborative representation learning device for open environments described in this application includes: Training module 401 is used to pre-map the multi-dimensional feature collaborative representation learning task of the open environment into a task model, and train the task model based on a comprehensive loss function; the task model includes a feature extraction module, a three-level adversarial constraint module, and a global prediction module; The receiving module 402 is used to receive multimodal perception data in an open environment based on the task model after training, divide the multimodal perception data into perception dimensions, and extract the dimensional features in each perception dimension through the feature extraction module. Input module 403 is used to collaboratively process the dimensional features through a preset three-level adversarial constraint module to obtain globally consistent collaborative features, and input the collaborative features into the global prediction module; The processing module 404 is used to process the collaborative features based on the global prediction module to obtain the target task prediction result in the open environment, so as to complete the multimodal and multidimensional feature collaborative representation learning task for the open environment.

[0069] In one feasible implementation, the input module includes: The three-level adversarial constraint module is pre-configured so that it has multiple units; different units have different processing methods. The processing method is executed so that the corresponding unit performs corresponding collaborative processing on the received dimensional features.

[0070] In one feasible implementation, the input module further includes: The distributed environment sensitivity discrimination unit identifies environment-specific patterns with single-dimensional features. Suppress the environment-specific patterns of the single-dimensional features to output environment-robust single-dimensional features.

[0071] In one feasible implementation, the input module also includes: The heterogeneous dimension constancy discrimination unit distinguishes the feature types of different perception dimensions. Suppress the specific modalities of each perceptual dimension to output dimension-constant single-dimensional features.

[0072] In one feasible implementation, the input module further includes: The fused feature is obtained by integrating all dimension-constant single-dimensional features and environment-robust single-dimensional features; Suppress environmentally sensitive information in the fused features and output globally consistent collaborative features.

[0073] In one feasible implementation, the processing module includes: A multilayer perceptron structure is used to map the collaborative features layer by layer, outputting the predicted probability distribution of each target task category. Based on the predicted probability distribution, the task category with the highest probability is selected as the target task prediction result.

[0074] In one feasible implementation, the training module includes: The task model performs feature extraction, three-level adversarial constraint processing, and global task prediction on the training data to obtain the process output results. A comprehensive loss function is constructed that integrates task prediction loss and level 3 adversarial constraint loss to calculate the comprehensive loss value from the output results of the process. Based on the comprehensive loss value, the parameters of each module of the task model are iteratively updated according to the parameter grouping rules until the task model converges.

[0075] Example 3 This application also provides an electronic device, such as Figure 5As shown, it includes: a processor 501, a memory 502, and a bus 503. The memory 502 stores machine-readable instructions that can be executed by the processor 501. When the electronic device is running, the processor 501 and the memory 502 communicate through the bus 503. When the machine-readable instructions are executed by the processor 501, the steps of any one of the following describes a multimodal multidimensional feature collaborative representation learning method for open environments.

[0076] Example 4 This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of any one of the methods for learning multimodal and multidimensional feature collaborative representations for open environments.

[0077] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0078] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0079] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0080] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a platform server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0081] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A multimodal, multidimensional feature collaborative representation learning method for open environments, characterized in that, The method includes: The multidimensional feature collaborative representation learning task of the open environment is pre-mapped as a task model, and the task model is trained based on a comprehensive loss function; the task model includes a feature extraction module, a three-level adversarial constraint module, and a global prediction module; Based on the trained task model, multimodal perception data in an open environment is received, the multimodal perception data is divided into perception dimensions, and the dimensional features in each perception dimension are extracted through the feature extraction module. The dimensional features are processed collaboratively by a pre-set three-level adversarial constraint module to obtain globally consistent collaborative features, and these collaborative features are then input into the global prediction module. Based on the processing of the collaborative features by the global prediction module, the target task prediction result in the open environment is obtained, so as to complete the multimodal and multidimensional feature collaborative representation learning task for the open environment.

2. The method according to claim 1, characterized in that, The process of obtaining globally consistent collaborative features by collaboratively processing the dimensional features through a pre-set three-level adversarial constraint module includes: The three-level adversarial constraint module is pre-configured so that it has multiple units; different units have different processing methods. The processing method is executed so that the corresponding unit performs corresponding collaborative processing on the received dimensional features.

3. The method according to claim 2, characterized in that, The unit includes at least a distributed environment sensitivity discrimination unit; The processing method described above is executed so that the corresponding unit performs corresponding collaborative processing on the received dimensional features, including: The distributed environment sensitivity discrimination unit identifies environment-specific patterns with single-dimensional features. Suppress the environment-specific patterns of the single-dimensional features to output environment-robust single-dimensional features.

4. The method according to claim 2, characterized in that, The unit includes at least a heterogeneous dimension constancy discrimination unit; The processing method described above is executed so that the corresponding unit performs corresponding collaborative processing on the received dimensional features, including: The heterogeneous dimension constancy discrimination unit distinguishes the feature types of different perception dimensions. Suppress the specific modalities of each perceptual dimension to output dimension-constant single-dimensional features.

5. The method according to claim 2, characterized in that, The unit includes at least a global scene collaborative discrimination unit; The processing method described above is executed so that the corresponding unit performs corresponding collaborative processing on the received dimensional features, including: The fused feature is obtained by integrating all dimension-constant single-dimensional features and environment-robust single-dimensional features; Suppress environmentally sensitive information in the fused features and output globally consistent collaborative features.

6. The method according to claim 1, characterized in that, Based on the processing of the collaborative features by the global prediction module, the target task prediction result in the open environment is obtained, including: The collaborative features are mapped layer by layer using a multilayer perceptron structure, and the predicted probability distribution of each target task category is output. Based on the predicted probability distribution, the task category with the highest probability is selected as the target task prediction result.

7. The method according to claim 1, characterized in that, The training of the task model based on the comprehensive loss function includes: The task model performs feature extraction, three-level adversarial constraint processing, and global task prediction on the training data to obtain the process output results. A comprehensive loss function is constructed that integrates task prediction loss and level 3 adversarial constraint loss to calculate the comprehensive loss value from the output results of the process. Based on the comprehensive loss value, the parameters of each module of the task model are iteratively updated according to the parameter grouping rules until the task model converges.

8. A multimodal, multidimensional feature collaborative representation learning device for open environments, characterized in that, The device includes: The training module is used to pre-map the multi-dimensional feature collaborative representation learning task of the open environment into a task model, and train the task model based on a comprehensive loss function; the task model includes a feature extraction module, a three-level adversarial constraint module, and a global prediction module; The receiving module is used to receive multimodal perception data in an open environment based on the task model after training, divide the multimodal perception data into perception dimensions, and extract the dimensional features in each perception dimension through the feature extraction module. The input module is used to collaboratively process the dimensional features through a pre-set three-level adversarial constraint module to obtain globally consistent collaborative features, and input the collaborative features into the global prediction module; The processing module is used to process the collaborative features based on the global prediction module to obtain the target task prediction result in the open environment, so as to complete the multimodal and multidimensional feature collaborative representation learning task for the open environment.

9. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of a multimodal, multidimensional feature collaborative representation learning method for open environments as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the multimodal, multidimensional feature collaborative representation learning method for open environments as described in any one of claims 1 to 7.

Images