Heterogeneous vehicle road agent collaborative cognitive calculation method based on visual prompt

Abstract

The invention discloses a heterogeneous vehicle road agent collaborative cognitive calculation method based on visual cue, which comprises a polymorphic feature conversion architecture composed of a dimension alignment module, a channel alignment module, a visual cue module and a space attention module. Adjusting and aligning the dimension of the intermediate feature map of the heterogeneous agent through a dimension alignment module; and the one-to-one alignment between each channel and the own vehicle feature map channel is realized through the channel alignment module. The visual prompt module adopts a lightweight feature modification mechanism, combines dual design of general prompt and specific prompt, realizes general capture of global feature distribution, provides basic global semantic guidance, and specifically adjusts model behaviors to optimize cognitive performance. And the refined spatial semantic information of the neighborhood features is mapped to the semantic space of the own vehicle features through a spatial attention module, so that accurate spatial level feature conversion is ensured. According to the method, the training and storage cost can be remarkably reduced while invariance and expansibility are ensured.

Description

Technical Field

[0001] The present invention relates to the field of vehicle networking and intelligent vehicle technology, and in particular to a heterogeneous vehicle-road intelligent agent collaborative cognitive computing method based on visual cues. Background Art

[0002] With the rapid development of intelligent driving and vehicle-to-vehicle communication technology, collaborative cognition between vehicles and between vehicles and infrastructure has become an important research direction in the field of intelligent transportation and autonomous driving. Collaborative cognition improves the environmental cognition ability of individual agents through the sharing and collaboration of perception data between vehicle agents and between vehicles and roadside agents, and can effectively solve the limitations of a single mode under conditions such as occlusion and bad weather.

[0003] Collaborative cognition is an important technology to ensure the driving safety of autonomous vehicles. Its goal is to improve each agent's understanding of the driving environment through the exchange of perception data, thereby solving challenges such as occlusion and adverse weather conditions. Collaborative cognition has been widely used in fields such as target detection, obstacle recognition, and occupancy prediction. Homogeneous cognitive networks are commonly mentioned in existing studies, and collaborative cognition is achieved by exchanging and fusing feature-level perception data that remain consistent in semantics and granularity. However, there are significant differences in cognitive networks between different models or manufacturers, resulting in the transmitted intermediate features being too heterogeneous and difficult to be understood by other agents. In addition, cognitive networks are crucial to driving safety and are tightly coupled with downstream tasks, which makes it difficult to replace or retrain cognitive networks, that is, each heterogeneous cognitive network needs to meet invariance, and the continuous development of new heterogeneous collaborative cognitive frameworks makes it more difficult to fuse heterogeneous features. The above limitations have led to an urgent need in the current collaborative cognition field for methods that are scalable, do not change the original network, and can achieve efficient collaborative cognition of heterogeneous agents.

[0004] Most current collaborative cognition assumes that the cognitive networks between agents are homogeneous, so that the cognitive models of multiple vehicles or roadside equipment remain consistent in semantics and granularity, that is, the intermediate feature maps generated by the cognitive network have the same meaning in terms of dimension and features at different locations and regions. However, in practical applications, the cognitive platforms of different vehicles often have invariant heterogeneity, which is specifically reflected in the following two aspects: On the one hand, due to differences in vehicle manufacturers, models and perception hardware, the intermediate cognitive features transmitted between different platforms have significant differences in semantics, format and granularity, resulting in the inability of existing feature fusion networks to accurately fuse the intermediate cognitive features provided by other agents, thereby creating a semantic gap in which perception data is difficult for other agents to understand; on the other hand, the cognitive network is strongly coupled with downstream tasks, and the cognitive network is also closely related to autonomous driving safety, so the cognitive network architecture of each agent is usually immutable.

[0005] In response to the above challenges, some studies have proposed methods to bridge the semantic gap through collaborative cognitive intermediate feature semantic converters. These methods aim to solve the heterogeneity problem between different cognitive platforms by designing feature semantic converter structures to perform semantic mapping and conversion on intermediate features generated by collaborative cognitive agents on different platforms. However, existing solutions based on semantic converters still have the following shortcomings:

[0006] Single-stage strategy: such as HEAL. This strategy involves the reconstruction of the entire cognitive network and is mainly composed of two stages. In the first stage, the three important collaborative cognitive modules of encoder, feature fusion network and detection head are trained through the perception data generated by an appropriate number of heterogeneous agents, and then a unified feature space is built. In the second stage, the parameters of the latter two modules are frozen to retrain the encoder (converter) module for each newly added heterogeneous agent, so that the generated intermediate cognitive features can be aligned to the unified feature space. This type of method requires separate training and storage of converters for each pair of heterogeneous collaborative cognitive agents (that is, each type of conversion from heterogeneous neighboring vehicles to the direction of the vehicle is regarded as a pair. If there are N types of heterogeneous agents, there are N*(N-1) pairs of agents). The training overhead is extremely high, and it is difficult to meet the needs of new equipment emerging in large-scale environments. In addition, this type of method also requires the reconstruction of the cognitive networks of all agents, which destroys the immutability of heterogeneous agents and increases the difficulty of practical application.

[0007] Two-stage strategy: For example, PnPDA designs a plug-in converter. With the help of the intermediate standard semantic space, the semantics are converted to the standard semantic space through the output converter of the heterogeneous neighboring vehicle agent, and then the semantics are converted from the standard semantic space to the own vehicle semantic space by the input converter of the own vehicle agent, thereby realizing the fusion of heterogeneous agents. This method does not need to change the original cognitive network architecture of the agent, meets the invariance, and only needs to train two converters for each agent (if there are N heterogeneous agents, only 2*N converters need to be trained), which reduces the training overhead to a certain extent and has good scalability. However, the two semantic mappings may bring about a large cumulative loss in semantic degree, which limits its practicality. Summary of the invention

[0008] The invention patent with application number 202210548591.0, "A method for fusing multi-source heterogeneous perception data for autonomous driving vehicles", mainly studies the heterogeneity of the original perception data of the intelligent agent, uses the self-attention mechanism to generate intermediate feature maps, and performs feature fusion based on knowledge distillation. Although it solves the collaborative cognition problem of heterogeneous intelligent agents to a certain extent, its method violates the five invariances of the intelligent agent cognitive network architecture and lacks scalability.

[0009] The invention patent with application number 202210410414.6, "A heterogeneous multi-task collaborative system in an autonomous driving scenario", mainly targets the downstream task scenarios of autonomous driving. It uses the cross-attention mechanism and the fusion of language information and visual features to eliminate the heterogeneous differences in multi-task learning, but does not consider the heterogeneous differences caused by the cognitive network architecture, and only considers single-vehicle multi-task scenarios, rather than the currently common multi-vehicle collaborative scenarios.

[0010] The invention patent with application number 202010403211.5, "A method for collaborative execution of computing tasks in a heterogeneous autonomous driving vehicle formation", mainly focuses on the heterogeneity caused by differences in vehicle performance in an autonomous driving vehicle formation, and proposes a method for collaborative execution of computing tasks. By allocating and recovering computing tasks and communication tasks, it effectively utilizes the computing and communication resources within the formation and minimizes the total task completion time. However, the entire invention is based on the isomorphism of cognitive networks between vehicles, and the assumptions are too simple.

[0011] The invention patent with application number 202110273101.6, "A reinforcement learning autonomous driving method based on heterogeneous fusion features", proposes a deep deterministic policy gradient reinforcement learning method based on heterogeneous fusion features. It reduces the dimension of the heterogeneous input information of autonomous driving through a neural network and uses a hierarchical integration connection method to fuse the reduced-dimensional features, thereby realizing collaborative cognition between vehicles. However, this method still requires the reconstruction of the entire network architecture from the original data to the final perception result, which violates the invariance of the intelligent agent network architecture, and this method does not reflect scalability.

[0012] In summary, the existing patents do not solve the scalability and invariance of the agent network architecture in the scenario of heterogeneous agent collaborative cognition. In addition, the methods proposed in the existing patents not only have extremely high training costs, but also have extremely high costs for application to existing agents. Summary of the invention

[0014] In order to solve the problems of poor collaborative cognition between heterogeneous agents due to different cognitive networks and architectures, high cost of retraining architecture and difficulty in modifying cognitive network architecture, the present invention proposes a heterogeneous vehicle-road agent collaborative cognition calculation method based on visual cues. Based on the polymorphic feature interpretation architecture (Polymorphic Feature Interpretation), it solves the heterogeneity and scalability problems in non-isomorphic collaborative cognition, which not only greatly reduces the training cost, but also makes it easy to apply it to existing agents on a large scale.

[0015] The polymorphic feature conversion architecture proposed in the present invention introduces a learnable visual cue module, which enables efficient adaptation for new heterogeneous agents by only training specific visual cue sub-modules, without the need to retrain the entire interpreter. While ensuring invariance and scalability, the cost of training and storage is significantly reduced. Based on this framework, agents can achieve efficient collaboration in heterogeneous cognitive networks, bridge the semantic gap, and significantly improve the scalability of collaborative cognition. At the same time, the polymorphic interpreter architecture only requires a single semantic conversion, without the need for double conversion with the help of an intermediate standard semantic space as in the two-stage strategy, which effectively reduces semantic loss and ensures the semantic consistency and information integrity of the perceived data during the collaborative process.

[0016] A visual cue-based heterogeneous vehicle-road agent collaborative cognitive computing method inserts a polymorphic feature conversion architecture between the encoder module and the feature fusion network module in the heterogeneous collaborative cognitive network. The polymorphic feature conversion architecture includes four modules: dimension alignment module, channel alignment module, visual cue module and spatial attention module.

[0017] The dimension alignment module is used to adjust and align the three dimensions of C, H, and W of the intermediate features to the corresponding dimensions of the feature map of the ego vehicle agent. At the same time, the alignment between the specific prompt dimension obtained by the visual prompt module in the neighboring agent and the intermediate feature dimension of the ego vehicle agent is achieved.

[0018] The channel alignment module uses cosine similarity to flatten the two matrices into one-dimensional vectors and then calculate the quotient of their dot product and norm, and then match the channels of the ego vehicle feature map and the current neighborhood agent feature map in pairs to generate a similarity matrix M. Each element of the matrix M represents the similarity score between the channels of the ego vehicle and the neighborhood agent feature map, ranging from 0 to 1. The higher the score, the more similar the meanings expressed by the corresponding two channels. The feature maps of all neighborhood agents and the specific prompts obtained from the visual module will be escaped through the channel selection matrix M, and each channel will be aligned one by one with each channel of the ego vehicle feature map.

[0019] The visual cue module adopts a lightweight feature modification mechanism, combining the dual design of general cues and specific cues. The general cues are responsible for capturing the commonality of global feature distribution and providing basic global semantic guidance. The specific cues focus on the characteristics of the current environment or specific input data, and specifically adjust the model behavior to optimize cognitive performance.

[0020] The spatial attention module uses a three-dimensional attention mechanism that integrates axial attention to calculate the local and global spatial similarities between the intermediate features of the ego vehicle and the intermediate features of the neighborhood refined by the channel alignment module and the visual cue module, and calculates the spatial similarity by performing attention operations at both local and global scales.

[0021] The training of the polymorphic feature conversion architecture of the above structure includes two training stages: basic model training and model generalization. In the basic model training stage, style loss and adversarial loss are introduced to capture shared semantic information and agent-specific semantic information. During the training process, all encoders and the detection head of the ego-vehicle agent are frozen, and only the parameters of the polymorphic feature converter are trained to ensure the training effect. In the generalization stage, based on the basic model training stage, the general prompt parts in the channel alignment module, spatial attention mechanism module, and visual prompt module are frozen, and the scalable specific prompt S related to the newly added agent is optimized. The same loss constraints as in the first stage are applied to normalize the specific feature map. The learning of is constrained by the collaborative object detection loss; The loss function formula is:

[0022]

[0023] The advantages of the present invention are:

[0024] 1. The present invention aims to design a plug-in feature converter architecture, which can be added to the existing collaborative cognitive network framework in a plug-and-play manner to provide group collaboration capabilities for heterogeneous intelligent agents with different cognitive network architectures. Existing patents currently focus on designing a new collaborative cognitive network framework, but this is not applicable to real scenarios. The cognitive network architectures of various manufacturers cannot be easily changed, and this also involves the safety of autonomous driving.

[0025] 2. The present invention emphasizes the efficient integration of new heterogeneous agents. The semantic information of the intermediate features of the heterogeneous agents can be quickly learned through the designed specific prompt module. Based on the basic model, only the specific prompt submodule needs to be retrained with low consumption to seamlessly integrate the new heterogeneous agents into the existing collaborative cognitive network. Existing patents all focus on the collaborative cognitive methods of known heterogeneous agents, and do not take into account the emergence of new heterogeneous agents, and cannot guarantee the scalability of the methods.

[0026] 3. The present invention designs a more efficient training scheme, which greatly improves the collaborative cognitive effect through the rational use of various modules, and reduces the number of parameters that need to be learned through the design and combination of multiple loss functions, speeding up the training while ensuring the effect. The existing patents do not study the training scheme, and the entire network needs to be completely retrained, which consumes a lot of resources and time.

[0027] 4. The present invention provides an innovative and comprehensive collaborative cognition method for heterogeneous agents, which not only solves the problem that heterogeneous agents cannot collaborate normally due to differences in intermediate features, but also provides a plug-in intermediate feature interpreter architecture that is plug-and-play, efficient, convenient, scalable and has low resource consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A block diagram of the collaborative cognitive computing method for heterogeneous vehicle-road agents based on visual cues.

[0029] Figure 2 A block diagram of the polymorphic feature conversion architecture designed for the present invention.

[0030] Figure 3 Flowchart of the training method for the polymorphic feature conversion architecture. DETAILED DESCRIPTION

[0031] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0032] The present invention designs a polymorphic feature conversion architecture based on visual cues for heterogeneous vehicle-road agent collaborative cognition computing, which is scalable, satisfies the invariance of heterogeneous agents, and has low cost and high performance. This architecture will act as a plug-in architecture in the heterogeneous collaborative cognitive network without changing the original collaborative cognitive network architecture of each agent. Figure 1 As described above, in the heterogeneous vehicle-road agent collaborative cognitive computing method based on visual cues of the present invention, the encoder module, feature fusion network module, and detection head are all collaborative cognitive architecture modules that are commonly used at present. The polymorphic feature conversion architecture designed by the present invention can be directly inserted as a plug-in between the encoder module and the feature fusion network module to process and map the intermediate features generated by the ego-vehicle agent and the neighboring agent, thereby converting the intermediate features of the neighboring agent into the semantic space of the ego-vehicle features, ensuring the efficient subsequent fusion of the intermediate features.

[0033] The channel alignment module calculates the similarity between the feature channels of the neighborhood agent and the self-vehicle agent through cosine similarity, and performs one-to-one mapping conversion through the generated similarity matrix to achieve accurate alignment of the channel dimension, thereby optimizing the fusion effect. Specifically, the channel selection module uses cosine similarity to flatten the two matrices into one-dimensional vectors and then calculates the quotient of their dot product and norm (module), and then matches the self-vehicle feature map with each channel of the current neighborhood agent feature map, generating a similarity matrix M. Each element of the matrix M represents the similarity score between each channel of the self-vehicle and neighborhood agent feature maps, ranging from 0 to 1. The higher the score, the more similar the meanings expressed by the corresponding two channels are. The feature maps of all neighborhood agents will be escaped through the channel selection matrix M, that is, along the channel dimension, the similarity of the corresponding channels in the matrix M is used as the weight to perform weighted matrix and operation, and each channel is aligned one by one with each channel of the self-vehicle feature map, and then enter the subsequent module for further operation.

[0034] like Figure 2 As shown, the polymorphic feature conversion architecture designed by the present invention mainly includes four modules, namely, dimension alignment module, channel alignment module, visual cue module and spatial attention module.

[0035] Since the intermediate features generated by the current general collaborative perception network are all BEV (Bird's Eye View) feature maps in the shape of H x W x C, H is the feature map height, W is the feature map width, and C is the number of feature map channels. However, due to the heterogeneity of the network architecture, the dimensions of the BEV feature maps generated by other agents may be H x W x C. ’ x W ’ x C ’ , that is, the corresponding three dimensions may not be exactly the same. Therefore, a dimension alignment module is designed in the present invention as the basis of the polymorphic feature conversion architecture. The number of channels C of the feature map of the heterogeneous agent is adjusted and aligned through a trainable one-dimensional convolutional layer. The adjustment and alignment of the height H and width W of the feature map use cropping, padding and nearest neighbor interpolation sampling techniques: the perception range of the neighboring agent is cropped or supplemented to be consistent with the self-vehicle agent through cropping or padding, and the feature map resolution of the neighboring agent is aligned with that of the self-vehicle agent through nearest neighbor interpolation sampling. Finally, the three dimensions C, H, and W of the intermediate features of the neighboring agent are adjusted and aligned to the corresponding dimensional sizes of the feature map of the self-vehicle agent, so as to achieve the alignment of the dimensions of the feature map of the neighboring agent and the feature map of the self-vehicle.

[0036] Although the intermediate features after the dimension alignment module have the same dimension, C represents the channel dimension, and different channels may represent different features recognized from the original data; for example, some channels of an intermediate feature represent the edge contour of the vehicle, and some channels represent the road contour. Therefore, in a heterogeneous agent network, the meanings of each channel of the intermediate features generated by encoders of different architectures may be different. For this reason, the present invention designs a channel alignment module to align two intermediate features along the channel dimension, so as to achieve better results when fused.

[0037] The prompt mechanism in the visual prompt module is a flexible and efficient parameterized mechanism that aims to guide the model to learn and adapt the semantic information of a specific task or scene. In the present invention, the prompt mechanism is extended from the guiding role in language expression to the field of multimodal feature representation, and is directly embedded in the intermediate feature space in the form of visual prompts to achieve refined optimization of feature mapping. Through the visual prompt module designed by the present invention, prior information can be provided to the model at the feature level, so that the subsequent spatial attention module can accurately parse and process the intermediate features of heterogeneous agents.

[0038] Different from traditional methods, the prompt mechanism in the visual prompt module of the present invention combines the dual design of general prompts and specific prompts. The general prompt module is responsible for capturing the commonalities of global feature distribution and providing basic global semantic guidance; the specific prompt focuses on the characteristics of the current environment or specific input data, and specifically adjusts the model behavior to optimize cognitive performance.

[0039] In order to achieve prompt generation better and faster, the present invention adopts two prompt initialization methods, one of which can be selected to initialize general prompts and specific prompts; the two prompt initialization methods are: sampling-based initialization and random initialization. The former generates prompts with semantic associations by using statistical information in the training data. Specifically, random sampling is performed from the training set to obtain N point cloud samples, which are aggregated by average pooling after the encoder forms intermediate features. It is usually used in the basic model training stage to generate general prompts and specific prompts; the latter effectively reduces the number of prompt parameters through low-rank decomposition technology, which is mainly used in the initialization of specific prompts in the generalization stage. The last two dimensions HxW of the specific prompt are low-rank decomposed, and the HxW matrix is ​​decomposed into HxR and RxW matrices, where the parameter R used to decompose the matrix is ​​much smaller than H and W, thereby reducing the number of parameters from CxHxW to CxRx(H+W), and further reducing the number of parameters of specific indications by proportionally reducing the depth of C (scaling factor is T), thereby minimizing the trainable parameters and greatly improving the training speed.

[0040] The dimensions of the general prompts in the visual prompt module of the above-mentioned neighboring agents are consistent with those of the ego vehicle agent, so they are aligned without the need for the dimension alignment module; while the dimensions of the specific prompts are consistent with those of the corresponding neighboring agents, so they also need to be dimensionally aligned with the intermediate features of the ego vehicle through the dimension alignment module in the same way as described above. At the same time, the specific prompts obtained in the visual modules of all neighboring agents will also be weighted by the matrix M in the channel alignment module to keep them semantically consistent with the ego vehicle feature map in the channel dimension.

[0041] In the aforementioned channel alignment module, alignment between channel semantic components is performed, but even in the feature map of the corresponding channel, there may still be heterogeneity between the position information of the HxW dimension, so it is necessary to learn and align the semantic relationships of different positions for this dimension. In order to align the intermediate features of the ego vehicle and the neighboring intelligent agent in the spatial dimension, the present invention designs a spatial attention module to use a three-dimensional attention mechanism that integrates axial attention to calculate the local and global spatial similarities between the intermediate features of the ego vehicle and the intermediate features of the neighborhood refined by the channel alignment module and the visual cue module. Specifically, the three-dimensional attention mechanism calculates spatial similarity by performing attention operations at both local and global scales; at the local scale, the feature map is divided into non-overlapping three-dimensional windows by the three-dimensional attention mechanism, and then the information of the local area is aggregated by the self-attention mechanism. At the global scale, the three-dimensional attention mechanism performs global attention through a sparse three-dimensional grid. Through this process, the spatial attention maps the spatial semantic information of the refined neighborhood feature map to the semantic space of the ego vehicle feature map, thereby ensuring accurate spatial level feature conversion.

[0042] The training method of the above polymorphic feature conversion architecture is as follows Figure 3 As shown in the figure, through the two training stages of basic model training and model generalization, the new heterogeneous agent only needs to train the specific prompts related to it with low consumption, and can be seamlessly integrated into the entire heterogeneous collaborative perception network. Each agent only needs to retain a single basic converter, and can automatically adapt to different heterogeneous agents through polymorphism, thereby realizing efficient heterogeneous agent collaborative perception tasks. The following is a detailed description of the training methods of the two training stages:

[0043] 1. Basic model training stage.

[0044] In the basic model training phase, the goal is to train a highly generalizable polymorphic feature transformer for the ego-car agent, which resides in the ego-car and can establish a foundation for the addition of new heterogeneous agents in the next phase of training. To achieve this goal, the information in the feature graph of the neighboring agents is decoupled into shared semantic information and agent-specific semantic information, and the general prompt G and specific prompt S will be responsible for learning these two aspects respectively. However, relying solely on end-to-end training and using a collaborative perception loss function may lead to overfitting to the neighboring agent encoder used in the first phase. Therefore, style loss and adversarial loss are introduced in the first phase of training to capture shared semantic information and agent-specific semantic information, which are used to generate refined specific features and general features (output by the channel selection module). During the training process, all encoders and the detection head of the ego-car agent will be frozen, and only the parameters of the polymorphic feature transformer will be trained, ensuring the training effect.

[0045] The specific methods for learning the above-mentioned specific semantics and shared semantics are as follows:

[0046] 1) Specific semantic learning

[0047] To ensure that a specific feature map ( is the feature map after the dimension alignment module, S r is a specific prompt after the dimension alignment module), that is, the neighborhood agent feature map refined by specific prompts captures the specific semantic information of each heterogeneous agent as much as possible, and the training goal is designed to enable it to perform well in the single-vehicle (non-collaborative cognition) target detection task, that is, the single-vehicle perception loss function is used as part of the global loss function, which aims to encourage the polymorphic feature converter to learn and retain the specific semantic information encoded by the unique encoder of each neighborhood agent.

[0048] The bicycle perception loss function is:

[0049]

[0050] in, is the single vehicle target detection loss; and Represent the bounding box classification loss and regression loss of a single agent respectively. Even in the absence of collaborative perception input, this mechanism allows specific prompts to fully utilize these features to complete independent target detection tasks; at the same time, in order to ensure the balance of feature distribution among neighboring agents. A style loss function is introduced to prevent the features of neighboring agents from being overly concentrated or overly dispersed compared to the features of the ego vehicle agent.

[0051] The style loss function is:

[0052]

[0053] in, is the style consistency loss; μ(·) and σ(·) represent the mean and standard deviation of the feature, respectively. ego It is the intermediate feature of the ego-vehicle agent.

[0054] The loss function is given by and Composition, expressed as:

[0055]

[0056] Among them, ω is a hyperparameter.

[0057] 2) Shared semantic learning

[0058] Refining common features Responsible for extracting shared semantic information from the features of neighboring agents. To this end, a discriminator D is designed to judge Whether it comes from the neighboring agent or the ego-agent. This adversarial framework forces the feature transformer to interpret domain-invariant information more efficiently.

[0059] The discriminator D is designed as:

[0060]

[0061] in, is to combat learning loss; and denotes the classification error of the discriminator in predicting whether the feature comes from the vehicle or the neighborhood; style loss is also applied Its loss function can be expressed as:

[0062]

[0063] In the formula, for The loss function of is the style loss function.

[0064] In summary, in the first stage of end-to-end training, the conversion ability of the polymorphic feature converter is enhanced by the collaborative target detection task. and Different constraints are imposed. The total loss It can be expressed as:

[0065]

[0066] in, represents the collaborative loss, which is calculated when the ego vehicle and the neighboring agent jointly perform the target detection task. Its calculation method is the same as Same as λ s With λ g are all hyperparameters.

[0067] 2. Generalization stage

[0068] The polymorphic feature converter trained in the basic model training phase is endowed with generalization capabilities, and supports the integration of new agents through its basic capabilities (feature conversion capabilities), and completes the integration using its specific prompts. Therefore, in the generalization phase, there is no need to retrain the entire polymorphic feature converter, but only to optimize the scalable specific prompts S related to the new agents. In this phase, the general prompt parts in the channel alignment module, spatial attention mechanism module, and visual prompt module will be frozen on the basis of the basic model training phase.

[0069] In order to refine specific features It can capture the specific semantic information of the new agent type and applies the same loss constraint as the first stage to regularize Furthermore, in order to be able to effectively interpret this information in the semantic space of the ego-car, The learning of is constrained by the collaborative object detection loss. Therefore, The loss function formula is:

[0070]

[0071] The polymorphic feature conversion architecture designed in the present invention supports cognitive networks provided by different vehicles or manufacturers, and provides an efficient and feasible method for collaborative cognition of heterogeneous collaborative cognitive networks through the refinement and conversion of feature-level information; and with its plug-and-play characteristics, it can be seamlessly integrated into any existing collaborative cognitive network architecture while maintaining the invariance of the original cognitive network architecture, thereby achieving efficient collaborative cognition between heterogeneous intelligent agents.

[0072] The visual prompt module used in the present invention adopts a mechanism of separating shared semantics from specific semantics, which improves the ability to parse the intermediate features of heterogeneous agents, thereby improving cognitive efficiency and accuracy, and supports the efficient integration of newly added heterogeneous agents. It only needs to train and optimize the specific prompts of the corresponding agents without retraining the entire converter. At the same time, the visual prompt module adopts a lightweight feature modification mechanism, which directly embeds the visual prompts into the intermediate feature space to achieve auxiliary optimization of feature mapping. In this module, the visual prompts are divided into two parts: general prompts and specific prompts. The general prompts are used to capture the global feature distribution and provide basic semantic guidance, while the specific prompts focus on the specific heterogeneous agent feature semantics and provide specific feature information for each heterogeneous agent. The spatial attention module calculates the local and global spatial similarity by integrating the three-dimensional attention mechanism of axial attention, and maps the intermediate features of the neighborhood agents refined by the prompt module and the channel selection module to the semantic space of the vehicle, eliminating the heterogeneity of the feature map space dimension and ensuring accurate spatial level feature interpretation.

[0073] The training and learning of the entire polymorphic feature conversion architecture is achieved through two stages: the first stage basic model training and the second stage generalization training. The goal of the first stage training is to build a polymorphic feature converter with high generalization ability for the ego-vehicle agent. To this end, the feature information is divided into shared semantic information and specific semantic information, which are learned by two sub-modules, general prompts and specific prompts. To avoid overfitting, two loss functions, style loss and adversarial loss, are introduced on the basis of single-vehicle cognitive loss and collaborative cognitive loss. The style loss constrains the distribution balance of the features of the neighborhood agents to avoid excessive concentration or dispersion. The adversarial loss extracts domain-invariant information through the discriminator to improve the parsing ability of shared semantics. At this stage, all encoders and detection heads are frozen, and only the converter parameters are trained to ensure efficient and accurate training. In the second stage, based on the generalized polymorphic interpreter architecture trained in the first stage, there is no need to retrain the entire interpreter. Only the specific prompt sub-module of the newly added heterogeneous agent is trained. Through the collaborative cognitive target detection loss and the specific prompt loss constraints, it is ensured that the specific features of the newly added agent can be represented by the trained specific prompt module, and then the intermediate features of the agent are efficiently integrated into the semantic space of the intermediate features of the vehicle.

[0074] In summary, the polymorphic feature conversion architecture designed in the present invention provides support for efficient collaborative cognition of heterogeneous agents in collaborative cognition scenarios in Internet of Vehicles environments. A highly innovative visual prompt module is designed in the polymorphic feature conversion architecture, which refines the specific feature information and shared feature information of different agents into corresponding prompters through training, and provides prior information when the intermediate features of heterogeneous agents are mapped to the self-vehicle space, thereby achieving efficient collaborative cognition. At the same time, the polymorphic feature conversion architecture can be integrated into any existing type of vehicle collaborative cognition network architecture as a plug-in without destroying its original structure; and it can also efficiently integrate new heterogeneous agents into the entire collaborative cognition network, greatly reducing the training resources consumed compared to existing similar converter architectures.

Claims

1. A method for collaborative cognitive computing of heterogeneous vehicle-road agents based on visual cues, characterized by: Insert a polymorphic feature transformation architecture between the encoder module and the feature fusion network module in the heterogeneous collaborative cognitive network; The polymorphic feature conversion architecture includes four modules: dimension alignment module, channel alignment module, visual cue module and spatial attention module; The dimension alignment module is used to adjust and align the three dimensions of C, H, and W of the intermediate features to the corresponding dimensions of the feature map of the ego vehicle agent; at the same time, the specific prompt dimension obtained by the visual prompt module in the neighboring agent is aligned with the intermediate feature dimension of the ego vehicle agent; The channel alignment module uses cosine similarity to flatten the two matrices into one-dimensional vectors and then calculate the quotient of their dot product and norm, and then match the channels of the feature map of the ego vehicle and the feature map of the current neighborhood agent in pairs to generate a similarity matrix M. Each element of the matrix M represents the similarity score between each channel of the feature map of the ego vehicle and the neighborhood agent, ranging from 0 to 1. The higher the score, the more similar the meanings expressed by the corresponding two channels are. The feature maps of all neighboring agents and the specific hints obtained from the vision module are transformed through the channel selection matrix M, aligning each channel with each channel of the feature map of the ego vehicle one by one; The visual cue module adopts a lightweight feature modification mechanism, combining the dual design of general cues and specific cues. The general cues are responsible for capturing the commonality of global feature distribution and providing basic global semantic guidance; the specific cues focus on the characteristics of the current environment or specific input data, and specifically adjust the model behavior to optimize cognitive performance. The spatial attention module uses a three-dimensional attention mechanism that integrates axial attention to calculate the local and global spatial similarities between the ego-vehicle intermediate features and the neighborhood intermediate features refined by the channel alignment module and the visual cue module, and calculates the spatial similarity by performing attention operations at both local and global scales; The training of the polymorphic feature conversion architecture of the above structure includes two training stages: basic model training and model generalization. In the basic model training stage, style loss and adversarial loss are introduced to capture shared semantic information and agent-specific semantic information. During the training process, all encoders and the detection head of the ego-vehicle agent are frozen, and only the parameters of the polymorphic feature converter are trained to ensure the training effect. In the generalization stage, based on the basic model training stage, the general prompt parts in the channel alignment module, spatial attention mechanism module, and visual prompt module are frozen, and the scalable specific prompt S related to the newly added agent is optimized. The same loss constraint as the first stage is applied to normalize the specific feature map. The learning of is constrained by the collaborative object detection loss; The loss function formula is:

2. The method for heterogeneous vehicle-road agent collaborative cognitive computing based on visual cues as claimed in claim 1, characterized in that: The polymorphic feature conversion architecture is a plug-and-play plug-in architecture that works in heterogeneous collaborative cognitive networks without changing the original collaborative cognitive network architecture of each intelligent agent.

3. The method for heterogeneous vehicle-road agent collaborative cognitive computing based on visual cues as claimed in claim 1, characterized in that: The dimensional alignment module adjusts and aligns the number of channels C of the feature maps of heterogeneous agents through a trainable one-dimensional convolutional layer, and uses cropping, padding, and nearest neighbor interpolation sampling techniques to adjust and align the height H and width W of the feature maps. The perception range of the neighboring agents is cropped or supplemented to be consistent with that of the ego vehicle agent through cropping or padding, and the feature map resolution of the neighboring agents is aligned with that of the ego vehicle agent through nearest neighbor interpolation sampling.

4. The method for heterogeneous vehicle-road agent collaborative cognitive computing based on visual cues as claimed in claim 1, characterized in that: The 3D attention mechanism of the spatial attention module divides the feature map into non-overlapping 3D windows at the local scale, and then aggregates the information of the local area through the self-attention mechanism; at the global scale, the 3D attention mechanism performs global attention through a sparse 3D grid.

5. The method for heterogeneous vehicle-road agent collaborative cognitive computing based on visual cues as claimed in claim 1, characterized in that: During the training process of the polymorphic feature conversion architecture, the specific methods of learning specific semantics and shared semantics are as follows: 1) Specific semantic learning To ensure that the specific feature map In order to capture the specific semantic information of each heterogeneous agent as much as possible, the training goal is designed to make it perform well in the single-vehicle target detection task. The single-vehicle perception loss function is used as part of the global loss function. The single-vehicle perception loss function is: in, is the single vehicle target detection loss; and Represent the bounding box classification loss and regression loss of a single agent respectively; At the same time, in order to ensure that the feature distribution between neighboring agents is flat, a style loss function is introduced: in, is the style consistency loss; μ(·) and σ(·) represent the mean and standard deviation of the feature, respectively. ego It is the intermediate feature of the ego-vehicle agent; Specific feature map The loss function is given by and Composition, expressed as: Among them, ω is a hyperparameter; 2) Shared semantic learning Refining common features Responsible for extracting shared semantic information from the features of neighboring agents and designing the discriminator D to judge Is it from the neighborhood agent or the self-car agent? The discriminator D is designed as: in, is to combat learning loss; and denotes the classification error of the discriminator in predicting whether the feature comes from the vehicle or the neighborhood; style loss is also applied Its loss function can be expressed as: In the formula, for The loss function of is the style loss function; In summary, in the end-to-end training of the basic model training stage, the conversion ability of the polymorphic feature converter is enhanced by the collaborative target detection task. and Imposing different constraints; total loss It is expressed as: in, represents the collaborative loss, which is calculated when the ego vehicle and the neighboring agent jointly perform the target detection task. Its calculation method is the same as Same as λ s With λ g are all hyperparameters.

Images