Cross-modal scene recognition method, device and medium from rgb image to event camera

Abstract

This invention relates to a cross-modal scene recognition method, device, and medium from RGB images to event cameras. The method utilizes a recognition model for scene recognition, including: for the same scene sample, extracting sample image content features, sample event content features, and motion features; calculating a contrast loss based on the sample image content features and sample event content features; generating a first pseudo-image and a second pseudo-image based on the sample image content features, sample event motion features, and sample event content features, and calculating a latent feature consistency loss; self-reconstructing the RGB image and event data and calculating the self-reconstruction loss; for each scene sample, constructing a geographic proximity-distant triplet and calculating the triplet loss and domain adversarial loss for the geographic proximity-distant triplet; optimizing the recognition model based on all losses, and using the recognition model to obtain the scene recognition result from the event data of the scene to be recognized. Compared with existing technologies, this invention provides more accurate and stable scene recognition.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and robot perception technology, and in particular to a method, device and medium for cross-modal scene recognition from RGB images to event cameras. Background Technology

[0002] With the widespread application of unmanned systems in complex environments, scene recognition has become one of the key technologies for achieving precise positioning and navigation. Traditional visual sensors (such as RGB cameras and LiDAR) are limited in performance under conditions of high-speed motion and drastic changes in lighting, making it difficult to provide stable and reliable environmental perception capabilities. Neuromorphic visual sensors, also known as event cameras, are a new type of bio-inspired visual sensor that can simulate the response mechanism of the human retina to dynamic scenes. These sensors trigger events by detecting whether the change in light intensity at each pixel location exceeds a certain threshold, thereby generating an asynchronous and sparse event stream. Unlike traditional RGB cameras that rely on a fixed frame rate to acquire a complete image, event cameras generate irregular and asynchronous sets of events. This characteristic gives them advantages such as high temporal resolution (microsecond level), high dynamic range (140dB), low latency, and low power consumption, making them particularly suitable for challenging scenarios such as high-speed motion, low illumination, or strong light interference. In recent years, event cameras have been widely used in optical flow estimation, target detection, visual inertial navigation, target tracking, and attitude recognition, achieving a series of cutting-edge results. However, event data differs fundamentally from traditional RGB image data in its representation, making it impossible to directly apply existing image-based vision algorithms to event data. Furthermore, event modalities and RGB modalities exhibit inherent structural differences in data representation, spatial distribution, and semantic features, which limits the application of event cameras in scene recognition tasks.

[0003] To address the aforementioned issues, Chinese patent application CN116188930A provides a scene recognition method based on a fused event camera. This method incorporates event camera data into the scene recognition task, including: fusing acquired high-speed, high-dynamic images and high-speed, high-dynamic event streams to obtain fused features; acquiring a preset reference image and extracting reference image features from it; and performing a search and matching process between the fused features and the reference image features to obtain the reference image recognition result. While this method achieves scene recognition through feature fusion and enhancement, it suffers from the following problems: 1) Lack of cross-modal capability; if the RGB image is completely invalid, such as under extreme lighting conditions, scene recognition cannot be performed; 2) Difficulty in semantic alignment; the method relies on simple splicing and fusion at the feature level, which may lead to information loss after fusion. Furthermore, the recognition network may not be able to fully understand the deep semantic relationships between the two modalities, resulting in inaccurate scene recognition results.

[0004] Therefore, providing a method that can fully utilize RGB images and event data, and can still perform scene recognition when RGB images are invalid, is a technical problem that needs to be solved. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art by providing a cross-modal scene recognition method, device and medium from RGB images to event cameras. This method can effectively reduce the modal differences between event data and traditional image data, and achieve highly robust scene recognition based on event data. In particular, when RGB images fail due to extreme lighting conditions, this method can still use event data to ensure high-precision scene recognition performance.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] According to a first aspect of the present invention, a cross-modal scene recognition method from RGB images to event cameras is provided, the method utilizing a recognition model for scene recognition, comprising:

[0008] For the same scene sample, extract sample image content features based on the corresponding sample RGB image, extract sample event content features and sample event motion features based on the corresponding sample event data, perform cross-modal contrast learning based on the sample image content features and sample event content features, and calculate contrast loss.

[0009] Based on the sample image content features and sample event motion features, pseudo-event data is generated by decoding. Based on the pseudo-event data, image reconstruction is performed to obtain a first pseudo-image. Cycle consistency loss is calculated based on the first pseudo-image and the sample RGB image. A second pseudo-image is generated based on the sample event content features. Latent feature consistency loss is calculated based on the pseudo-event data, the second pseudo-image, the sample RGB image, and the sample event data.

[0010] The sample RGB images and sample event data are reconstructed separately, and the self-reconstruction loss is calculated based on the reconstruction results.

[0011] For each scene sample, a descriptor is obtained based on the corresponding sample image content features and sample event content features; several scene samples are randomly selected as query samples, and for each query sample, a geographic proximity-far distance triplet is constructed, and the triplet loss and domain adversarial loss of the geographic proximity-far distance triplet are calculated based on the descriptor; the descriptor includes image descriptors and event descriptors;

[0012] The recognition model is optimized based on the contrast loss, cycle consistency loss, latent feature consistency loss, self-reconstruction loss, triplet loss, and domain adversarial loss.

[0013] The event data of the scene to be identified is input into the optimized recognition model to generate a scene descriptor, and the scene recognition result is obtained based on the scene descriptor.

[0014] As a preferred technical solution, the method for calculating the contrast loss is as follows:

[0015] For each scene sample, the sample image content features and sample event content features are projected onto the same low-dimensional projection space to obtain image projection features and event projection features.

[0016] For scene sample i, its image projection features and event projection features are used to construct a positive sample pair, and its image content features are used to construct a negative sample pair with the event projection features or image projection features of the other scene samples.

[0017] The contrastive loss is calculated based on the aforementioned positive and negative sample pairs, and its expression is as follows:

[0018]

[0019] Where N represents the total number of scene samples; Represents the image projection features of scene sample i; The event projection features represent scene sample i; Represents positive sample pairs; sim(·) represents similarity calculation; τ represents temperature hyperparameter; Represents the image projection features of sample scene j; The event projection features represent scene sample j; and All of these represent negative samples.

[0020] As a preferred technical solution, the method for calculating the cycle consistency loss is as follows:

[0021] The sample image content features and sample event motion features are decoded to generate pseudo-event data, and the pseudo-event content features are extracted from the pseudo-event data.

[0022] The pseudo-event content features are decoded to obtain a first pseudo-image. Based on the first pseudo-image, the cycle consistency loss is calculated, and its expression is as follows:

[0023]

[0024] Among them, I i This represents the RGB image of the sample corresponding to scene sample i; represents the first pseudo-image; ||·||1 represents the calculation of the 1-norm.

[0025] As a preferred technical solution, the method for calculating the potential feature consistency loss is as follows:

[0026] Based on the second pseudo-image, extract the content features of the second pseudo-image; based on the pseudo-event data, extract the motion features of the pseudo-events.

[0027] The latent feature consistency loss is calculated based on the second pseudo-image content features and pseudo-event motion features, and its expression is as follows:

[0028]

[0029] in, The sample image content features representing scene sample i; The second pseudo-image content feature represents scene sample i; The sample event content features represent the characteristics of scene sample i; Indicates the characteristics of pseudo-event content; λ ζ Indicates the equilibrium hyperparameters; Represents the motion features of sample events in scene sample i; This indicates the motion characteristics of pseudo-events.

[0030] As a preferred technical solution, the method for calculating the self-reconstruction loss is as follows:

[0031]

[0032] Among them, I i Represents the RGB image of the corresponding scene sample i; This represents the reconstructed RGB image of the sample obtained from self-reconstruction of the sample RGB image, and has... E img (·) represents the image encoding operation, D img (·) represents an image decoding operation; ||·||1 represents the 1-norm calculation; E i This represents the sample event data corresponding to scene sample i; This represents the reconstructed event data obtained by self-reconstruction based on sample event data, and E event (·) indicates an event coding operation, D event (·) indicates an event decoding operation.

[0033] As a preferred technical solution, the method for calculating the triplet loss is as follows:

[0034] For each sample scenario, the sample image content features and sample event content features are encoded using advanced semantics to obtain advanced image semantic features and advanced event semantic features; feature aggregation is then performed based on the advanced image semantic features and advanced event semantic features to obtain image descriptors and event descriptors.

[0035] Geographic information of each scene sample is obtained. For a selected query sample, nearby scene samples are selected as positive samples and distant scene samples are selected as negative samples based on the geographic information to construct a triple. The nearby scene samples are scene samples whose distance from the query sample does not exceed a first preset distance, and the distant scene samples are scene samples whose distance from the query sample is greater than a second preset distance.

[0036] Select the hard triples from the constructed triples, where the negative samples in the hard triples satisfy: q represents the query sample; This represents the j-th sample farthest from the scene in the query sample q; d des (,) represents the Euclidean distance between corresponding descriptors, where the descriptor is an image descriptor or an event descriptor; This represents the i-th best-matching neighboring scene sample of query sample q; m represents the marginal value;

[0037] The recognition model is trained using the aforementioned difficult triples, and the triple loss is calculated as follows:

[0038]

[0039] As a preferred technical solution, the method for calculating the domain adversarial loss is as follows:

[0040]

[0041] Where N represents the total number of scene samples; y i This represents the domain label of the input data for the recognition model. When the input data is an image, y i =1, when the input data is an event. i =0; D domain (·) indicates a scalar probability value calculation operation; d i The descriptor refers to either an image descriptor or an event descriptor.

[0042] As a preferred technical solution, the method for obtaining the scene recognition result includes:

[0043] For all RGB images in the image database of known locations, the optimized recognition model is used to generate scene descriptors, which are stored together with the corresponding geographic labels to construct a scene descriptor database.

[0044] A nearest neighbor search is performed in the scene descriptor database to select the label scene descriptor with the smallest Euclidean distance to the scene descriptor. The geographical location corresponding to this label scene descriptor is the scene recognition result.

[0045] According to a second aspect of the present invention, an electronic device is provided for performing cross-modal scene recognition from RGB images to event cameras, comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement the method.

[0046] According to a third aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described thereon.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] 1) This invention provides a method for scene recognition directly based on event data. Specifically, when training and optimizing the recognition model, this invention ensures that the feature representation learned by the recognition model is insensitive to modal differences through cross-modal contrastive learning, effectively solving the problem that the model cannot rely on a single event feature for scene recognition. In addition, this invention introduces adversarial training when constructing scene identifiers, and constructs a modality-invariant shared semantic space by combining contrastive learning. This forces the recognition model to learn this semantic space, making the features encoded by the same scene (whether it is an RGB image or an event stream) similar. Ultimately, this enables the recognition model to learn to extract the common, abstract scene essence from different data sources. Therefore, when an RGB image fails due to extreme lighting, the recognition model can use only its embedded event encoder to process valid event data and generate a high-quality scene descriptor. Since this descriptor is in the same semantic space and has similarity with the descriptors generated by RGB images under normal lighting in the database, the recognition model can still accurately match the correct scene, greatly improving the robustness and all-weather working capability of the system.

[0049] 2) This invention introduces bidirectional cross-modal translation during the training of the recognition model, including feature encoding and decoding processes from image to event and back to image, as well as encoding and decoding processes from event to image. This ensures that the features learned by the recognition model can achieve deep semantic alignment, and that different modal features of the same scene can be understood and converted without obstacles. In addition, a cycle consistency loss is constructed to ensure that the information is not distorted after translating from image features to event features and back to image features. Through the above mechanism, the model is forced to learn a more essential mapping relationship between the two modalities. Compared with simple feature splicing, it can better guarantee the semantic alignment and information fidelity of data from different modalities, and significantly improve the accuracy and stability of the final scene recognition. Attached Figure Description

[0050] Figure 1 This is a flowchart of the method of the present invention;

[0051] Figure 2 This is a schematic diagram illustrating the overall architecture and loss function calculation of the present invention;

[0052] Figure 3 This is a schematic diagram of the cross-modal contrastive learning process of the present invention;

[0053] Figure 4 This is a schematic diagram of the cross-modal translation and reconstruction process of the present invention;

[0054] Figure 5 This is a schematic diagram of the supervised learning process of the present invention. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0056] To address the problems existing in current technologies, a cross-modal scene recognition method from RGB images to event cameras is proposed. This method utilizes a recognition model for scene recognition, and the recognition model constructed in this invention includes an image encoder-decoder E. img and D img Event Encoder-Decoder E event and D event Feature aggregation layer Agg and high-level semantic encoder E advA cross-modal contrastive learning mechanism is introduced during the training of the recognition model to shorten the distance between matching image-event features in the projection space, while increasing the distance between non-matching features, thereby constructing a modality-invariant shared semantic space. A bidirectional translation network and cycle consistency constraints are used to achieve semantic alignment and information fidelity between image and event modal data. A supervised learning task based on triplet loss and domain adversarial loss is employed to generate highly discriminative scene descriptors through a feature aggregation layer for scene recognition. The process is as follows: Figure 1 As shown, the training framework of the recognition model is as follows: Figure 2 As shown.

[0057] In detail, the entire process includes:

[0058] S1. For the same scene sample, extract sample image content features based on the corresponding sample RGB image, extract sample event content features and sample event motion features based on the corresponding sample event data, perform cross-modal contrastive learning based on the sample image content features and sample event content features, and calculate the contrastive loss.

[0059] The purpose of this step is to ensure that the encoder can extract high-dimensional feature vectors that can represent the semantic content of the scene from the input image and event data, and to extract additional motion features specific to the event data.

[0060] S11. For each scene sample, project the sample image content features and sample event content features to the same low-dimensional projection space to obtain image projection features and event projection features.

[0061] S111, Extract image content features.

[0062] Sample RGB image Input Image Encoder E img H and W represent the height and width of the sample RGB image, with 3 channels. This encoder uses a modified ResNet50 architecture as the backbone network, removes the original model's classification head, and adds a fully connected layer after the global average pooling layer, ultimately outputting a fixed-dimensional content feature vector.

[0063]

[0064] Where, d c The dimension of the sample image content features; This represents the features of the sample image.

[0065] ResNet50 is used as the backbone to leverage its proven and powerful capabilities in image feature extraction, ensuring the richness and discriminativeness of the extracted features. It aims to encode static scene content information of images.

[0066] S112. Extract the content features and motion features of the sample events.

[0067] Sample event data The sample event data is represented in the form of event frames or voxel grids, where C is the number of channels representing the event, and the input is the event encoder E. event .

[0068] Encoder E event Designed with a two-branch structure, the first branch is the content feature extraction branch, which is connected to the image encoder E. img By sharing the backbone network structure and weights, it is ensured that the content features of the two modalities can be mapped to a compatible feature space, thereby outputting sample event content feature vectors. The second branch is the motion feature extraction branch, which consists of a lightweight convolutional neural network containing several convolutional layers and residual blocks. This network is specifically designed to parse the motion patterns of the camera or object from an asynchronous event stream, outputting a motion feature vector of the sample events. Where d m The dimension of motion features for sample events.

[0069] The entire extraction process is represented as follows:

[0070]

[0071] Where, d c Indicates the feature dimension.

[0072] By sharing content branch weights and establishing independent motion branches, content features and motion features are extracted separately. This ensures that the recognition model maintains consistency across modal content while preserving the unique dynamic information of event data.

[0073] S113, Feature projection.

[0074] Extract all content features High-dimensional features are mapped to a lower-dimensional projection space, which is more suitable for computing contrastive loss, through a shared projection head Proj. This projection head is implemented by a multilayer perceptron containing several fully connected layers and using the ReLU activation function and batch normalization layers.

[0075] This process can be represented as:

[0076]

[0077] Where, d p For the projection feature dimension; Represents image projection features; This represents the event projection characteristics.

[0078] Through the above projection transformation, the features are mapped to a more compact space, while the learnable parameters of the projection head itself can enhance the expressive power of the model.

[0079] S12. For scene sample i, construct a positive sample pair by combining the image projection features and event projection features obtained after feature dimensionality reduction, and construct a negative sample pair by combining the image content features obtained after feature dimensionality reduction with the event projection features or image projection features of the other scene samples.

[0080] This step introduces a contrastive learning mechanism to shorten the distance between paired image and event content features in the projection space, while simultaneously widening the distance between unpaired features. This forces the encoder to learn a shared semantic feature space that is insensitive to modal differences. See [link to previous section]. Figure 3 .

[0081] S13. Calculate the contrastive loss based on positive and negative sample pairs.

[0082] The purpose of constructing a contrastive loss function is to teach the model to determine which image-event pairs come from the same scene (positive pairs) and which come from different scenes (negative pairs). Its expression is:

[0083]

[0084] Where N represents the total number of scene samples; Represents the image projection features of scene sample i; The event projection features represent scene sample i; Represents positive sample pairs; sim(·) represents similarity calculation; τ represents temperature hyperparameter; Represents the image projection features of sample scene j; The event projection features represent scene sample j; and All of these represent negative samples.

[0085] This loss function maximizes the similarity of positive sample pairs while minimizing the similarity with negative sample pairs. By optimizing this loss, the image encoder and event encoder are forced to learn to ignore modality-specific low-level appearances such as texture and brightness, and instead focus on shared high-level scene semantics, thereby achieving deep cross-modal feature alignment.

[0086] S2. Based on the content features of the sample image and the motion features of the sample event, decode to generate pseudo-event data, reconstruct the image based on the pseudo-event data to obtain the first pseudo-image, calculate the cycle consistency loss based on the first pseudo-image and the sample RGB image; generate the second pseudo-image based on the content features of the sample event, and calculate the latent feature consistency loss based on the pseudo-event data, the second pseudo-image, the sample RGB image and the sample event data.

[0087] The purpose of this step is to ensure that the cross-modal translation process is information-preserving by constructing a bidirectional translation loop and imposing consistency constraints, thereby further enhancing the quality of feature representations and the robustness of the model. See [link to relevant documentation]. Figure 4 .

[0088] S21. Calculate the cycle consistency loss.

[0089] S211. Decode the sample image content features and sample event motion features to generate pseudo-event data, and extract pseudo-event content features and pseudo-event motion features from the pseudo-event data. This process is represented as follows:

[0090]

[0091] in, This represents pseudo-event data. Indicates the characteristics of pseudo-event content. Describing the motion characteristics of pseudo-events, d c and d m Indicates the feature dimension.

[0092] This step improves the quality of decoded events by adding reliable motion features, ensuring information consistency during the translation process.

[0093] S212. Decode the pseudo-event content features to obtain the first pseudo-image, and calculate the cycle consistency loss based on the first pseudo-image.

[0094]

[0095] Among them, I i This represents the RGB image of the sample corresponding to scene sample i; represents the first pseudo-image; ||·||1 represents the L1 norm calculation, which is used to encourage the model to generate clearer results and be insensitive to outliers.

[0096] This loss constraint constrains the "image → event → image" loop process, ensuring that no key scene content information is lost during the translation process.

[0097] S22. Calculate the consistency loss of latent features.

[0098] This step executes a reverse loop of "event → image", due to the event decoder D. event Since both content features and motion features are received simultaneously, this loop no longer proceeds to the "image → event" part to avoid reuse. This leads to event decoder D event It becomes dependent on motion features, thus neglecting content features.

[0099] S221. Extract the content features of the second pseudo-image based on the second pseudo-image.

[0100]

[0101] in, Indicates the second pseudo-image; This represents the content features of the second pseudo-image.

[0102] S222. Calculate the latent feature consistency loss based on the second pseudo-image content features and pseudo-event motion features. The expression is as follows:

[0103]

[0104] in, The sample image content features representing scene sample i; The second pseudo-image content feature represents scene sample i; The sample event content features represent the characteristics of scene sample i; Indicates the characteristics of pseudo-event content; λ ζ Indicates the equilibrium hyperparameters; Represents the motion features of sample events in scene sample i; This indicates the motion characteristics of pseudo-events.

[0105] Training the model using latent feature consistency loss forces the encoder output features to maintain high stability after undergoing complex cross-modal translation transformations. This is more direct and profound than pixel-level cyclic consistency constraints, and can effectively improve the structural rationality of the feature space and the reliability of cross-modal translation.

[0106] S3. Perform self-reconstruction on the sample RGB image and sample event data respectively, and calculate the self-reconstruction loss based on the reconstruction results.

[0107] The purpose of this step is to provide additional self-supervised constraints for feature learning by constructing the encoder path and calculating the self-reconstruction error, thereby further enhancing the consistency of feature representations and the reconstruction capability of the decoder. By requiring the encoder-decoder pair to reconstruct the original input, it provides important regularization constraints for feature learning, which helps to improve the integrity and expressive power of features. The process is described in [link to process details]. Figure 4 This includes the following steps:

[0108] S31. Construct an autoencoder pathway.

[0109] Autoencoder paths for images and events are constructed separately, and self-reconstruction of the data is achieved through an encoder-decoder structure. The input image I... i via image encoder E img Features are extracted and then processed by the image decoder D. img Reconstruction is performed to obtain an autoencoded reconstructed image. Right now:

[0110] Input event data E i via event encoder E event Features are extracted and then processed by the event decoder D. event Reconstruction is performed to obtain the self-encoded reconstruction event. Right now:

[0111] S32. The method for calculating self-reconstruction loss is as follows:

[0112]

[0113] Among them, I i Represents the RGB image of the corresponding scene sample i; This represents the reconstructed RGB image of the sample obtained from self-reconstruction of the sample RGB image, and has... E img (·) represents the image encoding operation, D img (·) represents an image decoding operation; ||·||1 represents the 1-norm calculation; E i This represents the sample event data corresponding to scene sample i; This represents the reconstructed event data obtained by self-reconstruction based on sample event data, and E event (·) indicates an event coding operation, D event (·) indicates an event decoding operation.

[0114] By minimizing the self-reconstruction loss, we can ensure that the encoder and decoder have basic information preservation capabilities, while providing additional regularization constraints for feature learning, stabilizing the training process and improving feature quality.

[0115] S4. For each scene sample, obtain descriptors based on the corresponding sample image content features and sample event content features. Randomly select several scene samples as query samples. For each query sample, construct a geographic proximity-far distance triplet and calculate the triplet loss and domain adversarial loss based on the descriptors. The descriptors include image descriptors and event descriptors.

[0116] The purpose of this step is to further optimize the alignment features learned in the previous steps into highly discriminative descriptors suitable for scene recognition tasks using geo-labeled data. Through a triplet loss training strategy, the network can retrieve the corresponding scene from the database based on the input feature descriptor, training the entire network to complete the final scene recognition task. Furthermore, a domain adversarial mechanism is introduced during descriptor generation to further reduce residual domain differences between modalities, forcing the encoder to generate more modality-invariant feature representations. The entire process is described in [link to documentation]. Figure 5 .

[0117] S41. For each sample scene, perform high-level semantic encoding on the sample image content features and sample event content features to obtain high-level image semantic features and high-level event semantic features; perform feature aggregation based on the high-level image semantic features and high-level event semantic features respectively to obtain image descriptors and event descriptors.

[0118] Specifically, this process utilizes a high-level semantic encoder E adv The processing, consisting of later stages of the ResNet50 architecture such as subsequent residual layers and global average pooling layers, is used to perform deeper nonlinear transformations on the coarse-grained content features, extracting more abstract and semantically rich high-level feature maps, which can be represented as:

[0119]

[0120] Where, d a For high-level semantic feature dimensions, For high-level image semantic features, These are high-level event semantic features; two different types of content features are mapped to the same semantic feature space after being encoded using high-level semantics.

[0121] The high-level features are input into the feature aggregation layer Agg. Because the NetVLAD layer (Vector of Locally Aggregated Descriptors) can effectively model the statistical distribution of features and generate scene representations robust to changes in viewpoint and appearance, it is very suitable for scene recognition tasks. Therefore, this invention uses the NetVLAD layer as the feature aggregation layer. It introduces a set of learnable cluster centers, calculates the soft-assigned weights of local feature descriptors relative to these centers, aggregates the residuals, and finally outputs a global, fixed-length scene descriptor vector, represented as:

[0122]

[0123] Where, d v For NetVLAD descriptor dimensions.

[0124] Through steps S42 to S44, scene descriptors generated from different modal data (images / events) from the same location are made as close as possible in the feature space, while descriptors from data from different locations are made as far apart as possible, enabling the model to learn to generate highly discriminative scene descriptors that are robust to viewpoint, lighting, occlusion, and modal changes.

[0125] S42. Obtain the geographic information of each scene sample. For the selected query sample, filter nearby scene samples as positive samples and far-away scene samples as negative samples based on the geographic information to construct a triplet. Nearby scene samples are scene samples whose distance from the query sample does not exceed the first preset distance, and far-away scene samples are scene samples whose distance from the query sample is greater than the second preset distance.

[0126] A scene sample is randomly selected from the scene samples as a query sample. For a given query sample q, a triplet for model training is constructed based on the geographical information attached to the sample, such as GPS coordinate information.

[0127] In the scene sample set, images that are geographically close to the query sample q (e.g., within 25 meters) are selected as potential positive samples p. q Images that are geographically far from the query sample q (e.g., 75 meters away) are selected as negative samples n. q Each triple contains a query sample q and potential positive samples that are geographically adjacent to q. And negative samples that are far from q

[0128] The triples used for training can be represented as A set of potential positive samples should contain at least one positive sample that matches the query sample. In this invention, the best-matching potential positive sample is used. It satisfies:

[0129]

[0130] S43. To improve model training efficiency, hard triplet mining can be performed based on feature distance. Samples closer to feature q in the dataset are selected as hard negative samples to improve training efficiency. The negative samples in hard triplets satisfy the following conditions: q represents the query sample; This represents the j-th sample farthest from the scene in the query sample q; d des (,) represents the Euclidean distance between corresponding descriptors, which can be image descriptors or event descriptors; This represents the i-th best-matching neighboring scene sample of the query sample q; m represents the marginal value.

[0131] S44. Train the recognition model using difficult triples.

[0132] By actively selecting difficult negative samples that the current model cannot yet distinguish, the model can obtain more effective gradient signals in each parameter update, avoiding wasting computational resources on a large number of simple samples, thereby accelerating the convergence speed. The triplet loss is calculated as follows:

[0133]

[0134] This loss function forces the query sample q to be similar to the best positive sample. The distance between the query sample and the positive sample must be at least a marginal value m smaller than the distance between the query sample and any difficult negative sample. By bringing the query sample closer to the positive sample and pushing it further away from the negative sample, the discriminativeness of the feature descriptor for scene recognition tasks is explicitly optimized.

[0135] Furthermore, to fully utilize cross-modal information, the triplet loss allows for cross-modal cross-combination. For example, if the query sample q selects an event descriptor, then the positive and negative samples p and n select image descriptors, and vice versa, resulting in a total of 2... 3 A variety of cross-modal cross-combination forms.

[0136] S45, Domain Confrontation Training.

[0137] This invention introduces a domain discriminator D domain It is a binary classifier consisting of fully connected layers, designed to distinguish the input feature descriptor d. i Is it from the image modality or the event modality; meanwhile, the feature encoder (E) img E event and E adv The feature aggregation layer Agg is trained through a gradient reversal layer (GRL) to generate modality-invariant feature descriptors that can fool the domain discriminator.

[0138] The gradient reversal layer is a key component for implementing adversarial training. During forward propagation, it performs an identity transformation, doing nothing. During backward propagation, it multiplies the gradients from the domain discriminator back to the feature encoder and feature aggregation layer by a negative coefficient λ. adver , λ adver It is a hyperparameter used to control the intensity of adversarial training.

[0139] The introduction of the gradient reversal layer improves the network's performance in the optimization domain discriminator D. domainMeanwhile, the gradient signs received by the feature encoder and feature aggregation layer are opposite to those of the domain discriminator, which motivates the feature encoder and feature aggregation layer to generate feature descriptors that can deceive the domain discriminator. This causes the feature descriptor distributions of events and images to gradually align and become indistinguishable, laying the groundwork for scene recognition based solely on event data when images fail.

[0140] In this invention, the domain adversarial loss is a binary cross-entropy loss, and its expression is:

[0141]

[0142] Where N represents the total number of scene samples; y i This represents the domain label of the input data for the recognition model. When the input data is an image, y i =1, when the input data is an event. i =0; D domain The · symbol represents a scalar probability value calculation operation, which outputs a scalar probability value p. i ∈[0,1], representing the probability that the input feature comes from the image domain; d i This represents a descriptor, which can be an image descriptor or an event descriptor.

[0143] S5. An optimized identification model is based on contrastive loss, cyclic consistency loss, latent feature consistency loss, self-reconstruction loss, triplet loss, and domain adversarial loss.

[0144] Specifically, the total loss function is constructed based on contrastive loss, cycle consistency loss, latent feature consistency loss, self-reconstruction loss, triplet loss, and domain adversarial loss, as follows:

[0145] L total =L triplet +λ con L con +λ cycle L cycle

[0146] +λ autoenc ·L autoenc +λ latent L latent +λ adver L adver

[0147] Where, λ con , λ cycle , λ autoenc , λ latent and λ adver This represents the hyperparameter that balances the weights of various losses.

[0148] During training, an end-to-end approach is adopted, using the gradient descent algorithm to jointly optimize all network parameters.

[0149] S6. Input the event data of the scene to be identified into the optimized recognition model to generate a scene descriptor, and obtain the scene recognition result based on the scene descriptor.

[0150] S61. For all RGB images in the image database of known locations, I db Using the image encoder E in the optimized recognition model img Advanced Semantic Encoder E adv The feature aggregation layer Agg is processed to generate tag scene descriptors, which are stored together with the corresponding geographic tags to build a scene descriptor database.

[0151] S62. Perform a nearest neighbor search in the scene descriptor database and select the label scene descriptor with the smallest Euclidean distance to the scene descriptor. The geographical location corresponding to this label scene descriptor is the scene recognition result.

[0152] Furthermore, this invention also provides an electronic device for cross-modal scene recognition from RGB images to event cameras, including a central processing unit (CPU) that can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) or loaded from a storage unit into random access memory (RAM). The RAM may also store various programs and data required for device operation. The CPU, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0153] Multiple components in the device are connected to the I / O interface, including: input units such as keyboards and mice; output units such as various types of displays and speakers; storage units such as disks and optical discs; and communication units such as network interface cards (NICs), modems, and wireless transceivers. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0154] The processing unit executes the various methods and processes described above, such as methods S1 to S6. For example, in some embodiments, methods S1 to S6 may be implemented as computer software programs tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the CPU, one or more steps of methods S1 to S6 described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute methods S1 to S6 by any other suitable means (e.g., by means of firmware).

[0155] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0156] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0157] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0158] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A cross-modal scene recognition method from RGB images to event cameras, characterized in that, The method utilizes a recognition model for scene recognition, including: For the same scene sample, extract sample image content features based on the corresponding sample RGB image, extract sample event content features and sample event motion features based on the corresponding sample event data, perform cross-modal contrast learning based on the sample image content features and sample event content features, and calculate contrast loss. Based on the sample image content features and sample event motion features, pseudo-event data is generated by decoding. Based on the pseudo-event data, image reconstruction is performed to obtain a first pseudo-image. Cycle consistency loss is calculated based on the first pseudo-image and the sample RGB image. A second pseudo-image is generated based on the sample event content features. Latent feature consistency loss is calculated based on the pseudo-event data, the second pseudo-image, the sample RGB image, and the sample event data. The sample RGB images and sample event data are reconstructed separately, and the self-reconstruction loss is calculated based on the reconstruction results. For each scene sample, a descriptor is obtained based on the corresponding sample image content features and sample event content features; several scene samples are randomly selected as query samples, and a geographic proximity-far distance triplet is constructed for each query sample. The triplet loss and domain adversarial loss of the geographic proximity-far distance triplet are calculated based on the descriptor; the descriptor includes image descriptors and event descriptors. The recognition model is optimized based on the contrast loss, cycle consistency loss, latent feature consistency loss, self-reconstruction loss, triplet loss, and domain adversarial loss. The event data of the scene to be identified is input into the optimized recognition model to generate a scene descriptor, and the scene recognition result is obtained based on the scene descriptor. The method for calculating the triplet loss is as follows: For each sample scenario, the sample image content features and sample event content features are encoded using advanced semantics to obtain advanced image semantic features and advanced event semantic features; feature aggregation is then performed based on the advanced image semantic features and advanced event semantic features to obtain image descriptors and event descriptors. Geographic information of each scene sample is obtained. For a selected query sample, nearby scene samples are selected as positive samples and distant scene samples are selected as negative samples based on the geographic information to construct a triple. The nearby scene samples are scene samples whose distance from the query sample does not exceed a first preset distance, and the distant scene samples are scene samples whose distance from the query sample is greater than a second preset distance. Select the hard triples from the constructed triples, where the negative samples in the hard triples satisfy: , Indicates the query sample; This represents the j-th sample that is far from the scene in the query sample q; This represents the Euclidean distance between corresponding descriptors, where the descriptor is either an image descriptor or an event descriptor; This represents the i-th best-matching neighboring scene sample of the query sample q; Indicates marginal value; The recognition model is trained using the aforementioned difficult triples, and the triple loss is calculated as follows: 。 2. The cross-modal scene recognition method from RGB image to event camera as described in claim 1, characterized in that, The method for calculating the contrast loss is as follows: For each scene sample, the sample image content features and sample event content features are projected onto the same low-dimensional projection space to obtain image projection features and event projection features. For scene sample i, its image projection features and event projection features are used to construct a positive sample pair, and its image content features are used to construct a negative sample pair with the event projection features or image projection features of the other scene samples. The contrastive loss is calculated based on the aforementioned positive and negative sample pairs, and its expression is as follows: ， in, Indicates the total number of scene samples; Represents the image projection features of scene sample i; The event projection features represent scene sample i; Indicates a positive sample pair; This indicates similarity calculation; Indicates temperature hyperparameter; Represents the image projection features of sample scene j; The event projection features represent scene sample j; and All of these represent negative samples.

3. The cross-modal scene recognition method from RGB image to event camera as described in claim 1, characterized in that, The method for calculating the cycle consistency loss is as follows: The sample image content features and sample event motion features are decoded to generate pseudo-event data, and the pseudo-event content features of the pseudo-event data are extracted. The pseudo-event content features are decoded to obtain a first pseudo-image. Based on the first pseudo-image, the cycle consistency loss is calculated, and its expression is as follows: ， in, This represents the RGB image of the sample corresponding to scene sample i; This represents the first pseudo-image; This indicates the calculation of the 1-norm.

4. The cross-modal scene recognition method from RGB image to event camera according to claim 3, characterized in that, The method for calculating the latent feature consistency loss is as follows: Based on the second pseudo-image, extract the content features of the second pseudo-image; based on the pseudo-event data, extract the motion features of the pseudo-events. The latent feature consistency loss is calculated based on the second pseudo-image content features and pseudo-event motion features, and its expression is as follows: ， in, The sample image content features representing scene sample i; The second pseudo-image content feature represents scene sample i; The sample event content features represent the characteristics of scene sample i; Indicates the characteristics of pseudo-event content; Indicates the equilibrium hyperparameters; Represents the motion features of sample events in scene sample i; This indicates the motion characteristics of pseudo-events.

5. The cross-modal scene recognition method from RGB image to event camera according to claim 1, characterized in that, The method for calculating the self-reconstruction loss is as follows: ， in, Represents the RGB image of the corresponding scene sample i; This represents the reconstructed RGB image of the sample obtained from self-reconstruction of the sample RGB image, and has... , Indicates image encoding operation, Indicates image decoding operation; Represents the calculation of the 1-norm; This represents the sample event data corresponding to scene sample i; This represents the reconstructed event data obtained by self-reconstruction based on sample event data, and , Indicates event coding operation, This indicates an event decoding operation.

6. The cross-modal scene recognition method from RGB image to event camera according to claim 1, characterized in that, The method for calculating the domain adversarial loss is as follows: ， in, Indicates the total number of scene samples; This represents the domain label of the input data for the recognition model; when the input data is an image. When the input data is an event ; This indicates a scalar probability value calculation operation; The descriptor refers to either an image descriptor or an event descriptor.

7. The cross-modal scene recognition method from RGB image to event camera according to claim 1, characterized in that, The methods for obtaining the scene recognition results include: For all RGB images in the image database of known locations, the optimized recognition model is used to generate scene descriptors, which are stored together with the corresponding geographic labels to construct a scene descriptor database. A nearest neighbor search is performed in the scene descriptor database to select the label scene descriptor with the smallest Euclidean distance to the scene descriptor. The geographical location corresponding to this label scene descriptor is the scene recognition result.

8. An electronic device for performing cross-modal scene recognition from RGB images to event cameras, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 7.

Images