Image recognition method and device, electronic equipment and storage medium

Abstract

The invention is applicable to the technical field of artificial intelligence, and provides an image recognition method, which comprises the following steps: acquiring a to-be-recognized image and auxiliary perception data of the to-be-recognized image; performing cross-modal attention fusion processing on the to-be-recognized image and the auxiliary perception data to obtain a multi-modal fusion feature map; and based on user input data, performing fuzzy matching on the multi-modal fusion feature map to obtain an identification result of the to-be-identified image. According to the invention, cross-modal attention fusion processing is carried out on the to-be-recognized image and the auxiliary perception data to obtain the multi-modal fusion feature map, and fuzzy matching is carried out on the multi-modal fusion feature map according to the user input data to obtain the recognition result of the to-be-recognized image, so that the problem that the existing image recognition method is poor in recognition accuracy under a complex shooting condition is solved. And the identification accuracy is low.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, and in particular relates to an image recognition method, device, electronic device and storage medium. Background Technology

[0002] In multimodal recognition scenarios (such as children's object recognition, intelligent visual interaction, and AI toy visual understanding), cameras often need to identify target objects under different lighting, distance, and angle conditions. Currently, existing visual recognition models often experience a decrease in recognition accuracy under backlighting, low illumination, blurred distant scenes, reflections, or occlusion conditions. Especially in consumer-grade embedded devices (such as children's cameras and educational companion robots), traditional visual algorithms struggle to consistently output high-confidence results due to limitations in sensor quality and computing resources. Furthermore, general image classification models are difficult to integrate with multimodal inputs such as semantics and speech in these scenarios, leading to a decrease in the accuracy of overall content generation (such as text generation and semantic association). Therefore, there is an urgent need for an image recognition method that improves the accuracy of object recognition to address the problem of low recognition accuracy under complex shooting conditions in existing image recognition methods. Summary of the Invention

[0003] This application provides an image recognition method that addresses the problem of low recognition accuracy in existing image recognition methods under complex shooting conditions. By performing cross-modal attention fusion processing on the image to be recognized and auxiliary sensing data, a multimodal fusion feature map is obtained. Based on user input data, fuzzy matching is performed on the multimodal fusion feature map to obtain the recognition result of the image to be recognized, thus solving the problem of low recognition accuracy in existing image recognition methods under complex shooting conditions.

[0004] In a first aspect, embodiments of this application provide an image recognition method, the method comprising the following steps:

[0005] Acquire the image to be identified and its auxiliary sensing data;

[0006] The image to be identified and the auxiliary perception data are subjected to cross-modal attention fusion processing to obtain a multimodal fusion feature map;

[0007] Based on user input data, fuzzy matching is performed on the multimodal fusion feature map to obtain the recognition result of the image to be recognized.

[0008] Optionally, the step of performing cross-modal attention fusion processing on the image to be identified and the auxiliary perception data to obtain a multimodal fusion feature map includes:

[0009] The image to be identified is subjected to image enhancement and depth estimation processing to obtain a preprocessed image to be identified;

[0010] The preprocessed image to be identified and the auxiliary perception data are fused together using a cross-modal feature fusion model to obtain a multimodal fusion feature map.

[0011] Optionally, before performing cross-modal attention fusion processing on the preprocessed image to be identified and the auxiliary perception data through the cross-modal feature fusion model, the method further includes training the cross-modal feature fusion model, wherein training the cross-modal feature fusion model specifically includes:

[0012] Obtain the model to be trained and the training dataset. The model to be trained includes a first branch network, a second branch network, and a feature fusion network. The training dataset includes sample image data and sample perception data.

[0013] The model to be trained is trained using the training dataset to obtain a trained cross-modal feature fusion model.

[0014] Optionally, training the model to be trained using the training dataset to obtain a trained cross-modal feature fusion model includes:

[0015] In the first training phase, the first branch network and the second branch network are trained using the training dataset;

[0016] In the second training phase, the first branch network and the second branch network are fixed, and the feature fusion network is trained.

[0017] In the third training phase, contrastive learning loss is used for end-to-end fine-tuning to obtain a trained cross-modal feature fusion model.

[0018] Optionally, the multimodal fusion feature map includes multiple candidate regions, and the step of performing fuzzy matching on the multimodal fusion feature map based on user input data to obtain the recognition result of the image to be recognized includes:

[0019] Convert the user input data into a semantic vector;

[0020] For each candidate region, the classifier outputs the top-N coarse categories;

[0021] For each coarse category, a sub-category is retrieved from a preset category tree, and the cosine similarity between the visual features of the sub-category and the semantic vector is calculated. The category tree includes the visual features of each sub-category.

[0022] Based on the cosine similarity, the recognition result of the image to be recognized is determined.

[0023] Optionally, determining the recognition result of the image to be recognized based on the cosine similarity includes:

[0024] If the confidence level of the recognition result is less than the confidence level threshold, or the image quality of the image to be recognized is less than the image quality threshold, then the target lighting conditions of the image to be recognized are determined.

[0025] The target illumination weight that matches the target illumination condition is found in the cache. The cache stores the correlation between illumination conditions and illumination weights. The illumination weights are obtained by predicting different illumination conditions through an illumination weight prediction model.

[0026] The recognition result is fused with the historical recognition result based on the target illumination weight to obtain the final recognition result.

[0027] Optionally, after obtaining the recognition result of the image to be recognized, the method further includes:

[0028] Obtain the scene-object relationship graph;

[0029] Based on the scene-object relationship graph, the recognition result of the image to be recognized is verified by a graph neural network.

[0030] Secondly, embodiments of this application provide an image recognition device, the image recognition device comprising:

[0031] The acquisition module is used to acquire the image to be identified and the auxiliary sensing data of the image to be identified;

[0032] The processing module is used to perform cross-modal attention fusion processing on the image to be identified and the auxiliary perception data to obtain a multimodal fusion feature map;

[0033] The fuzzy matching module is used to perform fuzzy matching on the multimodal fusion feature map based on user input data to obtain the recognition result of the image to be recognized.

[0034] Thirdly, embodiments of the present invention provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the image recognition method provided in embodiments of the present invention.

[0035] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the image recognition method provided in the embodiments of the invention.

[0036] The above-described solution of this application has the following beneficial effects: acquiring the image to be recognized and its auxiliary sensing data; performing cross-modal attention fusion processing on the image to be recognized and the auxiliary sensing data to obtain a multimodal fusion feature map; and performing fuzzy matching on the multimodal fusion feature map based on user input data to obtain the recognition result of the image to be recognized. This invention solves the problem of low recognition accuracy in existing image recognition methods under complex shooting conditions by performing cross-modal attention fusion processing on the image to be recognized and the auxiliary sensing data to obtain a multimodal fusion feature map, and then performing fuzzy matching on the multimodal fusion feature map based on user input data to obtain the recognition result of the image to be recognized.

[0037] Other beneficial effects of this application will be described in detail in the following detailed description section. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 A flowchart illustrating an image recognition method provided in one embodiment of this application;

[0040] Figure 2 This is a schematic diagram of the structure of an image recognition device provided in an embodiment of this application;

[0041] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0042] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0043] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0044] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0045] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0046] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0047] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0048] like Figure 1 As shown, Figure 1 This is a flowchart of an image recognition method provided by an embodiment of the present invention. The image recognition method includes the following steps:

[0049] 101. Acquire the image to be recognized and its auxiliary perception data.

[0050] In this embodiment of the invention, the image recognition method described above can be applied to an image recognition platform, which can be built on a server-based or distributed platform. The image recognition platform includes a data interface (for sensors or users to upload data), a knowledge database, and a knowledge database construction program. The data interface can be used to obtain a first feature vector to be searched, and the knowledge database construction program can be used to construct the knowledge database. The knowledge database is specifically used to provide additional association information for the identified data entities, thereby improving the depth of the data recognition system's understanding of the content.

[0051] The image to be identified can be an RGB image or image sequence containing the target object captured by a visible light camera. The image to be identified can be image data under complex scenes such as low light, backlight, backlight, strong reflection, occlusion, or bad weather.

[0052] The auxiliary sensing data for the image to be identified can be other modal data that are spatiotemporally associated with the image to be identified and used to supplement information loss or quality degradation of the image under specific physical conditions. For example, auxiliary sensing data can be infrared image data acquired by an infrared sensor, depth information acquired by a depth sensor, voice description information about the image to be identified acquired by an audio acquisition device, or semantic context information acquired by a text input device.

[0053] 102. Perform cross-modal attention fusion processing on the image to be recognized and the auxiliary perception data to obtain a multimodal fusion feature map.

[0054] In this embodiment of the invention, the aforementioned cross-modal attention fusion processing can be a process of analyzing the feature representation of the image to be recognized, dynamically extracting complementary information from the feature representation of the auxiliary sensing data, and performing weighted fusion of the image to be recognized and the auxiliary sensing data. Specifically, the features of the image to be recognized can be used as a 'query', and the features of the auxiliary sensing data can be used as both a 'key' and a 'value' to calculate the cross-modal attention weight. The cross-modal attention weight characterizes the degree of contribution of the auxiliary sensing data features to the features of the image to be recognized at different spatial locations or feature channels during the fusion process. Based on the cross-modal attention weight, the features of the auxiliary sensing data and the features of the image to be recognized are fused to generate multimodal fusion features.

[0055] Furthermore, a cross-modal attention model based on the CrossModal Transformer architecture can be used to perform cross-modal attention fusion processing on the image to be recognized and the auxiliary perception data. Specifically, the formula for cross-modal attention fusion processing is:

[0056] Q = W_q * F_visible

[0057] K = W_k * F_infrared

[0058] V = W_v * F_infrared

[0059] Attention = Softmax(Q·K^T / √d_k) · V

[0060] F_fused = LayerNorm(F_visible + Attention)

[0061] Where Q represents the query vector; W_q represents the parameter matrix, which maps visible light features to the semantic space used for the query; F_visible represents the feature map extracted from the visible light image branch; K represents the key vector; W_k represents the weight matrix, which is responsible for mapping infrared features to the key space that matches Q; F_infrared represents the feature map extracted from the infrared image branch; V represents the value vector; W_v represents the parameter matrix, which is responsible for mapping infrared features to the value space; Attention represents supplementary information from the infrared modality after intelligent filtering and weighting; Q·K^T represents the relevance calculation, which calculates the matching degree between each "question" of the visible light feature and all "indices" of the infrared feature; √d_k represents the scaling operation, where d_k is the dimension of the K vector; V represents weighted aggregation; F_fused represents the multimodal fused feature map; and F_visible + Attention represents residual fusion.

[0062] The visible light image features extracted by the backbone network can be used as the query vector Q, and the features of the auxiliary perception data can be used as both the key vector K and the value vector V. The attention-weighted auxiliary features are obtained by calculating Softmax(Q·K^T / √d_k)·V, and then added to the visible light image features and normalized by the layers to output a multimodal fusion feature map.

[0063] The CrossModal Transformer (CMT) described above is an architecture for multimodal learning. The core idea of ​​CMT is to achieve cross-modal fusion by implicitly encoding 3D positional information into multimodal features, thereby avoiding biases caused by explicit feature alignment and improving the model's robustness. The CMT architecture includes a backbone network, a coordinate encoding module, a position-guided query generator, an encoder, and a masking module for training. The backbone network processes data from different modalities separately, extracting initial feature labels. Both the visible light and infrared branches use EfficientNet-B3 as the backbone network. The coordinate encoding module encodes 3D positional information into multimodal labels. For images, position-aware features are implicitly encoded using 3D points sampled in frustum space; for LiDAR point clouds, BEV (bird's-eye view) coordinates are directly encoded. The position-guided query generator generates queries based on 3D reference points. These queries are transformed to different modal spaces for relative coordinate encoding, enabling the model to focus on cross-modal associations. The decoder employs a Transformer decoder architecture, interacting with multimodal tags to update the representation, ultimately used to predict 3D bounding boxes and categories. Masked modality training is a training strategy that randomly masks some modalities during training, enabling the model to operate robustly in single or mixed modalities and enhancing its tolerance to sensor failures.

[0064] The aforementioned multimodal fusion feature map can be obtained by performing cross-modal attention fusion processing on the image to be recognized and the auxiliary perception data. The multimodal fusion feature is a joint feature representation that integrates the visual features of the image to be recognized and the complementary features of the auxiliary perception data.

[0065] 103. Based on user input data, perform fuzzy matching on the multimodal fusion feature map to obtain the recognition result of the image to be recognized.

[0066] In this embodiment of the invention, the aforementioned user input data can be input data actively provided by the user through an interactive device to guide or constrain the current object recognition task. It can be voice data, text input data, interactive instruction data, etc. Voice data can be user-generated spoken descriptions or instructions collected through a microphone; text input data can be descriptive keywords or statements entered via a keyboard or touchscreen; interactive instruction data can be areas of interest or objects specified by the user in the image interface through clicking, selecting, or drawing lines.

[0067] The aforementioned fuzzy matching can be a technique that allows comparing different list items and considering them matches when they are similar. Candidate regions can be generated using Grad-CAM-based saliency detection, with the region score formula: S_region = Σ(M_attention ⊙ F_feature) / Σ(M_attention), where S_region represents the region score; M_attention represents the attention mask, a matrix generated by Grad-CAM, where each element represents the attention weight at the corresponding spatial location; and F_feature represents the multimodal fusion feature map.

[0068] Furthermore, fuzzy matching can be performed for each candidate region, where the classifier outputs the top-N coarse categories, and for each coarse category, relevant subcategories are retrieved in the Trie tree. The cosine similarity between the visual features and the semantic vector is calculated to obtain a comprehensive score. For example, when the comprehensive score is <0.7, the fuzzy matching process is triggered to reduce visual ambiguity through semantic constraints.

[0069] Specifically, user input data can be transformed into high-dimensional semantic vectors through a semantic encoding model, and fuzzy matching can be performed in the multimodal fusion feature map to match all relevant subcategories, forming an extended candidate set. The cosine similarity between the visual features of the target region in the multimodal fusion feature map and the semantic vectors corresponding to each subcategory in the extended candidate set is calculated. The visual confidence and semantic similarity are fused according to preset weights to obtain a comprehensive score for each candidate category. The candidate category with the highest comprehensive score is determined as the recognition result of the image to be recognized.

[0070] The aforementioned recognition result can be the optimal output obtained after fuzzy matching based on user input data, using a robust feature base provided by a multimodal fusion feature map. The recognition result represents the optimal understanding of the image to be recognized in complex scenes.

[0071] In this embodiment of the invention, the present invention can adopt the CrossModal Transformer architecture to perform cross-modal attention fusion processing on the image to be recognized and the auxiliary perception data to obtain a multimodal fusion feature map. Based on the user input data, fuzzy matching is performed on the multimodal fusion feature map to obtain the recognition result of the image to be recognized. This can solve the problem of low recognition accuracy of existing image recognition methods under complex shooting conditions.

[0072] In this embodiment of the invention, an image to be identified and its auxiliary sensing data are acquired; the image to be identified and the auxiliary sensing data are subjected to cross-modal attention fusion processing to obtain a multimodal fusion feature map; based on user input data, fuzzy matching is performed on the multimodal fusion feature map to obtain the recognition result of the image to be identified. This invention solves the problem of low recognition accuracy in existing image recognition methods under complex shooting conditions by performing cross-modal attention fusion processing on the image to be identified and the auxiliary sensing data to obtain a multimodal fusion feature map, and then performing fuzzy matching on the multimodal fusion feature map based on user input data to obtain the recognition result of the image to be identified.

[0073] It is understood that in the specific implementation of this application, data such as image data, assisted perception data, multimodal data, and user data are involved. When the embodiments in this application are applied to specific products or technologies, user permission or consent is required. Furthermore, the collection, use, and processing of related data, as well as the training, deployment, and invocation of algorithm models, must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0074] Optionally, in the step of performing cross-modal attention fusion processing on the image to be identified and the auxiliary perception data to obtain a multimodal fusion feature map, image enhancement processing and depth estimation processing can be performed on the image to be identified to obtain a preprocessed image to be identified; the preprocessed image to be identified and the auxiliary perception data can be performed cross-modal attention fusion processing through a cross-modal feature fusion model to obtain a multimodal fusion feature map.

[0075] In this embodiment of the invention, the image enhancement processing described above can be a process of enhancing the image to be recognized. Image enhancement processing aims to highlight features of interest (such as edges and contrast) in the image and suppress irrelevant noise or interference, thereby improving the readability and analytical value of the image. It can employ adaptive illumination compensation combined with multi-frame temporal fusion technology (HDR) to restore dark details in backlit or low-light scenes by fusing the Retinex model with HDR. The aforementioned adaptive illumination compensation can be a technique that dynamically adjusts the output of the image acquisition device through algorithms or hardware (e.g., the Retinex model), aiming to eliminate or reduce the impact of changes in ambient light on image quality, thereby maintaining the consistency of brightness, contrast, and recognizability of the image under different lighting conditions. The aforementioned multi-frame temporal fusion technology is a method to improve visual perception quality by processing multiple frames of images in a continuous time series. The core idea of ​​multi-frame temporal fusion technology is to utilize redundant information and complementary features in the temporal dimension to integrate visual data from different times to enhance the dynamic range of the image, suppress noise, or improve detail representation. The Retinex model, proposed by Edwin H. Land in 1963, is a theory for calculating color constancy, designed to explain the human visual system's ability to stably perceive the color of objects under different lighting conditions. The core idea of ​​Retinex theory is that an object's color is determined by its ability to reflect long-wavelength, medium-wavelength, and short-wavelength light, rather than the absolute value of the reflected light intensity. This means that the human visual system can automatically separate the effects of lighting and identify the inherent color attributes of objects. HDR (High Dynamic Range), mentioned above, is an imaging technique designed to achieve a wider dynamic range of exposure than ordinary digital imaging techniques. It can simultaneously capture and display details in both extremely bright and extremely dark areas, thus more realistically reproducing real-world lighting conditions. Furthermore, through image enhancement processing, the inherent defects of the image to be identified under conditions such as backlighting and low illumination are initially and specifically improved.

[0076] The aforementioned depth estimation processing can be a process of evaluating the depth range that can be clearly focused in the enhanced image to be identified. Depth estimation processing can change the color intensity and brightness of objects according to their distance from the observer. In other words, it is a function that reduces the color and brightness of objects as they move away from the observer. For example, as an object moves further away from the observer's line of sight, it appears increasingly blurry.

[0077] Depth information can be estimated from the image to be recognized using a lightweight DepthNet network (based on a MobileNetV3 encoder and a transposed convolutional decoder). The network loss function combines scale-invariant log loss and gradient matching loss.

[0078] L_depth = λ1 * L_silog + λ2 * L_grad

[0079] L_silog = 1 / n * Σ(log d_i - log d_i^*)^2 - 1 / n^2 * (Σ(log d_i - logd_i^*))^2

[0080] L_grad = 1 / n * Σ(|∇x(d_i - d_i^*)| + |∇y(d_i - d_i^*)|)

[0081] Where L_depth represents the total loss; λ1 and λ2 represent hyperparameters; L_silog represents the scale-invariant logarithmic loss; L_grad represents the gradient matching loss; n represents the total number of pixels in the image to be identified; d_i represents the predicted depth value of the i-th pixel; d_i^* represents the true depth value of the i-th pixel; ∇x(d_i - d_i^*) represents the gradient of the difference between the predicted depth map and the true depth map in the x-direction; and ∇y(d_i - d_i^*) represents the gradient of the difference between the predicted depth map and the true depth map in the y-direction.

[0082] Under backlighting conditions, an attention mask is generated using the estimated depth map to locally enhance the contrast of foreground object regions. Adaptive histogram equalization is applied only to the foreground region to avoid amplifying background noise. Multi-frame fusion employs weighted fusion after optical flow alignment.

[0083] I_fused = Σ(w_i * I_i) / Σ(w_i)

[0084] Where w_i = exp(-|I_i - I_median| / σ).

[0085] Where I_fused represents the fused output image; I_i represents the i-th frame in a sequence of consecutive multi-frame images; w_i represents the weight of the i-th frame; I_median represents a "median image" obtained by taking the median at each pixel position in the frame image sequence; and σ represents the scale parameter.

[0086] The preprocessed image to be identified can be obtained by performing image enhancement and depth estimation processing on the image to be identified.

[0087] The aforementioned cross-modal feature fusion models can be built based on deep learning or machine learning, such as CrossModal Transformer (CMT) and Vision Transformer (VIT). CMT is an architecture for multimodal learning. Its core idea is to achieve cross-modal fusion by implicitly encoding 3D positional information into multimodal features, thereby avoiding biases caused by explicit feature alignment and improving model robustness. VIT is a deep learning model that applies the Transformer architecture to computer vision tasks. VIT achieves goals such as image recognition by segmenting images into small patches and using a self-attention mechanism to capture global dependencies.

[0088] The aforementioned cross-modal attention fusion processing can be a process of analyzing the feature representation of the preprocessed image to be recognized through a cross-modal feature fusion model, dynamically extracting complementary information from the feature representation of the auxiliary sensing data, and then performing a weighted fusion of the image to be recognized and the auxiliary sensing data.

[0089] The aforementioned multimodal fusion feature map can be obtained by performing cross-modal attention fusion processing on the preprocessed image to be recognized and the auxiliary perception data through a cross-modal feature fusion model. The multimodal fusion feature is a joint feature representation that integrates the visual features of the image to be recognized and the complementary features of the auxiliary perception data.

[0090] Optionally, before performing cross-modal attention fusion processing on the preprocessed image to be recognized and the auxiliary perception data through the cross-modal feature fusion model, the cross-modal feature fusion model can be trained. The training of the cross-modal feature fusion model specifically includes: obtaining the model to be trained and the training dataset. The model to be trained includes a first branch network, a second branch network, and a feature fusion network. The training dataset includes sample image data and sample perception data. The model to be trained is trained using the training dataset to obtain the trained cross-modal feature fusion model.

[0091] In this embodiment of the invention, the aforementioned cross-modal feature fusion model can be a cross-modal feature fusion model built based on deep learning or machine learning, such as CrossModal Transformer, VIT, etc. The cross-modal feature fusion model aims to integrate models from different modalities (such as text, images, audio, video, etc.) by associating and unifying multi-source heterogeneous information to improve the machine's ability to understand complex data and its task performance.

[0092] The above training can be supervised training. The basic idea of ​​supervised training is to use a set of training data with known inputs and corresponding outputs (i.e. labels) to train the model, so that the model can learn the mapping relationship from input to output.

[0093] The model to be trained includes a first branch network, a second branch network, and a feature fusion network. The first branch network can be a visible light image branch, with EfficientNet-B3 as the backbone network. This first branch network is specifically responsible for extracting high-quality visual features (color, texture, etc.) from visible light images. The second branch network can be an infrared image branch, with EfficientNet-B3 as the backbone network. This second branch network is specifically responsible for extracting complementary features from infrared images (auxiliary perception data). The feature fusion network receives features from both branch networks and uses an attention mechanism to allow the features of the two branch networks to dynamically interact and complement each other, ultimately outputting fused features.

[0094] The training dataset includes sample image data and sample perception data. The sample image data can be visible light images acquired under various complex scenes (backlighting, low illumination, occlusion, etc.). The sample perception data can be auxiliary perception data corresponding to the sample image data.

[0095] The trained cross-modal feature fusion model described above can identify the multimodal feature fusion map of the image to be identified and its auxiliary perceptual data.

[0096] Specifically, during training, the optimization objective can be to minimize the loss function. The model parameters of the cross-modal feature fusion model are adjusted using the backpropagation algorithm. This adjustment process is iterated until the total error loss is less than a preset value, or the number of iterations reaches a preset number. The training process then ends, resulting in a well-trained cross-modal feature fusion model. The loss function measures the difference between the model's prediction and the actual result. Its purpose is to minimize the loss function value by adjusting the model parameters, thereby improving prediction accuracy. The backpropagation algorithm described above can be understood as an algorithm that updates the weights by calculating the gradient of the loss function to minimize the loss.

[0097] Optionally, in the step of training the model to be trained using the training dataset to obtain a trained cross-modal feature fusion model, the first branch network and the second branch network can be trained using the training dataset in the first training stage; in the second training stage, the first branch network and the second branch network can be fixed and the feature fusion network can be trained; in the third training stage, the contrastive learning loss can be used for end-to-end fine-tuning to obtain a trained cross-modal feature fusion model.

[0098] In this embodiment of the invention, the training dataset includes sample image data and sample perception data. The sample image data can be visible light images acquired under various complex scenes (backlighting, low illumination, occlusion, etc.). The sample perception data can be auxiliary perception data corresponding to the sample image data.

[0099] The first branch network mentioned above can be a visible light image branch, with the backbone network being EfficientNet-B3. The first branch network is specifically responsible for extracting high-quality visual features (color, texture, etc.) from visible light images.

[0100] The second branch network mentioned above can be an infrared image branch, with the backbone network being EfficientNet-B3. The second branch network is specifically responsible for extracting complementary features from infrared images (auxiliary sensing data).

[0101] Understandably, in the first training phase, training the first and second branch networks with the training dataset allows the two branch networks to learn to extract the most useful and discriminative features from their respective modalities.

[0102] The aforementioned feature fusion network can receive features from two branch networks and use an attention mechanism to allow the features of the two branch networks to dynamically interact and complement each other, ultimately outputting fused features.

[0103] In the second training phase, all parameters of the two pre-trained branch networks from the first phase are fixed, and only the parameters of the feature fusion network are allowed to be updated through training data. This allows the network to focus on learning fusion, enabling it to learn how to establish relationships between the features of the two branch networks and dynamically determine the information exchange method. This reduces the number of parameters that need to be optimized simultaneously, lowers the training difficulty and uncertainty, and makes the training of the feature fusion network faster and more stable.

[0104] The contrastive learning loss mentioned above can be the objective function for deep metric learning. The core purpose of the contrastive learning loss is to narrow the distance between matched cross-modal feature pairs and widen the distance between mismatched feature pairs, thereby achieving alignment between visible light image features and auxiliary perception data features in the deep semantic space.

[0105] The aforementioned end-to-end fine-tuning can be a training phase in which the parameters of the cross-modal feature fusion model are finely adjusted by comparing the learning loss after the first and second training phases are completed.

[0106] The trained cross-modal feature fusion model described above can be obtained by training the model to be trained in stages using a training dataset. The trained cross-modal feature fusion model can identify the multimodal feature fusion map of the image to be identified and its auxiliary perceptual data.

[0107] It should be noted that in the third training stage, the cross-modal feature fusion model can be trained using the training dataset, and end-to-end fine-tuning can be performed by comparing the learning loss to achieve synergistic optimization of the model's overall representation ability and semantic alignment of cross-modal features.

[0108] It should be noted that end-to-end fine-tuning can effectively compensate for the insufficient adaptation between modules caused by staged training, and help the model find parameter configurations with better global performance.

[0109] Optionally, the multimodal fusion feature map includes multiple candidate regions. In the step of performing fuzzy matching on the multimodal fusion feature map based on user input data to obtain the recognition result of the image to be recognized, the user input data can be converted into a semantic vector. For each candidate region, the top-N coarse categories are output by the classifier. For each coarse category, the sub-categories are retrieved from the preset category tree, and the cosine similarity between the visual features of the sub-categories and the semantic vector is calculated. The category tree includes the visual features of each sub-category. Based on the cosine similarity, the recognition result of the image to be recognized is determined.

[0110] In this embodiment of the invention, the candidate region may be a spatial location or feature fragment containing the target object.

[0111] The aforementioned user input data can be input data actively provided by the user through the interactive device to guide or constrain the current object recognition task. It can be voice data, text input data, interactive instruction data, etc.

[0112] Furthermore, user input data can be transformed into semantic vectors using a semantic encoding model. These semantic vectors represent the core semantics of the user input data. The aforementioned semantic encoding model can be a deep learning model used to generate sentence vector representations, employing the Sentence-BERT structure. Sentence-BERT is based on the BERT architecture and incorporates a Siamese network structure, aiming to optimize sentence-level semantic similarity calculation. The core idea of ​​Sentence-BERT is to fine-tune the BERT model so that its output sentence vectors can better capture semantic similarity.

[0113] For each candidate region, a classifier is used to classify the features of each candidate region, outputting a list of the top N categories with the highest probabilities, which can quickly narrow down the search scope. The classifier can be a pre-trained neural network, and its function is to map the input candidate region features to a set of preset category probabilities.

[0114] The above top-N can be understood as the top N coarse categories with the highest output probability (highest confidence), such as top-3, top-5, etc.

[0115] The aforementioned preset category tree can be a pre-built, hierarchical knowledge base, or a Trie tree. A Trie tree can contain over 200,000 Chinese object names, supporting prefix matching and pinyin error tolerance. The upper level of the category tree consists of coarse categories, and the lower level contains a massive number of subcategories. Each subcategory is not only associated with a name but also has its typical visual features pre-stored. The category tree includes the visual features of each subcategory. The aforementioned Trie tree, also known as a dictionary tree or prefix tree, is a tree-like data structure primarily used for efficiently storing and retrieving string sets. The core idea of ​​a Trie tree is to utilize the common prefixes of strings to share storage space, thereby achieving high efficiency in string matching, prefix lookup, and other operations.

[0116] The aforementioned cosine similarity can be the cosine similarity between the visual features and semantic vectors of a subcategories. Cosine similarity measures the directional consistency between the visual features and semantic vectors of a subcategories in high-dimensional space, transforming vague natural language descriptions from users into precise quantitative scores for specific visual candidate categories. It can be calculated as the cosine of the angle θ between the visual features and semantic vectors of a subcategories, where a value closer to 1 indicates an angle close to 0 degrees, signifying greater similarity between the visual features and semantic vectors, while a value closer to -1 indicates an angle close to 180 degrees, signifying less similarity.

[0117] It should be noted that for each candidate region, the classifier outputs the top-N coarse categories. For each coarse category, relevant subcategories are retrieved in the Trie tree. The cosine similarity between the visual features and the semantic vector is calculated to obtain a comprehensive score. For example, the comprehensive score = 0.6 * visual confidence + 0.4 * semantic similarity. Fuzzy matching is triggered when the comprehensive score is <0.7. This can reliably select the answer data that best matches the user from multiple visually similar candidate regions, thereby solving the ambiguity problem in complex scenarios.

[0118] Optionally, in the step of determining the recognition result of the image to be recognized based on cosine similarity, if the confidence level of the recognition result is less than the confidence level threshold, or the image quality of the image to be recognized is less than the image quality threshold, then the target illumination condition of the image to be recognized is determined; the target illumination weight that matches the target illumination condition is searched in the cache, the cache stores the correlation between illumination conditions and illumination weights, and the illumination weights are predicted for different illumination conditions by the illumination weight prediction model; the recognition result is fused with the historical recognition results based on the target illumination weights to obtain the final recognition result.

[0119] In this embodiment of the invention, the confidence threshold can be a pre-set confidence threshold, such as a confidence threshold of 0.6. The image quality threshold can also be a pre-set image quality threshold, such as an image quality threshold of 0.5.

[0120] The aforementioned target lighting conditions can be vectors or labels describing the current ambient lighting. They can also be obtained by analyzing the image to be identified and quantifying its lighting attributes. Furthermore, the target lighting conditions of the image to be identified can be obtained by calculating the global light intensity, color temperature, and contrast statistics of the image.

[0121] The aforementioned cache can be a pre-configured cache by the system, storing the correlation between lighting conditions and lighting weights. The cache can be a three-level cache architecture, including a first-level cache, a second-level cache, and a third-level cache. Specifically, the first-level cache stores the most recent M recognition results, where M can be 100, 110, etc.; the second-level cache stores feature templates under typical lighting conditions; and the third-level cache stores the user's personalized recognition history.

[0122] The aforementioned target illumination weights can be obtained by predicting the target illumination conditions using an illumination weight prediction model.

[0123] The aforementioned illumination weight prediction model can employ a 3-layer MLP, taking ambient light intensity, color temperature, and contrast statistics as inputs, and outputting recognition weights under different illumination conditions. The MLP (Multilayer Perceptron) is a feedforward neural network comprising an input layer, hidden layers, and an output layer. Each layer consists of multiple neurons, and all layers are fully connected.

[0124] The aforementioned illumination weight prediction model can be obtained by training a pre-trained illumination weight prediction model using a training dataset. This training dataset includes sample image data, corresponding illumination annotation data, and illumination weight annotation data. The sample image data can be image data collected under different illumination conditions. The annotation data refers to the process of adding labels or structured information to raw data, transforming it into a form that machine learning algorithms can recognize and understand. Through annotation data, the model can learn features from the data and improve recognition accuracy. The aforementioned pre-trained illumination weight prediction model can be an illumination weight prediction model built based on deep learning or machine learning, such as an MLP or DNN. An MLP (Multilayer Perceptron) is a feedforward neural network containing an input layer, hidden layers, and an output layer. Each layer consists of multiple neurons, and the layers are fully connected. The aforementioned DNN (Deep Neural Network) model learns complex feature representations of data through multi-layer nonlinear transformations. Its core principle is to simulate the working method of neurons in the human brain, consisting of an input layer, multiple hidden layers, and an output layer, each containing multiple neurons.

[0125] The training described above can be supervised training. The basic idea of ​​supervised training is to use a set of training data with known inputs and corresponding outputs (i.e., labels) to train the model, enabling the model to learn the mapping relationship from inputs to outputs.

[0126] Furthermore, during training, the optimization objective can be to minimize the loss function. The model parameters of the pre-trained illumination weight prediction model can be adjusted using the backpropagation algorithm. This adjustment process is iterated until the total error loss is less than a preset value, or the number of iterations reaches a preset number, at which point the training process ends, resulting in a well-trained illumination weight prediction model. The loss function measures the difference between the model's prediction and the actual result. Its purpose is to minimize the loss function value by adjusting the model parameters, thereby improving prediction accuracy. The backpropagation algorithm described above can be understood as an algorithm that updates the weights by calculating the gradient of the loss function to minimize the loss.

[0127] The loss function described above can be cross-entropy loss plus weight distribution regularization. Cross-entropy loss ensures that the predicted weight distribution is consistent with the true reliability label distribution. Weight distribution regularization prevents the model from predicting overly extreme (all 0s or all 1s) weights, encourages a smooth and reasonable weight distribution, and improves generalization ability.

[0128] The above recognition result can be obtained by performing fuzzy matching on a multimodal fusion feature map based on user input data.

[0129] The aforementioned historical recognition results can be retrieved from the cache or saved from previously successfully recognized instances.

[0130] The aforementioned fusion can be a weighted fusion process that combines the recognition result with historical recognition results based on the target illumination weight. Further, based on the target illumination weight, the first illumination weight corresponding to the current recognition result and the second illumination weight corresponding to the historical recognition results can be obtained. Then, the current recognition result is multiplied by the first illumination weight, the historical recognition result is multiplied by the second illumination weight, and the two are added together to obtain the final recognition result. Specifically, the final recognition result can be expressed as:

[0131] Final_result = (R1 * (1 – w1) + R2 * w2)

[0132] Where Final_result represents the final recognition result; R1 represents the recognition result; w1 represents the first weight of the recognition result; R2 represents the historical recognition result; and w2 represents the second weight corresponding to the historical recognition result.

[0133] The final response data mentioned above can be obtained by weighting and fusing the recognition results with historical recognition results according to the illumination weight.

[0134] In one possible implementation, for example, when the confidence threshold is 0.6 and the image quality threshold is 0.5, if the confidence of the recognition result is less than the confidence threshold of 0.6, or the image quality of the image to be recognized is less than the image quality threshold of 0.5, multi-frame fusion re-recognition is enabled, and templates with similar lighting conditions are queried in the cache. If the cache is hit, the current result and the cached result are fused using weighted voting to obtain the final answer data.

[0135] It's important to note that the aforementioned first-level and second-level caches can employ an LRU (Least Recently Used) policy. LRU is a commonly used cache eviction algorithm. The core idea of ​​LRU is based on the principle of temporal locality, which assumes that recently accessed data has a higher probability of being accessed again in the future, while data that hasn't been used for a long time may no longer be needed. Therefore, when cache space is insufficient, the LRU algorithm will prioritize evicting the least recently used data to make room for new data.

[0136] Optionally, after obtaining the recognition result of the image to be recognized, a scene-object relationship graph can be obtained; based on the scene-object relationship graph, the recognition result of the image to be recognized can be verified by a graph neural network.

[0137] In this embodiment of the invention, the aforementioned scene-object relationship graph can be a pre-set scene-object relationship graph by the system. The scene-object relationship graph can be a knowledge base used to represent common-sense associations between scenes and objects. The scene-object relationship graph is stored in a graph structure, where nodes represent scene or object categories, and edges represent the semantic relationships and relationship strength between them. The aforementioned scene-object relationship graph can be a scene-object relationship graph containing 5000+ common-sense relationships.

[0138] The above verification can be a process of using graph neural networks to perform reasonableness checks based on the scene-object relationship graph. Reasonableness checks can filter out recognition results that do not conform to common sense.

[0139] Furthermore, the scene-object relationship graph is used to verify the reasonableness of the preliminary identification results. Specifically, the object categories identified by the graph neural network and the estimated scene categories are queried in the scene-object relationship graph. If the query result is a strong negative association (such as 'unreasonable' or 'impossible') or the association degree is lower than a preset threshold, the identification result is determined to be inconsistent with common sense.

[0140] The aforementioned graph neural network can employ the T5-base model to generate conditional text from the recognition results. The input templates are [identified object][scene description][time information], and the training data uses text-image description pairs from the educational domain. Accuracy and engaging content are balanced through temperature sampling and kernel sampling. The diversity of generated data is controlled to ensure that the output is both accurate and appropriate for children's comprehension levels. The T5-Base model is a basic version in Google's Text-to-Text Transfer Transformer series, employing an encoder-decoder architecture to unify all Natural Language Processing (NLP) tasks into a text-to-text conversion problem.

[0141] The temperature sampling and kernel sampling (Top-p sampling) mentioned above can be two key methods for controlling the diversity and creativity of text generated by control graph neural networks, achieving a balance through different mechanisms. Temperature sampling controls randomness by adjusting the "sharpness" of the probability distribution. Kernel sampling (Top-p) balances flexibility and naturalness by dynamically selecting a set of candidate words.

[0142] It should be noted that the recognition results can be verified by using a graph neural network based on the scene-object relationship graph. This can avoid obvious errors caused by the limitations of the model, making the recognition results not only based on data statistics but also more in line with cognitive logic, thereby further improving the accuracy and reliability of the recognition results at the semantic level.

[0143] like Figure 2 As shown, an embodiment of the present invention provides an image recognition device, which includes:

[0144] The acquisition module 201 is used to acquire the image to be identified and the auxiliary perception data of the image to be identified;

[0145] Processing module 202 is used to perform cross-modal attention fusion processing on the image to be identified and the auxiliary perception data to obtain a multimodal fusion feature map;

[0146] The fuzzy matching module 203 is used to perform fuzzy matching on the multimodal fusion feature map based on user input data to obtain the recognition result of the image to be recognized.

[0147] Optionally, the processing module 202 is further configured to perform image enhancement processing and depth estimation processing on the image to be identified to obtain a preprocessed image to be identified; and to perform cross-modal attention fusion processing on the preprocessed image to be identified and the auxiliary perception data through a cross-modal feature fusion model to obtain a multimodal fusion feature map.

[0148] Optionally, the device is further configured to train a cross-modal feature fusion model. The training of the cross-modal feature fusion model specifically includes: acquiring a model to be trained and a training dataset, wherein the model to be trained includes a first branch network, a second branch network, and a feature fusion network, and the training dataset includes sample image data and sample perception data; and training the model to be trained using the training dataset to obtain a trained cross-modal feature fusion model.

[0149] Optionally, the device is further configured to train the first branch network and the second branch network using a training dataset in a first training phase; train the feature fusion network while fixing the first branch network and the second branch network in a second training phase; and fine-tune the model end-to-end using contrastive learning loss in a third training phase to obtain a trained cross-modal feature fusion model.

[0150] Optionally, the fuzzy matching module 203 is further configured to convert the user input data into a semantic vector; for each candidate region, output the top-N coarse categories through a classifier; for each coarse category, retrieve the sub-categories in a preset category tree, calculate the cosine similarity between the visual features of the sub-categories and the semantic vector, wherein the category tree includes the visual features of each sub-category; and determine the recognition result of the image to be recognized based on the cosine similarity.

[0151] Optionally, the processing module 202 is further configured to: determine the target illumination condition of the image to be identified if the confidence level of the recognition result is less than a confidence level threshold, or the image quality of the image to be identified is less than an image quality threshold; search in a cache for a target illumination weight that matches the target illumination condition, wherein the cache stores the correlation between illumination conditions and illumination weights, and the illumination weights are obtained by predicting different illumination conditions through an illumination weight prediction model; and fuse the recognition result with historical recognition results based on the target illumination weights to obtain the final recognition result.

[0152] Optionally, the device is further configured to acquire a scene-object relationship graph; and based on the scene-object relationship graph, to verify the recognition result of the image to be recognized using a graph neural network.

[0153] like Figure 3 As shown, this embodiment of the invention also provides an electronic device, including a processor, which can execute any of the above-described image recognition methods.

[0154] Specifically, it includes a processor 301 and a memory 302, as well as a computer program stored in the memory 302 and capable of running on the processor 301 to perform an image recognition method, wherein:

[0155] The processor 301 executes the calculator program for the image recognition method stored in the memory 302, performing the following steps:

[0156] Acquire the image to be identified and its auxiliary sensing data;

[0157] The image to be identified and the auxiliary perception data are subjected to cross-modal attention fusion processing to obtain a multimodal fusion feature map;

[0158] Based on user input data, fuzzy matching is performed on the multimodal fusion feature map to obtain the recognition result of the image to be recognized.

[0159] Optionally, the processor 301 performs cross-modal attention fusion processing on the image to be identified and the auxiliary perception data to obtain a multimodal fusion feature map, including:

[0160] The image to be identified is subjected to image enhancement and depth estimation processing to obtain a preprocessed image to be identified;

[0161] The preprocessed image to be identified and the auxiliary perception data are fused together using a cross-modal feature fusion model to obtain a multimodal fusion feature map.

[0162] Optionally, before performing cross-modal attention fusion processing on the preprocessed image to be recognized and the auxiliary perception data through the cross-modal feature fusion model, the method executed by the processor 301 further includes training the cross-modal feature fusion model, wherein training the cross-modal feature fusion model specifically includes:

[0163] Obtain the model to be trained and the training dataset. The model to be trained includes a first branch network, a second branch network, and a feature fusion network. The training dataset includes sample image data and sample perception data.

[0164] The model to be trained is trained using the training dataset to obtain a trained cross-modal feature fusion model.

[0165] Optionally, the processor 301 performs the step of training the model to be trained using the training dataset to obtain a trained cross-modal feature fusion model, including:

[0166] In the first training phase, the first branch network and the second branch network are trained using the training dataset;

[0167] In the second training phase, the first branch network and the second branch network are fixed, and the feature fusion network is trained.

[0168] In the third training phase, contrastive learning loss is used for end-to-end fine-tuning to obtain a trained cross-modal feature fusion model.

[0169] Optionally, the multimodal fusion feature map includes multiple candidate regions. The processor 301 performs fuzzy matching on the multimodal fusion feature map based on user input data to obtain the recognition result of the image to be recognized, including:

[0170] Convert the user input data into a semantic vector;

[0171] For each candidate region, the classifier outputs the top-N coarse categories;

[0172] For each coarse category, a sub-category is retrieved from a preset category tree, and the cosine similarity between the visual features of the sub-category and the semantic vector is calculated. The category tree includes the visual features of each sub-category.

[0173] Based on the cosine similarity, the recognition result of the image to be recognized is determined.

[0174] The step of determining the recognition result of the image to be recognized based on the cosine similarity, performed by an optional processor 301, includes:

[0175] If the confidence level of the recognition result is less than the confidence level threshold, or the image quality of the image to be recognized is less than the image quality threshold, then the target lighting conditions of the image to be recognized are determined.

[0176] The target illumination weight that matches the target illumination condition is found in the cache. The cache stores the correlation between illumination conditions and illumination weights. The illumination weights are obtained by predicting different illumination conditions through an illumination weight prediction model.

[0177] The recognition result is fused with the historical recognition result based on the target illumination weight to obtain the final recognition result.

[0178] Optionally, after obtaining the recognition result of the image to be recognized, the method executed by the processor 301 further includes:

[0179] Obtain the scene-object relationship graph;

[0180] Based on the scene-object relationship graph, the recognition result of the image to be recognized is verified by a graph neural network.

[0181] This invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the image recognition method provided in this invention and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0182] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An image recognition method, characterized in that, The method includes the following steps: Acquire the image to be identified and its auxiliary sensing data; The image to be identified and the auxiliary perception data are subjected to cross-modal attention fusion processing to obtain a multimodal fusion feature map; Based on user input data, fuzzy matching is performed on the multimodal fusion feature map to obtain the recognition result of the image to be recognized.

2. The image recognition method as described in claim 1, characterized in that, The step of performing cross-modal attention fusion processing on the image to be identified and the auxiliary perception data to obtain a multimodal fusion feature map includes: The image to be identified is subjected to image enhancement and depth estimation processing to obtain a preprocessed image to be identified; The preprocessed image to be identified and the auxiliary perception data are fused together using a cross-modal feature fusion model to obtain a multimodal fusion feature map.

3. The image recognition method as described in claim 2, characterized in that, Before performing cross-modal attention fusion processing on the preprocessed image to be identified and the auxiliary perception data through the cross-modal feature fusion model, the method further includes training the cross-modal feature fusion model, which specifically includes: Obtain the model to be trained and the training dataset. The model to be trained includes a first branch network, a second branch network, and a feature fusion network. The training dataset includes sample image data and sample perception data. The model to be trained is trained using the training dataset to obtain a trained cross-modal feature fusion model.

4. The image recognition method as described in claim 3, characterized in that, The step of training the model to be trained using the training dataset to obtain a trained cross-modal feature fusion model includes: In the first training phase, the first branch network and the second branch network are trained using the training dataset; In the second training phase, the first branch network and the second branch network are fixed, and the feature fusion network is trained. In the third training phase, contrastive learning loss is used for end-to-end fine-tuning to obtain a trained cross-modal feature fusion model.

5. The image recognition method as described in claim 4, characterized in that, The multimodal fusion feature map includes multiple candidate regions. The step of performing fuzzy matching on the multimodal fusion feature map based on user input data to obtain the recognition result of the image to be recognized includes: Convert the user input data into a semantic vector; For each candidate region, the classifier outputs the top-N coarse categories; For each coarse category, a sub-category is retrieved from a preset category tree, and the cosine similarity between the visual features of the sub-category and the semantic vector is calculated. The category tree includes the visual features of each sub-category. Based on the cosine similarity, the recognition result of the image to be recognized is determined.

6. The image recognition method as described in claim 5, characterized in that, The step of determining the recognition result of the image to be recognized based on the cosine similarity includes: If the confidence level of the recognition result is less than the confidence level threshold, or the image quality of the image to be recognized is less than the image quality threshold, then the target lighting conditions of the image to be recognized are determined. The target illumination weight that matches the target illumination condition is found in the cache. The cache stores the correlation between illumination conditions and illumination weights. The illumination weights are obtained by predicting different illumination conditions through an illumination weight prediction model. The recognition result is fused with the historical recognition result based on the target illumination weight to obtain the final recognition result.

7. The image recognition method according to any one of claims 1 to 6, characterized in that, After obtaining the recognition result of the image to be recognized, the method further includes: Obtain the scene-object relationship graph; Based on the scene-object relationship graph, the recognition result of the image to be recognized is verified by a graph neural network.

8. An image recognition device, characterized in that, The image recognition device includes: The acquisition module is used to acquire the image to be identified and the auxiliary sensing data of the image to be identified; The processing module is used to perform cross-modal attention fusion processing on the image to be identified and the auxiliary perception data to obtain a multimodal fusion feature map; The fuzzy matching module is used to perform fuzzy matching on the multimodal fusion feature map based on user input data to obtain the recognition result of the image to be recognized.

9. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the image recognition method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the image recognition method as described in any one of claims 1 to 7.

Images