Image recognition method, device, apparatus and storage medium

Abstract

The application provides an image recognition method, device and equipment and a storage medium, and can be applied to the field of remote sensing target detection and recognition. The method comprises the following steps: performing feature extraction on an optical image and a synthetic aperture radar image representing a target ground area respectively to obtain optical semantic features and synthetic aperture semantic features, wherein the optical semantic features and the synthetic aperture semantic features respectively represent object categories of objects to be recognized in the optical image and the synthetic aperture radar image; processing the optical semantic features and the synthetic aperture semantic features based on an attention mechanism to obtain fused semantic features; and performing object recognition on the target ground area based on an image recognition model according to the fused semantic features to obtain an object recognition result.

Description

Technical Field

[0001] This invention relates to the field of remote sensing target detection and recognition, and more specifically to an image recognition method, apparatus, device, and storage medium. Background Technology

[0002] With the continuous development of remote sensing satellite constellation systems and on-board computing platforms, space information acquisition is gradually evolving towards multi-source and multi-modal approaches. However, existing multimodal remote sensing image recognition suffers from problems such as strong modal interference and poor information fusion, resulting in low image recognition accuracy. Summary of the Invention

[0003] In view of the above problems, the present invention provides an image recognition method, apparatus, device and storage medium.

[0004] According to a first aspect of the present invention, an image recognition method is provided, comprising: extracting features from an optical image and a synthetic aperture radar image representing a target surface region to obtain optical semantic features and synthetic aperture radar semantic features, wherein the optical semantic features and synthetic aperture radar semantic features represent the object categories of the objects to be identified in the optical image and the synthetic aperture radar image, respectively; processing the optical semantic features and synthetic aperture radar semantic features based on an attention mechanism to obtain fused semantic features; and performing object recognition on the target surface region based on an image recognition model according to the fused semantic features to obtain an object recognition result.

[0005] According to an embodiment of the present invention, optical semantic features and synthetic aperture semantic features are processed based on an attention mechanism to obtain fused semantic features, including: processing optical semantic features and synthetic aperture semantic features using a gating network to obtain attention weights; and using the attention weights to perform weighted fusion of optical semantic features and synthetic aperture semantic features to obtain fused semantic features.

[0006] According to an embodiment of the present invention, an attention weight is obtained by processing optical semantic features and synthetic aperture semantic features using a gating network, including: fusing optical semantic features and synthetic aperture semantic features to obtain intermediate fused features; and processing the intermediate fused features using an activation function to obtain attention weights.

[0007] According to an embodiment of the present invention, the image recognition model is trained as follows: Feature extraction is performed on optical sample images and synthetic aperture radar (SAR) sample images representing the surface region of the sample to obtain sample optical features and sample SAR features. The sample optical features include sample optical semantic features and sample optical modal features, and the sample SAR features include sample SAR semantic features and sample SAR modal features. The sample optical modal features and sample SAR modal features represent the image styles in the optical sample images and SAR sample images, respectively. The sample optical semantic features and sample SAR semantic features are processed based on an attention mechanism to obtain sample fusion semantic features. The sample optical modal features, sample SAR modal features, sample fusion semantic features, and sample image labels are processed based on a multivariate loss function to obtain a multivariate loss value. Based on the multivariate loss value, an initial image classification model is trained to obtain a trained image recognition model.

[0008] According to an embodiment of the present invention, the multivariate loss function is determined based on a triplet loss term, an orthogonal constraint loss term, an optical sample image loss term, a synthetic aperture radar sample image loss term, and a sample fusion semantic feature loss term. The triplet loss term is used to ensure the boundary between the sample optical modal features and the sample synthetic aperture modal features, and the orthogonal constraint loss term is used to reduce the linear correlation between the sample optical modal features and the sample synthetic aperture modal features.

[0009] According to an embodiment of the present invention, the multivariate loss value is determined based on the triplet loss value of the triplet loss term, the orthogonal constraint value of the orthogonal constraint loss term, the optical sample image loss value of the optical sample image loss term, the synthetic aperture radar sample image loss value of the synthetic aperture radar sample image loss term, and the fusion loss value of the sample fusion semantic feature loss term. The triplet loss value, based on the triplet loss term, represents the difference between the Euclidean distance between the anchor sample image modal features and the positive sample image modal features, and the Euclidean distance between the positive sample image modal features and the negative sample image modal features. The anchor sample image modal features and the positive sample image modal features are determined based on sample images of the same category, while the anchor sample image modal features and the negative sample image modal features are determined based on sample images of different categories. The categories of the sample images include optical image categories and synthetic aperture radar image categories.

[0010] According to an embodiment of the present invention, the orthogonal constraint value is obtained by orthogonalizing the sample optical modal features and the sample synthetic aperture modal features; the optical sample image loss value is determined based on the optical sample image loss term, according to the image classification results and sample image labels of the optical sample images; the synthetic aperture radar sample image loss value is determined based on the synthetic aperture radar sample image loss term, according to the image classification results and sample image labels of the synthetic aperture radar sample images; and the fusion loss value is determined based on the sample fusion semantic feature loss term, according to the image classification results and sample image labels of the sample fusion semantic features.

[0011] A second aspect of the present invention provides an image recognition device, comprising: a feature extraction module for extracting features from an optical image and a synthetic aperture radar image representing a target surface area, respectively, to obtain optical semantic features and synthetic aperture semantic features, wherein the optical semantic features and synthetic aperture semantic features respectively represent the object categories of objects to be identified in the optical image and the synthetic aperture radar image; a fusion module for processing the optical semantic features and synthetic aperture semantic features based on an attention mechanism to obtain fused semantic features; and an object recognition module for performing object recognition on the target surface area based on an image recognition model and according to the fused semantic features to obtain object recognition results.

[0012] A third aspect of the present invention provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0013] A fourth aspect of the present invention also provides a computer-readable storage medium storing a computer program or instructions which, when executed by a processor, implement the steps of the above-described method.

[0014] According to embodiments of the present invention, by extracting features from optical images and synthetic aperture radar images representing the target surface area, optical semantic features and synthetic aperture radar semantic features can be obtained. The optical semantic features and synthetic aperture radar semantic features represent the object categories of the objects to be identified in the optical images and synthetic aperture radar images, respectively. By processing the optical semantic features and synthetic aperture radar semantic features based on an attention mechanism, fused semantic features can be obtained, which align and strengthen key semantic dimensions in a shared space, thereby improving the consistency and discriminativeness of the fused semantics. Based on an image recognition model, object recognition is performed on the target surface area according to the fused semantic features, and object recognition results are obtained. This solves the problems of strong modal interference and insufficient fusion effect due to multiple modalities, and improves the accuracy of object recognition. Attached Figure Description

[0015] The above-mentioned contents, as well as other objects, features and advantages of the present invention, will become clearer from the following description of embodiments of the present invention with reference to the accompanying drawings.

[0016] Figure 1 The illustration shows an application scenario of the image recognition method and image recognition device according to an embodiment of the present invention.

[0017] Figure 2 A flowchart of an image recognition method according to an embodiment of the present invention is shown.

[0018] Figure 3 A schematic diagram of modal signal decoupling according to an embodiment of the present invention is shown.

[0019] Figure 4 A flowchart illustrating a training method for an image recognition model according to an embodiment of the present disclosure is shown.

[0020] Figure 5 A diagram of a multi-task learning framework according to an embodiment of the present invention is shown.

[0021] Figure 6 A structural block diagram of an image recognition device according to an embodiment of the present invention is shown.

[0022] Figure 7 A block diagram of an electronic device suitable for implementing an image recognition method according to an embodiment of the present invention is shown. Detailed Implementation

[0023] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the invention. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the invention for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0024] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0025] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0026] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0027] Most existing multimodal remote sensing image recognition methods rely on feature-level or decision-level fusion strategies. Feature-level fusion directly concatenates or combines feature vectors extracted from different modalities before feeding them into the classifier, which easily introduces redundant interference between modalities. Decision-level fusion classifies each modality separately and then fuses the prediction results, resulting in limited information integration capabilities. Furthermore, with the introduction of deep neural networks, some deep fusion frameworks for multimodal images have emerged, but these models are often large in scale and have numerous parameters, requiring powerful computing resources for both training and inference, making them difficult to deploy on resource-constrained satellite processing platforms. Simultaneously, existing methods generally lack mechanisms for modeling differences in modal characteristics, leading to difficulties in accurately aligning key semantic features between modalities, and the fusion results failing to reflect the true target attributes.

[0028] With the continuous development of remote sensing satellite constellation systems and onboard computing platforms, spatial information acquisition is gradually evolving towards multi-source, multi-modal, and high-time-efficiency approaches. Multi-satellite collaborative observation has become an important means to improve the accuracy of remote sensing interpretation, especially in tasks such as monitoring ships on the sea surface and identifying anomalies in marine areas. Different types of sensors, such as optical, synthetic aperture radar (SAR), and infrared sensors, are widely used to perform near-temporal or synchronous imaging of the same target area to acquire multi-modal remote sensing image data.

[0029] Optical images offer advantages such as high resolution and good visibility, facilitating the extraction of visual features like target shape and color. However, they are susceptible to interference from factors like cloud cover, lighting variations, and camouflage. SAR images, on the other hand, can stably image under all weather and time conditions, exhibiting strong anti-interference capabilities and effectively capturing target geometric contours and strong scattering features. However, they suffer from drawbacks such as high image noise, weak texture features, and relatively low resolution. Therefore, integrating and fusing optical and SAR images has become an important direction for improving the accuracy of remote sensing target classification.

[0030] In view of this, the present invention provides an image recognition method, an image recognition device, and an apparatus. The method includes: extracting features from an optical image and a synthetic aperture radar image representing a target surface region to obtain optical semantic features and synthetic aperture radar semantic features, whereby the optical semantic features and synthetic aperture radar semantic features represent the object categories of the objects to be identified in the optical image and the synthetic aperture radar image, respectively; processing the optical semantic features and synthetic aperture radar semantic features based on an attention mechanism to obtain fused semantic features; and performing object recognition on the target surface region based on an image recognition model and the fused semantic features to obtain an object recognition result.

[0031] It should be noted that the image recognition method and image recognition device provided by the present invention can be used in the field of remote sensing target detection and recognition, and can also be used in any field other than remote sensing target detection and recognition, such as the field of multimodal image fusion and on-board intelligent processing technology. Therefore, the application field of the image recognition method and image recognition device provided by the present invention is not limited.

[0032] Figure 1 The illustration shows an application scenario diagram of the image recognition method and image recognition device according to an embodiment of the present invention.

[0033] like Figure 1 As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0034] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0035] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0036] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0037] It should be noted that the image recognition method provided in the embodiments of the present invention can generally be executed by server 105. Correspondingly, the image recognition device provided in the embodiments of the present invention can generally be located in server 105. The image recognition method provided in the embodiments of the present invention can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the image recognition device provided in the embodiments of the present invention can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0038] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0039] Figure 2 A flowchart of an image recognition method according to an embodiment of the present invention is shown.

[0040] like Figure 2 As shown, the image recognition method of this embodiment includes operations S210 to S230, and the image recognition method can be executed by an electronic device.

[0041] In operation S210, features are extracted from the optical image and synthetic aperture radar image representing the target surface area to obtain optical semantic features and synthetic aperture radar semantic features.

[0042] In operation S220, optical semantic features and synthetic aperture semantic features are processed based on an attention mechanism to obtain fused semantic features.

[0043] In operation S230, based on the image recognition model, object recognition is performed on the target surface area according to the fused semantic features to obtain the object recognition result.

[0044] In a remote sensing constellation composed of multiple satellites, different satellites can carry heterogeneous payloads, including optical and SAR (Synthetic Aperture Radar) payloads, to simultaneously or near-temporally image the target surface area, acquiring multimodal image data. Each satellite platform performs preliminary image preprocessing and target detection on-orbit, obtaining standardized target slice images. Each standardized target slice image includes a unique object to be identified. The standardized target slice images include both optical and synthetic aperture radar (SAR) images. Each image is transmitted via inter-satellite communication links to a fusion satellite platform with fusion inference capabilities, where object identification is performed using the method described in this invention.

[0045] Feature extraction is performed on optical images (Red Green Blue, RGB) and synthetic aperture radar images representing the target surface area. Through a modal signal decoupling mechanism, the image features of each modality can be divided into semantic information features and modal information features.

[0046] Feature extraction is performed on optical images and synthetic aperture radar (SAR) images representing the target surface region to obtain optical semantic features and SAR semantic features. These features represent the object categories of the objects to be identified in the optical and SAR images, respectively.

[0047] After extracting and decoupling RGB and SAR images through a residual network, optical semantic features and synthetic aperture semantic features can be obtained, respectively. In the feature extraction stage, semantically related information is separated from modality-specific information, enabling the model to model semantic features and modality style separately, thereby improving the expressive power and classification performance of multimodal fusion.

[0048] By processing optical semantic features and synthetic aperture semantic features using an attention mechanism, fused semantic features can be obtained. These fused semantic features retain the advantages of multimodal semantic alignment while achieving stronger class discrimination capabilities under independent supervision.

[0049] Based on image recognition models, objects are identified in target surface areas by fusing semantic features, resulting in object recognition results, such as ship type classification tasks.

[0050] The method in this invention is applicable to remote sensing constellation systems with inter-satellite communication capabilities. It can complete high-precision fusion recognition of multimodal target images under conditions of limited on-board computing resources and limited communication bandwidth. It is especially suitable for recognition tasks of structurally stable ground objects such as ship targets and vehicle targets.

[0051] Figure 3 A schematic diagram of modal signal decoupling according to an embodiment of the present invention is shown.

[0052] like Figure 3As shown, the ResNet50 network can be used to process optical images or synthetic aperture radar images to obtain features with semantic signals and features with modal signals. Taking optical images as an example, it can be demonstrated that the 2048-dimensional feature vector is divided into optical semantic features and optical modal features to effectively avoid intermodal interference.

[0053] According to embodiments of the present invention, by extracting features from optical images and synthetic aperture radar images representing the target surface area, optical semantic features and synthetic aperture radar semantic features can be obtained. The optical semantic features and synthetic aperture radar semantic features represent the object categories of the objects to be identified in the optical images and synthetic aperture radar images, respectively. By processing the optical semantic features and synthetic aperture radar semantic features based on an attention mechanism, fused semantic features can be obtained, which align and strengthen key semantic dimensions in a shared space, thereby improving the consistency and discriminativeness of the fused semantics. Based on an image recognition model, object recognition is performed on the target surface area according to the fused semantic features, and object recognition results are obtained. This solves the problems of strong modal interference and insufficient fusion effect due to multiple modalities, and improves the accuracy of object recognition.

[0054] According to an embodiment of the present invention, optical semantic features and synthetic aperture semantic features are processed based on an attention mechanism to obtain fused semantic features, including: processing optical semantic features and synthetic aperture semantic features using a gating network to obtain attention weights; and using the attention weights to perform weighted fusion of optical semantic features and synthetic aperture semantic features to obtain fused semantic features.

[0055] To fuse optical semantic features and synthetic aperture semantic features, a gating network can be used to process the optical and synthetic aperture semantic features, obtaining attention weights to control the weighting ratio of the two features. These attention weights can then be used to weight and fuse the optical and synthetic aperture semantic features, resulting in fused semantic features. The fused semantic features are represented as follows:

[0056] (1);

[0057] in, To integrate semantic features, Indicates attention weights, Representing optical semantic features, This represents the semantic features of the synthetic aperture.

[0058] According to embodiments of the present invention, the semantic information carried by optical semantic features and synthetic aperture semantic features is significantly complementary. By utilizing attention weights, the two semantic features of different modalities are fused, aligning them in a shared space and strengthening the semantic dimension, thereby enhancing the model's ability to understand and classify multimodal inputs.

[0059] According to an embodiment of the present invention, an attention weight is obtained by processing optical semantic features and synthetic aperture semantic features using a gating network, including: fusing optical semantic features and synthetic aperture semantic features to obtain intermediate fused features; and processing the intermediate fused features using an activation function to obtain attention weights.

[0060] Optical semantic features and synthetic aperture semantic features can be fused to obtain intermediate fused features. The attention weights can be obtained by processing the intermediate fused features using an activation function. Specifically, the semantic features of the two modalities can be concatenated and fed into a two-layer fully connected network. First, a nonlinear transformation is performed, and then the attention weights are output through the sigmoid activation function, as shown in formula (2).

[0061] (2);

[0062] in, and For learnable parameters, This represents the Sigmoid function, whose output value is naturally limited to the interval [0, 1]. This indicates a modified linear unit.

[0063] By using attention weights, the relative contributions of the two modalities in the current task can be adaptively determined based on the feature state of the current image, ensuring that information fusion is targeted and scene-adaptive.

[0064] Figure 4 A flowchart illustrating a training method for an image recognition model according to an embodiment of the present disclosure is shown.

[0065] like Figure 4 As shown, the training method of the image recognition model in this embodiment includes operations S410 to S440, and the image recognition method can be executed by an electronic device.

[0066] In operation S410, feature extraction is performed on the optical sample image and synthetic aperture radar sample image representing the sample surface area to obtain sample optical features and sample synthetic aperture features. The sample optical features include sample optical semantic features and sample optical modal features, and the sample synthetic aperture features include sample synthetic aperture semantic features and sample synthetic aperture modal features.

[0067] In operation S420, the sample optical semantic features and sample synthetic aperture semantic features are processed based on the attention mechanism to obtain the sample fusion semantic features.

[0068] In operation S430, the sample optical modal features, sample synthetic aperture modal features, sample fusion semantic features, and sample image labels are processed based on the multivariate loss function to obtain the multivariate loss value.

[0069] In operation of S440, an initial image classification model is trained based on multivariate loss values ​​to obtain a trained image recognition model.

[0070] The sample optical features can include sample optical semantic features and sample optical modal features. The sample synthetic aperture features can include sample synthetic aperture semantic features and sample synthetic aperture modal features. The sample optical modal features and sample synthetic aperture modal features can respectively represent the image style in the optical sample image and the synthetic aperture radar sample image.

[0071] Modal information of a SAR image refers to information determined by the SAR imaging mechanism, and may specifically include band information, scattering mechanism, imaging geometry, or phase information. Modal information of an RGB image may include imaging band, three-channel sampling, color or texture information, etc., but is not limited thereto. The embodiments of this disclosure do not limit the modal information of SAR images or RGB images.

[0072] Feature extraction is performed on optical sample images and synthetic aperture radar sample images that characterize the surface area of ​​the sample to obtain sample optical features and sample synthetic aperture features. Specifically, the input image is a remote sensing image of a certain mode, which may be an optical image (RGB image) or a SAR image, and is only a single mode input.

[0073] For example, if the input image is an optical image, global features are first extracted through a pre-trained residual backbone network to obtain a 2048-dimensional high-dimensional feature vector, as shown in formula (3).

[0074] (3);

[0075] in, The optical semantic features of an optical sample image are high-level abstract features that are closely related to the target category and have consistency across different modalities. The optical modal features of the optical sample image represent the sample optical modal characteristics. Represents an optical image.

[0076] By processing sample optical semantic features and sample synthetic aperture semantic features based on the attention mechanism, sample fusion semantic features can be obtained.

[0077] By processing sample optical modal features, sample synthetic aperture modal features, sample fusion semantic features, and sample image labels using a multivariate loss function, a multivariate loss value can be obtained. Based on the multivariate loss value, an initial image classification model can be trained to obtain a trained image recognition model.

[0078] According to embodiments of the present invention, all tasks are optimized in the same backpropagation process, which can determine the consistency of the image recognition model in understanding multimodal information. In addition, the image recognition model training method can enhance the classification ability of each modality and its fused semantic features. It has a clear structure, few parameters, and low computational complexity, and is suitable for deployment and operation in constellation remote sensing systems with limited on-orbit computing resources. It can achieve high-precision multimodal target recognition while ensuring real-time response.

[0079] According to an embodiment of the present invention, the multivariate loss function is determined based on a triplet loss term, an orthogonal constraint loss term, an optical sample image loss term, a synthetic aperture radar sample image loss term, and a sample fusion semantic feature loss term. The triplet loss term is used to ensure the boundary between the sample optical modal features and the sample synthetic aperture modal features, and the orthogonal constraint loss term is used to reduce the linear correlation between the sample optical modal features and the sample synthetic aperture modal features.

[0080] In multimodal remote sensing image classification tasks, different modalities of data (such as RGB and SAR) exhibit significant differences in feature distribution due to variations in imaging principles, resolution, and signal-to-noise ratio. These modalities have distinct "styles," and direct fusion may lead to semantic information contamination, affecting the final classification performance.

[0081] A triplet loss term can be used to ensure the boundary between sample optical modal features and sample synthetic aperture modal features. This not only improves the discrimination boundary between modalities but also enhances the model's robustness to modal disturbances. In actual training, a batch size of 16 can be used to construct a combined structure containing 8 categories, each category containing both RGB and SAR modal samples, ensuring that the triplet samples are balanced between modalities and within categories.

[0082] To further enhance the independence between semantic features and modal style features and improve the decoupling effect of modal signals, an orthogonal constraint mechanism between semantic features and modal style features is introduced. This mechanism aims to suppress the interference of modal redundancy information on semantic modeling and enhance the structurality and discriminativeness of feature representation.

[0083] The orthogonal constraint loss term is used to reduce the linear correlation between sample optical modal features and sample optical semantic features, as well as between sample synthetic aperture modal features and sample synthetic aperture semantic features.

[0084] According to embodiments of the present invention, the boundary clarity between modes can be enhanced by using a triplet loss term, thereby improving the robustness of the model to modal interference. Furthermore, the use of an orthogonal constraint loss term can further separate semantic information from modal style, effectively avoiding cross-interference.

[0085] According to an embodiment of the present invention, the multivariate loss value is determined based on the triplet loss value of the triplet loss term, the orthogonal constraint value of the orthogonal constraint loss term, the optical sample image loss value of the optical sample image loss term, the synthetic aperture radar sample image loss value of the synthetic aperture radar sample image loss term, and the fusion loss value of the sample fusion semantic feature loss term. The triplet loss value, based on the triplet loss term, represents the difference between the Euclidean distance between the anchor sample image modal features and the positive sample image modal features, and the Euclidean distance between the positive sample image modal features and the negative sample image modal features. The anchor sample image modal features and the positive sample image modal features are determined based on sample images of the same category, while the anchor sample image modal features and the negative sample image modal features are determined based on sample images of different categories. The categories of the sample images include optical image categories and synthetic aperture radar image categories.

[0086] For the sample optical modal features and sample synthetic aperture modal features extracted and decoupled by the residual network, a triplet sample set can be constructed in each training batch based on the triplet loss term, where: This represents the modal features of the anchor sample image. This indicates that the modal features of the positive sample image belong to the same modality as the modal features of the anchor sample image (e.g., both are RGB or both are SAR). The negative sample image modal features belong to a different modality than the anchor sample image modal features (i.e., one is RGB and the other is SAR). The objective is to minimize the distance between the anchor sample image modal features and the positive sample image modal features, and make it less than at least one marginal value of the distance between the anchor sample image modal features and the negative sample image modal features. This brings the modal style features of samples with the same modality closer together, while widening the feature representations of samples with different modalities. The loss function is defined as follows:

[0087] (4);

[0088] in, This represents the ternary loss value. Represents Euclidean distance (or other optional distance metrics, such as cosine distance). The hyperparameter, called the margin, is used to constrain the minimum distance difference between positive and negative samples. When the loss is zero, it means that the relationship between the samples has met the modal structure separation requirements for classification.

[0089] The modal features of anchor sample images and positive sample images are determined based on sample images of the same category, while the modal features of anchor sample images and negative sample images are determined based on sample images of different categories. The categories of sample images can include optical image categories and synthetic aperture radar image categories.

[0090] To optimize the structural distribution of modal style features and enhance the independence between semantic features and modal style features, triplet loss and orthogonal constraint loss are introduced. Finally, by integrating all subtasks, a multivariate loss value is determined based on the triplet loss value of the triplet loss term, the orthogonal constraint value of the orthogonal constraint loss term, the optical sample image loss value of the optical sample image loss term, the synthetic aperture radar sample image loss value of the synthetic aperture radar sample image loss term, and the fusion loss value of the sample fusion semantic feature loss term. The multivariate loss function is constructed as follows:

[0091] (5);

[0092] in, ~ The weighting coefficients for each loss term can be tuned through experimental setup or cross-validation. Represents the multivariate loss value. Represents the optical sample image loss value. This represents the loss value of the synthetic aperture radar sample image. This represents the fusion loss value. This represents the triplet loss value. This represents the orthogonal constraint value.

[0093] According to embodiments of the present invention, a triplet loss value can be obtained based on the triplet loss term, which can further optimize the modal feature information of each sample, ensuring that modal styles maintain clear boundaries in representation. Based on orthogonal constraint values, the linear correlation between semantic features and modal features in each modality can be minimized, improving the modality separation effect. While maintaining a lightweight, efficient, and deployable model structure, the object recognition accuracy is improved, making it suitable for remote sensing intelligent recognition tasks that require both response time and accuracy, such as ship type classification tasks, including ship target recognition.

[0094] According to an embodiment of the present invention, the orthogonal constraint value is obtained based on the sample optical modal features and the sample synthetic aperture modal features; the optical sample image loss value is determined based on the optical sample image loss term, according to the image classification results and sample image labels of the optical sample images; the synthetic aperture radar sample image loss value is determined based on the synthetic aperture radar sample image loss term, according to the image classification results and sample image labels of the synthetic aperture radar sample images; and the fusion loss value is determined based on the sample fusion semantic feature loss term, according to the image classification results and sample image labels of the sample fusion semantic features.

[0095] During the training phase, classification task supervision is applied to optical modalities (RGB), SAR modalities, and their fused features, respectively. Simultaneously, modal style optimization and feature orthogonality constraints are introduced to collaboratively enhance feature representation capabilities from multiple perspectives. Let the semantic features obtained after modal decoupling (sample optical semantic features in optical sample images and sample synthetic aperture semantic features in SAR sample images) be... Modal style features (sample optical modal features in optical sample images, and sample synthetic aperture modal features in SAR sample images) are The two can be kept orthogonal in the feature space by minimizing their inner product squared terms. The specific orthogonality constraint loss function is defined as follows:

[0096] (6);

[0097] in, This represents the orthogonal constraint loss value. Characterization transpose, Characterizes orthogonal functions.

[0098] Geometrically, this loss function encourages and Orthogonal orientations in high-dimensional space minimize linear correlations between modalities, improving modality separation. This constraint can be directly embedded into the end-to-end training of the overall network. Semantic features can include sample optical semantic features and sample synthetic aperture semantic features. Modal style features can include sample optical modal features and sample synthetic aperture modal features.

[0099] The optical sample image loss value is determined based on the optical sample image loss term, according to the image classification results and sample image labels of the optical sample images, as shown in formula (7); the synthetic aperture radar sample image loss value is determined based on the synthetic aperture radar sample image loss term, according to the image classification results and sample image labels of the synthetic aperture radar sample images, as shown in formula (8); the fusion loss value is determined based on the sample fusion semantic feature loss term, according to the image classification results and sample image labels of the sample fusion semantic features, as shown in formula (9).

[0100] Specifically, let and These are the semantic features extracted from RGB and SAR images after modal signal decoupling, respectively. To fuse semantic features into samples, an independent classification branch is constructed for each of these three feature classes, outputting the predicted class probability. The corresponding cross-entropy loss functions are as follows:

[0101] (7);

[0102] (8);

[0103] (9);

[0104] in, For real category labels, Represents the optical sample image loss value. This represents the loss value of the synthetic aperture radar sample image. This represents the fusion loss value.

[0105] By independently supervising optical sample images, synthetic aperture radar sample images, and fused semantic features, each type of semantic feature possesses a clear class discrimination capability, avoiding information dilution or shift during the fusion process. By introducing a semantic and modal feature separation structure, combined with an attention mechanism and a triplet loss function, the adaptability and accuracy of the multimodal classification model in the space environment are improved.

[0106] Figure 5 A diagram of a multi-task learning framework according to an embodiment of the present invention is shown.

[0107] like Figure 5As shown, feature extraction can be performed on optical sample images to obtain sample optical semantic features and sample optical modal features. Similarly, feature extraction can be performed on synthetic aperture radar sample images to obtain sample synthetic aperture semantic features and sample synthetic aperture modal features. Based on the sample optical semantic features and sample optical modal features, an optical orthogonal loss value can be obtained. Based on the sample synthetic aperture semantic features and sample synthetic aperture modal features, a synthetic aperture orthogonal loss value can be obtained. The optical orthogonal loss value and the synthetic aperture orthogonal loss value can be fused to obtain an orthogonal constraint value.

[0108] The triplet loss term is used to process the optical modal features and synthetic aperture radar (SAP) modal features of the samples, yielding the triplet loss value. Based on the optical sample image loss term, the optical sample image loss value is determined according to the image classification results and sample image labels. Based on the SAP sample image loss term, the SAP sample image loss value is determined according to the image classification results and sample image labels. Based on the sample fusion semantic feature loss term, the fusion loss value is determined according to the image classification results and sample image labels of the sample fusion semantic features.

[0109] The orthogonal constraint value, triplet loss value, optical sample image loss value, synthetic aperture radar sample image loss value, and fusion loss value can be fused to obtain a multivariate loss value. The initial image classification model is trained using the multivariate loss value to obtain a trained image recognition model.

[0110] Figure 6 A structural block diagram of an image recognition device according to an embodiment of the present invention is shown.

[0111] like Figure 6 As shown, the image recognition device in this embodiment includes a feature extraction module 610, a fusion module 620, and an object recognition module 630.

[0112] The feature extraction module 610 is used to extract features from the optical image and the synthetic aperture radar image representing the target surface area, respectively, to obtain optical semantic features and synthetic aperture semantic features. The optical semantic features and synthetic aperture semantic features represent the object categories of the objects to be identified in the optical image and the synthetic aperture radar image, respectively.

[0113] The fusion module 620 is used to process optical semantic features and synthetic aperture semantic features based on an attention mechanism to obtain fused semantic features.

[0114] The object recognition module 630 is used to identify objects in the target surface area based on the image recognition model and the fused semantic features, and to obtain the object recognition result.

[0115] According to embodiments of the present invention, by extracting features from optical images and synthetic aperture radar images representing the target surface area, optical semantic features and synthetic aperture radar semantic features can be obtained. The optical semantic features and synthetic aperture radar semantic features represent the object categories of the objects to be identified in the optical images and synthetic aperture radar images, respectively. By processing the optical semantic features and synthetic aperture radar semantic features based on an attention mechanism, fused semantic features can be obtained, which align and strengthen key semantic dimensions in a shared space, thereby improving the consistency and discriminativeness of the fused semantics. Based on an image recognition model, object recognition is performed on the target surface area according to the fused semantic features, and object recognition results are obtained. This solves the problems of strong modal interference and insufficient fusion effect due to multiple modalities, and improves the accuracy of object recognition.

[0116] According to an embodiment of the present invention, the fusion module 620 includes: an attention weighting unit and a weighted fusion unit.

[0117] The attention weight unit is used to process optical semantic features and synthetic aperture semantic features using a gating network to obtain attention weights.

[0118] The weighted fusion unit is used to perform weighted fusion of optical semantic features and synthetic aperture semantic features using attention weights to obtain fused semantic features.

[0119] According to an embodiment of the present invention, the attention weighting unit includes: an intermediate fusion subunit and an attention weighting subunit.

[0120] The intermediate fusion subunit is used to fuse optical semantic features and synthetic aperture semantic features to obtain intermediate fusion features.

[0121] The attention weight subunit is used to process intermediate fused features using an activation function to obtain attention weights.

[0122] According to an embodiment of the present invention, the object recognition module 630 includes: a sample feature extraction unit, a sample fusion unit, a multivariate loss value unit, and a training unit.

[0123] The sample feature extraction unit is used to extract features from optical sample images and synthetic aperture radar sample images that characterize the sample surface area, respectively, to obtain sample optical features and sample synthetic aperture features. The sample optical features include sample optical semantic features and sample optical modal features, and the sample synthetic aperture features include sample synthetic aperture semantic features and sample synthetic aperture modal features. The sample optical modal features and sample synthetic aperture modal features represent the image style in the optical sample image and the synthetic aperture radar sample image, respectively.

[0124] The sample fusion unit is used to process the optical semantic features and synthetic aperture semantic features of the samples based on the attention mechanism to obtain the sample fused semantic features.

[0125] The multivariate loss value unit is used to process sample optical modal features, sample synthetic aperture modal features, sample fusion semantic features, and sample image labels based on the multivariate loss function to obtain multivariate loss values.

[0126] The training unit is used to train an initial image classification model based on multivariate loss values ​​to obtain a trained image recognition model.

[0127] According to an embodiment of the present invention, the multivariate loss value unit includes a multivariate loss function determined based on a triplet loss term, an orthogonal constraint loss term, an optical sample image loss term, a synthetic aperture radar sample image loss term, and a sample fusion semantic feature loss term. The triplet loss term is used to ensure the boundary between the sample optical modal features and the sample synthetic aperture modal features, and the orthogonal constraint loss term is used to reduce the linear correlation between the sample optical modal features and the sample synthetic aperture modal features.

[0128] According to an embodiment of the present invention, the multivariate loss value unit further includes a multivariate loss value determined based on the triplet loss value of the triplet loss term, the orthogonal constraint value of the orthogonal constraint loss term, the optical sample image loss value of the optical sample image loss term, the synthetic aperture radar sample image loss value of the synthetic aperture radar sample image loss term, and the fusion loss value of the sample fusion semantic feature loss term.

[0129] According to an embodiment of the present invention, the multivariate loss value unit further includes a triplet loss value, which is based on a triplet loss term and represents the difference between the Euclidean distance between the anchor sample image modal features and the positive sample image modal features, and the Euclidean distance between the positive sample image modal features and the negative sample image modal features. The anchor sample image modal features and the positive sample image modal features are determined based on sample images of the same category, while the anchor sample image modal features and the negative sample image modal features are determined based on sample images of different categories. The categories of sample images include optical image categories and synthetic aperture radar image categories.

[0130] According to embodiments of the present invention, the multivariate loss value unit further includes an orthogonal constraint value obtained by orthogonalizing the sample optical modal features and the sample synthetic aperture modal features. The optical sample image loss value is determined based on the optical sample image loss term, according to the image classification results and sample image labels of the optical sample images. The synthetic aperture radar sample image loss value is determined based on the synthetic aperture radar sample image loss term, according to the image classification results and sample image labels of the synthetic aperture radar sample images. The fusion loss value is determined based on the sample fusion semantic feature loss term, according to the image classification results and sample image labels of the sample fusion semantic features.

[0131] According to embodiments of the present invention, any plurality of modules among the feature extraction module 610, fusion module 620, and object recognition module 630 may be combined into one module, or any one of these modules may be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules may be combined with at least a portion of the functionality of other modules and implemented in one module. According to embodiments of the present invention, at least one of the feature extraction module 610, fusion module 620, and object recognition module 630 may be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the feature extraction module 610, fusion module 620, and object recognition module 630 may be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0132] Figure 7 A block diagram of an electronic device suitable for implementing an image recognition method according to an embodiment of the present invention is shown.

[0133] like Figure 7 As shown, an electronic device 700 according to an embodiment of the present invention includes a processor 701, which can perform various appropriate actions and processes according to a program stored in ROM 702 (i.e., read-only memory) or a program loaded from storage portion 708 into RAM 703 (i.e., random access memory). The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0134] RAM 703 stores various programs and data required for the operation of electronic device 700. Processor 701, ROM 702, and RAM 703 are interconnected via bus 704. Processor 701 executes various operations of the method flow according to embodiments of the present invention by executing programs in ROM 702 and / or RAM 703. It should be noted that programs may also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 may also execute various operations of the method flow according to embodiments of the present invention by executing programs stored in one or more memories.

[0135] According to an embodiment of the present invention, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to a bus 704. The electronic device 700 may also include one or more of the following components connected to the input / output (I / O) interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output (I / O) interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0136] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0137] According to embodiments of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM 703), read-only memory (ROM 702), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of the present invention, the computer-readable storage medium may include ROM 702 and / or RAM 703 and / or one or more memories other than ROM 702 and RAM 703 described above.

[0138] Embodiments of the present invention also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of the present invention.

[0139] When the computer program is executed by the processor 701, it performs the functions defined in the system / apparatus of this invention. According to embodiments of the invention, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0140] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 709, and / or installed from a removable medium 711. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0141] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, it performs the functions defined in the system of this embodiment of the invention. According to embodiments of the invention, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0142] According to embodiments of the present invention, program code for executing the computer programs provided in the embodiments of the present invention can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0143] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0144] Those skilled in the art will understand that the features described in the various embodiments of the present invention can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention can be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or combinations fall within the scope of the present invention.

[0145] The embodiments of the present invention have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of the invention. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of the invention, and all such substitutions and modifications should fall within the scope of the invention.

Claims

1. An image recognition method characterized by, The method comprises: respectively performing feature extraction on an optical image and a synthetic aperture radar image representing a target ground surface area to obtain optical semantic features and synthetic aperture semantic features, the optical semantic features and the synthetic aperture semantic features representing object categories of objects to be identified in the optical image and the synthetic aperture radar image respectively; processing the optical semantic features and the synthetic aperture semantic features based on an attention mechanism to obtain fused semantic features; performing object identification on the target ground surface area based on an image recognition model according to the fused semantic features to obtain an object identification result, the image recognition model being obtained by training in the following manner: respectively performing feature extraction on an optical sample image and a synthetic aperture radar sample image representing a sample ground surface area to obtain sample optical features and sample synthetic aperture features, the sample optical features comprising sample optical semantic features and sample optical modal features, and the sample synthetic aperture features comprising sample synthetic aperture semantic features and sample synthetic aperture modal features, the sample optical modal features and the sample synthetic aperture modal features representing image styles in the optical sample image and the synthetic aperture radar sample image respectively; processing the sample optical semantic features and the sample synthetic aperture semantic features based on an attention mechanism to obtain sample fused semantic features; processing the sample optical modal features, the sample synthetic aperture modal features, the sample fused semantic features, and sample image labels based on a multi-element loss function to obtain a multi-element loss value, the multi-element loss function being determined according to a triple loss term, an orthogonal constraint loss term, an optical sample image loss term, a synthetic aperture radar sample image loss term, and a sample fused semantic feature loss term, the triple loss term being used to ensure a boundary between the sample optical modal features and the sample synthetic aperture modal features, and the orthogonal constraint loss term being used to reduce linear correlation between the sample optical modal features and the sample synthetic aperture modal features; training an initial image classification model based on the multi-element loss value to obtain the trained image recognition model.

2. The method of claim 1, wherein, The processing of the optical semantic features and the synthetic aperture semantic features based on the attention mechanism to obtain the fused semantic features comprises: processing the optical semantic features and the synthetic aperture semantic features by using a gating network to obtain attention weights; performing weighted fusion on the optical semantic features and the synthetic aperture semantic features by using the attention weights to obtain the fused semantic features.

3. The method of claim 2, wherein, The processing of the optical semantic features and the synthetic aperture semantic features by using the gating network to obtain the attention weights comprises: fusing the optical semantic features and the synthetic aperture semantic features to obtain intermediate fused features; processing the intermediate fused features by using an activation function to obtain the attention weights.

4. The method of claim 1, wherein, The multi-element loss value is determined according to a triplet loss value of the triplet loss term, an orthogonal constraint value of the orthogonal constraint loss term, an optical sample image loss value of the optical sample image loss term, a synthetic aperture radar sample image loss value of the synthetic aperture radar sample image loss term, and a fusion loss value of the sample fusion semantic feature loss term. The triplet loss value is based on the triplet loss term, represents a difference between a Euclidean distance between an anchor sample image modal feature and a positive sample image modal feature and a Euclidean distance between the positive sample image modal feature and a negative sample image modal feature, the anchor sample image modal feature and the positive sample image modal feature are determined according to sample images of the same category, the anchor sample image modal feature and the negative sample image modal feature are determined based on sample images of different categories, and the categories of the sample images include an optical image category and a synthetic aperture radar image category.

5. The method of claim 4, wherein, The orthogonal constraint value is obtained according to the sample optical modal feature and the sample synthetic aperture modal feature. The optical sample image loss value is determined according to an image classification result of the optical sample image and a sample image label based on the optical sample image loss term; the synthetic aperture radar sample image loss value is determined according to an image classification result of the synthetic aperture radar sample image and a sample image label based on the synthetic aperture radar sample image loss term; and the fusion loss value is determined according to an image classification result of the sample fusion semantic feature and a sample image label based on the sample fusion semantic feature loss term.

6. An image recognition apparatus characterized by comprising: The method comprises: a feature extraction module configured to perform feature extraction on an optical image and a synthetic aperture radar image representing a target ground surface area to obtain optical semantic features and synthetic aperture semantic features, the optical semantic features and the synthetic aperture semantic features representing object categories of objects to be identified in the optical image and the synthetic aperture radar image, respectively; a fusion module configured to process the optical semantic features and the synthetic aperture semantic features based on an attention mechanism to obtain fusion semantic features; an object identification module configured to perform object identification on the target ground surface area based on an image recognition model and the fusion semantic features to obtain an object identification result, the image recognition device being obtained according to the method of any one of claims 1-5.

7. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1-5.

8. A computer readable storage medium having stored thereon a computer program or instructions, characterized in that, The computer program or instructions are executed by the processor to implement the steps of the method according to any one of claims 1-5.

Images