Target detection and material identification method and system based on multispectral image

Abstract

The invention discloses a target detection and material identification method based on a multispectral image, and the method comprises the following steps: 1, building an imaging system, and obtaining an RGB image and a grayscale image through the imaging system; step 2, aligning the RGB image with a target image in the grayscale image; step 3, marking the accurate contour of the target in the RGB image by using a data marking platform; 4, taking the RGB image marked with the accurate contour and the gray level image as input, and obtaining target contour information and a target category through a pre-trained deep learning target detection network; step 5, identifying a target material type through a pre-trained material identification network by taking the grayscale image and the target contour information as input; step 6, fusing the deep learning target detection network and the material identification network to obtain a target detection and material identification integrated network; and step 7, using the trained target detection and material identification integrated network to complete identification of a target category and a material category at the same time.

Description

Technical Field

[0001] The present invention relates to the field of image processing and computer vision technology, and in particular to a method and system for target detection and material recognition based on multispectral images. Background Art

[0002] Existing object detection and material recognition methods typically rely on RGB images or single spectral band information for target location and classification. However, traditional methods are often affected by factors such as lighting variations, material differences, and image noise in complex environments, resulting in low detection and classification accuracy. Although multispectral imaging technology has been widely used in remote sensing and industrial inspection, existing multispectral object detection and material recognition methods mostly rely on simple spectral classification methods, which cannot fully exploit the spectral characteristics of different bands in the image, resulting in an inability to accurately distinguish complex materials.

[0003] In addition, existing target detection and material recognition technologies usually process these two tasks separately, failing to effectively combine the spatial location information and spectral characteristics of the target, resulting in low system efficiency and inability to fully utilize the spectral information in the target area for more accurate material classification.

[0004] Therefore, how to fuse the information of RGB images and multi-band grayscale images, adopt more accurate sampling and deep learning methods, and simultaneously achieve target detection and material recognition in the same system has become a major challenge in the current technical field. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this paper proposes a method and system for target detection and material recognition based on multispectral imaging. This system utilizes RGB images and multiple grayscale images at different wavelengths to efficiently detect and classify targets. The system can extract spatial information about targets in dynamic scenes and classify materials based on their spectral signatures.

[0006] In order to solve the above technical problems, the technical solution of the present invention is:

[0007] A method for target detection and material recognition based on multispectral images includes the following steps:

[0008] Step 1: Build an imaging system and use it to acquire RGB images and grayscale images;

[0009] Step 2: Align the target image in the RGB image and the grayscale image;

[0010] Step 3: Use the data annotation platform to mark the precise outline of the target in the RGB image;

[0011] Step 4: Use the RGB image and grayscale image annotated with precise contours as input to obtain target contour information and target category through a pre-trained deep learning target detection network. The pre-trained deep learning target detection network obtains spatial features and retains spectral features based on the spatial features.

[0012] Step 5: Use the grayscale image and target contour information as input to identify the target material type through a pre-trained material recognition network;

[0013] Step 6: Use the spatial feature encoder and the spectral feature encoder to map the spatial and spectral features to the same vector space, thereby fusing the deep learning target detection network and the material recognition network to obtain an integrated target detection and material recognition network. The RGB image, grayscale image, target outline information, target category, and target material category are used as training data to train the integrated target detection and material recognition network.

[0014] Step 7: Apply the trained object detection and material recognition network to simultaneously identify the object category and material category.

[0015] Preferably, seven types of imaging data are acquired through the imaging system, the imaging data being RGB images and grayscale images at wavelengths of 450 nm, 540 nm, 590 nm, 600 nm, 650 nm, and 690 nm;

[0016] Preferably, in step 2, image alignment is performed using a checkerboard image to ensure that the target positions in images of different wavelengths are consistent.

[0017] Assume there is an RGB image I RGB (x, y) and grayscale images of different wavelengths I λ (x, y) where λ represents the wavelength of the image. Image registration is performed using a checkerboard image. The registration process can be represented by an image transformation model: T(x, y) = (x′, y′) = s·(x, y) + (tx, ty) where (x, y) are the coordinates of the original image, (x′, y′) are the coordinates of the transformed target image, s is the scaling factor, and (tx, ty) is the translation parameter. This ensures that the target position in images of different wavelengths is consistent. For each set of data, we need to segment each frame of video into several images and align the images of different wavelengths using the above method.

[0018] Preferably, in step 3, the Label Studio data annotation platform is used to annotate the precise outline of the target in the RGB image. Since the grayscale image and the RGB image are aligned, the same set of spectra share a set of target outline information.

[0019] Preferably, in step 5, a material recognition network (MS-MatNet) is used to implement spectral material recognition. Based on grayscale images at six different wavelengths and the target's contour information, the network is trained by sampling multiple spectral points to identify the target's material type. During training, the material recognition network samples a set of data multiple times (typically 5 to 10 times). This effectively utilizes the spectral information of more points within the contour and improves the network's robustness. During prediction, only a single sampling is performed to speed up network processing.

[0020] As a preference, N sets of data are obtained through the imaging system and the data processing system, and each imaging of each set of data has a total of M n images, each set of data is 7×M i images, N sets of data in total images.

[0021] As a preferred method, the basic architecture of the material recognition network (MS-MatNet) takes as input the randomly sampled sample matrix Mat = 10×10×6 and the spatial context information Ploygon of the target. It passes through the spectral feature extraction module, the spatial-spectral fusion module and the material classifier in sequence to complete the material recognition of the target area using spectral information.

[0022] As a preferred method, the deep learning target detection network (MS-DetNet) takes 9 single-channel images as input = 9 × Img width ×Img height , the spatial features and band features are extracted by the feature extraction module and fused into the joint feature F fused The three-branch target detection module respectively predicts the target category, outer border and polygon contour points, thereby realizing target category recognition and fine contour segmentation.

[0023] The present invention also provides a target detection and material recognition system based on multispectral images, which includes an imaging system, a motion control system, a motion guide rail, a stage and a central processing unit loaded with the above-mentioned target detection and material recognition method based on multispectral images, and the stage can be movably mounted on the motion guide rail.

[0024] Preferably, the imaging system includes an RGB camera, a six-hole filter wheel, a grayscale camera and a spectrometer, and the spectrometer is used to split the incident light and capture the reflected light of the target within different wavelength ranges.

[0025] Preferably, the motion control system is used to control the stage on which the target is placed to move at a constant speed on a guide rail in front of the imaging system.

[0026] The present invention has the following characteristics and beneficial effects:

[0027] The above-mentioned technical solution is adopted, which uses the RGB camera to work in conjunction with a six-hole filter wheel and a spectrometer to efficiently collect multi-band images, adapt to dynamic scenes (such as uniform motion of the stage), and ensure data acquisition stability. It accurately controls the movement of the stage and supports continuous shooting of multi-spectral videos, which is suitable for real-time detection scenarios such as industrial assembly lines. Through video splitting, inter-frame alignment and batch labeling technology, it supports multi-spectral data acquisition and processing of dynamic targets, expanding the application scope of the system in scenarios such as real-time monitoring and industrial inspection. The system collects RGB images and grayscale images of six specific wavelengths of 450nm, 540nm, 590nm, 600nm, 650nm, and 690nm, covering the range of blue light to deep red light. It can capture the differences in reflective characteristics of different materials in specific bands (such as blue light enhances plastic / alloy contrast, and red light improves metal detection capabilities), significantly improving the accuracy of material classification in complex environments.

[0028] For imaging results, checkerboard images and affine transformation models (scaling and translation) are used to align images of different wavelengths, eliminating spatial offsets between multispectral images, ensuring target position consistency, and reducing errors in subsequent detection and classification. RGB image contours are then annotated and shared with the multispectral image to avoid duplicate annotations and improve data consistency. The full-stride averaging method automatically calculates inter-frame annotation displacements, efficiently generating batch annotation data and reducing labor costs and errors.

[0029] In the target detection network, multi-band spectral features and spatial information are integrated to enhance the weight distribution of key features and improve the robustness of target detection in complex backgrounds. Compared with traditional bounding boxes, it accurately divides the target contour, reduces background interference, and is suitable for the detection of irregularly shaped targets. When training the material recognition network, the target area is sampled multiple times (5 to 10 times) to fully capture the internal spectral distribution characteristics of the material and enhance the model's sensitivity to local differences. The spectral data of 100 sampling points in the target area are organized into a 10×10×6 matrix, and the convolutional layer is used to extract the spectral features of the target area. The self-attention mechanism is combined to capture the global distribution law and enhance the robustness of material classification. Combined with spatial context information such as target shape and position, the ability to distinguish similar materials (such as wood and plastic) is improved. The target detection and material recognition networks are integrated to share a multimodal encoder to reduce repeated calculations; the joint loss function (L=L det +αL mat )Optimize multi-task training, promote the mutual promotion of the two tasks, and improve the overall efficiency and accuracy of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0031] Figure 1 This is a schematic diagram of the integrated architecture of a multispectral image-based target detection and material recognition system according to an embodiment of the present invention;

[0032] Figure 2 for Figure 1 Schematic diagram of the imaging device;

[0033] Figure 3 Schematic diagram of the basic architecture of the material recognition network (MS-MatNet) in a method for target detection and material recognition based on multispectral images according to an embodiment of the present invention;

[0034] Figure 4 Schematic diagram of the infrastructure of a deep learning target detection network (MS-DetNet) in a method for target detection and material recognition based on multispectral images according to an embodiment of the present invention;

[0035] Figure 5 A schematic diagram of affine alignment feature point matching using an embodiment of the present invention;

[0036] Figure 6 A diagram showing batch labeling results using an embodiment of the present invention;

[0037] Figure 7 This figure shows the results of target detection using an embodiment of the present invention. DETAILED DESCRIPTION

[0038] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0039] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0040] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0041] Example 1

[0042] This embodiment provides a method for target detection and material recognition based on multispectral images, including the following steps:

[0043] Step 1: Build an imaging system and use it to obtain RGB images and grayscale images.

[0044] Specifically, seven types of imaging data are acquired: RGB images and grayscale images with wavelengths of 450 nm, 540 nm, 590 nm, 600 nm, 650 nm, and 690 nm.

[0045] 450nm (blue light): Used to obtain target details, especially on low-contrast materials to enhance image recognition. Blue light is very sensitive to the reflective properties of certain plastics and alloys.

[0046] 540nm (green light): The green wavelength has a more balanced reflective property, which can provide better distinguishability on many materials. It can effectively distinguish between wood and alloy materials.

[0047] 590nm (yellow light): It has good resolution for wood, especially in terms of reflection characteristics, which are significantly different from plastic.

[0048] 600nm (orange light): Usually used to enhance the detail contrast of materials, especially for identifying red and yellow materials (such as wooden objects).

[0049] 650nm (red light): Red light can effectively capture the reflective characteristics of alloy materials such as plastics and metals, and is a commonly used wavelength in alloy material identification.

[0050] 690nm (deep red light): This wavelength has a strong reflection on metals and can enhance material recognition capabilities in darker environments. It is suitable for detecting alloys and plastics.

[0051] Step 2: Align the target images in the RGB image and the grayscale image, and perform image registration through the checkerboard image to ensure that the target positions in the images of different wavelengths are consistent. In the data processing process, the target is first aligned to ensure that the target positions between the images of different wavelengths are consistent. Assume that there is an RGB image I RGB (x, y) and grayscale images of different wavelengths I λ (x, y), where λ represents the wavelength of the image. Image registration is performed using a checkerboard pattern. The registration process can be represented by an image transformation model: T(x, y) = (x′, y′) = s·(x, y) + (tx, ty), where (x, y) are the coordinates of the original image, (x′, y′) are the coordinates of the transformed target image, s is the scaling factor, and (tx, ty) is the translation parameter. This ensures that the target position is consistent across images with different wavelengths.

[0052] It can be understood that for each set of data, each frame of video needs to be divided into several pictures, and the above method is used to align images of different wavelengths.

[0053] In this embodiment, different target objects are imaged by the imaging system to clearly capture the object motion video of each channel: V i-RGB and V i-λ , i∈[1, N], where i is the number of the video group, and a total of N groups are shot.

[0054] Specifically, first split N groups of videos into images and get images, The first video after the i-th group of videos is split The number of images in channels,

[0055] Then align the image using IB RGB and IB λ To determine the transformation affine matrix for aligning each wavelength image to the RGB image, the specific process is as follows:

[0056] (1) Grayscale value normalization: Where G(x, y) is the original image and G′(x, y) is the normalized image;

[0057] (2) Initialize the affine parameters: p = [s, tx, ty] = [1, 0, 0], where s is the scaling factor, 1 means no scaling, tx and ty represent the translation in the x and y directions, and 0 means no displacement;

[0058] (3) Calculation error: Calculate the mean square error between the single-wavelength grayscale image and the RGB grayscale image: Where i is the wavelength of the current grayscale image;

[0059] (4) Update parameters: Use the Powell algorithm to update the parameters, iteratively update the MSE so that the mean square error approaches 0, and record the best performing parameters;

[0060] (5) Construct the transformation matrix: Perform affine transformation on the target grayscale image to obtain a grayscale image aligned with the RGB image. The transformation matrix information obtained by this system is:

[0061]

[0062]

[0063] (6) Screening and renaming: Ensure that the number of images in each channel of the same group is the same, the starting spatial position of the target is the same, and the ending spatial position of the target is the same, and rename the images of different channels in the same group starting from 1 to obtain images, I i is the number of images under any channel in the i-th group of images.

[0064] Alignment results, such as Figure 5 As shown, the points with the same features in the left and right parts are connected by the same line.

[0065] Step 3: Use the data annotation platform to mark the precise outline of the target in the RGB image.

[0066] Specifically, in this embodiment, key frames are selected: for each set of RGB images, several complete target images are selected at equal intervals as key frames and sequentially imported into the annotation platform. In this embodiment, it is assumed that the step size between key frames is 10, such as taking 150, 160, 170, 180, 190, 200, etc., a total of N frames as key frames.

[0067] Manual annotation: Use polygons to manually annotate the first frame of each group, then copy the annotation to other frames in the same group and manually align them to ensure that the annotation information of each frame is correct.

[0068] Calculate frame spacing: Export the annotated keyframe annotation data and use the full-stride averaging method to calculate the annotation movement distance between adjacent single frames. The calculation formula of the full-stride averaging method is: Where step is the step size. Using the full-step averaging method can effectively utilize all key frames and reduce errors.

[0069] Calculate the specific location of each frame label: Based on the gap and the position of the initial key frame in the original data, the label location information of all frames can be calculated. For frames where the target is not entirely in the field of view, the area intersection method can be used to obtain the area information within the field of view.

[0070] Batch annotation: Use the open API interface to update the calculated annotation information to the annotation platform to check whether the batch annotation results are accurate.

[0071] The marking results are as follows Figure 6 As shown in the example, an RGB image containing an M1A2 tank is labeled.

[0072] Step 4: Use the RGB image and grayscale image annotated with precise contours as input to obtain the target contour information and target category through a pre-trained deep learning target detection network.

[0073] Specifically, the deep learning target detection network (MS-DetNet) integrates a deep neural network of RGB and multi-band grayscale images. The network inputs seven images, uses an encoder to extract the spatial and spectral features of each band, and uses a polygon annotation auxiliary module: polygon annotation provides training supervision to ensure that the model can achieve high-precision detection in complex backgrounds, and realizes information fusion through a multi-head attention module, and finally outputs the precise polygon outline and category information of the target. The design of MS-DetNet is as follows: Figure 4 As shown, the details are as follows:

[0074] Input layer: The input includes one RGB image and six grayscale images at different wavelengths (450nm, 540nm, 590nm, 600nm, 650nm, and 690nm), for a total of seven images. Each image is processed through a separate convolutional encoder to extract spatial features and spectral information.

[0075] It can be understood that the RGB image is split into grayscale images of 3 channels, so the input layer is provided with 9 convolution encoders, and the 9 convolution encoders perform convolution processing on the grayscale images of 3 channels and 6 grayscale images in the RGB image respectively to extract spatial features, take points at the same position on each layer of spatial features, and then combine the points on each layer of spatial features to obtain spectral features.

[0076] The feature extraction module includes 9 spatial feature encoders and a spatial-spectral feature fusion module. The three channels of the RGB image and each band corresponding to the 6 grayscale images use their own spatial feature encoders to extract their own spatial features and output F R 、F G 、F B 、F 450 、F 540 、F 590 、F 600 、F 650 、F 690 ;

[0077] The spatial-spectral feature fusion device uses a multi-head attention mechanism to calculate the correlation between different bands, enhance the contribution of key band information to target detection, and generate fusion features F fused .

[0078] It should be noted that each band refers to the grayscale image of 3 channels and 6 grayscale images obtained by splitting the RGB image.

[0079] Object detection module: Based on the detection head, a structure similar to Mask R-CNN is designed: classification branch - predicts the target category; bounding box branch - predicts the preliminary bounding box of the target; polygon branch - generates the precise polygon outline of the target through vertex regression.

[0080] Loss function: Joint use of classification loss L cls , bounding box loss L box and polygon regression loss L ploy , the final total loss is: L det =L cls +β1L box +β2L ploy , where β1 and β2 are weight hyperparameters.

[0081] Step 5: Use the grayscale image and target contour information as input to identify the target material type through a pre-trained material recognition network.

[0082] Specifically, 100 points are randomly sampled within the spatial position of the target, and the spectral data of these points in the range of 450nm to 690nm are extracted to form a spectral matrix of Mat = 100×6 = 10×10×6, which is called a single sampling. Multiple sampling is required during the training process, generally 5 to 10 times, and only one sampling is required during prediction. The material recognition network (MS-MatNet) is a deep learning network based on spectral features and spatial distribution features. It uses a convolutional layer to process the spectral matrix, combines the spatial context information of the target, and outputs the material category of the target (such as alloy, plastic, wood, paper, etc.). The MS-MatNet network design is as follows Figure 3 As shown below:

[0083] Input layer: The input includes the spectral matrix M∈R of 100 points randomly sampled from the detected target area 100 ×6 , where each point contains the grayscale values of six bands (450nm, 540nm, 590nm, 600nm, 650nm, and 690nm). The spatial contextual information of the target (such as regional shape and location) is used as additional input to enhance the contextual awareness of material classification. The target region is the closed area enclosed by the target outline information.

[0084] The spectral feature extraction module includes local convolution feature extraction: the convolution layer extracts local spectral features on the spectral matrix and learns the relationship between the bands of the material. The feature is represented by F spec ∈R 100×d , where d is the feature dimension; Global context fusion: Use the self-attention mechanism to calculate the correlation between sampling points and extract the global feature F of the spectral distribution global .

[0085] The spatial-spectral fusion module includes spatial feature encoding: extracting the spatial features F of the target area space , including the shape descriptor of the target (such as area, side length) and polygon vertex distribution information; fusion network: the spectral feature F spec and spatial characteristics F space Fusion is performed to generate the final feature representation F for material classification through a fully connected network mat .

[0086] Classifier: The classification head uses a fully connected layer and a Softmax function to output the probability distribution of the target material, supporting multiple types of materials (such as alloy, plastic, wood, paper, etc.).

[0087] Loss function: cross entropy loss function L is used mat =-∑ c y c log(p c ), where y cis the true label, p c is the probability of the network output.

[0088] Step 6: Use the spatial feature encoder and the spectral feature encoder to map the spatial and spectral features to the same vector space, thereby fusing the deep learning target detection network and the material recognition network to obtain a target detection and material recognition integrated network, and use the RGB image, grayscale image, target contour information, target category and target material category as training data to train the target detection and material recognition integrated network; use the joint loss function L = L det +αL mat , where L det is the target detection loss, L mat is the material recognition loss, and α is the weight coefficient.

[0089] Step 7: Apply the trained target detection and material recognition network to simultaneously identify target categories and material categories.

[0090] This embodiment provides a method for target detection and material recognition based on multispectral imaging. This method utilizes RGB images and multiple grayscale images at different wavelengths to efficiently detect and classify targets. This method can extract spatial information about targets in dynamic scenes and classify materials based on spectral characteristics. This article uses indoor scenes as an example to illustrate the imaging process. Similar steps can be used to image real-world outdoor scenes. The proposed algorithm can be used to detect and identify targets in both types of scenes.

[0091] Example 2

[0092] This embodiment provides a target detection and material recognition system based on multispectral images, such as Figure 1 and Figure 2 As shown, the system includes an imaging system, a motion control system, a motion guide rail, a stage, and a central processing unit loaded with the method for target detection and material recognition based on multispectral imaging provided in Example 1. The stage is movably mounted on the motion guide rail. The imaging system includes an RGB camera, a six-hole filter wheel, a grayscale camera, and a spectrometer. The spectrometer splits the incident light and captures the reflected light of the target within different wavelength ranges. The motion control system is used to control the stage on which the target is placed to move at a constant speed on the guide rail in front of the imaging system.

[0093] Furthermore, in this embodiment, a black and white chessboard is placed on the stage, ensuring that it is positioned in the middle of the imaging field of view, and the black and white chessboard images of each channel are clearly captured: IB RGB and IB λ,λ∈{450,540,590,600,650,690}; Place different target objects on the stage in various postures, use the motion controller to control them to pass from left to right at a constant speed, and clearly capture the object motion video of each channel: V i-RGB and V i-λ , i∈[1, N], where i is the number of the video group, and a total of N groups are shot

[0094] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. It will be apparent to those skilled in the art that various changes, modifications, substitutions, and variations of these embodiments, including components, without departing from the principles and spirit of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for target detection and material recognition based on multispectral images, characterized in that: The steps include: Step 1: Build an imaging system and use it to acquire RGB images and grayscale images; Step 2: Align the target image in the RGB image and the grayscale image; Step 3: Use the data annotation platform to mark the precise outline of the target in the RGB image; Step 4: Use the RGB image and grayscale image annotated with precise contours as input to obtain target contour information and target category through a pre-trained deep learning target detection network. The pre-trained deep learning target detection network obtains spatial features and retains spectral features based on the spatial features. Step 5: Use the grayscale image and target contour information as input to identify the target material type through a pre-trained material recognition network; Step 6: Use the spatial feature encoder and the spectral feature encoder to map the spatial and spectral features to the same vector space, thereby fusing the deep learning target detection network and the material recognition network to obtain an integrated target detection and material recognition network. The RGB image, grayscale image, target outline information, target category, and target material category are used as training data to train the integrated target detection and material recognition network. Step 7: Apply the trained target detection and material recognition network to simultaneously identify target categories and material categories.

2. The method for target detection and material recognition based on multispectral images according to claim 1, characterized in that: In step 1, an RGB image and grayscale images with wavelengths of 450 nm, 540 nm, 590 nm, 600 nm, 650 nm, and 690 nm are acquired through an imaging system.

3. The method for target detection and material recognition based on multispectral images according to claim 2, characterized in that: The method for aligning the RGB image and the grayscale image is: First define the RGB image as I RGB (x, y) and grayscale images of different wavelengths are I λ (x, y), where x and y represent the horizontal and vertical coordinates of the image respectively, and λ represents the wavelength of the image. Then the image transformation model is used for alignment, and the expression is as follows: T(x,y)=(x′,y′)=s·(x,y)+(tx,ty) Where (x, y) is the coordinate of the original image, (x′, y′) is the coordinate of the target image after transformation, s is the scaling factor, and (tx, ty) is the translation parameter.

4. The method for target detection and material recognition based on multispectral images according to claim 1, characterized in that: The data labeling platform adopts the Label Studio data labeling platform.

5. The method for target detection and material recognition based on multispectral images according to claim 2, characterized in that: The deep learning target detection network includes a first input layer, a feature extraction module and a target detection module. The first input layer includes 9 convolutional encoders, which perform convolution processing on the three channels of the RGB image and the 6 grayscale images separately, but sum the convolution results of all channels to extract the spatial features of each layer and output the spatial features of each of the 9 channels; The feature extraction module includes 9 spatial feature encoders and a spatial-spectral feature fusion module. The three channels of the RGB image and each band corresponding to the 6 grayscale images use their own spatial feature encoders to extract their own spatial features and output F R 、F G 、F B 、F 450 、F 540 、F 590 、F 600 、F 650 、F 690 ; The spatial-spectral feature fusion device uses a multi-head attention mechanism to calculate the correlation between spatial features of different bands and generate fusion features F fused ; The target detection module includes three detection heads, namely a classification branch for predicting the target category, a bounding box branch for predicting the preliminary bounding box of the target, and a polygon branch for generating the precise polygon outline of the target through vertex regression.

6. The method for target detection and material recognition based on multispectral images according to claim 5, characterized in that: The method for retaining the spectral features is: taking points at the same position on each layer of spatial features, and then combining the points on each layer of spatial features to obtain the spectral features.

7. The method for target detection and material recognition based on multispectral images according to claim 5, characterized in that: When the deep learning target detection network is trained, the loss function is to use the classification loss L cls , bounding box loss L box and polygon regression loss L ploy , the final total loss is: L det =L cls +β1L box +β2L ploy , where β1 and β2 are weight hyperparameters.

8. The method for target detection and material recognition based on multispectral images according to claim 6, characterized in that: The material recognition network includes a second input layer, a spectral feature extraction module, a spatial-spectral fusion module and a classifier; The second input layer is used to input the spectral matrix M∈R of 100 points randomly sampled from the target area 100×6 , where each point contains the grayscale values of six bands, and the spatial context information of the target is used as an additional input, and the target area is a closed area surrounded by the target contour information; The spectral feature extraction module is used to extract local spectral features on the spectral matrix through the convolution layer and learn the relationship between the bands of the material. The feature is represented by F spec ∈R 100×d , where d is the feature dimension, and the self-attention mechanism is used to calculate the correlation between the sampling points and extract the global feature F of the spectral distribution global ; The spatial-spectral fusion module is used to extract the spatial features F of the target area through the spatial feature encoder space , including the shape descriptor of the target and the polygon vertex distribution information; and through the fusion network, the spectral feature F spac and spatial characteristics F space Fusion is performed to generate the final feature representation F for material classification through a fully connected network mat ; The classifier is used to output the probability distribution of the target material by using a fully connected layer and a Softmax function.

9. The method for target detection and material recognition based on multispectral images according to claim 6, characterized in that: The cross entropy loss function is used when training the material recognition network: where y c is the true label, p c is the probability of the network output.

10. The method for target detection and material recognition based on multispectral images according to claim 8, characterized in that: When training the target detection and material recognition integrated network, a joint loss function is used: L=L det +αL mat Among them L det is the target detection loss, L mat is the material recognition loss, and α is the weight coefficient.

11. A target detection and material recognition system based on multispectral images, characterized in that: It comprises an imaging system, a motion control system, a motion guide rail, a loading platform and a central processing unit loaded with a target detection and material recognition method based on multispectral images as described in any one of claims 1 to 9, and the loading platform can be movably mounted on the motion guide rail.

12. The target detection and material recognition system based on multispectral images according to claim 10, characterized in that: The imaging system includes an RGB camera, a six-hole filter wheel, a grayscale camera and a spectrometer. The spectrometer is used to split the incident light and capture the reflected light of the target within different wavelength ranges.

13. The target detection and material recognition system based on multispectral images according to claim 11, characterized in that: The motion control system is used to control the object carrier on which the object is placed to move at a uniform speed on the guide rail in front of the imaging system.

Images