Target detection method and device based on feature fusion and related equipment

Abstract

The embodiment of the invention provides a target detection method and device based on feature fusion and related equipment, and belongs to the technical field of computer vision. The method comprises the steps of obtaining a target detection image and a target detection model; inputting the target detection image into an image feature coding network of the target detection model, and performing multi-scale feature extraction on the target detection image to obtain image feature spectrums of different feature scales; inputting the image feature spectrums into a segmentation feature dynamic decoding network of a target detection model, and performing multi-level semantic segmentation on the target detection image according to the image feature spectrums of different feature scales to obtain segmentation feature spectrums of different semantic levels; and inputting the image characteristic spectrum and the segmentation characteristic spectrum into a detection network of a target detection model, and obtaining a target detection result according to the image characteristic spectrum and the segmentation characteristic spectrum. According to the embodiment of the invention, the feature discrimination can be improved, and the target detection accuracy is improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a target detection method, device and related equipment based on feature fusion. Background Art

[0002] Object detection involves identifying all objects of interest from a given image or video and determining their category, location, and size. This requires not only being able to accurately classify objects in the image, but also accurately locating the bounding box of each object.

[0003] In related technologies, target detection methods mainly rely on deep convolutional neural networks, which only focus on the basic features of the target and fail to deeply explore more complex detail information. That is, they only consider the rough and abstract information in the deep convolutional neural network. As a result, when faced with complex scenes such as target background interference, diverse appearances, and occlusion between targets, they are unable to obtain complete target information and effectively distinguish the features between different targets, resulting in a decrease in feature discriminability and a reduction in the accuracy of target detection. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a target detection method, device and related equipment based on feature fusion, which can improve feature discrimination and improve the accuracy of target detection.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a target detection method based on feature fusion, the method comprising: acquiring a target detection image and a target detection model;

[0006] Inputting the target detection image into the image feature encoding network of the target detection model, performing multi-scale feature extraction on the target detection image, and obtaining image feature spectra of different feature scales;

[0007] Inputting the image feature spectrum into the segmentation feature dynamic decoding network of the target detection model, performing multi-level semantic segmentation on the target detection image according to the image feature spectrum of different feature scales, and obtaining the segmentation feature spectrum of different semantic levels;

[0008] The image feature spectrum and the segmentation feature spectrum are input into the detection network of the target detection model, and the target detection result is obtained according to the image feature spectrum and the segmentation feature spectrum.

[0009] In some embodiments, inputting the image feature spectrum and the segmentation feature spectrum into the detection network of the target detection model, and obtaining the target detection result according to the image feature spectrum and the segmentation feature spectrum, comprises:

[0010] Inputting the image feature spectrum and the segmentation feature spectrum into a detection network of the target detection model;

[0011] Embedding the segmentation feature spectra at different semantic levels into the image feature spectra at different feature scales to generate a target detection feature spectrum;

[0012] The target detection result is obtained according to the target detection feature spectrum.

[0013] In some embodiments, the image feature encoding network includes a convolutional layer and a residual layer connected in sequence, wherein the residual layer includes a plurality of residual blocks connected in sequence with decreasing feature extraction scales in sequence, and the multi-scale feature extraction of the target detection image is performed to obtain image feature spectra of different feature scales, including:

[0014] Inputting the target detection image into the convolution layer for feature extraction to obtain detection input data;

[0015] The detection input data is input into the residual layer, and feature extraction is performed based on a plurality of residual blocks connected in sequence and having successively decreasing feature extraction scales, so as to obtain the image feature spectrum corresponding to each residual block.

[0016] In some embodiments, the segmentation feature dynamic decoding network includes a dynamic skip-layer fusion unit, and the target detection image is subjected to multi-level semantic segmentation according to the image feature spectra of different feature scales to obtain the segmentation feature spectra of different semantic levels, including:

[0017] Determining the resolution ranking based on the high and low relationship between the characteristic scales corresponding to the image characteristic spectrum;

[0018] For the image feature spectrum of the lowest feature scale in the resolution sorting, a convolution operation is performed on the image feature spectrum of the lowest feature scale based on the dynamic layer skipping fusion unit to obtain the segmentation feature spectrum corresponding to the image feature spectrum;

[0019] For the image feature spectrum of the non-lowest feature scale in the resolution sorting, the image feature spectrum of the non-lowest feature scale is aggregated with the segmentation feature spectrum corresponding to the corresponding next-level feature scale based on the dynamic skip-layer fusion unit to obtain the segmentation feature spectrum corresponding to the image feature spectrum of the non-lowest feature scale.

[0020] In some embodiments, the detection network includes a multi-level context embedding unit, and embedding the segmentation feature spectra at different semantic levels into the image feature spectra at different feature scales to generate the target detection feature spectra includes:

[0021] For the image feature spectrum at each feature scale, cascading the image feature spectrum with the segmentation feature spectrum at the corresponding semantic level to obtain a cascade feature spectrum;

[0022] Performing a convolution operation on the cascade characteristic spectrum to obtain an initial characteristic spectrum;

[0023] Inputting the initial feature spectrum into the multi-level context embedding unit to obtain a calibration feature spectrum;

[0024] A target detection characteristic spectrum is obtained according to the calibration characteristic spectrum and the cascade characteristic spectrum.

[0025] In some embodiments, the multi-level context embedding unit includes an attention convolution layer, a relational reasoning convolution layer, and a feature allocation convolution layer, and the inputting the initial feature spectrum into the multi-level context embedding unit to obtain a calibration feature spectrum includes:

[0026] Obtaining a contextual attention spectrum according to the attention convolution layer and the cascade feature spectrum, and obtaining contextual features according to the contextual attention spectrum and the initial feature spectrum;

[0027] Obtaining a contextual relationship spectrum based on the relationship reasoning convolution layer and the contextual features, and obtaining enhanced contextual features according to the contextual relationship spectrum and the contextual features;

[0028] A feature allocation spectrum is obtained according to the feature allocation convolution layer and the cascade feature spectrum, and the calibration feature spectrum is obtained according to the feature allocation spectrum and the enhanced context feature.

[0029] In some embodiments, the pre-trained object detection model is jointly optimized based on the segmentation loss function and the object detection loss function, specifically including:

[0030] Obtaining a training data set and an original detection model; wherein the training data set includes a plurality of training sample images;

[0031] Performing target detection on the training data set based on the original detection model to obtain a sample detection result for each of the training sample images;

[0032] Calculating the target classification loss function of the original detection model based on the classification result of each detection target in each sample detection result;

[0033] Calculating the regression loss function of the original detection model based on the predicted box position of each detection target in each sample detection result;

[0034] Calculating the segmentation loss function of the original detection model based on the classification result of each pixel in each sample detection result;

[0035] Calculate the target detection loss function based on the target classification loss function and the regression loss function;

[0036] The original detection model is jointly optimized based on the target detection loss function and the segmentation loss function to obtain the target detection model.

[0037] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a target detection device based on feature fusion, comprising: an acquisition module, used to acquire a target detection image and a target detection model;

[0038] An image feature spectrum module is used to input the target detection image into the image feature encoding network of the target detection model, perform multi-scale feature extraction on the target detection image, and obtain image feature spectra of different feature scales;

[0039] A segmentation feature spectrum module is used to input the image feature spectrum into the segmentation feature dynamic decoding network of the target detection model, perform multi-level semantic segmentation on the target detection image according to the image feature spectrum of different feature scales, and obtain the segmentation feature spectrum of different semantic levels;

[0040] The detection result module is used to input the image feature spectrum and the segmentation feature spectrum into the detection network of the target detection model, and obtain the target detection result according to the image feature spectrum and the segmentation feature spectrum.

[0041] To achieve the above-mentioned purpose, the third aspect of the embodiments of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the method described in the above-mentioned first aspect embodiment when executing the computer program.

[0042] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium, and the storage medium stores a computer program. When the computer program is executed by a processor, the method described in the embodiment of the first aspect is implemented.

[0043] The target detection method, device and related equipment based on feature fusion proposed in the present application have the following beneficial effects: inputting the target detection image into the image feature encoding network of the target detection model, performing multi-scale feature extraction on the target detection image, and obtaining image feature spectra of different feature scales; inputting the image feature spectrum into the segmentation feature dynamic decoding network of the target detection model; performing multi-level semantic segmentation on the target detection image according to the image feature spectra of different feature scales, and obtaining segmentation feature spectra of different semantic levels; inputting the image feature spectrum and the segmentation feature spectrum into the detection network of the target detection model, and obtaining the target detection result according to the image feature spectrum and the segmentation feature spectrum. The target detection image is feature extracted through the image feature encoding network to obtain image feature spectra of different feature scales, so that the model can capture local details (low-level features) and global context (high-level features) in the target detection image. The segmentation feature dynamic decoding network generates segmentation feature spectra of different semantic levels according to the image feature spectra of different feature scales. This multi-level segmentation feature spectrum can provide more detailed pixel-level information, which helps the model to better understand the image content, especially when there is occlusion or similar background between multiple target objects in the target detection image. The detection network embeds the segmentation feature spectra of different semantic levels into the image feature spectra of different scales to generate the target detection feature spectrum, making full use of the contextual information of the target detection image, enhancing the model's ability to distinguish the target object in complex scenes, improving feature discriminability, and thus improving the accuracy of target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flow chart of a target detection method based on feature fusion provided in an embodiment of the present application;

[0045] Figure 2 is another flow chart of the target detection method based on feature fusion provided in an embodiment of the present application;

[0046] Figure 3 is a schematic diagram of a target detection model provided in an embodiment of the present application;

[0047] Figure 4 is another flow chart of the target detection method based on feature fusion provided in an embodiment of the present application;

[0048] Figure 5 is a schematic diagram of a dynamic layer hopping fusion unit provided in an embodiment of the present application;

[0049] Figure 6 is another flow chart of the target detection method based on feature fusion provided in an embodiment of the present application;

[0050] Figure 7 is another flow chart of the target detection method based on feature fusion provided in an embodiment of the present application;

[0051] Figure 8 It is a schematic diagram of the process of target detection characteristic spectrum provided in the embodiment of the present application;

[0052] Fig. 9 is another flow chart of the target detection method based on feature fusion provided in an embodiment of the present application;

[0053] Fig.10 is a schematic diagram of a multi-level context embedding unit provided in an embodiment of the present application;

[0054] Fig.11 is a schematic diagram of a target detection method based on feature fusion provided in an embodiment of the present application;

[0055] Fig.12 is another flow chart of the target detection method based on feature fusion provided in an embodiment of the present application;

[0056] Fig.13 This is an optional structural diagram of a target detection device based on feature fusion provided in an embodiment of the present application;

[0057] Fig.14 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0059] It should be noted that although the functional modules are divided in the device schematic and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0061] First, some nouns involved in this application are analyzed:

[0062] Object Detection: It is an important task in the field of computer vision. It aims to identify and locate one or more target objects from images or video frames. Object detection requires not only identifying what the target objects in the image are (i.e. classification), but also determining the specific locations of these target objects in the image (i.e. positioning). Object detection is the basis of many advanced computer vision applications, such as autonomous driving, video surveillance, face recognition, medical image analysis, etc.

[0063] Object detection involves identifying all objects of interest from a given image or video and determining their category, location, and size. This requires not only being able to accurately classify objects in the image, but also accurately locating the bounding box of each object.

[0064] In related technologies, target detection methods mainly rely on deep convolutional neural networks, which only focus on the basic features of the target and fail to deeply explore more complex detail information. That is, they only consider the rough and abstract information in the deep convolutional neural network. As a result, when faced with complex scenes such as target background interference, diverse appearances, and occlusion between targets, they are unable to obtain complete target information and effectively distinguish the features between different targets, resulting in a decrease in feature discriminability and a reduction in the accuracy of target detection.

[0065] Based on this, the embodiments of the present application provide a target detection method, apparatus and related equipment based on feature fusion, which can improve feature discriminability and thus improve the accuracy of target detection.

[0066] The target detection method, device and related equipment based on feature fusion provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the target detection method based on feature fusion in the embodiments of the present application is described.

[0067] The target detection method based on feature fusion in the embodiments of the present application can be illustrated by the following embodiments.

[0068] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first. For example, when obtaining user stored data and the user's cached data access request, the user's permission or consent will be obtained first. Moreover, the collection, use and processing of these data will comply with the relevant laws, regulations and standards of the relevant countries and regions. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or separate consent will be obtained by means of a pop-up window or jumping to a confirmation page. After clearly obtaining the user's separate permission or separate consent, the necessary user-related data used to enable the normal operation of the embodiment of the present application will be obtained.

[0069] Exemplarily, the target detection method in the embodiment of the present application is based on multi-level context feature fusion technology, which can not only mine fine pixel segmentation features, but also recalibrate harmful noise features with the help of the context information of the scene and the relationship between features. The low-level feature spectrum is helpful for small-scale target detection due to the presence of more detailed information; while the high-level feature spectrum has a larger receptive field, which is more beneficial for large-scale target detection and can be applied to a variety of specific application scenarios, including autonomous driving, video surveillance and other fields. In the field of autonomous driving, the target detection method based on feature fusion is used to identify and locate pedestrians, vehicles, traffic signs and other obstacles around the vehicle in real time. The embodiment of the present application can improve the vehicle's understanding of the surrounding environment through precise feature extraction and context analysis, thereby making safer and more accurate driving decisions.

[0070] Figure 1 is an optional flow chart of a target detection method based on feature fusion provided in an embodiment of the present application. Figure 1 The method in the embodiment may include but is not limited to steps 101 to 104. Figure 1 The order of step 101 to step 104 is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0071] Step 101, obtaining a target detection image and a target detection model.

[0072] The following is a detailed explanation of step 101.

[0073] It is understood that the target detection image refers to the image to be analyzed, which may contain one or more target objects whose categories and locations need to be identified and located. The pre-trained target detection model is a deep learning model that has been trained with a large amount of data and can automatically extract features, classify and locate the target detection image.

[0074] In some embodiments, the target detection model includes an image feature encoding network, a segmentation feature dynamic decoding network and a detection network. The image feature encoding network is used to extract features from the target detection image to obtain image feature spectra of different feature scales. The segmentation feature dynamic decoding network is used to generate segmentation feature spectra of different semantic levels based on image feature spectra of different feature scales. The detection network is used to embed segmentation feature spectra of different semantic levels into image feature spectra of different scales to generate target detection feature spectra, and obtain target detection results based on the target detection feature spectra.

[0075] Step 102: input the target detection image into the image feature encoding network of the target detection model, perform multi-scale feature extraction on the target detection image, and obtain image feature spectra of different feature scales.

[0076] Step 102 is explained in detail below.

[0077] Exemplarily, the image feature encoding network is a component in the target detection model, which is responsible for extracting image feature spectra of different feature scales from the input target detection image, including image texture, color, shape and other information.

[0078] It can be understood that the target detection image is input into the pre-trained target detection model, and multi-scale feature extraction, that is, analyzing the target detection image at different scales to extract features, enables the target detection model to capture a wide range of information from local details to global context, and obtains image feature spectra of different feature scales. At different network levels, the target detection model extracts image feature spectra of different scales. For example, the lower layer may focus on capturing local features (such as edges and textures) of the target detection image, while the higher layer captures more abstract semantic information. When the target detection image is a medical image, the more abstract semantic information can be the shape and position of the organ. The low-level image feature spectrum has a higher spatial resolution and is suitable for capturing small-scale structures in the image. The high-level image feature spectrum has a lower spatial resolution, but can provide a wider range of contextual information, which helps to identify large-scale structures and overall layout in the image.

[0079] Step 103, input the image feature spectrum into the segmentation feature dynamic decoding network of the target detection model, perform multi-level semantic segmentation on the target detection image according to the image feature spectra of different feature scales, and obtain segmentation feature spectra of different semantic levels.

[0080] Step 103 is explained in detail below.

[0081] Exemplarily, the segmentation feature dynamic decoding network is responsible for further processing the low-level and high-level image feature spectra extracted by the image feature encoding network to generate segmentation feature spectra at different semantic levels, which can more finely describe the target object in the target detection image.

[0082] It can be understood that the target detection image is subjected to multi-level semantic segmentation according to the image feature spectra of different feature scales, different objects in the image are identified, and the semantic information of these objects is understood, such as which category they belong to (for example, people, vehicles, buildings, etc.). The segmentation feature spectrum is a set of features generated during the image segmentation process. These features describe the probability or confidence that each pixel in the target detection image belongs to a specific category. The segmentation feature spectra of different semantic levels represent the basic pixel-level features to the complex object-level features.

[0083] Step 104: input the image feature spectrum and the segmentation feature spectrum into the detection network of the target detection model, and obtain the target detection result according to the image feature spectrum and the segmentation feature spectrum.

[0084] Step 104 is explained in detail below.

[0085] Exemplarily, the detection network combines the segmentation feature spectrum generated by the segmentation feature dynamic decoding network with the image feature spectrum of the image feature encoding network to generate a final target detection feature spectrum, and locates the target object based on it.

[0086] It can be understood that the image feature spectrum is a multi-scale feature extracted from the image feature encoding network, and the segmentation feature spectrum is a feature generated by the segmentation feature dynamic decoding network, which provides detailed information that each pixel in the image belongs to a specific category. The multi-scale image feature spectrum obtained from the image feature encoding network is fused with the segmentation feature spectrum generated by the segmentation feature dynamic decoding network. This fusion can be achieved through cascading, weighted summation or other feature fusion techniques. According to the image feature spectrum and the segmentation feature spectrum, the target detection result is obtained. The target detection result includes information such as the category, position (bounding box), and confidence of each target in the target detection image.

[0087] Through the above steps 101 to 104, the target detection image is input into the image feature encoding network of the target detection model, multi-scale feature extraction is performed on the target detection image to obtain image feature spectra of different feature scales, the image feature spectrum is input into the segmentation feature dynamic decoding network of the target detection model, multi-level semantic segmentation is performed on the target detection image according to the image feature spectra of different feature scales, and segmentation feature spectra of different semantic levels are obtained; the image feature spectrum and the segmentation feature spectrum are input into the detection network of the target detection model, and the target detection result is obtained according to the image feature spectrum and the segmentation feature spectrum. The target detection image is feature extracted through the image feature encoding network to obtain image feature spectra of different feature scales, so that the model can capture local details (low-level features) and global context (high-level features) in the target detection image. The segmentation feature dynamic decoding network generates segmentation feature spectra of different semantic levels according to the image feature spectra of different feature scales. This multi-level segmentation feature spectrum can provide more detailed pixel-level information, which helps the model to better understand the image content, especially when there is occlusion or similar background between multiple target objects in the target detection image. The detection network embeds the segmentation feature spectra of different semantic levels into the image feature spectra of different scales to generate the target detection feature spectrum, making full use of the contextual information of the target detection image, enhancing the model's ability to distinguish the target object in complex scenes, improving feature discriminability, and thus improving the accuracy of target detection.

[0088] See also Figure 2 In some embodiments, the image feature encoding network includes sequentially connected convolutional layers and residual layers, wherein the residual layer includes sequentially connected residual blocks with successively decreasing feature extraction scales, and multi-scale feature extraction is performed on the target detection image to obtain image feature spectra of different feature scales, which may include the following steps 201 to 202.

[0089] Step 201: Input the target detection image into the convolution layer for feature extraction to obtain detection input data.

[0090] Step 202 , input the detection input data into the residual layer, perform feature extraction based on a plurality of residual blocks connected in sequence and having successively decreasing feature extraction scales, and obtain an image feature spectrum corresponding to each residual block.

[0091] Steps 201 to 202 are described in detail below.

[0092] It can be understood that the image feature encoding network includes sequentially connected convolutional layers and residual layers. The residual layer is the core of the image feature encoding network and is composed of multiple residual blocks. The residual block can contain a series of convolution operations, batch normalization, and activation functions. As the data flows through more residual blocks, the scale of the feature (i.e., spatial resolution) gradually decreases, while the depth of the feature gradually increases, so that the deep residual blocks can capture higher-level abstract features, and the output image feature spectrum has richer semantic information. Finally, the output of each residual block is collected to form a series of image feature spectra of different feature scales, which will be used in the subsequent segmentation feature dynamic decoding network and detection network for more refined semantic segmentation and target detection.

[0093] In some embodiments, a residual network ResNet is used as an image feature encoding network, and the image feature spectrum F is as follows:

[0094] F=ResNet(I)={F1,F2,...,F n}

[0095] Among them, F n ∈R C×H×W Represents the image feature spectrum output by the residual network ResNet in the residual block, where C×H×W is the feature scale, C is the number of channels, H is the height, W is the width, and I is the target detection image.

[0096] See also Figure 3 , Figure 3 : is a schematic diagram of a target detection model provided in an embodiment of the present application, wherein a residual network ResNet is used as an image feature encoding network, and the image feature encoding network includes sequentially connected convolutional layers and residual layers. The residual layer shown in the figure includes four sequentially connected residual blocks with decreasing feature extraction scales. Therefore, image feature spectra of different feature scales can be expressed as {F2, F3, F4, F5}, wherein F2 is the feature spectrum output by the first residual block after the convolutional layer, F3 is the feature spectrum output by the second residual block after the convolutional layer, F4 is the feature spectrum output by the third residual block after the convolutional layer, and F5 is the feature spectrum output by the fourth residual block after the convolutional layer. The resolution of image feature spectra {F2, F3, F4, F5} of different feature scales is successively reduced to {0.5 2 ,0.5 3 ,0.5 4 ,0.5 5}.

[0097] Through the above steps 201 to 202, the target detection image is input into the convolution layer for feature extraction to obtain detection input data, and the detection input data is input into the residual layer. Feature extraction is performed based on a plurality of residual blocks connected in sequence with decreasing feature extraction scales, and an image feature spectrum corresponding to each residual block is obtained. Rich multi-scale features are extracted from the target detection image, which can capture a wide range of information from local details to global context.

[0098] See also Figure 4 In some embodiments, the segmentation feature dynamic decoding network includes a dynamic skip-layer fusion unit, which performs multi-level semantic segmentation on the target detection image according to image feature spectra of different feature scales to obtain segmentation feature spectra of different semantic levels, which may include the following steps 301 to 303.

[0099] Step 301, determining the resolution ranking based on the high and low relationship between the image feature spectrum and the feature scale.

[0100] Step 302 , for the image feature spectrum with the lowest feature scale in the resolution sorting, a convolution operation is performed on the image feature spectrum with the lowest feature scale based on a dynamic skip layer fusion unit to obtain a segmentation feature spectrum corresponding to the image feature spectrum.

[0101] Step 303, for the image feature spectrum of the non-lowest feature scale in the resolution sorting, the image feature spectrum of the non-lowest feature scale is aggregated with the segmentation feature spectrum corresponding to the corresponding next-level feature scale based on the dynamic skip-layer fusion unit to obtain the segmentation feature spectrum corresponding to the image feature spectrum of the non-lowest feature scale.

[0102] Steps 301 to 303 are described in detail below.

[0103] For example, based on the high and low relationship between the image feature spectrum and the feature scale, the resolution ranking is determined, and the resolutions of the image feature spectra {F2, F3, F4, F5} of different feature scales are successively reduced to {0.5 2 ,0.5 3 ,0.5 4 ,0.5 5}, that is, the resolution is sorted from high to low as {F2, F3, F4, F5}, where F5 is the image feature spectrum with the lowest feature scale.

[0104] In some embodiments, see Figure 3After the target detection image is input into the image feature encoding network, the image feature spectrum with the resolution sorting from high to low is obtained as {F2, F3, F4, F5}. For the image feature spectrum F5 with the lowest feature scale in the resolution sorting, the image feature spectrum with the lowest feature scale is convolved based on the dynamic skip layer fusion unit to obtain the segmentation feature spectrum S5 corresponding to the image feature spectrum F5. For the image feature spectrum with a non-lowest feature scale in the resolution sorting, such as the image feature spectrum F4, the image feature spectrum F4 is aggregated with the segmentation feature spectrum S5 corresponding to the corresponding next-level feature scale based on the dynamic skip layer fusion unit to obtain the segmentation feature spectrum S4 corresponding to the image feature spectrum F4.

[0105] It can be understood that considering the semantic importance of image feature spectra of features at different scales, it is proposed to aggregate the image feature spectra of the current scale based on the dynamic skip-layer fusion unit F i With the adjacent segmentation feature spectrum S i+1 , thereby decoding the segmentation feature spectra S at different semantic levels i (i=2,3,4), after the dynamic skip layer fusion unit, the segmentation feature spectrum S obtained by the segmentation layer i (i=2,3,4,5) is the category score for each pixel in the target detection image, so as to achieve the refined pixel analysis of the target object in the target detection image, such as Figure 3 As shown, the image segmentation result is obtained based on the segmentation prediction model.

[0106] In some embodiments, a convolution operation is performed on the image feature spectrum of the lowest feature scale based on a dynamic skip layer fusion unit to obtain a segmentation feature spectrum S5 corresponding to the image feature spectrum F5. The image feature spectrum F5 can be sent to a 1×1 convolution layer and a 3×3 convolution layer to obtain the segmentation feature spectrum S5 corresponding to the image feature spectrum F5.

[0107] See also Figure 5 , Figure 5 The image feature spectrum F of the current scale is aggregated based on the dynamic layer skipping fusion unit provided in the embodiment of the present application. i With the adjacent segmentation feature spectrum S i+1 , thereby decoding the segmentation feature spectra S at different semantic levels i In order to facilitate the fusion of feature spectra of different dimensions, firstly, the low-resolution segmentation feature spectrum S i+1 ∈R C ×H×W , C×H×W is the feature scale, C is the number of channels, H is the height, and W is the width. Bilinear interpolation is used for upsampling (the scale factor is 2) to ensure that the width and height of different feature spectra are consistent; for high-resolution image feature spectra F i, a 1×1 convolutional layer with a channel number of C is used to reduce the number of channels of the image feature spectrum to obtain the same i+1 Image feature spectrum F with the same channel dimension i '∈R C×H×W Then, the image feature spectrum F of the adjacent scales is i ' and segmentation characteristic spectrum S i+1 Cascade and get the characteristic spectrum U i ∈R 2C×H×W , where 2C×H×W is the characteristic scale, and 2C is the characteristic spectrum U i The number of channels is H, the height is W, and the width is W. Subsequently, according to the designed dynamic soft aggregation unit, the importance of pixels at each spatial position in different feature spectra is calculated to control the information flow of different scale features in the feature spectrum Ui, thereby avoiding the semantic confusion problem of different objects. Among them, the dynamic soft aggregation unit consists of two 1×1 convolutional layers with a channel number of C and a Sigmoid activation function. The specific calculation is as follows:

[0108]

[0109] in, are the different semantic feature spectra after dynamic soft aggregation. represents element-wise addition, ⊙ represents the Hadamard product operation, W1 and W2 are the learnable weight matrices of two 1×1 convolutional layers with C channels, and b1 and b2 are the biases of two 1×1 convolutional layers with C channels. After obtaining the different semantic feature spectra after dynamic soft aggregation, a 3×3 convolutional layer with C channels is used to transform the feature spectra. Information interaction is carried out between all feature channels of , thereby further reducing the semantic gap and decoding a more refined segmentation feature spectrum S i .

[0110] Through the above steps 301 to 303, the resolution sorting is determined based on the high-low relationship between the feature scales corresponding to the image feature spectrum. For the image feature spectrum of the non-lowest feature scale in the resolution sorting, the image feature spectrum of the non-lowest feature scale and the segmentation feature spectrum corresponding to the corresponding next-level feature scale are aggregated based on the dynamic skip-layer fusion unit to obtain the segmentation feature spectrum corresponding to the image feature spectrum of the non-lowest feature scale. The dynamic skip-layer fusion unit allows effective information exchange between feature spectra of different scales. The generated segmentation feature spectrum provides rich semantic information at the pixel level, so that the model can perform high-precision image segmentation.

[0111] See also Figure 6In some embodiments, the image feature spectrum and the segmentation feature spectrum are input into the detection network of the target detection model, and the target detection result is obtained according to the image feature spectrum and the segmentation feature spectrum, which may specifically include the following steps 401 to 403.

[0112] Step 401: input the image feature spectrum and the segmentation feature spectrum into the detection network of the target detection model.

[0113] Step 402, embedding segmentation feature spectra at different semantic levels into image feature spectra at different feature scales to generate target detection feature spectra;

[0114] Step 403, obtaining a target detection result according to the target detection feature spectrum.

[0115] Steps 401 to 403 are described in detail below.

[0116] It can be understood that by embedding segmentation feature spectra of different semantic levels into image feature spectra of different feature scales, more discriminative target detection feature spectra can be generated by mining the effective information in the segmentation feature spectra, thereby improving the accuracy of target detection. Figure 3 The detection network includes a multi-level context embedding unit, based on which the target detection feature spectrum {D2, D3, D4, D5} of the detection layer is obtained. For example, the segmentation feature spectrum S2 is embedded in the image feature spectrum F2 to obtain the target detection feature spectrum D2.

[0117] See also Figure 7 In some embodiments, the detection network includes a multi-level context embedding unit, which embeds segmentation feature spectra of different semantic levels into image feature spectra of different feature scales to generate target detection feature spectra, which may include the following steps 501 to 504.

[0118] Step 501 : for each feature scale image feature spectrum, a cascade feature spectrum is obtained by concatenating the image feature spectrum with a segmentation feature spectrum at a corresponding semantic level.

[0119] Step 502: Perform a convolution operation on the cascaded feature spectrum to obtain an initial feature spectrum.

[0120] Steps 501 to 502 are described in detail below.

[0121] Understandably, see Figure 8 , for each feature scale image feature spectrum F i , the image feature spectrum F i and the segmentation feature spectrum S of the corresponding semantic level i The cascade characteristic spectrum is obtained by cascading, and the cascade characteristic spectrum is convolved, that is, after a channel number C rThe 1×1 convolution layer reduces the number of channels of the feature spectrum to C r , and obtain the initial characteristic spectrum X=[X1,X2,...,X Cr ] T ∈R Cr×H×W .

[0122] Step 503: input the initial feature spectrum into a multi-level context embedding unit to obtain a calibrated feature spectrum.

[0123] The following is a detailed description of step 503.

[0124] See also Fig. 9 In some embodiments, the multi-level context embedding unit includes an attention convolution layer, a relational reasoning convolution layer, and a feature allocation convolution layer. The initial feature spectrum is input into the multi-level context embedding unit to obtain a calibrated feature spectrum, which may include the following steps 601 to 603.

[0125] Step 601, obtain a contextual attention spectrum according to the attention convolution layer and the cascade feature spectrum, and obtain contextual features according to the contextual attention spectrum and the initial feature spectrum.

[0126] The following is a detailed description of step 601.

[0127] See also Figure 8 In some embodiments, the concatenated feature spectrum is input into an attention convolution layer with N channels to obtain an N-dimensional feature spectrum A = [A1, A2, ..., A i ,...,A n ] T ∈R N×HW , where A i is a row vector of HW dimensions, for each dimension of the characteristic spectrum A i , obtain the corresponding context attention spectrum W through the Softmax layer Ai , the calculation is as follows:

[0128]

[0129] Among them, a ii W represents the eigenvalue at position (i, j) in the feature spectrum A, i∈[1,N], j∈[1,HW], and N represents the number of channels in the feature spectrum. Ai =[W A1 ,W A2 ,...,W AN ] T ∈R N×HW Represents the generated context attention spectrum, which has different focus points. When W AiWhen focusing on close areas (such as local objects or parts), it is used to capture local context information; when W Ai When the information is scattered in a long-distance space (such as multiple objects or surrounding scenes), it is used to perceive non-local context information. Finally, the context attention spectrum W is used Ai The context feature is obtained by performing a weighted summation on each position of the initial feature spectrum X. The calculation is as follows:

[0130]

[0131] Where Y=[Y1,Y2,...,Y Cr ]∈R Cr×N Represents context features. Through step 601, the multi-level context information in the image is collected using the context attention spectrum.

[0132] Step 602: obtain a contextual relationship spectrum based on the relational reasoning convolutional layer and the contextual features, and obtain enhanced contextual features according to the contextual relationship spectrum and the contextual features.

[0133] Step 602 is described in detail below.

[0134] In some embodiments, based on the multi-level context information captured above, the relationships between them are inferred, such as the relationship between different object regions, the relationship between objects and scenes, etc., to further enhance the expressiveness of these context features. In order to model the dependency relationship of these features, the present application embodiment shows two relationship reasoning convolutional layers. Figure 8 In the relational reasoning convolutional layer part, method 1 uses dot product operation to calculate the semantic similarity relationship of context features in the embedding space. The relational reasoning convolutional layer includes two embedding functions implemented by convolution operations, namely two 1×1 convolutional layer operations with N channels. The contextual relationship spectrum calculation based on the relational reasoning convolutional layer and context features is as follows:

[0135] R ij = Θ(Y i ) T ·Φ(Y j )

[0136]

[0137] Where Θ and Φ represent the embedding functions implemented by the two convolutional layers, i∈[1,N], j∈[1,HW]. ij Represents the similarity between the i-th and j-th context features. is the similarity weight normalized by the Softmax function, that is, the contextual relationship spectrum. The sum of is 1, The closer it is to 1, the more important the i-th context feature is to the j-th context feature.

[0138] See also Figure 8 Method 2 simplifies the dot product operation in the embedding space of the above method 1, replaces the two 1×1 convolutional layer operations with N channels in the relational reasoning convolutional layer with one 1×1 convolutional layer operation with N channels, and directly realizes the relational reasoning between contextual features.

[0139] Understandably, participation Figure 8 , according to the context spectrum Adaptively weight and aggregate all relevant context features Y to enhance their feature representation capabilities and obtain enhanced context features The specific calculation is as follows:

[0140]

[0141] Step 603: obtain a feature allocation spectrum according to the feature allocation convolution layer and the cascade feature spectrum, and obtain a calibration feature spectrum according to the feature allocation spectrum and the enhanced context feature.

[0142] The following is a detailed description of step 603.

[0143] Understandably, see Figure 8 The feature allocation convolution layer is a 1×1 convolution operation with N channels. The cascaded feature spectrum is input into the feature allocation convolution layer to obtain the importance feature spectrum B, B = [B1, B2, ..., B HW ]∈R N×HW represents the importance feature spectrum of the output of the feature allocation convolution layer, B j is an N-dimensional row vector, and then B is transformed into k Normalized to the range of (0,1), the feature distribution spectrum W is obtained Bij , the calculation is as follows:

[0144]

[0145] Among them, the higher the value of the feature allocation spectrum, the more important the i-th context feature is to the j-th local feature, which can help enhance the feature semantics of the location.

[0146] Exemplarily, these rich context information are used to supplement each local feature, and these context features are more flexibly allocated and applied to each position in combination with the feature requirements of the current position. The calibration feature spectrum is obtained according to the feature allocation spectrum and the enhanced context features. The specific calculation is as follows:

[0147]

[0148] in, represents the calibration characteristic spectrum after recalibration.

[0149] Through the above steps 601 to 603, refer to Fig.10 Through three stages, multiple contextual attention spectra are used to collect multi-level contextual information in the image. The contextual feature Y is obtained according to the contextual attention spectrum and the initial feature spectrum. The dependency between the contextual information is modeled. The enhanced contextual feature is obtained through the contextual relationship spectrum and the contextual feature. The contextual information is allocated and applied to each local feature. The calibrated feature spectrum is obtained through the feature allocation spectrum and the enhanced contextual feature to suppress noise interference and enhance the discriminability of the target feature.

[0150] Step 504, obtaining a target detection characteristic spectrum according to the calibration characteristic spectrum and the cascade characteristic spectrum.

[0151] Step 504 is described in detail below.

[0152] Understandably, see Figure 8 , the calibration feature spectrum is input into a 1×1 convolution layer with a channel number of 2C and then added to the cascade feature spectrum, and then input into a 3×3 convolution layer with a channel number of C to obtain the target detection feature spectrum D i , the characteristic dimensions are C×H×W.

[0153] Through the above steps 501 to 504, the segmentation feature spectra of different semantic levels are embedded in the image feature spectra of different feature scales to generate the final target detection feature spectrum. By combining the segmentation feature spectra of different semantic levels, the target detection feature spectrum can more comprehensively express the image content, including local details and global context information, thereby improving the accuracy of target detection.

[0154] See also Fig.11 In some embodiments, a target detection image is input into a pre-trained target detection model, such as a convolutional neural network (CNN), to obtain an image feature spectrum and a segmentation feature spectrum. According to the segmentation feature spectrum, a pixel segmentation result of the target detection image can be obtained, and the category probability of each pixel in the image is given; segmentation feature spectra of different semantic levels are embedded into image feature spectra of different feature scales to generate target detection feature spectra to achieve multi-level context embedding, and finally, the target detection result is obtained according to the target detection feature spectrum.

[0155] See also Fig.12 In some embodiments, the pre-trained target detection model is jointly optimized based on the segmentation loss function and the target detection loss function, which may include steps 701 to 707:

[0156] Step 701, obtain a training data set and an original detection model.

[0157] The training data set includes multiple training sample images.

[0158] Step 702: Perform target detection on the training data set based on the original detection model to obtain a sample detection result for each training sample image.

[0159] Step 703, calculating the target classification loss function of the original detection model based on the classification result of each detection target in each sample detection result.

[0160] Step 704, calculating the regression loss function of the original detection model based on the predicted box position of each detection target in each sample detection result.

[0161] Step 705: Calculate the segmentation loss function of the original detection model based on the classification result of each pixel in each sample detection result.

[0162] Step 706: Calculate the target detection loss function based on the target classification loss function and the regression loss function.

[0163] Step 707: jointly optimize the original detection model based on the target detection loss function and the segmentation loss function to obtain a target detection model.

[0164] Steps 701 to 707 are described in detail below.

[0165] It is understandable that the training data set includes multiple training sample images, and the target detection loss function L det And the segmentation loss function L seg The original detection model is jointly trained to achieve end-to-end network optimization. The overall loss function is:

[0166] L=L det +λ seg L seg

[0167] Among them, λ seg Used to control the target detection loss L det With pixel segmentation loss L seg balance between.

[0168] It can be understood that the regression loss function of the original detection model is calculated based on the predicted box position of each detection target in each sample detection result, and the target classification loss function of the original detection model is calculated based on the classification result of each detection target in each sample detection result. cls And the regression loss function L reg The target detection loss function is obtained, and the specific calculation is as follows:

[0169]

[0170] Among them, the target classification loss function uses classification cross entropy loss to optimize the category of the target object of interest. The regression loss function uses SmoothL1 loss to regress the offset position of the bounding box. i,l represents the probability predicted by the original detection model that the i-th prediction box belongs to the l-th class object, t i is the corresponding bounding box position offset. i,l ∈{0,1} and t i * ∈{x,y,w,h} represents the real manually annotated target category label and bounding box position respectively. i =1 means that the i-th prediction box belongs to the l-th class object, otherwise, it does not belong to this class object. x, y, w, h represent the center point coordinates, width, and height of the bounding box respectively. N cls and N reg Represents the number of samples N for classification and regression respectively l Represents the number of target categories contained in the dataset. SmoothL1 loss can be expressed as:

[0171]

[0172] The pixel segmentation loss function uses pixel-level cross entropy loss to optimize the prediction of each pixel category in the image. The specific calculation is as follows:

[0173]

[0174] Where T represents the number of pixels contained in the training sample image. is the pixel segmentation label. It means that the t-th pixel belongs to the l-th class object. Otherwise, it does not belong to this class object. It represents the probability that the t-th pixel predicted by the segmentation feature dynamic decoding network belongs to the l-th class object.

[0175] Through the above steps 701 to 707, joint optimization is performed based on the segmentation loss function and the target detection loss function to pre-train the target detection model, allowing the model to simultaneously learn target detection and pixel-level segmentation tasks within a unified framework, thereby achieving end-to-end training. The introduction of the target detection task provides additional contextual information for the segmentation task, which helps the model to better understand the relationship between the target and the background, thereby improving the segmentation performance.

[0176] See also Fig.13, the embodiment of the present application also provides a target detection device based on feature fusion, which can implement the above-mentioned target detection method based on feature fusion, and the device includes: an acquisition module, used to acquire a target detection image and a target detection model;

[0177] An image feature spectrum module is used to input the target detection image into the image feature encoding network of the target detection model, perform multi-scale feature extraction on the target detection image, and obtain image feature spectra of different feature scales;

[0178] The segmentation feature spectrum module is used to input the image feature spectrum into the segmentation feature dynamic decoding network of the target detection model, perform multi-level semantic segmentation on the target detection image according to the image feature spectrum of different feature scales, and obtain segmentation feature spectrum of different semantic levels;

[0179] The detection result module is used to input the image feature spectrum and the segmentation feature spectrum into the detection network of the target detection model, and obtain the target detection result according to the image feature spectrum and the segmentation feature spectrum.

[0180] The specific implementation of the target detection device based on feature fusion is basically the same as the specific implementation of the target detection method based on feature fusion described above, and will not be repeated here. On the premise of meeting the requirements of the embodiments of the present application, the target detection device based on feature fusion can also be provided with other functional modules to implement the target detection method based on feature fusion in the above embodiments.

[0181] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned target detection method based on feature fusion when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0182] See also Fig.14 , Fig.14 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0183] The processor 1401 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0184] The memory 1402 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1402 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1402, and the processor 1401 calls and executes the target detection method based on feature fusion in the embodiment of this application;

[0185] Input / output interface 1403, used to implement information input and output;

[0186] The communication interface 1404 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0187] A bus 1405 that transmits information between the various components of the device (e.g., the processor 1401, the memory 1402, the input / output interface 1403, and the communication interface 1404);

[0188] The processor 1401 , the memory 1402 , the input / output interface 1403 and the communication interface 1404 are connected to each other in communication within the device via a bus 1405 .

[0189] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned target detection method based on feature fusion is implemented.

[0190] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0191] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0192] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0193] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0194] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0195] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0196] It should be understood that in the present application, "at least one (item)" and "several" refer to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0197] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0198] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0199] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0200] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0201] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A target detection method based on feature fusion, characterized in that: include: Obtain target detection images and target detection models; Inputting the target detection image into the image feature encoding network of the target detection model, performing multi-scale feature extraction on the target detection image, and obtaining image feature spectra of different feature scales; Inputting the image feature spectrum into the segmentation feature dynamic decoding network of the target detection model, performing multi-level semantic segmentation on the target detection image according to the image feature spectrum of different feature scales, and obtaining the segmentation feature spectrum of different semantic levels; The image feature spectrum and the segmentation feature spectrum are input into the detection network of the target detection model, and the target detection result is obtained according to the image feature spectrum and the segmentation feature spectrum.

2. The target detection method based on feature fusion according to claim 1, characterized in that: The step of inputting the image feature spectrum and the segmentation feature spectrum into the detection network of the target detection model and obtaining a target detection result according to the image feature spectrum and the segmentation feature spectrum comprises: Inputting the image feature spectrum and the segmentation feature spectrum into a detection network of the target detection model; Embedding the segmentation feature spectra at different semantic levels into the image feature spectra at different feature scales to generate a target detection feature spectrum; The target detection result is obtained according to the target detection feature spectrum.

3. The target detection method based on feature fusion according to claim 1, characterized in that: The image feature encoding network includes a convolutional layer and a residual layer connected in sequence, wherein the residual layer includes a plurality of residual blocks connected in sequence with successively decreasing feature extraction scales, and the multi-scale feature extraction is performed on the target detection image to obtain image feature spectra of different feature scales, including: Inputting the target detection image into the convolution layer for feature extraction to obtain detection input data; The detection input data is input into the residual layer, and feature extraction is performed based on a plurality of residual blocks connected in sequence and having successively decreasing feature extraction scales, so as to obtain the image feature spectrum corresponding to each residual block.

4. The target detection method based on feature fusion according to claim 1, characterized in that: The segmentation feature dynamic decoding network includes a dynamic skip-layer fusion unit, and the target detection image is subjected to multi-level semantic segmentation according to the image feature spectra of different feature scales to obtain the segmentation feature spectra of different semantic levels, including: Determining the resolution ranking based on the high and low relationship between the characteristic scales corresponding to the image characteristic spectrum; For the image feature spectrum of the lowest feature scale in the resolution sorting, a convolution operation is performed on the image feature spectrum of the lowest feature scale based on the dynamic layer skipping fusion unit to obtain the segmentation feature spectrum corresponding to the image feature spectrum; For the image feature spectrum of the non-lowest feature scale in the resolution sorting, the image feature spectrum of the non-lowest feature scale is aggregated with the segmentation feature spectrum corresponding to the corresponding next-level feature scale based on the dynamic skip-layer fusion unit to obtain the segmentation feature spectrum corresponding to the image feature spectrum of the non-lowest feature scale.

5. The target detection method based on feature fusion according to claim 2, characterized in that: The detection network includes a multi-level context embedding unit, and the embedding of the segmentation feature spectra at different semantic levels into the image feature spectra at different feature scales to generate the target detection feature spectra includes: For the image feature spectrum at each feature scale, cascading the image feature spectrum with the segmentation feature spectrum at the corresponding semantic level to obtain a cascade feature spectrum; Performing a convolution operation on the cascade characteristic spectrum to obtain an initial characteristic spectrum; Inputting the initial feature spectrum into the multi-level context embedding unit to obtain a calibration feature spectrum; A target detection characteristic spectrum is obtained according to the calibration characteristic spectrum and the cascade characteristic spectrum.

6. The target detection method based on feature fusion according to claim 5, characterized in that: The multi-level context embedding unit includes an attention convolution layer, a relational reasoning convolution layer and a feature allocation convolution layer, and the initial feature spectrum is input into the multi-level context embedding unit to obtain a calibration feature spectrum, including: Obtaining a contextual attention spectrum according to the attention convolution layer and the cascade feature spectrum, and obtaining contextual features according to the contextual attention spectrum and the initial feature spectrum; Obtaining a contextual relationship spectrum based on the relationship reasoning convolution layer and the contextual features, and obtaining enhanced contextual features according to the contextual relationship spectrum and the contextual features; A feature allocation spectrum is obtained according to the feature allocation convolution layer and the cascade feature spectrum, and the calibration feature spectrum is obtained according to the feature allocation spectrum and the enhanced context feature.

7. The target detection method based on feature fusion according to claim 1, characterized in that: The pre-trained target detection model is jointly optimized based on the segmentation loss function and the target detection loss function, specifically including: Obtaining a training data set and an original detection model; wherein the training data set includes a plurality of training sample images; Performing target detection on the training data set based on the original detection model to obtain a sample detection result for each of the training sample images; Calculating the target classification loss function of the original detection model based on the classification result of each detection target in each sample detection result; Calculating the regression loss function of the original detection model based on the predicted box position of each detection target in each sample detection result; Calculating the segmentation loss function of the original detection model based on the classification result of each pixel in each sample detection result; Calculate the target detection loss function based on the target classification loss function and the regression loss function; The original detection model is jointly optimized based on the target detection loss function and the segmentation loss function to obtain the target detection model.

8. A target detection device based on feature fusion, characterized in that: include: An acquisition module, used to acquire target detection images and target detection models; An image feature spectrum module is used to input the target detection image into the image feature encoding network of the target detection model, perform multi-scale feature extraction on the target detection image, and obtain image feature spectra of different feature scales; A segmentation feature spectrum module is used to input the image feature spectrum into the segmentation feature dynamic decoding network of the target detection model, perform multi-level semantic segmentation on the target detection image according to the image feature spectrum of different feature scales, and obtain the segmentation feature spectrum of different semantic levels; The detection result module is used to input the image feature spectrum and the segmentation feature spectrum into the detection network of the target detection model, and obtain the target detection result according to the image feature spectrum and the segmentation feature spectrum.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the target detection method based on feature fusion according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the target detection method based on feature fusion according to any one of claims 1 to 7 is implemented.

Images