Significant target detection method based on characteristic decomposition and image reunion
By using feature decomposition and image reconciliation methods in significant object detection, combined with ResNet50 and Fourier transform technology, the problems of detection accuracy and accuracy in complex scenarios in the prior art are solved, and more efficient multi-objectives are achieved Task detection.
Patent Information
- Application Number
- CN202411382337.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-30
- Publication Date
- 2025-02-18
AI Technical Summary
Existing significant object detection methods are difficult to achieve accurate and efficient detection when facing complex and dense object categories, especially in scenarios with higher segmentation accuracy and more accurate classification.
Using a method based on feature decomposition and image reconciliation, the object detection model is constructed through the ResNet50 backbone network combining two-dimensional discrete Fourier transform, spectrum centralization, low-frequency filtering, inverse Fourier transform, attention mechanism and feature encoder. Achieve the acquisition of the target instance segmentation map.
The detection accuracy and accuracy of multi-objective tasks in complex scenarios are improved. Through feature decomposition and image reconciliation technology, useful feature information is extracted, and feature information is enhanced through attention mechanism, which improves the detection effect.
Smart Images
Figure CN119478341A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to a salient object detection method based on feature decomposition and image re-aggregation. Background Art
[0002] Salient Object Detection (SOD) aims to detect the most attention-grabbing objects in an image. Salient objects have very obvious feature edges. Since the properties of the salient objects to be detected are highly separated from the background properties, there is an obvious distinction in human visual perception. Salient object detection is widely studied in the field of computer vision. For example, it has strong practical value in the fields of general object retrieval, image classification, object recognition, modality matching, intelligent classification and instance segmentation. However, facing the complex and dense object categories in image information, it is a very challenging task to accurately and efficiently detect different salient objects in the image. Salient objects have relatively obvious characteristic attributes such as size, color, shape and appearance. Although conventional object detection algorithms can easily detect the contour of objects, it is difficult to achieve accurate segmentation and object classification. Better detection methods are needed for scenes with higher segmentation accuracy and more accurate classification.
[0003] Salient object detection has extremely high academic and application value in many fields. Through model prediction, many salient objects can be classified and cut. For example, salient object detection can be used for model cutting in the manufacturing field, image separation in the biomedical field, plant labeling in the ecological agriculture field, and garbage classification in daily life. Therefore, salient object detection has extremely high application value in intelligent classification and instance segmentation. At present, the mainstream salient object detection methods include boundary-guided detection, lightweight coding detection, channel attention module detection, etc. These detection methods largely analyze the characteristics of salient objects from the time domain. It is difficult to achieve higher and more accurate detection in the face of more complex scenes. Summary of the invention
[0004] In view of this, the present invention provides a salient object detection method based on feature decomposition and image re-aggregation to solve the above problems.
[0005] The present invention provides a salient target detection method based on feature decomposition and image re-aggregation, comprising: obtaining an input image, preprocessing the input image; using ResNet50 as a backbone network, constructing a target detection model based on two-dimensional discrete Fourier transform, spectrum centering, low-frequency filtering, two-dimensional discrete Fourier inverse transform, attention mechanism and feature encoder; inputting the preprocessed image into the target detection model to obtain a target instance segmentation map, comprising: processing the input image through the ResNet50 backbone network, outputting a feature map, and performing Two-dimensional discrete Fourier transform processing; performing spectrum centering, low-frequency filtering and inverse transform processing on the result of the two-dimensional discrete Fourier transform processing to obtain a new feature map; encoding the new feature map through the feature encoder to obtain a coarse instance segmentation map; strengthening the feature texture and edge information of the coarse instance segmentation map and the new feature map through the attention mechanism to obtain a standard instance segmentation map; cascading and splicing the standard instance segmentation map and the new feature map from deep features to shallow features, and outputting them after passing through an activation function to obtain the target instance segmentation map.
[0006] In another implementation of the present invention, the preprocessing includes image cropping, tensor processing, and normalization processing.
[0007] In another implementation of the present invention, the two-dimensional discrete Fourier transform is expressed as:
[0008]
[0009] Among them, W and H represent the width and height of the feature information, f i (x,y) is a mapping of feature information.
[0010] In another implementation of the present invention, the inverse transformation is expressed as:
[0011]
[0012] Among them, F CNi (x, y) is the mapping of feature information after two-dimensional discrete Fourier transform.
[0013] In another implementation of the present invention, it also includes: using the two-class problem classification loss function and the intersection-over-union loss function to construct the loss function of the image to be detected in the target detection process:
[0014]
[0015] The classification loss function for the two-class problem is expressed as:
[0016]
[0017] The intersection-over-union loss function is expressed as:
[0018]
[0019] Among them, GT(x,y) represents the true value label and P(x,y) represents the predicted probability.
[0020] In another implementation of the present invention, it also includes: verifying and evaluating the performance of the target detection model through the structural similarity S index, the mean absolute error, the E index of human visual perception, and the weighted F index, and the values of the four evaluation indicators are [0, 1].
[0021] The salient target detection method based on feature decomposition and image re-aggregation of the present invention uses two-dimensional discrete Fourier transform to convert feature information from time domain to frequency domain, extracts useful feature information through the frequency size of the feature information in the frequency domain, uses inverse Fourier transform to restore the useful feature information into an information feature map, and then further enhances the feature information through an attention mechanism module. It also introduces model evaluation indicators to verify the performance of the model, and improves the detection precision and accuracy of multi-target tasks in complex scenes through the combination of multiple technologies. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. By reading the detailed description of the following implementations, the advantages and benefits of the solutions become clear to those skilled in the art. The drawings are only used to illustrate the preferred implementations and are not to be considered as limiting the present invention.
[0023] In the attached picture:
[0024] Figure 1 The figure is a flow chart of a salient object detection method based on feature decomposition and image re-aggregation according to an embodiment of the present invention.
[0025] Figure 2 The figure is a schematic diagram of the structure of a network framework according to an embodiment of the present invention.
[0026] Figure 3 It is a schematic diagram of feature decomposition and feature aggregation according to an embodiment of the present invention.
[0027] Figure 4 Schematic diagram of an attention mechanism module according to an embodiment of the present invention.
[0028] Figure 5 A schematic diagram of a frequency spectrum and a centralization of a frequency spectrum according to an embodiment of the present invention.
[0029] Figure 6 The figure is a schematic diagram of the comparison between the prediction result and the real image according to an embodiment of the present invention. DETAILED DESCRIPTION
[0030] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be described clearly and in detail below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in the field based on the embodiments in the embodiments of the present invention should fall within the scope of protection of the embodiments of the present invention.
[0031] Figure 1 A flowchart of a salient object detection method based on feature decomposition and image regrouping provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, this embodiment mainly includes:
[0032] S101: Obtain an input image and pre-process the input image.
[0033] S102. ResNet50 is used as the backbone network, and a target detection model is constructed based on two-dimensional discrete Fourier transform, spectrum centering, low-frequency filtering, two-dimensional inverse discrete Fourier transform, attention mechanism and feature encoder.
[0034] For example, Figure 2 As shown, the input image I i (i=1,2,3,……,N) The initial state is W×H×C, where W represents width, H represents height, and C represents the number of channels. After image cropping, tensor processing, normalization and other preprocessing operations, it enters the ResNet50 network model cascade Stage 1, Stage 2, Stage 3, Stage 4, and Stage 5 in turn, and each stage outputs feature samples X in turn. 1 , X 2 , X 3 , X 4 , X 5 The stage cascade reduces the resolution of the feature samples output by the next stage and increases the number of channels.
[0035] Feature Sample X i The resolution of (i=1,2,3,4,5) is expressed as:
[0036]
[0037] Feature Sample C i The number of channels (i=1,2,3,4,5) is expressed as:
[0038] Ci =2 k (k=6,8,9,10,11)
[0039] Specifically, the training of the target detection model is as follows:
[0040] Collect salient object training datasets and download salient target datasets MSRA10K, ASD, DUT-OMRON, DUTS, ECSSD / CSSD, HKU-IS, PASCAL-S, SOC, SOD.
[0041] The data sets collected in the present invention are as follows: the MSRA10K data set contains a total of 10,000 images of different scenes. The ADS data set contains a total of 1,000 images of different scenes. The image data of the DUT-OMRON data set also has one or more significant targets and a relatively complex background, and the data set contains a total of 5,168 images. The DUT data set contains a total of 15,572 images. The HKU-IS data set contains a total of 4,447 images. The CSSD data set contains a total of 200 images. The ECSSD data set is an extended data set of the CSSD data set, containing a total of 1,000 images. The SOD data set is a collection of significant object boundaries of the base BSD data set, containing a total of 300 images.
[0042] The training sets in the DUT-OMRON, DUTS, ECSSD / CSSD, HKU-IS, PASCAL-S, SOC, and SOD datasets are used to train the models, and the test sets are used to predict the effects.
[0043] With ResNet50 as the backbone network, the target detection model is constructed using two-dimensional discrete Fourier transform (TDFT), spectrum centering, low-frequency filtering, two-dimensional inverse discrete Fourier transform (TDIFT), attention mechanism (AM), and feature encoder (FT).
[0044] S103, inputting the preprocessed image into the target detection model to obtain a target instance segmentation map, including:
[0045] S1031. Process the input image through the ResNet50 backbone network, output a feature map, and perform a two-dimensional discrete Fourier transform on the feature map.
[0046] Exemplarily, two-dimensional discrete Fourier transform, frequency spectrum centering, and low-frequency filtering are used to obtain the required feature information from the frequency domain space. Feature sample X 2 , X 3 , X 4 , X 5 Perform a two-dimensional discrete Fourier transform to obtain a frequency spectrum (such as Figure 3The TDFT shown in Figure 1 shows that the feature samples are converted from the spatial state to the frequency state. The white bright information in the frequency spectrum represents low-frequency features, with higher energy and slow gradient changes. These low-frequency features represent the parameters such as color and brightness of the original feature samples. The black dark feature information in the frequency spectrum represents high-frequency features, with lower energy and obvious gradient changes. These high-frequency features represent the parameters such as texture and edge in the original feature samples.
[0047] S1032, performing spectrum centering, low-frequency filtering and inverse transformation on the result of the two-dimensional discrete Fourier transform processing to obtain a new feature spectrum.
[0048] Exemplarily, a two-dimensional discrete Fourier transform is performed and then a frequency spectrum centering operation is performed (e.g. Figure 3 As shown in FSC). Figure 5 As shown in the middle right figure, after the transformation, the low-frequency information is concentrated in the center of the spectrum, and the high-frequency information is distributed around the spectrum. The required feature information is the edge and texture details of the high-frequency part. Therefore, after the spectrum is centralized, the low-frequency features will be concentrated in the center of the spectrum, and the high-frequency features will be distributed around the spectrum.
[0049] After the spectrum is centered, a new frequency spectrum F is obtained C , spectrum F C The low-frequency features at the center position are reduced in brightness to obtain the frequency spectrum F CN .
[0050] Frequency spectrum F CN After two-dimensional discrete Fourier inverse transform (such as Figure 3 The TDIFT shown in the figure is restored to obtain a new feature sample FX k (k=1,2,3,4,5), new feature sample FX k Relative to feature sample X i It has more obvious feature texture and edge information.
[0051] S1033, using the feature encoder to generate the new feature map FX k Perform encoding processing to obtain a rough instance segmentation map.
[0052] For example, the new feature sample FX 2 FX 3 FX 4 FX 5 Enter the feature encoder module in turn to obtain a rough instance segmentation map P 1 .
[0053] S1034. Strengthen the feature texture and edge information of the rough instance segmentation map and the new feature map through the attention mechanism to obtain a standard instance segmentation map.
[0054] For example, Figure 4 As shown, adding attention mechanism further enhances the new feature sample FX k The idea of the attention mechanism is similar to the addressing operation. Given a query for an element in the target, the weight coefficient of the value corresponding to each key is obtained by calculating the similarity or correlation between the query and each key, and then the value is weighted and summed to get the final Attention value. The attention mechanism imitates the way humans focus on key information when processing information, focusing only on the required feature information, and greatly reducing irrelevant feature information from entering the field of view.
[0055]
[0056] Among them, L x =||Source||, indicating the length of Source.
[0057] S1035. Concatenate the standard instance segmentation map and the new feature map from deep features to shallow features, and output them after passing through an activation function to obtain the target instance segmentation map.
[0058] For example, the rough instance segmentation spectrogram is combined with the new feature sample FX k The target map is output by cascading deep features to shallow features, and the target map is then combined with the previous new feature sample FX k-1 The cascade outputs the previous target map, and finally the final target instance segmentation map is obtained after the activation function.
[0059] Specifically, the coarse instance segmentation atlas P 1 , P 1 With FX 5 After sampling and splicing operations, we can get a further target map P 2 , P 2 With FX 4 The further target map P after sampling and splicing operations 3 , P 3 With FX 3 The further target map P after sampling and splicing operations 4 , P 4 With FX 2 The further target map P after sampling and splicing operations 5 , P 5 After the Sigmoid activation function and smoothing filter, the prediction result P is output. r .
[0060] The salient target detection method based on feature decomposition and image re-aggregation of the present invention uses two-dimensional discrete Fourier transform to convert feature information from time domain to frequency domain, extracts useful feature information through the frequency size of the feature information in the frequency domain, uses inverse Fourier transform to restore the useful feature information into an information feature map, and then further enhances the feature information through an attention mechanism module. It also introduces model evaluation indicators to verify the performance of the model, and improves the detection precision and accuracy of multi-target tasks in complex scenes through the combination of multiple technologies.
[0061] In another implementation of the present invention, the preprocessing includes image cropping, tensor processing, and normalization processing.
[0062] In another implementation of the present invention, the two-dimensional discrete Fourier transform, that is, the feature information X 2 , X 3 , X 4 , X 5 The transformed result is expressed as:
[0063]
[0064] Among them, W and H represent the width and height of the feature information, f i (x,y) is the feature information X i 's mapping.
[0065] In another implementation of the present invention, the inverse transformation is expressed as:
[0066]
[0067] Among them, F CNi (x, y) is the mapping of feature information after two-dimensional discrete Fourier transform.
[0068] In another implementation of the present invention, it also includes: using a two-class problem classification loss function (BCE) and an intersection-over-union loss function (IoU) to construct a loss function of the image to be detected in the target detection process:
[0069]
[0070] BCE means that there are two kinds of prediction results. When the prediction result is true, the output probability is p, and when the prediction result is false, the output probability is 1-p. The classification loss function of the two-class problem is expressed as:
[0071]
[0072] The intersection-over-union loss function is expressed as:
[0073]
[0074] Among them, GT(x,y) represents the true value label and P(x,y) represents the predicted probability.
[0075] It should be understood that the loss function is calculated from the first round of model training to the last round. The value of the loss function converges with the increase of iteration rounds. When the value of the loss function converges approximately to 0, the training model has reached the best state.
[0076] In another implementation of the present invention, it also includes: verifying and evaluating the performance of the target detection model through the structural similarity S index (S-measure), the mean absolute error (MAE), the E index (E-measure) of human visual perception, and the weighted F index (Weighted F-measure), and the values of the four evaluation indicators are [0, 1].
[0077] For example, the S index refers to extracting three key features from the image, namely brightness, contrast, and structure. σ represents the contrast composition function, μ represents the estimation function of brightness measurement, and C 1 C 2 To avoid μ tending to a constant of 0, L=255, K<<1 is used to indicate stability.
[0078]
[0079] The E index refers to verifying the predictive performance of the model from a human perspective, FM Represents the enhanced alignment matrix used to capture pixel-level matching and image-level matching of binary maps.
[0080]
[0081] The weighted F index refers to a comprehensive evaluation index with weight added.
[0082] pressure represents the precision rate. Recall represents the recall rate.
[0083] TP represents true positive, FP represents false positive, and FN represents false negative.
[0084]
[0085] The absolute mean error refers to the difference between the predicted result and the true value label minus the average value, y pre Represents the predicted value, y GT Represents the true value label.
[0086]
[0087] For the four evaluation indicators, the larger the values of the S, E, and F indicators, the better. The closer the indicator data is to 1, the more accurate the model prediction is. The better the MAE is required to be, the closer the MAE index converges to 0, the smaller the difference between the prediction result and the true value label, and the more accurate and complete the instance segmentation effect is.
[0088] In another implementation of the present invention, the training model starts training with epoch=100, lr=0.0001, batch_size=16, and train_size=352×352 as basic parameters, where epoch represents the training round, lr represents the learning rate, batch_size represents the batch unit, and train_size represents the training size.
[0089] Download the MSRA10K, ASD, DUT-OMRON, DUTS, ECSSD / CSSD, HKU-IS, PASCAL-S, SOC, and SOD data sets, and sort out their respective training sets and test sets. The training set is used to train the network model and obtain the training weight file, and the test set is used to test the training effect of the network model, so as to modify the subsequent parameters to obtain the best training model and model evaluation.
[0090] The organized data sets are stored in the specified path of the project file.
[0091] Debug the training model and start training, and wait for the training to end to obtain the training weights.
[0092] Debug the prediction model and start prediction, wait for log output, check the prediction results and prediction errors, adjust the training parameters again by comparing the predicted image with the original image and the true value label, and start the next training. The final model parameters are determined through multiple rounds of training and prediction.
[0093] The performance of the model is verified through various evaluation indicators.
[0094] like Figure 6 As shown, the test set image prediction and comparison results of the present invention are shown.
[0095] In the present invention, ResNet50 is used as the backbone network to extract the feature information of the image at different stages, and the feature information in different stages is converted into a frequency spectrum using a two-dimensional discrete Fourier transform and the frequency spectrum is centered. The high-frequency feature information and the low-frequency feature information in the frequency spectrum are respectively filtered using a high-frequency filter (HFF) and a low-frequency filter (LFF); after the spectrum is centered, the brightness of the center of the spectrum is reduced by a low-frequency filter, and the feature spectrum is reconstructed by a two-dimensional discrete Fourier inverse transform; the feature information of the reconstructed feature spectrum is enhanced by a feature enhancement module; a feature encoder is used to obtain a rough instance outline of the deep feature information, and the deep feature instance outline is spliced with the feature information output by the feature enhancement module to restore the feature instance step by step, and finally the feature instance is activated to output the detected salient objects.
[0096] The present invention can be applied to model cutting in the field of manufacturing, image separation in the field of biomedicine, plant labeling in the field of ecological agriculture, and garbage classification in daily life.
[0097] In another aspect of the present invention, an electronic device includes: a processor, a memory, a communication bus, and a communication interface.
[0098] in:
[0099] The image processor GPU, memory and communication interface communicate with each other through a communication bus.
[0100] Communication interface, used to communicate with other electronic devices or servers.
[0101] The image processor GPU is used for training and predicting the target detection model, and specifically can execute the steps of any one of the salient target detection methods based on feature decomposition and image re-aggregation in the above embodiments.
[0102] Specifically, the program may include a program code including a computer high-level language.
[0103] The image processor may be an RTX series graphics card. The one or more image processors included in the smart device may be processors of the same type, such as one or more RTX3070s;
[0104] The memory is used to store programs and data sets. The memory may include high-speed RAM memory and SSD solid state disk, for example, at least one disk memory.
[0105] The program can be specifically used to enable the processor to execute the steps of any of the salient target detection methods based on feature decomposition and image reunion described in the embodiments. The specific implementation of each step in the program can refer to the corresponding descriptions in the steps and units executed by any of the salient target detection methods based on feature decomposition and image reunion in the above steps, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the aforementioned method embodiment.
[0106] The above method according to the embodiment of the present invention can be implemented in a server equipped with a central processing unit CPU and an image processing unit COU.
[0107] Thus far, specific embodiments of the present invention have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results.
[0108] It should be noted that all directional indications in the embodiments of the present invention (such as up, down, left, right, back, etc.) are only used to explain the relative position relationship between the components in a certain specific order (as shown in the accompanying drawings). If the specific order changes, the directional indication will also change accordingly.
[0109] In the description of the present invention, the terms "first" and "second" are only used to facilitate the description of different components or names, and cannot be understood as indicating or implying a sequential relationship, relative importance, or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features.
[0110] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.
[0111] It should be noted that although the specific embodiments of the present invention are described in detail in conjunction with the accompanying drawings, it should not be understood as limiting the scope of protection of the present invention. Within the scope described in the claims, various modifications and variations that can be made by those skilled in the art without creative work still belong to the scope of protection of the present invention.
[0112] The examples of the embodiments of the present invention are intended to concisely illustrate the technical features of the embodiments of the present invention so that those skilled in the art can intuitively understand the technical features of the embodiments of the present invention, and are not intended to be improper limitations of the embodiments of the present invention.
[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A salient object detection method based on feature decomposition and image re-aggregation, characterized in that: include: Acquire an input image, and preprocess the input image; ResNet50 is used as the backbone network, and the target detection model is constructed based on two-dimensional discrete Fourier transform, spectrum centering, low-frequency filtering, two-dimensional inverse discrete Fourier transform, attention mechanism and feature encoder; The preprocessed image is input into the target detection model to obtain a target instance segmentation map, including: The input image is processed by the ResNet50 backbone network, a feature map is output, and a two-dimensional discrete Fourier transform is performed on the feature map; Performing spectrum centering, low-frequency filtering and inverse transformation on the result of the two-dimensional discrete Fourier transform processing to obtain a new feature spectrum; The new feature map is encoded by the feature encoder to obtain a rough instance segmentation map; Strengthen the feature texture and edge information of the rough instance segmentation map and the new feature map through the attention mechanism to obtain a standard instance segmentation map; The standard instance segmentation map and the new feature map are cascaded and spliced from deep features to shallow features, and output after passing through an activation function to obtain the target instance segmentation map.
2. The method according to claim 1, characterized in that The preprocessing includes image clipping, tensor processing, and normalization processing.
3. The method according to claim 1, characterized in that The two-dimensional discrete Fourier transform is expressed as: Among them, W and H represent the width and height of the feature information, f i (x,y) is a mapping of feature information.
4. The method according to claim 3, characterized in that The inverse transform is expressed as: Among them, F CNi (x, y) is the mapping of feature information after two-dimensional discrete Fourier transform.
5. The method according to claim 4, characterized in that Also includes: The loss function of the image to be detected in the target detection process is constructed using the two-class problem classification loss function and the intersection-over-union loss function: The classification loss function for the two-class problem is expressed as: The intersection-over-union loss function is expressed as: Among them, GT(x,y) represents the true value label and P(x,y) represents the predicted probability.
6. The method according to claim 1, characterized in that Also includes: The performance of the target detection model is verified and evaluated by the structural similarity S index, mean absolute error, human visual perception E index, and weighted F index. The values of the four evaluation indicators are [0, 1].