A sketch-based gabor feature based object detection method

Abstract

The application relates to the technical field of image processing, in particular to a target detection method based on sketch Gabor features, which comprises the following steps: drawing an object sketch; performing discrete Fourier transform-Gabor filter-discrete inverse Fourier transform on the sketch in a data-free mode to obtain a sketch Gabor feature; performing edge extraction and subgraph segmentation on a robot shooting picture, and performing discrete Fourier transform-Gabor filter-discrete inverse Fourier transform on the segmented subgraph to obtain a robot shooting picture Gabor feature; and comparing the Gabor features of the picture through similarity measurement. In the data-free mode, the Gabor feature can well retain the sketch detail features; in the data set mode, different Gabor filter parameters are trained for different categories of objects through an adaptive Gabor parameter network, and better inference performance can be achieved under the condition of fewer data sets.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a target detection method based on sketch Gabor features. Background Technology

[0002] Currently, there are two main methods for robot object search: one is for humans to manually control the robot to search for objects in a scene; the other is to use object detection algorithms, where images of the objects to be searched are input into the algorithm for training, and the algorithm then searches for the specified objects. However, in real-world scenarios, there are often no images of the objects to be searched, or the number of samples is too small, causing the robot to be unable to automatically search for the specified objects.

[0003] Currently, most sketch feature extraction algorithms rely on traditional methods with manually designed parameters. Because sketches are highly abstract, traditional point and line features cannot extract complex sketch features. Furthermore, deep neural network-based algorithms require large datasets for training and deep layers to learn abstract features from images. Therefore, it is necessary to address the limitations of traditional algorithms in extracting abstract sketch features and the issues of deep neural networks requiring large datasets and exhibiting poor inference performance. Summary of the Invention

[0004] To address the shortcomings of existing algorithms, this invention proposes a model based on both dataset-based and dataset-less modes. In the dataset-less mode, Gabor features can effectively preserve sketch details. In the dataset-based mode, an adaptive Gabor parameter network is used to train different Gabor filter parameters for different object categories, achieving better inference performance with less dataset.

[0005] The technical solution adopted in this invention is: a target detection method based on sketch Gabor features, comprising the following steps:

[0006] Step 1: Draw a sketch of the object;

[0007] Step 2: In data-free mode, perform Discrete Fourier Transform-Gabor Filter-Inverse Discrete Fourier Transform on the sketch to obtain Gabor features of the sketch; perform edge extraction and sub-image segmentation on the robot-captured image, and perform Discrete Fourier Transform-Gabor Filter-Inverse Discrete Fourier Transform on the segmented sub-image to obtain Gabor features of the robot-captured image; compare the Gabor features of the images by similarity measurement to determine whether the robot-captured image includes the objects in the sketch.

[0008] Furthermore, it also includes: Step 3: In the data-rich mode, perform Discrete Fourier Transform-Gabor Filter-Inverse Discrete Fourier Transform on the sketch to obtain the sketch Gabor features; input the sketch into a Gabor parameter network to train and obtain Gabor filter parameters; the sketch Gabor features are processed through a feature transformation residual network to obtain the sketch feature vector; input the object image and the robot-captured image into a feature extraction convolutional network; the feature vector of the object image is used to update the parameters of the feature extraction convolutional network; and the feature vector of the robot-captured image is output; compare the feature vectors of the images through a similarity metric to determine whether the robot-captured image includes the object in the sketch.

[0009] Furthermore, step two includes:

[0010] S1. Draw a sketch of the object to be detected. s The sketch needs to include the object's outline and main geometric features. Before using the Gabor filter, the sketch I... s Perform a two-dimensional discrete Fourier transform to convert the image to the frequency domain;

[0011] The formula for the two-dimensional discrete Fourier transform is:

[0012]

[0013] Where (x,y) represents the pixel coordinates of the image, W and H are the width and length of the image, respectively, and F(u,v) represents the value corresponding to the transformed pixel coordinates;

[0014] S2. Use k two-dimensional Gabor filters to obtain the Gabor feature vector g of the object sketch. si The formula for a two-dimensional Gabor filter is as follows:

[0015]

[0016] Where, the coordinates (u', v') represent the original coordinates (u, v) after rotation by an angle θ in the standard coordinate system, ω represents the peak response frequency of the filter, and σ x σ represents the frequency bandwidth. y Represents angular bandwidth, used to extract sketch features in different directions; ω, σ x σ y The parameters are preset according to the object features to be searched, and k Gabor filters are used to extract sketch feature vectors g in different directions. si Where i∈{1,2,...,k}; #

[0017] S3. Convolve the sketch feature vectors after the two-dimensional discrete Fourier transform using a Gabor filter. This process can be represented as:

[0018] F i '(u,v)=g si *F(u,v),i∈(1,2,...,k) (3)

[0019] Among them, g si Let be the Gabor eigenvectors of the sketch, and F(u,v) be the two-dimensional discrete Fourier transform.

[0020] After convolution, a two-dimensional inverse Fourier transform is used to restore the image, resulting in k Gabor feature maps f of the sketch. i The formula for the two-dimensional inverse Fourier transform is:

[0021]

[0022] S4. The robot takes pictures of the object to obtain images, extracts the edge contours from the object images using the Canny algorithm, and uses anchor boxes to segment the contour map into m sub-images I. cn n∈(1,2,...,m), the size of the sub-image is set according to the size of the object to be found; the size of the sub-image is set according to the size, dimensions and pixels of the captured image;

[0023] Canny algorithm for obtaining image edge contours: Canny JA Computational Approach to Edge Detection[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,1986,PAMI-8(6):679-698.

[0024] S5. Calculate subgraph I using the methods in steps S2 and S3. cn Gabor features h ni ', where ni represents the Gabor feature in the i-th direction of the n-th subgraph;

[0025] S6. Use cosine similarity to compare sketch Gabor features f i 'Subgraph Gabor features h ni The similarity between ', when f i 'and h ni When the cosine similarity between two subgraphs exceeds the threshold α, the item corresponding to that subgraph is considered to be the object to be found.

[0026] Furthermore, the formula for calculating cosine similarity is:

[0027]

[0028] Where ||·|| represents the modulus of the vector;

[0029] If all subgraphs have Gabor features h ni 'With sketch Gabor features f i If the cosine similarity between the two objects is less than the threshold α, it means that there is no object to be searched in the robot's current field of vision, and the robot moves its chassis or gimbal to search for the next area.

[0030] If there are multiple subgraphs with Gabor features h ni 'With sketch Gabor features f i If the cosine similarity between the two subgraphs is greater than the threshold α, and the subgraphs are adjacent, it means that the dividing line will just separate the object to be searched when the subgraphs are segmented. In this case, the image of the merged adjacent subgraphs will be used as the object to be searched.

[0031] Furthermore, step three includes:

[0032] A1. Draw a sketch of the object to be detected. s The sketch is input into the Gabor parameter network to train the Gabor filter parameters;

[0033] Furthermore, the Gabor parameter network is used to learn Gabor filter parameters for different object categories, including peak response frequency ω and frequency bandwidth σ. x and angular bandwidth σ y The number of Gabor filters is k. The Gabor parameter network consists of convolutional layers with a kernel size of 3, ReLU activation layers, and max pooling layers with a parameter of 2. There are three layers for each type. Finally, 3k Gabor filter parameters are output through a fully connected layer. It can adaptively generate Gabor filters with different parameters according to different object categories.

[0034] A2. Sketch I s For sketch I s Perform a two-dimensional discrete Fourier transform to convert the image to the frequency domain;

[0035] A3. Use k two-dimensional Gabor filters to obtain the Gabor feature vector of the sketch;

[0036] A4. Use a Gabor filter to process the sketch eigenvector g after its two-dimensional discrete Fourier transform. si Convolution is performed, and then a two-dimensional inverse Fourier transform is used to restore the image, resulting in k Gabor feature maps f of the sketch. i ';

[0037] A5. Extract the k Gabor feature maps f from the sketch. i'Input feature transformation residual network, which includes but is not limited to using ResNet50 residual network, to transform abstract multi-level Gabor features into feature vectors and output N-dimensional sketch feature vectors;

[0038] A6. The dataset includes a set of object images corresponding to the sketches and images taken by the robot. The object image set has corresponding labels for the objects. Input the object images and robot images into a feature extraction convolutional network. The feature extraction convolutional network includes, but is not limited to, ResNet101, to obtain the feature vector of the N-dimensional robot image.

[0039] The feature extraction convolutional network is trained using a set of object images and its parameters are updated. Objects with similarity higher than the first similarity threshold (threshold tending to 1) are considered the same object, while objects with similarity lower than the second similarity threshold (threshold tending to 0) are considered different objects. Images taken by the robot are input into the trained feature extraction convolutional network to obtain an N-dimensional feature vector for similarity measurement.

[0040] In the dataset mode, the cosine similarity between two feature vectors of the same dimension is compared. The closer the value is to 1, the more similar the geometric features are between the sub-graph and the sketch. When training with a dataset, the similarity of the same objects in the label is 1, and the similarity of different objects is 0. The neural network parameters are updated during the training process.

[0041] A7. Use cosine similarity to compare the similarity between the feature vector of the N-dimensional sketch and the feature vector of the N-dimensional robot image, and compare the cosine similarity between the sketch and the robot image with the threshold α to confirm whether the corresponding object in the sketch has been found.

[0042] Two feature vectors are obtained from the robot's captured images and hand-drawn sketches after training. A threshold α is set based on the sketch drawing level and the imaging effect of the robot's sensor. If the sub-images are not adjacent, such as if there is only one object to be searched, the sub-image with the highest cosine similarity is selected as the object to be searched. Otherwise, all sub-images with similarity exceeding the threshold α are marked as the object to be searched.

[0043] The beneficial effects of this invention are:

[0044] 1. Using Gabor filters to extract sketch features can extract more abstract sketch features compared to traditional methods such as SIFT;

[0045] 2. It solves the problems of deep neural networks relying on large datasets and poor training and inference performance by using an adaptive Gabor filter, which can quickly extract sketch features in real time;

[0046] 3. This invention takes into account both scenarios with and without datasets, making the algorithm universally applicable. Attached Figure Description

[0047] Figure 1 This is a flowchart of the target detection method based on sketch Gabor features of the present invention;

[0048] Figure 2 This is a flowchart of the object detection method based on sketch Gabor features without a dataset according to the present invention.

[0049] Figure 3 This is a flowchart of the object detection method based on sketch Gabor features with a dataset according to the present invention.

[0050] Figure 4 This is a diagram of the Gabor parameter network structure of the present invention;

[0051] Figure 5 This is a comparison diagram of the similarity between the subgraph features and Gabor features of the present invention. Detailed Implementation

[0052] The present invention will be further described below with reference to the accompanying drawings and embodiments. The drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0053] like Figure 1 As shown, a target detection method based on sketch Gabor features includes the following steps:

[0054] Step 1: Draw a sketch of the object;

[0055] Step 2: In data-free mode, perform Discrete Fourier Transform-Gabor Filter-Inverse Discrete Fourier Transform on the sketch to obtain Gabor features of the sketch; perform edge extraction and sub-image segmentation on the robot-captured image, and perform Discrete Fourier Transform-Gabor Filter-Inverse Discrete Fourier Transform on the segmented sub-image to obtain Gabor features of the robot-captured image; compare the Gabor features of the images by similarity measurement to determine whether the robot-captured image includes the objects in the sketch.

[0056] Step 3: In data-rich mode, perform Discrete Fourier Transform-Gabor Filter-Inverse Discrete Fourier Transform on the sketch to obtain Gabor features. Input the sketch into a Gabor parameter network to train Gabor filter parameters. The sketch Gabor features are then processed through a feature transformation residual network to obtain sketch feature vectors. Input the object image and the robot-captured image into a feature extraction convolutional network. The feature vector of the object image is used to update the parameters of the feature extraction convolutional network, outputting the feature vector of the robot-captured image. Compare the feature vectors of the images using a similarity metric to determine whether the robot-captured image includes the object in the sketch.

[0057] like Figure 2 The flowchart for object detection methods using sketch Gabor features without a dataset includes:

[0058] S1. Draw a sketch of the object to be detected. s The sketch needs to include the object's outline and main geometric features. Before using the Gabor filter, the sketch I... s Perform a two-dimensional discrete Fourier transform to convert the image to the frequency domain;

[0059] The formula for the two-dimensional discrete Fourier transform is:

[0060]

[0061] Where (x,y) represents the pixel coordinates of the image, W and H are the width and length of the image, respectively, and F(u,v) represents the value corresponding to the transformed pixel coordinates;

[0062] S2. Use k two-dimensional Gabor filters to obtain the Gabor feature vector g of the object sketch. si The formula for a two-dimensional Gabor filter is as follows:

[0063]

[0064] Where, the coordinates (u', v') represent the original coordinates (u, v) after rotation by an angle θ in the standard coordinate system, ω represents the peak response frequency of the filter, and σ x σ represents the frequency bandwidth. y Represents angular bandwidth, ω, σ x σ y The parameters are preset according to the object features to be searched, and k Gabor filters are used to extract sketch feature vectors g in different directions. si where i∈{1,2,...,k};

[0065] In this embodiment, to balance accuracy and computational complexity, k is typically set to 6 or 8, meaning a Gabor filter is used to extract sketch features in 6 or 8 directions. Based on the characteristics of hand-drawn sketches, and without a dataset, the peak response frequency ω is set to 0.1, and the frequency bandwidth σ... x The value is 1e-3, and the angular bandwidth σ y The value is 3×1e-3, and the value needs to be modified according to the characteristics of the hand-drawn sketch and the features of the object being searched.

[0066] S3. Convolve the sketch feature vectors after the two-dimensional discrete Fourier transform using a Gabor filter. This process can be represented as:

[0067] F i '(u,v)=g si*F(u,v),i∈(1,2,...,k) (3)

[0068] The image was restored using a two-dimensional inverse Fourier transform, yielding k Gabor feature maps f of the sketch. i The formula for the two-dimensional inverse Fourier transform is:

[0069]

[0070] S4. The robot obtains an image of the object, extracts the edge contour from the image using the Canny algorithm, and divides the contour image into m sub-images using anchor boxes. cn For n∈(1,2,...,m), the size of the subgraph can be set according to the size of the object to be found, such as Figure 5 The image of the vase is divided into 12 sub-images;

[0071] S5. Calculate subgraph I using steps S2 and S3. cn Gabor features h ni ', where ni represents the Gabor feature in the i-th direction of the n-th subgraph;

[0072] S6. Use cosine similarity to compare sketch Gabor features f i 'Subgraph Gabor features h ni The similarity between ' is determined by setting a threshold α based on the sketch drawing level and the imaging effect of the robot sensor, when f i 'and h ni When the cosine similarity between two objects exceeds the threshold α, the object corresponding to that subgraph is considered to be the object to be found.

[0073] The formula for calculating cosine similarity is:

[0074]

[0075] Where ||·|| represents the modulus of the vector.

[0076] If all subgraphs have Gabor features h ni 'With sketch Gabor features f i If the cosine similarity between the two objects is less than the threshold α, it means that there is no object to be searched in the robot's current field of vision, and the robot moves its chassis or gimbal to search for the next area.

[0077] like Figure 5As shown, the vase image is segmented into six sub-images, and the cosine similarity between the features of each sub-image and the Gabor features exceeds a threshold. Therefore, the six adjacent sub-images are merged as the object to be searched. If the sub-images are not adjacent, such as if there is only one object to be searched, the sub-image with the highest cosine similarity is selected as the object to be searched. Otherwise, the sub-images with similarity exceeding the threshold α are all marked as the objects to be searched.

[0078] like Figure 3 The flowchart for a target detection method using sketch Gabor features with a dataset includes the following steps:

[0079] Some datasets learn image features of different categories through neural networks to improve recognition accuracy. The sketch dataset uses SHREC22, and the real scene image dataset uses COCO. The neural network includes: Gabor parameter network, feature transformation residual network, and feature extraction network.

[0080] A1. Draw a sketch of the object to be detected. s The sketch is input into the Gabor parameter network to train the Gabor filter parameters;

[0081] like Figure 4 This is a diagram of the Gabor parameter network structure. The Gabor parameter network is used to learn the Gabor filter parameters for different object categories, including the peak response frequency ω and the frequency bandwidth σ. x and angular bandwidth σ y The number of Gabor filters is k. The Gabor parameter network consists of convolutional layers with a kernel size of 3, ReLU activation layers, and max pooling layers with a parameter of 2. There are three layers for each type. Finally, 3k Gabor filter parameters are output through a fully connected layer. It can adaptively generate Gabor filters with different parameters according to different object categories.

[0082] A2. Sketch I s For sketch I s Perform a two-dimensional discrete Fourier transform to convert the image to the frequency domain;

[0083] A3. Use k two-dimensional Gabor filters to obtain the Gabor feature vector of the sketch;

[0084] A4. Use a Gabor filter to process the sketch eigenvector g after its two-dimensional discrete Fourier transform. si Convolution is performed, and then a two-dimensional inverse Fourier transform is used to restore the image, resulting in k Gabor feature maps f of the sketch. i ';

[0085] A5. Extract the k Gabor feature maps f from the sketch. i'Input feature transformation residual network, which uses ResNet50 residual network, transforms abstract multi-level Gabor features into feature vectors and outputs N-dimensional sketch feature vectors;

[0086] A6. The dataset includes a set of object images corresponding to the sketch and images taken by the robot. The set of object images has corresponding labels. Input the object images and images taken by the robot into a feature extraction convolutional network. The feature extraction convolutional network includes, but is not limited to, ResNet101, to obtain the feature vector of the N-dimensional robot image.

[0087] A7. Use cosine similarity to compare the similarity between the feature vector of the N-dimensional sketch and the feature vector of the N-dimensional robot image. When the cosine similarity between the sketch and the object photo exceeds the threshold, the object corresponding to the sketch is considered to be the object to be found.

[0088] If the cosine similarity between the feature vector of the N-dimensional sketch and the feature vector of the N-dimensional robot image is less than the threshold α, it means that there is no object to be searched in the robot's current field of vision, and the robot moves its chassis or gimbal to search for the next area.

[0089] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A target detection method based on sketch Gabor features, characterized in that, Includes the following steps: Step 1: Draw a sketch of the object; Step 2: In data-free mode, perform Discrete Fourier Transform-Gabor Filter-Inverse Discrete Fourier Transform on the sketch to obtain the Gabor features of the sketch; Edge extraction and sub-image segmentation are performed on images captured by the robot. The segmented sub-images are then subjected to Discrete Fourier Transform-Gabor filter-Inverse Discrete Fourier Transform to obtain Gabor features of the robot-captured images. By comparing the Gabor features of images using similarity metrics, it can be determined whether the images taken by the robot include objects from the sketch. Step two includes the following details: S1. Draw a sketch of the object to be detected. On the sketch Perform a two-dimensional discrete Fourier transform to convert the image to the frequency domain; S2. Use k Gabor filters to obtain Gabor feature vectors of the object sketch. ; S3. Convolve the Gabor feature vectors of the sketch after the two-dimensional discrete Fourier transform using a Gabor filter to obtain the transformed sketch; then restore the transformed sketch using a two-dimensional inverse Fourier transform to obtain k Gabor feature maps. ; S4. The robot obtains an image of the object, extracts the edge contour from the image using the Canny algorithm, and divides the contour image into m sub-images using anchor boxes. ; S5. Calculate m subgraphs using steps S2 and S3. Gabor feature map ,in, ni Represents the nth subgraph. i Gabor features in each direction; S6. Use cosine similarity to compare sketch Gabor features. Subgraph Gabor Features The similarity, and through similarity and threshold Compare and confirm whether the corresponding object in the sketch has been found; The formula for cosine similarity is: （5）； in This represents the magnitude of the vector; If Gabor features of all subgraphs With sketch Gabor features The cosine similarity between them is less than the threshold. This indicates that there is no object to be searched in the robot's current field of vision.

2. The target detection method based on sketch Gabor features according to claim 1, characterized in that, It also includes the following steps: Step 3: In data-rich mode, perform Discrete Fourier Transform-Gabor Filter-Inverse Discrete Fourier Transform on the sketch to obtain Gabor features. Input the sketch into a Gabor parameter network to train Gabor filter parameters. The sketch Gabor features are then processed through a feature transformation residual network to obtain sketch feature vectors. Input the object image and the robot-captured image into a feature extraction convolutional network. The feature vector of the object image is used to update the parameters of the feature extraction convolutional network, outputting the feature vector of the robot-captured image. Compare the feature vectors of the images using a similarity metric to determine whether the robot-captured image includes the object in the sketch.

3. The target detection method based on sketch Gabor features according to claim 1, characterized in that, The formula for the Gabor filter is: （2） Where, coordinates Represents the original coordinates Rotation angle in standard coordinate system The coordinates after, Indicates the peak response frequency. Indicates frequency bandwidth. Indicates angular bandwidth.

4. The target detection method based on sketch Gabor features according to claim 3, characterized in that, The values ​​of the peak response frequency, the frequency bandwidth, and the angular bandwidth are preset values ​​in the no-dataset mode.

5. The target detection method based on sketch Gabor features according to claim 1, characterized in that, The formula for convolving the Gabor eigenvectors of the sketch is: （3） in, This represents the Gabor eigenvectors of the sketch. It is a two-dimensional discrete Fourier transform.

6. The target detection method based on sketch Gabor features according to claim 2, characterized in that, Step three includes: A1. Draw a sketch of the object to be detected. The sketch is input into the Gabor parameter network to train the Gabor filter parameters; A2. On the sketch Perform a two-dimensional discrete Fourier transform to convert the image to the frequency domain; A3. Use k two-dimensional Gabor filters to obtain the Gabor feature vector of the sketch; A4. Using a Gabor filter Convolve the sketch feature vectors after the 2D Discrete Fourier Transform, and then use the 2D Inverse Fourier Transform to restore the image, obtaining k Gabor feature maps of the sketch. ; A5. Extract k Gabor feature maps from the sketch. Input a feature transformation residual network and output an N-dimensional sketch feature vector. A6. Input the object image and the robot image into the feature extraction convolutional network, train the feature extraction convolutional network using the object image set, update the parameters of the feature extraction convolutional network, input the robot image into the trained feature extraction convolutional network, and obtain the feature vector of the N-dimensional robot image. A7. Use cosine similarity to compare the similarity between the feature vectors of the N-dimensional sketch and the feature vectors of the N-dimensional robot image, and then compare the cosine similarity between the sketch and the robot image with a threshold. Compare and confirm whether the corresponding object in the sketch has been found.

7. The target detection method based on sketch Gabor features according to claim 2, characterized in that, The sketch Gabor parameter network consists of convolutional layers with a kernel size of 3, ReLU activation layers, and max pooling layers with a parameter of 2. There are 3 convolutional layers, 3 activation layers, and 3 fully connected layers to output 3k Gabor filter parameters.

Images