Target classification method and device, computer equipment and storage medium
By performing dual-channel feature extraction and fusion processing on multiple targets in the image to be processed, the problem of low classification accuracy in fine-grained target classification is solved, and higher classification accuracy and efficiency are achieved.
Patent Information
- Application Number
- CN202411683345.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-22
- Publication Date
- 2025-02-07
AI Technical Summary
The problem of low classification accuracy in the prior art in fine-grained target classification.
By performing dual-channel feature extraction of multiple targets in the image to be processed, semantic feature information and spatial feature information are obtained, and fusion processing is performed, the feature information to be processed is determined, and finally classification is performed based on these feature information.
It improves the accuracy of fine-grained classification results, reduces classification errors caused by manual participation, speeds up classification speed, and improves classification efficiency.
Smart Images
Figure CN119399549A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a target classification method, device, computer equipment and storage medium. Background Art
[0002] With the development of deep learning technology, deep learning-based object processing algorithms have been widely used in various fields. Taking the object classification algorithm as an example, the object classification algorithm is used to classify different objects in the image to be processed to obtain multiple different types of object sets.
[0003] In practical applications, there are often situations where the visual features of multiple different targets have high similarity. Therefore, it is particularly important to perform fine-grained classification of such targets.
[0004] However, the implementation of fine-grained classification in related technologies has the problem of low classification accuracy. Summary of the invention
[0005] Based on this, it is necessary to provide a target classification method, device, computer equipment and storage medium to address the above technical problems, which can improve the accuracy of fine-grained classification results.
[0006] In a first aspect, an embodiment of the present application provides a method for classifying a target, the method comprising:
[0007] Perform dual-channel feature extraction on multiple targets in the image to be processed to obtain semantic feature information and spatial feature information corresponding to each target; the difference value between different types of targets in each of the multiple targets is less than a preset difference threshold;
[0008] The semantic feature information and the corresponding spatial feature information of each target are fused and processed to determine the feature information to be processed of each target;
[0009] Based on the feature information to be processed of each target, each target is classified into different types of target sets to complete the classification processing.
[0010] In one embodiment, dual-channel feature extraction is performed on multiple targets in the image to be processed to obtain semantic feature information and spatial feature information corresponding to each target, including:
[0011] Perform preprocessing according to each target and determine multiple preprocessed targets;
[0012] Perform grayscale normalization on each preprocessed target to obtain each conversion target;
[0013] The first encoder is used to extract semantic features of each conversion target to obtain semantic feature information of each target; and the second encoder is used to extract spatial features of each conversion target to obtain spatial feature information of each target.
[0014] In one embodiment, preprocessing is performed according to each target to determine a plurality of preprocessed targets, including:
[0015] Performing target transformation processing according to each target to determine multiple targets to be processed; target enhancement processing includes rotation processing, scaling processing and cropping processing;
[0016] De-noising is performed on each target to be processed to obtain each pre-processed target.
[0017] In one embodiment, grayscale normalization is performed on each preprocessed target to obtain each conversion target, including:
[0018] For any preprocessed target, grayscale the preprocessed target to obtain the corresponding grayscale target;
[0019] The initial pixel value of each pixel in the grayscale target is subtracted from the minimum initial pixel value among the pixels, and then divided by the total number of pixels to obtain the current pixel value of each pixel;
[0020] According to the current pixel value of each pixel point, the conversion target corresponding to the preprocessed target is obtained.
[0021] In one of the embodiments, the first encoder is an encoder for semantic feature extraction based on local self-attention mechanism and global self-attention mechanism; the second encoder is an encoder based on local spatial feature extraction and global spatial feature extraction.
[0022] In one embodiment, the semantic feature information and the corresponding spatial feature information of each target are fused to determine the feature information to be processed of each target, including:
[0023] Cascade processing is performed according to the semantic feature information of each target and the corresponding spatial feature information to obtain cascade feature information;
[0024] Performing feature fusion processing on the cascade feature information to obtain feature fusion information of each target; the feature fusion processing includes a first depth-separable convolution operation;
[0025] The feature fusion information is subjected to feature extraction processing to determine feature information to be processed of the target; the feature extraction processing includes a second depthwise separable convolution operation, and the convolution kernel sizes of the first depthwise separable convolution operation and the second depthwise separable convolution operation are different.
[0026] In one embodiment, based on the feature information to be processed of each target, each target is correspondingly classified into different types of target sets to complete the classification processing, including:
[0027] The feature information to be processed and each target are input into the classifier for target classification processing to obtain each target set.
[0028] In a second aspect, an embodiment of the present application provides a target classification device, the device comprising:
[0029] A feature extraction module is used to perform dual-channel feature extraction on multiple targets in the processed image to obtain semantic feature information and spatial feature information corresponding to each target; the difference value between different types of targets in each of the multiple targets is less than a preset difference threshold;
[0030] A fusion processing module is used to fuse the semantic feature information and the corresponding spatial feature information of each target to determine the feature information to be processed of each target;
[0031] The classification processing module is used to classify each target into different types of target sets based on the feature information to be processed of each target to complete the classification processing.
[0032] In a third aspect, an embodiment of the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method of any one of the embodiments of the first aspect when executing the computer program.
[0033] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method of any embodiment in the above-mentioned first aspect are implemented.
[0034] In a fifth aspect, an embodiment of the present application further provides a computer program product, the computer program product comprising a computer program, which, when executed by a processor, implements the steps of the method of any embodiment of the first aspect above.
[0035] The embodiment of the present application provides a target classification method, device, computer equipment and storage medium, the method comprising: performing dual-channel feature extraction on multiple targets in a to-be-processed image, obtaining semantic feature information and spatial feature information corresponding to each target, fusing the semantic feature information and corresponding spatial feature information of each target, determining the to-be-processed feature information of each target, and classifying each target into different types of target sets based on the to-be-processed feature information of each target to complete the classification process, wherein the difference value between different types of targets in the multiple targets is less than a preset difference threshold. The above method can be applied to fine-grained classification scenarios, so that the difficulty of fine-grained classification is reduced, and in the processing process, multi-type feature information of each target, that is, semantic feature information and spatial feature information corresponding to each target, can be obtained, and then accurate classification can be achieved based on the multi-scale feature information of each target, which can improve the accuracy of fine-grained classification; at the same time, the above method does not require manual participation, thereby reducing the classification error caused by manual participation, greatly improving the correctness of fine-grained classification, and also speeding up the speed of fine-grained classification and improving the efficiency of fine-grained classification. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] FIG1 is a diagram of an application environment of a target classification method according to an embodiment;
[0037] FIG2 is a schematic diagram of a process flow of a target classification method in one embodiment;
[0038] FIG3 is a schematic diagram of a flow chart of a target classification method in another embodiment;
[0039] FIG4 is a schematic diagram of a flow chart of a target classification method in another embodiment;
[0040] FIG5 is a schematic diagram of a flow chart of a target classification method in another embodiment;
[0041] FIG6 is a block diagram of a first encoder in one embodiment;
[0042] FIG7 is a block diagram of a second encoder in one embodiment;
[0043] FIG8 is a schematic diagram of a flow chart of a target classification method in another embodiment;
[0044] FIG9 is a block diagram of a target classification device in one embodiment;
[0045] FIG. 10 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0046] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0047] In the field of artificial intelligence, target processing algorithms based on deep learning are widely used in various fields. Taking the target classification algorithm as an example, the target classification algorithm is used to classify different targets in the image to be processed to obtain multiple different types of target sets. In practical applications, there are often situations where the visual features of multiple different targets have a high degree of similarity. Therefore, it is particularly important to perform fine-grained classification on such targets. In the related art, the main method is to first extract feature information of different targets in the image to be processed, and then perform fine-grained classification processing on each target according to the feature information of each target to obtain a fine-grained classification result. However, the implementation of fine-grained classification in the related art will have the problem of low classification accuracy. Based on this, an embodiment of the present application provides a target classification method that can improve the accuracy of fine-grained classification results.
[0048] The target classification method provided in the embodiment of the present application can be applied to the application environment diagram shown in Figure 1. The application environment includes a target classification system, and the target classification system may include a computer device and at least one image acquisition device. Optionally, the computer device and each image acquisition device can be connected for communication, and the communication method can be Wi-Fi, mobile network or Bluetooth connection, etc. Optionally, the above-mentioned computer device can be implemented by, but not limited to, a personal computer, a desktop computer, a laptop computer, a smart phone, and a smart bracelet, and each image acquisition device can be implemented by, but not limited to, a camera, a smart phone, a computer, and a camera. This embodiment does not limit the specific form of the computer device and each image acquisition device. Among them, Figure 1 is an example of a computer device being a personal computer, a target classification system including an image acquisition device, and an image acquisition device being a smart phone. The following introduces the specific process of the target classification method by taking the computer device as the execution subject of the target classification method.
[0049] As shown in FIG2 , it is a schematic diagram of a process flow of a target classification method provided in an embodiment of the present application. The method can be implemented by the following steps:
[0050] S100, performing dual-channel feature extraction on multiple targets in the image to be processed to obtain semantic feature information and spatial feature information corresponding to each target, wherein the difference value between different types of targets in the multiple targets is less than a preset difference threshold.
[0051] In an embodiment of the present application, the above-mentioned image to be processed may be one or more; each image to be processed may include at least one target. Among them, the multiple targets in the above-mentioned image to be processed may be multiple targets in the same image to be processed, or multiple targets in multiple images to be processed; the multiple targets can be understood as regional images of plants, animals, pedestrians, buildings, lesions, electronic devices, etc. in the corresponding image to be processed. Optionally, the types of multiple targets may be the same or different. Optionally, the type of the target may be plants, animals, pedestrians, buildings, electronic devices, etc., which is not limited in the embodiment of the present application.
[0052] It should be noted that if the difference between different types of targets in the multiple targets is less than the preset difference threshold, it can be understood that the difference between the targets of different classes is large and the similarity is large. Optionally, the preset difference threshold can be user-defined or determined based on historical experience.
[0053] In one implementation, a computer device may pre-train a dual-channel feature extraction model, and then input multiple targets in the image to be processed into the dual-channel feature extraction model to perform dual-channel feature extraction on the multiple targets in the image to be processed, and the dual-channel feature extraction model outputs semantic feature information and spatial feature information corresponding to each target. Optionally, the dual-channel feature extraction model may be implemented by at least one of a convolutional neural network model, a fully connected neural network model, a recurrent neural network model, a long short-term memory neural network model, a residual neural network model, and the like.
[0054] In another implementation, the computer device may use a feature extraction algorithm to perform dual-channel feature extraction on multiple targets in the processed image to obtain semantic feature information and spatial feature information corresponding to each target. Optionally, the feature extraction algorithm may be implemented by at least two of a corner detection algorithm, a scale-invariant feature transform, an edge detection algorithm, a texture detection algorithm, a shape feature detection algorithm, a position detection algorithm, and the like.
[0055] S200: Fusing the semantic feature information and the corresponding spatial feature information of each target to determine the feature information to be processed of each target.
[0056] Specifically, for any target, the computer device can use a feature information fusion method to fuse the semantic feature information and the corresponding spatial feature information of the target to obtain the feature information to be processed of the target. Optionally, the feature information fusion method can be a weighted fusion method, a stacked fusion method, a linear fusion method, a convolution fusion method, an attention fusion method, etc.
[0057] S300 , based on the feature information to be processed of each target, classify each target into target sets of different types to complete classification processing.
[0058] In practical applications, the computer device can use a similarity method to perform similarity processing on the feature information to be processed between two targets in each target, and then classify the corresponding targets whose similarity values between the feature information to be processed between two targets are within a first preset threshold range into the same category, classify the corresponding targets whose similarity values between the feature information to be processed between two targets are within a second preset threshold range into the same category, and classify the corresponding targets whose similarity values between the feature information to be processed between two targets are within a third preset threshold range into the same category, that is, classify each target into different types of target sets to complete the classification processing.
[0059] Optionally, the similarity method may be a correlation coefficient method, a Euclidean distance method, a cosine similarity method, a Manhattan example method, etc., which is not limited in the present embodiment of the application. Wherein, the first preset threshold range, the second preset threshold range and the third preset threshold range are all different.
[0060] It should be noted here that the above target classification method is applicable to the scenario where the difference value between different types of targets among multiple targets is less than a preset difference threshold, that is, the above target classification method can be applicable to fine-grained classification scenarios.
[0061] In one embodiment, the step of classifying each target into different types of target sets based on the feature information to be processed of each target in the above S300 to complete the classification processing may include: inputting the feature information to be processed and each target into a classifier for target classification processing to obtain each target set.
[0062] In the embodiment of the present application, the above-mentioned classifier may be a linear classifier.
[0063] The technical solution in the embodiment of the present application performs dual-channel feature extraction on multiple targets in the image to be processed, obtains semantic feature information and spatial feature information corresponding to each target, fuses the semantic feature information and corresponding spatial feature information of each target, determines the feature information to be processed of each target, and classifies each target into different types of targets based on the feature information to be processed to complete the classification processing, wherein the difference value between different types of targets in the multiple targets is less than a preset difference threshold; the above method can be applied to fine-grained classification scenarios, so that the difficulty of fine-grained classification is reduced, and in the processing process, multi-type feature information of each target, that is, semantic feature information and spatial feature information corresponding to each target, can be obtained, and then accurate classification can be achieved based on the multi-scale feature information of each target, which can improve the accuracy of fine-grained classification; at the same time, the above method does not require manual participation, thereby reducing the classification error caused by manual participation, greatly improving the correctness of fine-grained classification, and also speeding up the speed of fine-grained classification and improving the efficiency of fine-grained classification.
[0064] The following is a description of the process of performing dual-channel feature extraction on multiple targets in the processed image to obtain semantic feature information and spatial feature information corresponding to each target. In one embodiment, as shown in FIG3 , the steps in S100 above can be implemented in the following manner:
[0065] S110, performing preprocessing according to each target, and determining a plurality of preprocessed targets.
[0066] Specifically, the computer device may preprocess each target to obtain a plurality of preprocessed targets by using image transformation, image correction, image registration and adjustment methods, etc. Optionally, the total number of the preprocessed targets may be greater than or equal to the total number of targets.
[0067] In one embodiment, as shown in FIG. 4 , the step of performing preprocessing on each target to determine a plurality of preprocessed targets in S110 may include:
[0068] S111, performing target transformation processing on each target to determine a plurality of targets to be processed, wherein the target enhancement processing includes rotation processing, scaling processing and cropping processing.
[0069] Specifically, for any target, the computer device can perform target transformation processing on the target to obtain a target to be processed corresponding to the target, which can increase the number of samples for subsequent feature extraction and improve the diversity of samples. Optionally, the target transformation processing may include at least one of geometric transformation processing, size transformation processing, etc. In an embodiment of the present application, the target enhancement processing includes random rotation processing, scaling processing, and cropping processing.
[0070] S112, performing denoising processing on each target to be processed to obtain each pre-processed target.
[0071] Furthermore, the computer device may perform denoising processing on each target to be processed to achieve image noise reduction and obtain a pre-processed target corresponding to each target to be processed.
[0072] In the embodiment of the present application, for any target to be processed, Gaussian filtering and smoothing processing can be performed on the target to be processed to complete denoising processing of the target to be processed.
[0073] Among them, the Gaussian filtering and smoothing processing of the target to be processed can be specifically described as: for any pixel point in the target to be processed, the pixel values of multiple pixel points around the pixel point are weighted averaged until the pixel value of each pixel point is weighted averaged, and it is determined that the Gaussian filtering and smoothing processing of the target to be processed is completed.
[0074] S120, performing grayscale normalization processing on each preprocessed target to obtain each converted target.
[0075] Specifically, the computer device can pre-train an algorithm model, and then input each preprocessed target into the algorithm model. The algorithm model performs grayscale normalization on each preprocessed target and outputs a conversion target corresponding to each preprocessed target, so that the pixel value of each pixel point in the conversion target is a balanced pixel value, which can enhance the contrast of the original target relative to the original target.
[0076] In addition, the computer device can also use grayscale transformation method, histogram correction method and normalization method to perform grayscale normalization processing on each preprocessed target to obtain the conversion target corresponding to each preprocessed target. In the embodiment of the present application, the pixel value of each pixel point in the conversion target can be any value between 0-1.
[0077] In one embodiment, as shown in FIG. 5 , the step of performing grayscale normalization processing on each preprocessed target to obtain each conversion target in S120 may include:
[0078] S121. For any preprocessed target, grayscale the preprocessed target to obtain a corresponding grayscale target.
[0079] Specifically, the computer device can call the cumulative distribution function, adopt the image grayscale processing method, grayscale the preprocessed target, and obtain the corresponding grayscale target. Optionally, the above image grayscale processing method can be at least one of the component method, maximum value method, average method, weighted average method, etc.
[0080] S122, subtract the initial pixel value of each pixel in the grayscale target from the minimum initial pixel value among the pixels, and then divide by the total number of pixels to obtain the current pixel value of each pixel.
[0081] Based on the grayscale targets obtained in the previous steps, for any grayscale target, the grayscale target can be normalized. Specifically, for any pixel in the grayscale target, the initial pixel value of the pixel in the grayscale target is subtracted from the minimum initial pixel value of each pixel in the grayscale target, and then the difference is divided by the total number of pixels in the grayscale target to obtain the current pixel value of the pixel.
[0082] In the embodiment of the present application, the initial pixel value of the pixel in the grayscale target refers to the pixel value of the pixel in the grayscale target. The minimum initial pixel value of each pixel in the grayscale target can be understood as the minimum value of the initial pixel values of each pixel in the grayscale image.
[0083] S123. Obtain a conversion target corresponding to the preprocessed target according to the current pixel value of each pixel point.
[0084] In practical applications, a computer device can create a blank image, and then fill the current pixel value of each pixel into the corresponding position of the blank image according to the position of each pixel in the corresponding grayscale target, that is, the number of rows and columns, to generate a conversion target corresponding to the corresponding preprocessed target of the grayscale target. The size of the blank image is the same as the grayscale target score size.
[0085] In addition, the computer device can directly generate a conversion target corresponding to the preprocessed target of the grayscale target according to the position of each pixel in the grayscale target in the corresponding grayscale target and the current pixel value of each pixel.
[0086] S130, using a first encoder to extract semantic features from each conversion target to obtain semantic feature information of each target; and using a second encoder to extract spatial features from each conversion target to obtain spatial feature information of each target.
[0087] In practical applications, the computer device may use a first encoder to extract semantic features of each conversion target to obtain semantic feature information of each target; at the same time, the computer device may use a second encoder to extract spatial features of each conversion target to obtain spatial feature information of each target.
[0088] Optionally, the first encoder may be one or more; the first encoder may be implemented by at least one of a dual encoder, an absolute encoder, an incremental encoder, etc. Meanwhile, the second encoder may be one or more; the second encoder may be implemented by at least one of an autoencoder, a transformer encoder, a convolutional neural network encoder, etc.
[0089] The technical solution in the embodiment of the present application performs preprocessing on each target to determine multiple preprocessed targets, performs grayscale normalization on each preprocessed target to obtain each conversion target, uses a first encoder to extract semantic features on each conversion target to obtain semantic feature information of each target, and uses a second encoder to extract spatial features on each conversion target to obtain spatial feature information of each target; the above method can first preprocess each target, so as to improve the accuracy of semantic feature extraction and spatial feature extraction, and at the same time, before semantic feature extraction and spatial feature extraction, grayscale normalization is performed on each preprocessed target, which can reduce the memory space occupied by the operation objects of semantic feature extraction and spatial feature extraction, thereby reducing the computing resource consumption in the process of semantic feature extraction and spatial feature extraction, and can improve the processing speed of semantic feature extraction and spatial feature extraction.
[0090] In one embodiment, the first encoder is an encoder for semantic feature extraction based on local self-attention mechanism and global self-attention mechanism; the second encoder is an encoder based on local spatial feature extraction and global spatial feature extraction.
[0091] It should be noted here that the first encoder may include three sequentially connected semantic information encoders constructed using a lightweight transformer. FIG6 is a block diagram showing the relationship between the processing flow of the first encoder and the semantic information encoder. In an embodiment of the present application, the process of the first encoder extracting semantic features from each conversion target can be described as follows: first, a first convolution kernel is used to perform a depth-separable convolution operation on each conversion target to crop the result of the depth-separable convolution operation (i.e., the corresponding image) into image blocks of the same size to implement a block operation, and then the result of the depth-separable convolution operation is input into the semantic information encoder 1, and then the output result of the semantic information encoder 1 is input into the semantic information encoder 2, and finally the output result of the semantic information encoder 2 is input into the semantic information encoder 3.
[0092] Among them, each semantic information encoder can be a transformer encoder based on a sparse attention mechanism, and the specific processing process is: first, the second convolution kernel is used to perform a depth-separable convolution operation on the result of the depth-separable convolution operation to achieve channel fusion, and then each image block is linearly mapped to a one-dimensional corresponding feature vector, and then each feature vector is input into the transformer encoder based on the sparse attention mechanism, and then the output result of the transformer encoder based on the sparse attention mechanism is converted. After that, since the one-dimensional vector sequence after the conversion is long, the third convolution kernel can be further used to perform a lightweight depth-separable convolution operation on the conversion result to achieve downsampling, thereby reducing the length of the input sequence of the next transformer encoder based on the sparse attention mechanism, thereby reducing the computational complexity and memory usage of the next transformer encoder based on the sparse attention mechanism; at the same time, the input and output of adjacent semantic information encoders can be connected by jump connection to enrich the feature information as much as possible and reduce the loss of important feature information.
[0093] Optionally, the depth-wise separable convolution operation can be implemented through channel-by-channel convolution operation and point-by-point convolution operation. Compared with the traditional standard convolution operation, the depth-wise separable convolution operation can reduce the amount of calculation and parameter of the standard convolution operation, and further improve the efficiency of vital sign extraction. At the same time, the size of the first convolution kernel can be larger than the size of the second convolution kernel and the size of the third convolution kernel, and the size of the second convolution kernel can be smaller than the size of the third convolution kernel. However, in an embodiment of the present application, the first convolution kernel can be a 7×7 convolution kernel, the second convolution kernel can be a 1×1 convolution kernel, and the third convolution kernel can be a 3×3 convolution kernel.
[0094] At the same time, the above-mentioned sparse attention mechanism is constructed based on the combination of local self-attention mechanism and global self-attention mechanism, which can ensure that each element in the one-dimensional feature vector can capture the local feature information of other surrounding elements and obtain important global feature information, while significantly reducing the computational complexity and memory usage of the transformer encoder based on the sparse attention mechanism. Among them, the specific approach of the above-mentioned sparse attention mechanism is to calculate the local attention of the input, limiting each element in the input to only pay attention to the elements around it instead of all elements. In addition, in order to retain a certain degree of globality, a global attention element is inserted into all elements of the input at regular intervals. During the processing process, the global attention element can pay attention to the relationship with all other elements in the input, so that the output result of the transformer encoder retains both locality and globality.
[0095] It should be noted here that the second encoder can be implemented by combining dilated convolution and standard convolution to extract multi-scale local feature information and global feature information of the input and minimize the loss of spatial information. Optionally, the second encoder can be composed of three spatial information encoders (i.e., spatial information encoder 1, spatial information encoder 2, and spatial information encoder 3) connected in sequence, as shown in Figure 7, which is a structural block diagram of the second encoder.
[0096] Among them, the processing process of the second encoder can be described as: first, the fourth convolution kernel is used to perform a standard convolution operation on each conversion target to achieve feature extraction and dimensionality reduction, so that the size of each conversion target is reduced, and then the fifth convolution kernel with different expansion rates (such as 2, 4, 8, etc.) is used to perform a hole convolution operation on the standard convolution operation results to achieve feature extraction, and obtain rich feature information at multiple scales while keeping the spatial size unchanged, further enriching the feature representation, and finally the sixth convolution kernel is used to perform a standard convolution operation on each hole convolution operation result to achieve channel information fusion.
[0097] In this embodiment of the present application, the size of the fourth convolution kernel is the same as that of the third convolution kernel, and the size of the sixth convolution kernel is the same as that of the second convolution kernel.
[0098] It should be noted here that the second encoder is implemented by combining standard convolution operations with dilated convolution operations, which can not only retain the receptive fields of different scales obtained by multi-scale dilated convolution operations, but also make full use of the ability of standard convolution operations to extract detail features. In addition, compared with the transformer encoder based on the sparse attention mechanism, the spatial information encoder does not compress the input into one-dimensional data, so that the spatial feature information of the input can be retained as much as possible.
[0099] Optionally, the above expansion rate can represent the spacing between elements in the fifth convolution kernel. For example, if the expansion rate d=1 represents a standard convolution kernel (i.e., a 1×1 convolution kernel), d=2 represents that an empty position is inserted between the elements of the standard convolution kernel, and the element in each empty position is expressed by formula (1):
[0100] (1)
[0101] Among them, I represents the standard convolution operation result of realizing the input of the hole convolution operation, represents the standard convolution kernel, d represents the dilation rate, Represents the element in the i-th row and j-th column in the output of the dilated convolution operation.
[0102] The technical solution in the embodiment of the present application can extract semantic feature information in the input based on a lightweight transformer semantic information encoder when extracting features for multiple targets with high similarity, and can extract multi-scale spatial feature information in the input as much as possible based on the spatial information encoder. Since the semantic information encoder adopts a lightweight transformer structure that combines deep separable convolution and sparse attention mechanism, it can obtain rich contextual semantic information as much as possible while introducing a small amount of parameters. At the same time, the spatial information encoder adopts a combination of dilated convolution and standard convolution with different expansion rates to extract multi-scale spatial feature information to compensate for the loss of spatial information caused by linear mapping in the transformer encoder. In this way, the semantic feature information and spatial feature information finally obtained are more accurate, providing reliable reference information for subsequent classification processing.
[0103] The following is an explanation of the process of fusing the semantic feature information of each target and the corresponding spatial feature information to determine the feature information to be processed of each target. In one embodiment, as shown in FIG8 , the steps in S200 may include:
[0104] S210 , performing cascade processing according to the semantic feature information of each target and the corresponding spatial feature information to obtain cascade feature information.
[0105] It should be noted that the above cascade processing can be understood as splicing processing, that is, a simple splicing or combining process. Specifically, the computer device can perform cascade processing on the semantic feature information of each target and the spatial feature information of each target to obtain the cascade feature information corresponding to each target.
[0106] S220, performing feature fusion processing on the cascade feature information to obtain feature fusion information of each target, wherein the feature fusion processing includes a first depthwise separable convolution operation.
[0107] In practical applications, the computer device can use the seventh convolution kernel to perform the first depth-separable convolution operation on the cascade feature information to achieve feature fusion processing and obtain feature fusion information of each target. The size of the seventh convolution kernel is the same as the size of the second convolution kernel.
[0108] S230, performing feature extraction processing on the feature fusion information to determine the feature information to be processed of each target, wherein the feature extraction processing includes a second depthwise separable convolution operation, and the convolution kernel sizes of the first depthwise separable convolution operation and the second depthwise separable convolution operation are different.
[0109] Further, the computer device can use the eighth convolution kernel to perform a second depth-separable convolution operation on the feature fusion information to achieve feature extraction processing and obtain the feature information to be processed of each target. Optionally, the size of the eighth convolution kernel is the same as the size of the third convolution kernel.
[0110] The technical solution in the embodiment of the present application performs cascade processing on the semantic feature information and the corresponding spatial feature information of each target to obtain cascade feature information, performs feature fusion processing on the cascade feature information to obtain feature fusion information of each target, and performs feature extraction processing on the feature fusion information to determine the feature information to be processed of each target; the above method can deeply fuse the semantic feature information and the corresponding spatial feature information of each target to obtain more comprehensive and rich feature information of multiple targets, so as to provide reliable reference information for subsequent accurate classification processing; at the same time, the above method can perform cascade processing on the multi-scale spatial feature information and semantic feature information obtained by dual-channel feature extraction in the channel dimension, that is, splicing processing, and then perform a deep separable convolution operation on the cascade feature information, which can fully fuse the feature information without introducing a large amount of parameters, and further perform a deep separable convolution operation to extract the fused feature information, thereby achieving lightweight and effective feature fusion.
[0111] In one embodiment, the present application also provides a target classification method, which is applied to a computer device. The method includes the following process:
[0112] (1) performing target transformation processing on each target in the image to be processed to determine a plurality of targets to be processed; the target enhancement processing includes rotation processing, scaling processing and cropping processing; the difference value between different types of targets in each of the plurality of targets is less than a preset difference threshold;
[0113] (2) De-noising each target to be processed to obtain each pre-processed target;
[0114] (3) For any preprocessed target, grayscale the preprocessed target to obtain the corresponding grayscale target;
[0115] (4) Subtract the initial pixel value of each pixel in the grayscale target from the minimum initial pixel value among all the pixels, and then divide it by the total number of pixels to obtain the current pixel value of each pixel;
[0116] (5) Obtain the conversion target corresponding to the preprocessed target according to the current pixel value of each pixel;
[0117] (6) using a first encoder to extract semantic features from each conversion target to obtain semantic feature information of each target; and using a second encoder to extract spatial features from each conversion target to obtain spatial feature information of each target; wherein the first encoder is an encoder for semantic feature extraction based on a local self-attention mechanism and a global self-attention mechanism; and the second encoder is an encoder based on local spatial feature extraction and global spatial feature extraction;
[0118] (7) Performing cascade processing based on the semantic feature information of each target and the corresponding spatial feature information to obtain cascade feature information;
[0119] (8) performing feature fusion processing on the cascade feature information to obtain feature fusion information of each target; the feature fusion processing includes a first depthwise separable convolution operation;
[0120] (9) performing feature extraction processing on the feature fusion information to determine feature information to be processed for each target; the feature extraction processing includes a second depthwise separable convolution operation, and the convolution kernel sizes of the first depthwise separable convolution operation and the second depthwise separable convolution operation are different;
[0121] (10) The feature information to be processed and each target are input into the classifier for target classification processing to obtain each target set.
[0122] The execution process of the above (1) to (10) can be specifically referred to the description of the above embodiment. The implementation principle and technical effect are similar and will not be repeated here.
[0123] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0124] Based on the same inventive concept, the embodiment of the present application also provides a target classification device for implementing the target classification method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more target classification device embodiments provided below can refer to the limitations on the target classification method above, and will not be repeated here.
[0125] In one embodiment, FIG9 is a schematic diagram of the structure of a target classification device in an embodiment of the present application. The target classification device provided in the embodiment of the present application can be applied to a computer device. As shown in FIG9 , the target classification device in the embodiment of the present application may include: a feature extraction module 11, a fusion processing module 12 and a classification processing module 13, wherein:
[0126] The feature extraction module 11 is used to perform dual-channel feature extraction on multiple targets in the processed image to obtain semantic feature information and spatial feature information corresponding to each target; the difference value between different types of targets in each of the multiple targets is less than a preset difference threshold;
[0127] A fusion processing module 12 is used to fuse the semantic feature information and the corresponding spatial feature information of each target to determine the feature information to be processed of each target;
[0128] The classification processing module 13 is used to classify each target into different types of target sets based on the feature information to be processed of each target to complete the classification processing.
[0129] The target classification device provided in the embodiment of the present application can be used to execute the technical solution in the above-mentioned target classification method embodiment of the present application. Its implementation principle and technical effect are similar and will not be repeated here.
[0130] In one embodiment, the feature extraction module 11 includes: a preprocessing unit, a grayscale normalization processing unit and a feature extraction unit, wherein:
[0131] A preprocessing unit, used for performing preprocessing according to each target and determining a plurality of preprocessed targets;
[0132] A grayscale normalization processing unit, used for performing grayscale normalization processing on each preprocessed target to obtain each conversion target;
[0133] A feature extraction unit is used to extract semantic features of each conversion target using a first encoder to obtain semantic feature information of each target; and to extract spatial features of each conversion target using a second encoder to obtain spatial feature information of each target;
[0134] Among them, the first encoder is an encoder for semantic feature extraction based on local self-attention mechanism and global self-attention mechanism; the second encoder is an encoder based on local spatial feature extraction and global spatial feature extraction.
[0135] The target classification device provided in the embodiment of the present application can be used to execute the technical solution in the above-mentioned target classification method embodiment of the present application. Its implementation principle and technical effect are similar and will not be repeated here.
[0136] In one embodiment, the preprocessing unit is specifically used for:
[0137] Performing target transformation processing according to each target to determine multiple targets to be processed; target enhancement processing includes rotation processing, scaling processing and cropping processing;
[0138] De-noising is performed on each target to be processed to obtain each pre-processed target.
[0139] The target classification device provided in the embodiment of the present application can be used to execute the technical solution in the above-mentioned target classification method embodiment of the present application. Its implementation principle and technical effect are similar and will not be repeated here.
[0140] In one embodiment, the grayscale normalization processing unit is specifically used for:
[0141] For any preprocessed target, grayscale the preprocessed target to obtain the corresponding grayscale target;
[0142] The initial pixel value of each pixel in the grayscale target is subtracted from the minimum initial pixel value among the pixels, and then divided by the total number of pixels to obtain the current pixel value of each pixel;
[0143] According to the current pixel value of each pixel point, the conversion target corresponding to the preprocessed target is obtained.
[0144] The target classification device provided in the embodiment of the present application can be used to execute the technical solution in the above-mentioned target classification method embodiment of the present application. Its implementation principle and technical effect are similar and will not be repeated here.
[0145] In one embodiment, the fusion processing module 12 is specifically used for:
[0146] Cascade processing is performed according to the semantic feature information of each target and the corresponding spatial feature information to obtain cascade feature information;
[0147] Performing feature fusion processing on the cascade feature information to obtain feature fusion information of each target; the feature fusion processing includes a first depth-separable convolution operation;
[0148] The feature fusion information is subjected to feature extraction processing to determine feature information to be processed of each target; the feature extraction processing includes a second depthwise separable convolution operation, and the convolution kernel sizes of the first depthwise separable convolution operation and the second depthwise separable convolution operation are different.
[0149] The target classification device provided in the embodiment of the present application can be used to execute the technical solution in the above-mentioned target classification method embodiment of the present application. Its implementation principle and technical effect are similar and will not be repeated here.
[0150] In one embodiment, the classification processing module 13 is specifically used for:
[0151] The feature information to be processed and each target are input into the classifier for target classification processing to obtain each target set.
[0152] For the specific definition of the target classification device, please refer to the definition of the target classification method above, which will not be repeated here. Each module in the above-mentioned target classification device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0153] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG10. The computer device includes a processor, a memory, and a network interface connected via a system bus. The processor of the computer device is used to provide processing capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store images to be processed and different types of target sets. The network interface of the computer device is used to communicate with an external endpoint via a network connection. When the computer program is executed by the processor, a target classification method is implemented.
[0154] Those skilled in the art will understand that the structure shown in FIG. 10 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.
[0155] In one embodiment, a computer device is also provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the technical solution in the above-mentioned target classification method embodiment of the present application is implemented. The implementation principle and technical effect are similar and will not be repeated here.
[0156] In one embodiment, a computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, the technical solution of the above-mentioned target classification method of the present application is implemented. The implementation principle and technical effect are similar and will not be repeated here.
[0157] In one embodiment, a computer program product is also provided, including a computer program, which, when executed by a processor, implements the technical solution of the above-mentioned target classification method of the present application. Its implementation principle and technical effect are similar and will not be repeated here.
[0158] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).
[0159] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0160] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
Claims
1. A target classification method, characterized in that: The method comprises: Perform dual-channel feature extraction on multiple targets in the image to be processed to obtain semantic feature information and spatial feature information corresponding to each of the targets; the difference value between different types of targets in each of the multiple targets is less than a preset difference threshold; The semantic feature information and the corresponding spatial feature information of each of the targets are fused to determine the feature information to be processed of each of the targets; Based on the feature information to be processed of each of the targets, each of the targets is correspondingly classified into target sets of different types to complete classification processing.
2. The method according to claim 1, characterized in that The dual-channel feature extraction of multiple targets in the image to be processed to obtain semantic feature information and spatial feature information corresponding to each target includes: Perform preprocessing according to each of the targets to determine a plurality of preprocessed targets; Performing grayscale normalization processing on each of the preprocessed targets to obtain each conversion target; A first encoder is used to extract semantic features of each conversion target to obtain semantic feature information of each target; and a second encoder is used to extract spatial features of each conversion target to obtain spatial feature information of each target.
3. The method according to claim 2, characterized in that The preprocessing is performed according to each of the targets to determine a plurality of preprocessed targets, including: Performing target transformation processing according to each of the targets to determine a plurality of targets to be processed; the target enhancement processing includes rotation processing, scaling processing and cropping processing; De-noising is performed on each of the targets to be processed to obtain each of the pre-processed targets.
4. The method according to claim 2, characterized in that: The grayscale normalization processing is performed on each of the pre-processed targets to obtain each of the conversion targets, including: For any preprocessed target, grayscale the preprocessed target to obtain a corresponding grayscale target; Subtract the initial pixel value of each pixel in the grayscale target from the minimum initial pixel value of the pixels, and then divide the difference by the total number of the pixels to obtain the current pixel value of each pixel; According to the current pixel value of each pixel point, a conversion target corresponding to the preprocessed target is obtained.
5. The method according to claim 2, characterized in that: The first encoder is an encoder for semantic feature extraction based on local self-attention mechanism and global self-attention mechanism; the second encoder is an encoder based on local spatial feature extraction and global spatial feature extraction.
6. The method according to any one of claims 1 to 4, characterized in that The fusing the semantic feature information and the corresponding spatial feature information of each target to determine the feature information to be processed of each target includes: Performing cascade processing on the semantic feature information and the corresponding spatial feature information of each target to obtain cascade feature information; Performing feature fusion processing on the cascade feature information to obtain feature fusion information of each target; the feature fusion processing includes a first depthwise separable convolution operation; Performing feature extraction processing on the feature fusion information to determine the feature information to be processed of each of the targets; the feature extraction processing includes a second depthwise separable convolution operation, and the convolution kernel sizes of the first depthwise separable convolution operation and the second depthwise separable convolution operation are different.
7. The method according to any one of claims 1 to 4, characterized in that Based on the feature information to be processed of each target, each target is correspondingly classified into different types of target sets to complete classification processing, including: The feature information to be processed and each of the targets are input into a classifier for target classification processing to obtain each of the target sets.
8. A target classification device, characterized in that: The device comprises: A feature extraction module is used to perform dual-channel feature extraction on multiple targets in the processed image to obtain semantic feature information and spatial feature information corresponding to each of the targets; the difference value between different types of targets in each of the multiple targets is less than a preset difference threshold; A fusion processing module, used for fusing the semantic feature information and the corresponding spatial feature information of each target to determine the feature information to be processed of each target; The classification processing module is used to classify each of the targets into target sets of different types based on the feature information to be processed of each of the targets to complete the classification processing.
9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.