A sketch rating method and model based on multi-scale feature fusion
Through the interaction of multi-scale feature fusion technology and feature maps, the problem of inaccurate rating of complex sketch works in existing technologies is solved, and higher rating accuracy and reliability are achieved.
Patent Information
- Application Number
- CN202310809641.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-03
- Publication Date
- 2025-09-30
- Estimated Expiration
- 2043-07-03
AI Technical Summary
Existing sketch rating methods are unable to effectively extract the features of complex sketches, resulting in inaccurate ratings and unreliable results.
Multi-scale feature fusion technology is used to generate multi-scale features through Resnet-50, and FPN is used to fuse high-resolution low-level features with abstract high-level features. The SE attention mechanism and inter-layer bilinear cross pooling are combined to fuse and interact feature maps. Dropout is used to prevent overfitting, and finally classification is performed through the softmax function.
The accuracy of sketch ratings and the reliability of results have been improved, enabling more accurate rating of complex sketches.
Smart Images
Figure CN116778182B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence, and in particular to the field of a sketch rating method based on multi-scale feature fusion. Background Art
[0002] With the increasing popularity of AI technology, education systems are also gradually incorporating it to improve their systematization. For example, the intelligent calligraphy grading system integrates image processing, deep learning, augmented reality, and intelligent robotics technologies to provide a complete solution for intelligent calligraphy learning and calligraphy grade exam training.
[0003] However, while various fields have entered the intelligent era, the current scoring of sketch works in art exams still relies on manual review, that is, manual scoring by judges. The judges first grade all sketch works into four grades: excellent, good, medium and poor. After determining the grade to which the work belongs, the subsequent scores of the works cannot exceed the score range of the grade. The judges then walk back and forth among the piles of works of each grade, holding a score sheet, next to the sketch work answer sheet spread out on the ground, scoring each work one by one. This is an extremely cumbersome method. This manual review method is still widely used in the admissions work of art colleges in my country. It puts a lot of pressure and overhead on the scoring work. At the same time, the workload of entering the score sheet later is very heavy and the cycle is very long. There are also problems with incorrect recording and omissions due to the illegible handwriting on the score sheet.
[0004] Therefore, in order to improve the efficiency of marking examination papers and protect examination works, the need to use artificial intelligence technology to quickly, accurately and intelligently mark and grade sketch works is imminent.
[0005] The problem of sketch rating can essentially be transformed into an image multi-classification problem. With the increasing application of machine learning in real-world scenarios, supervised learning methods require large amounts of labeled data to train models, but obtaining labeled data is a time-consuming and tedious task. Deep learning algorithms are one of the mainstream artificial intelligence algorithms. They provide a method for automatically learning pattern features and integrate feature learning into the model building process, thereby reducing the incompleteness caused by manually designed features. Under specific conditions, certain machine learning applications based on deep learning have achieved recognition or classification performance that surpasses existing algorithms. Based on this, existing technologies are generally constructed based on convolutional neural networks.
[0006] Existing sketch rating methods include:
[0007] VGG-16-based bilinear convolution feature method: Using the AlexNet model pre-trained on the ImageNet dataset, the output of its sixth fully connected layer is extracted as the convolution feature. Bilinear pooling is performed on this basis, and the tensor sketch is used to reduce the dimensionality of the obtained high-dimensional features, and the final features are classified.
[0008] Transfer learning method based on Resnet50: Through transfer learning, the high-level and low-level features of the model are integrated to enhance the model's ability to perceive image details.
[0009] Although the above methods can grade and classify sketch works, they also have their own shortcomings, mainly as follows:
[0010] (1) The classification algorithm used in the bilinear convolution feature method based on VGG-16 is linear SVM. The SVM theory uses a fixed penalty coefficient C. However, the losses caused by the two errors of positive and negative samples are different. The accuracy is not high enough, and the labeling of the data set is not mentioned, resulting in unreliable final results.
[0011] (2) The dataset for the transfer learning method based on Resnet50 is relatively small, and the dataset was annotated by only one art teacher, which resulted in strong subjectivity and unreliable results. The results of each fold obtained using five-fold cross-validation were unstable and varied greatly.
[0012] The above methods cannot fully and effectively extract the features of complex sketches, and therefore cannot be adapted to accurately rate complex sketches. Summary of the Invention
[0013] Therefore, the present invention proposes a sketch rating method based on multi-scale feature fusion, which is based on multi-scale feature fusion technology to complete the fusion and interaction between features at different scales of sketches, so that the obtained feature map can enable the high-level feature map to utilize the detailed information of the underlying feature map, thereby accurately rating the sketches.
[0014] This application is achieved through the following technical solutions:
[0015] On the one hand, the present application provides a sketch rating method based on multi-scale feature fusion, which includes:
[0016] Preprocessing sketches in a data set and performing feature extraction on the preprocessed sketches to obtain a first feature set C = {C1, C2, ..., Cn} of sketches at different resolutions, where n = 1, 2, ..., k, wherein k is a positive integer, and sorting the first features in the first feature set from high to low resolution;
[0017] Convolve the first feature Cn with the lowest resolution in the first feature set to obtain the corresponding second feature Mn;
[0018] After convolution of the mth first feature, it is fused with the upsampled second feature Mm+1 to obtain the corresponding second feature Mm; where the initial value of m is n-1;
[0019] Determine whether m is less than or equal to 1. If yes, complete feature extraction; if not, assign m to m-1 and repeat the previous step;
[0020] performing smoothing processing on each second feature in the second feature set one by one to obtain a third feature set;
[0021] Perform element-wise product between each pair of third features in the third feature set, and all the obtained features constitute the fourth feature set;
[0022] Pooling each fourth feature in the fourth feature set to achieve feature flattening;
[0023] All fourth features after the fourth feature concentration pooling are concatenated to obtain a final feature;
[0024] The final features are processed by the softmax function to obtain the work classification.
[0025] Furthermore, before pooling each fourth feature in the fourth feature set to achieve feature flattening, the method further includes:
[0026] Assigning a weight to each fourth feature in the fourth feature set through an attention mechanism.
[0027] Furthermore, the first feature set does not include the first feature C1 with the highest resolution.
[0028] Furthermore, during the model training phase, before the final features are processed by the softmax function, the following steps are also included:
[0029] The final features are processed by dropout, which randomly turns off some neurons during training.
[0030] Furthermore, the second features of different resolutions in the second feature set are upsampled through a feature pyramid.
[0031] On the other hand, this application also provides a sketch rating model based on multi-scale feature fusion, which includes:
[0032] Preprocessing layer: Preprocess the sketches in a dataset and perform feature extraction on the preprocessed sketches to obtain a first feature set C = {C1, C2...Cn} of sketches at different resolutions, where n = 1, 2, ....k...., where k is a positive integer. The first features in the first feature set are sorted from high to low resolution.
[0033] Feature convolution layer: used to convolve the first feature Cn with the lowest resolution in the first feature set to obtain the corresponding second feature Mn;
[0034] Feature fusion layer: used to perform feature fusion processing on the mth first feature after convolution with the second feature Mm+1 after upsampling to obtain the corresponding second feature Mm; where the initial value of m is n-1;
[0035] Judgment layer: used to determine whether m is less than or equal to 1. If yes, feature extraction is completed; if not, m is assigned m-1 and the previous step is repeated;
[0036] Smoothing processing layer: used for smoothing each second feature in the second feature set one by one to obtain a third feature set;
[0037] Feature capture layer: used to perform element-wise product between each pair of third features in the third feature set, and all the features obtained constitute the fourth feature set;
[0038] Pooling layer: pooling each fourth feature in the fourth feature set to achieve feature flattening;
[0039] Feature splicing layer: used to splice all the fourth features after the fourth feature is concentrated and pooled to obtain the final feature;
[0040] Classification layer: The final features are processed by the softmax function to obtain the work classification.
[0041] Furthermore, it also includes:
[0042] Weight update layer: used to assign a weight to each fourth feature in the fourth feature set through an attention mechanism.
[0043] Furthermore, during the model training phase, the model further includes:
[0044] Random inactivation layer: used for final features after dropout processing, randomly shutting down some neurons during training.
[0045] On the other hand, the present application also provides a computer device comprising:
[0046] at least one memory and at least one processor;
[0047] The memory is used to store one or more programs;
[0048] When the one or more programs are executed by the at least one processor, the at least one processor implements the steps of a sketch rating method based on multi-scale feature fusion as described in any one of the above items.
[0049] On the other hand, the present application also provides a computer-readable storage medium, which stores a computer program, characterized in that when the computer program is executed by a processor, it implements the steps of a sketch rating method based on multi-scale feature fusion as described in any one of the above items.
[0050] The present application provides a sketch rating method and model based on multi-scale feature fusion, which is used to score complex sketches. It uses the multi-scale features generated by Resnet-50 at various stages, and fuses high-resolution underlying features with positioning information and abstract high-level features with semantic information based on FPN (feature pyramid structure). It fully considers the characteristics of complex sketch images and captures target features at different scales, thereby improving classification accuracy. In addition, the performance of the model can be improved by fusing feature maps of different scales together, providing more comprehensive target features after fusion, thereby more accurately classifying. After weighting the channel dimensions of these feature maps using the SE attention mechanism, the information of the feature maps is aggregated using the inter-layer bilinear cross pooling method, and finally feature splicing is used to fuse multiple features for multi-classification output, while dropout is used to prevent the model from overfitting. Through this multi-scale feature fusion method, the accuracy of the rating is ultimately improved, and the results are reliable.
[0051] For better understanding and implementation of this application, the present invention is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] Figure 1 A flowchart of a sketch rating method based on multi-scale feature fusion provided in this application;
[0053] Figure 2 This is a structural diagram of a sketch rating model based on multi-scale feature fusion provided in this application;
[0054] Figure 3 This is an optimized flowchart of an exemplary multi-scale feature fusion sketch rating method provided in this application;
[0055] Figure 4 This is a structural diagram of an exemplary multi-scale feature fusion sketch rating model training process provided in this application. DETAILED DESCRIPTION
[0056] The existing sketch rating method model is unable to fully and effectively extract features from complex sketch works, resulting in an inability to accurately rate complex sketch works. This application will use multi-scale feature fusion technology to complete the fusion and interaction between features at different scales of sketch works, so that the obtained feature map can allow high-level feature maps to utilize the detailed information of low-level feature maps, thereby accurately rating sketch works. Figure 1 and Figure 2 , Figure 1 A flowchart of a sketch rating method based on multi-scale feature fusion provided in this application; Figure 2 To execute Figure 1 A structural block diagram of a sketch rating model based on multi-scale feature fusion. The present invention relates to a sketch rating method based on multi-scale feature fusion, specifically comprising:
[0057] S10: Preprocess the sketches in a dataset and perform feature extraction on the preprocessed sketches to obtain a first feature set C = {C1, C2, ..., Cn} of different resolutions for the sketches, where n = 1, 2, ..., k, where k is a positive integer. The first features in the first feature set are sorted from high to low resolution. Step S10 is performed by the preprocessing layer 10.
[0058] A dataset containing multiple sketches to be scored and classified is input. Preprocessing includes random cropping, horizontal flipping, rotation, normalization, random erasing, light and dark changes, and tonal changes to expand the data volume. A new set of sketches is obtained, and each sketch is input into a multi-scale feature network. In this application, Resnet50 is used as the feature extraction network. After layer-by-layer convolution, the first features of each sketch can be extracted in order: C1, C2, C3, C4, and C5 at different resolutions. C1 has the highest resolution, and C5 has the lowest resolution.
[0059] S20: Convolve the first feature Cn with the lowest resolution in the first feature set to obtain the corresponding second feature Mn. Step S20 is performed by the feature convolution layer 20.
[0060] Cn is the scale with the lowest resolution in the corresponding first feature set. The corresponding second feature Mn is obtained by performing 1×1 convolution on Cn to reduce the number of channels. In this application, C5 is convolved to obtain M5.
[0061] S30: After convolution of the mth first feature, the feature is fused with the upsampled second feature Mm+1 to obtain the corresponding second feature Mm, where the initial value of m is n-1. Step S30 is performed by the feature fusion layer 30.
[0062] As for C4, we first perform 1×1 convolution to reduce the number of channels, and then perform feature fusion with the upsampled second feature M5 to obtain the corresponding second feature M4. In this application, we use the Feature Pyramid Network to upsample the second feature M4. FPN itself allows high-level feature maps to interact with low-level feature maps by adding additional connections to the low-level feature maps, so that high-level feature maps can utilize the detailed information in the low-level feature maps. Upsampling increases the size of the low-level feature maps to the same as the high-level feature maps, which further strengthens the interaction and information transfer between feature maps.
[0063] S40: Determine whether m is less than or equal to 1. If yes, complete feature extraction; if not, assign m a value of m-1 and repeat step S30. Step S40 is executed by the judgment layer 40.
[0064] In order to complete the interaction between the high-level feature map and the low-level feature map for each feature in the first feature set and strengthen the information transfer between the feature maps, the first features of different scales in the first feature map are sequentially subjected to convolution transformation and feature pyramid upsampling, thereby fusing the information between the feature maps of different scales. In this application, a first feature is fused with an intermediate feature after upsampling of a second feature in a second feature set whose resolution is lower than the first feature and whose resolution is closest to the first feature, to obtain a second feature corresponding to the first feature, i.e., repeating steps S20 and S30 until all the first features in all the first feature sets have completed the interaction.
[0065] S50: Smoothing each second feature in the second feature set one by one to obtain a third feature set. Step 50 is performed by the smoothing processing layer 50.
[0066] The nearest neighbor interpolation algorithm causes grayscale discontinuity, and obvious jaggedness may appear at the location of grayscale changes. By smoothing, the feature values between the feature grayscale change areas are continuous, making the feature map more accurate.
[0067] S60: Perform element-wise product between each pair of third features in the third feature set, and all the obtained features form a fourth feature set. Step S60 is performed by the feature capture layer 60.
[0068] The element-wise product computes the product of the elements at corresponding positions in two eigenvectors. This operation preserves the correlation and proportionality of the eigenvectors at each position. By computing the product of two eigenvectors at corresponding positions, we can capture the interaction patterns and correlations between them. The element-wise product is used to capture spatial information between features.
[0069] S70: Pooling each fourth feature in the fourth feature set to achieve feature flattening. Step S70 is performed by the pooling layer 70.
[0070] The fourth feature set after pooling can be easily input into the fully connected layer with its feature structure.
[0071] S80: All fourth features after pooling in the fourth feature set are concatenated to obtain a final feature. Step S80 is performed by the feature concatenation layer 80.
[0072] S90: The final features are processed by the softmax function to obtain the work classification. Step S90 is performed by the classification layer 90.
[0073] See also Figure 3 and Figure 4 In another preferred embodiment, all features in the fourth feature set are pooled before information aggregation is completed, and step S70.a is further included: assigning a weight to each fourth feature in the fourth feature set through an attention mechanism. Step S70.a is performed by weight update layer 70.a.
[0074] The weight ratio between different features is obtained through the attention mechanism, and the features of different channels are adaptively recalibrated according to the weight ratio. In this way, the model can continuously optimize itself in the classification of sketch works, so as to increase the model's attention to useful features, thereby improving the performance of the model and improving the accuracy of sketch classification.
[0075] See also Figure 3 and Figure 4 In another preferred embodiment, the sketch rating model based on multi-scale feature fusion is further comprised of:
[0076] Step S90.a: The final features are processed by dropout, which randomly turns off some neurons during training. Step S90.a is performed by the random dropout layer 90.a.
[0077] Scaling activation values proportionally in forward and back propagation reduces network complexity and cooperative adaptation, avoids overfitting of neural networks, and increases network robustness and generalization ability.
[0078] In another embodiment, the first feature set does not include the first feature C1 with the highest resolution.
[0079] The sketch works are subjected to feature extraction by Resnet50 to obtain the first feature C1 with the highest resolution. Its resolution is too high and contains little semantic information, which has little impact on the scoring and classification of the works. However, its consumption of computing resources is consistent with other first features. Therefore, it is removed from the first feature set to improve resource utilization.
[0080] The present application provides a sketch rating method and model based on multi-scale feature fusion, which is used to score complex sketches. It uses the multi-scale features generated by Resnet-50 at various stages, and fuses high-resolution underlying features with positioning information and abstract high-level features with semantic information based on FPN (feature pyramid structure). It fully considers the characteristics of complex sketch images and captures target features at different scales, thereby improving classification accuracy. In addition, the performance of the model can be improved by fusing feature maps of different scales together, providing more comprehensive target features after fusion, thereby more accurately classifying. After weighting the channel dimensions of these feature maps using the SE attention mechanism, the information of the feature maps is aggregated using the inter-layer bilinear cross pooling method, and finally feature splicing is used to fuse multiple features for multi-classification output, while dropout is used to prevent the model from overfitting. Through this multi-scale feature fusion method, the accuracy of the rating is ultimately improved, and the results are reliable.
[0081] Based on the same inventive concept described above, this application also provides an electronic device, which can be a terminal device such as a server, desktop computing device, or mobile computing device (e.g., laptop computing device, handheld computing device, tablet computer, netbook, etc.). The device includes one or more processors and a memory, wherein the processor is configured to execute a program to implement the above-mentioned method for rating sketches based on multi-scale feature fusion; and the memory is configured to store a computer program executable by the processor.
[0082] Based on the same inventive concept, the present application also provides a computer-readable storage medium, corresponding to the embodiment of the aforementioned method for rating sketch works based on multi-scale feature fusion. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of a method for rating sketch works based on multi-scale feature fusion recorded in any of the aforementioned embodiments.
[0083] The present application may take the form of a computer program product implemented on one or more storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing program code. Computer-usable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. The information can be computer-readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include but are not limited to: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.
[0084] The above-described embodiments merely represent several implementations of the present invention. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous modifications and improvements without departing from the spirit of the present invention, and the present invention is intended to encompass such modifications and variations.
Claims
1. A sketch rating method based on multi-scale feature fusion, characterized by: include: Preprocessing sketches in a data set and performing feature extraction on the preprocessed sketches to obtain a first feature set C = {C1, C2, ..., Cn} of sketches at different resolutions, where n = 1, 2, ..., k, wherein k is a positive integer, and sorting the first features in the first feature set from high to low resolution; Convolving the first feature Cn with the lowest resolution in the first feature set to obtain a corresponding second feature Mn, wherein the first feature set does not include the first feature C1 with the highest resolution; After convolution of the mth first feature, it is fused with the upsampled second feature Mm+1 to obtain the corresponding second feature Mm; where the initial value of m is n-1; Determine whether m is less than or equal to 1. If yes, complete feature extraction; if not, assign m to m-1 and repeat the previous step; Performing smoothing on each second feature in the second feature set one by one to obtain a third feature set; Perform element-wise product between each pair of third features in the third feature set, and all the obtained features constitute the fourth feature set; Assigning a weight to each fourth feature in the fourth feature set through an attention mechanism; Pooling is performed on each fourth feature after weight assignment to achieve feature flattening; All fourth features after the fourth feature concentration pooling are concatenated to obtain a final feature; The final features are processed by dropout, which randomly turns off some neurons during training; The final features after randomly closing a part of neurons are processed by the softmax function to obtain the work classification.
2. The sketch rating method based on multi-scale feature fusion according to claim 1, characterized in that: The second features of different resolutions in the second feature set are upsampled through a feature pyramid.
3. A sketch rating system based on multi-scale feature fusion, characterized by: include: Preprocessing layer: Preprocess the sketches in a dataset and perform feature extraction on the preprocessed sketches to obtain a first feature set C = {C1, C2...Cn} of sketches at different resolutions, where n = 1, 2, ....k...., where k is a positive integer. The first features in the first feature set are sorted from high to low resolution. Feature convolution layer: used to convolve the first feature Cn with the lowest resolution in the first feature set to obtain the corresponding second feature Mn, where the first feature set does not include the first feature C1 with the highest resolution; Feature fusion layer: used to perform feature fusion processing on the mth first feature after convolution with the second feature Mm+1 after upsampling to obtain the corresponding second feature Mm; where the initial value of m is n-1; Judgment layer: used to determine whether m is less than or equal to 1. If yes, feature extraction is completed; if not, m is assigned m-1 and the previous step is repeated; Smoothing layer: used to smooth each second feature in the second feature set one by one to obtain a third feature set; Feature capture layer: used to perform element-wise product between each pair of third features in the third feature set, and all the features obtained constitute the fourth feature set; Weight update layer: used to assign a weight to each fourth feature in the fourth feature set through an attention mechanism; Pooling layer: used to pool the fourth feature after each weight is assigned to achieve feature flattening; Feature splicing layer: used to splice all the fourth features after the fourth feature is concentrated and pooled to obtain the final feature; Random inactivation layer: used for final features after dropout processing, randomly shutting down some neurons during training; Classification layer: It is used to process the final features after randomly closing some neurons through the softmax function to obtain the classification of the work.
4. A computer device, characterized in that: include: at least one memory and at least one processor; The memory is used to store one or more programs; When the one or more programs are executed by the at least one processor, the at least one processor implements the steps of a sketch rating method based on multi-scale feature fusion as described in any one of claims 1 to 2.
5. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of a sketch rating method based on multi-scale feature fusion as described in any one of claims 1 to 2 are implemented.