A method for detecting sub-graphs of academic papers based on multi-scale features

Abstract

The present application relates to the field of academic paper subgraph detection, and particularly relates to an academic paper subgraph detection method based on multi-scale features, comprising the following steps: reading an original image, obtaining local feature information and global feature information of the original image, the local feature information being local feature information with multiple scale levels extracted by a convolutional neural network, and the global feature information being global feature information with multiple scale levels extracted by a swin Transformer of a variable Patch; based on the local feature information and the global feature information, obtaining multi-scale feature fusion information by using an adaptive aggregation distribution network; based on the multi-scale feature fusion information, obtaining subgraph positioning information and subgraph category information; the method provided by the present application can simultaneously capture different scale local and global dependencies of a paper image, efficiently fuse multi-scale features, improve generalization capability, and realize high-precision detection of academic paper subgraphs.

Description

Technical Field

[0001] This invention relates to the field of subgraph detection in academic papers, and specifically to a method for detecting subgraphs in academic papers based on multi-scale features. Background Technology

[0002] Academic images are a direct representation of academic achievements and are crucial for showcasing research content, experimental approaches, and results. They not only help readers understand scientific ideas but also serve as an important means of verifying experimental results. However, academic misconduct, particularly the issue of fabricated academic images, is on the rise and has attracted widespread attention in the international academic community. Conducting large-scale, batch-based academic image detection can assist in identifying academic misconduct, establishing a research integrity supervision mechanism, and promoting a healthy research environment both domestically and internationally.

[0003] The first step in detecting academic image fraud is to extract sub-images from the paper for analysis to determine if it has been altered or plagiarized. Detection techniques can be categorized into traditional methods and deep learning-based methods.

[0004] Traditional methods for detecting subgraphs in academic papers rely heavily on manually designed rules such as handcrafted features and threshold strategies. These methods can lead to problems such as over-segmentation and under-segmentation, and they are also unable to cope with the diverse layouts of academic paper images. They have poor generalization capabilities, resulting in extremely low subgraph detection accuracy.

[0005] Existing feature extraction methods based on convolutional neural networks in deep learning can only extract local features, resulting in a limited range of extracted features and difficulty in uncovering subtle inter-class differences. Academic images, however, have relatively uniform pixel counts, resulting in less contrast and greater intra-class differences compared to inter-class differences, thus requiring in-depth analysis to uncover these subtle inter-class variations. Therefore, this type of feature extraction method is prone to classification errors.

[0006] Existing deep learning-based feature extraction methods, such as those using convolutional neural networks and transformers, can extract local details and long-range dependencies, which can compensate to some extent for the limitation of convolutional neural networks in extracting only local details. However, the extracted features are limited to a single scale. Academic papers often depict subgraphs with diverse scales, and these methods do not simultaneously consider local details and long-range dependencies at different scales. This makes it difficult to uncover subtle differences between classes at different scales, which can easily lead to subsequent classification errors.

[0007] Existing deep learning-based feature fusion methods, such as classic bottom-up and top-down approaches, can only fully fuse features from adjacent layers. For features from other layers, they can only be obtained indirectly and recursively. Due to the numerous paths and indirect interactions within the network, this leads to drawbacks such as cross-layer information exchange and loss. Furthermore, the scale of subgraphs in academic papers varies greatly, and the aforementioned feature fusion methods can cause overfitting to subgraphs at specific scales, resulting in low subgraph detection accuracy. Additionally, search graph-based methods have high search costs and significant overhead. Summary of the Invention

[0008] In view of the above problems, the present invention provides a method for detecting subgraphs in academic papers based on multi-scale features, which solves the problems of limited features extracted, difficulty in mining subtle differences between classes at different scales, and information loss when fusing multi-scale features in the existing technology for detecting subgraphs in academic papers.

[0009] This invention provides a method for detecting subgraphs in academic papers based on multi-scale features, comprising the following steps:

[0010] Step S1: The feature extraction module reads the original image and obtains the local feature information and global feature information of the original image. The local feature information is the local feature information with multiple scale levels extracted by the convolutional neural network, and the global feature information is the global feature information with multiple scale levels extracted by the variable patch swin Transformer.

[0011] Step S2: The feature fusion module obtains multi-scale feature fusion information based on the local and global feature information using an adaptive aggregation and distribution network;

[0012] Step S3: The feature processing module obtains sub-image localization information and sub-image category information based on the multi-scale feature fusion information.

[0013] Preferably, the local feature information with multiple scale levels extracted by the convolutional neural network is obtained by the convolutional neural network performing multiple convolution processes, multiple spatial and channel attention module processes, and multiple feature pyramid module processes on the input original image to obtain the local feature information with multiple scale levels.

[0014] The method for obtaining the global feature information with multiple scale levels extracted by the variable patch swin Transformer is as follows: the input original image is divided into variable patches, processed by multiple linear encoding processes, processed by multiple swinTransformer processes, and processed by multiple patch merging processes to obtain the global feature information with multiple scale levels.

[0015] Preferably, the convolutional neural network processes the input original image through multiple convolutions, multiple spatial and channel attention modules, and a feature pyramid module to obtain the local feature information with multiple scale levels, specifically including:

[0016] Step S1-1-1: Perform convolution processing on the input image;

[0017] Step S1-1-2: Process the output of step S1-1-1 using the spatial and channel attention module to obtain the output information of the current scale level;

[0018] Step S1-1-3: Repeat steps S1-1-1 and S1-1-2 to obtain the output information corresponding to multiple scale levels;

[0019] Step S1-1-4: Based on the output information of the multiple scale levels, obtain local feature information of the multiple scale levels.

[0020] Preferably, step S1-1-1 specifically includes: performing convolution processing on the input image with kernel size k=3, stride s=2, and padding p=1 to achieve downsampling operation;

[0021] Step S1-1-2 specifically includes: processing the output of step S1-1-1 using a CSPGAM attention module. The CSPGAM processing steps include: inputting the input feature map into a convolution with kernel size k=1, stride s=1, and padding margin p=0, halving the number of channels in the feature map; then dividing the channels of the feature map into two parts using a splitting function, one part of which is input into n BottleGAM modules, concatenating the outputs of all BottleGAM modules with the other part of the channels in the feature map, and then performing a convolution with kernel size k=1, stride s=1, and padding margin p=0, halving the number of channels in the feature map again, which is then used as the output information for the current scale level.

[0022] The specific steps S1-1-3 include: repeating the operations in steps S1-1-1 and S1-1-2 three times. During the repetition, the number n of BottleGAM modules in step S1-1-2 is set to 3, 6, 6, and 3 respectively, to obtain output information at multiple scale levels.

[0023] Step S1-1-4 specifically includes: taking the output information of the first three scale levels as the local feature information of the first three scale levels respectively, performing max pooling processing on the output information of the fourth scale level three times with a 5*5 kernel, and concatenating the results of each max pooling processing as the local feature information of the fourth scale level.

[0024] Preferably, the input original image undergoes variable patch partitioning, multiple linear encoding processes, multiple SwinTransformer processes, and multiple patch merging processes to obtain the global feature information with multiple scale levels, specifically including:

[0025] Step S1-2-1: Determine the prediction parameters of the patch, including: pre-dividing the image into multiple patches, and predicting the position and size of all patches. The prediction method is as follows:

[0026]

[0027]

[0028]

[0029]

[0030] δx, δy = Tanh(W offest *f p (A))

[0031] S w S h =RELU(Tanh(W scale *f p (A)+b scale ))

[0032] Where (x1, y1) and (x2, y2) are the coordinates of the top-left and bottom-right corners of the Patch rectangle, respectively. ct y ct ) represents the center coordinates of each patch, (δx, δy) represents the offset of the patch center coordinates, and (S h S w ) represents the height and width of the patch, f p () is a linear layer, A is the input feature map, Tanh() and ReLU() are activation functions, and W is a linear layer. offest For the offset weight, W scale To scale the weights, b scale For scaling offset;

[0033] Step S1-2-2: Based on the prediction parameters of the Patch, obtain multiple Patches after division, including: based on the position and size information of the Patch, obtain the coordinate information of the rectangular region where the Patch is located, uniformly sample m*m points in the rectangular region where each Patch is located, and calculate the features of the current point through bilinear interpolation.

[0034] Step S1-2-3: Perform linear encoding on the input information, and pre-set the dimension of the vector to a value that the Transformer can accept;

[0035] Step S1-2-4: Perform single-sample feature normalization on the output feature map of step S1-2-3. The W-MSA module first divides the feature map into M*M windows, and then encodes each window individually using a self-attention mechanism. The SW-MSA module first shifts the window elements, and then divides the window into windows of size 7. MSA operations are then performed within the windows. The window attention mechanism performs MSA operations within the 7*7 windows. This process only updates the weights and does not change the data dimensions. Then, the data is restored to the input image size, and the previously shifted windows are shifted in the reverse direction.

[0036] Step S1-2-5: Perform patch merging processing on the output feature map of step S1-2-4, including: dividing each 2x2 adjacent pixel into a patch, then concatenating the pixels at the same position in each patch to obtain 4 feature maps, concatenating these four feature maps in the depth direction, then performing normalization layer processing, and finally using a fully connected layer to perform linear transformation in the depth direction of the feature map to halve the depth of the feature map;

[0037] Step S1-2-6: Repeat steps S1-2-3, S1-2-4, and S1-2-5 three times. During the repetition, obtain the global feature information of the corresponding scale level.

[0038] Preferably, step S2 specifically includes:

[0039] Step S2-1: Concatenate the local feature information and global feature information according to their corresponding levels;

[0040] Step S2-2: Adaptively aggregate the concatenated feature information, including:

[0041] Align the feature sizes of multiple scale levels in the spliced ​​feature information with the scale level to be distributed, and assign different spatial weights to the features of different levels to perform adaptive feature weighting, so as to obtain the weighted fused features.

[0042] Step S2-3: Perform information fusion on the adaptively clustered features;

[0043] Step S2-4: Based on the information obtained in step S2-3, the fused features are combined with the local and global feature information obtained in step S1 to perform feature distribution operations, and the results are distributed to the corresponding scale level.

[0044] Step S2-5: Perform adaptive spatial fusion operation on the features distributed to the corresponding scale level.

[0045] Preferably, the step of concatenating the local feature information and global feature information according to the hierarchy includes: concatenating the local feature information of the four scale levels extracted by the convolutional neural network and the global feature information of the four scale levels extracted by the swin Transformer of the variable patch.

[0046] Aligning the feature sizes of multiple scale levels in the spliced ​​feature information to the scale level to be distributed includes: if the feature of the current scale level is higher than the scale level to be distributed, then upsampling is performed using bilinear interpolation; if the feature of the current scale level is lower than the scale level to be distributed, then downsampling is performed using downsampling average pooling.

[0047] The specific method for information fusion of the adaptively aggregated features is as follows: the result of step S2-2 is subjected to convolution operation, and then processed by the RepConvBlocks module to obtain the extracted gradient flow information;

[0048] The adaptive spatial fusion operation is implemented using an adaptive feature pooling network.

[0049] Preferably, in step S2-4, the feature distribution operation based on the fused features obtained in step S2-3 and the concatenated local and global feature information obtained in step S1 specifically includes:

[0050] Step S2-4-1: For the adjacent layer feature information after concatenating the local feature information and global feature information at a scale level, perform size alignment and concatenation, and then perform two 1*1 convolutions.

[0051] Step S2-4-2: Perform a 1*1 convolution on the fused features, then calculate the sigmoid function, then perform average pooling downsampling, and multiply the result of step S2-4-1.

[0052] Step S2-4-3: Perform a 1*1 convolution on the fused features, then perform average pooling downsampling, add the result of step S2-4-2, and then perform RepConvBlocks module calculation to obtain the distribution features of the corresponding scale level.

[0053] Preferably, step S3 specifically includes: obtaining positioning coordinates based on a positioning branch network, wherein the positioning branch network is specifically configured as follows: the positioning branch first undergoes two convolution operations with convolution size k=3, stride s=1, and padding p=1, and then undergoes one convolution operation with convolution size k=1, stride s=1, and padding p=0; the number of channels in the third convolution is set to the form 4*reg_max, where the value of the maximum regularization parameter reg_max is 16 by default; and the loss function of the positioning branch is set to include two parts: CIoU loss function and DFL loss function. Obtaining classification categories based on a classification branch network, wherein the classification branch network is specifically configured as follows: undergoing two convolution operations with convolution size k=3, stride s=1, and padding p=1, and then undergoing one convolution operation with convolution size k=1, stride s=1, and padding p=0; the number of channels is set to the number of categories to be classified; and the loss function of the classification branch is set to the binary cross-entropy function.

[0054] Compared with the prior art, the present invention has at least the following beneficial effects:

[0055] The feature extraction module provided by this invention is a dual-stream feature extraction network based on variable patches. Unlike current state-of-the-art methods, this invention extracts local and global features at different scales separately. The variable patch partitioning method of this invention can be data-driven and adaptively partition the data. Based on the GAM attention mechanism and CSP concept, this invention provides a novel CSPGAM module that can learn key local features. Using the feature extraction network proposed in this invention, local details and long-distance dependencies at different scales can be extracted, and subtle differences between classes can be deeply mined, thereby improving the classification accuracy of subgraphs.

[0056] The feature fusion module of this invention is an adaptive aggregation and distribution network based on an adaptive aggregation and distribution mechanism. Unlike current advanced methods, after the feature extraction network extracts features at different scales, it first performs an adaptive alignment and weighting operation on all layers of features. This allows it to directly obtain feature information from adjacent and non-adjacent layers. Then, it performs an information fusion operation to enrich gradient flow information and finally distributes it to each layer. This method can efficiently fuse multi-scale features, reduce information loss, address the problem of large scale changes, improve subgraph localization accuracy, and enhance generalization ability, thereby achieving high-precision detection of subgraphs in academic papers. Attached Figure Description

[0057] The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of the invention.

[0058] Figure 1 This is a flowchart of the academic paper subgraph detection method based on multi-scale features disclosed in this invention;

[0059] Figure 2 This is a flowchart of the sub-image detection process for a single paper image disclosed in this invention;

[0060] Figure 3 This is a diagram showing the overall model structure of the paper subgraph detection method disclosed in this invention;

[0061] Figure 4 This is a detailed diagram of a local branch of a CNN disclosed in this invention;

[0062] Figure 5 This is a detailed diagram illustrating the implementation of CSPGAM in a local branch of a CNN as disclosed in this invention.

[0063] Figure 6 This is a detailed diagram of the swin transformer block in the global branch of the Transformer disclosed in this invention;

[0064] Figure 7 This is a detailed diagram of the adaptive aggregation module in the adaptive aggregation distribution fusion network disclosed in this invention;

[0065] Figure 8 This is a detailed diagram of the information fusion module in the adaptive aggregation and distribution fusion network disclosed in this invention;

[0066] Figure 9 This is a detailed diagram of the feature distribution module in the adaptive aggregation distribution fusion network disclosed in this invention. Detailed Implementation

[0067] To better understand the above-described objectives, features, and advantages of the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other. Furthermore, the present invention can be implemented in other ways different from those described herein; therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0068] This invention designs a dual-stream feature extraction network based on variable patches, simultaneously capturing multi-scale local details and long-distance dependencies, uncovering subtle inter-class differences, and improving classification accuracy. It also proposes a novel adaptive aggregation and distribution network based on an adaptive aggregation and distribution mechanism, which efficiently fuses multi-scale features, reduces information loss, addresses the problem of large scale variations, improves subgraph localization accuracy, and enhances generalization ability. Figure 2 , Figure 3 The diagram shown illustrates the detection process and overall model structure of this invention.

[0069] The dual-stream feature extraction network based on variable patches mainly consists of two branches: a local branch of a convolutional neural network (CNN) and a global branch of a Transformer (a network model based on a self-attention mechanism). The local branch of the CNN uses convolutional modules for downsampling operations and the CSPGAM (Cross-Stage Partial Global Attention Module) provided in this invention to extract local detail information; the global branch of the Transformer uses an improved Swin Transformer (a hierarchical visual model using a shift window) network to extract long-range dependencies. Each branch contains four stages.

[0070] The adaptive aggregation and distribution network based on the adaptive aggregation and distribution mechanism first aligns the feature information extracted by the feature extraction network at different scales, performs adaptive weighting operations, assigns different spatial weights to features at different levels, enhances the importance of key levels, and reduces conflicting information from different objects; then it uses the RepConvBlocks module to fuse richer gradient flow information; finally, it distributes the features to different levels for subsequent feature processing.

[0071] To illustrate the effectiveness of the method proposed in this invention, the following detailed description of the above technical solution is provided through a specific embodiment. Please refer to [link / reference]. Figure 1 This invention provides a method for detecting subgraphs in academic papers based on multi-scale features, comprising the following steps:

[0072] Step S1: Read the original image and obtain the local feature information and global feature information of the original image. The local feature information is the local feature information with multiple scale levels extracted by the convolutional neural network, and the global feature information is the global feature information with multiple scale levels extracted by the Transformer.

[0073] Step S2: Based on the local feature information and global feature information, use an adaptive aggregation and distribution network to obtain multi-scale feature fusion information;

[0074] Step S3: Based on the multi-scale feature fusion information, obtain sub-graph localization information and sub-graph category information.

[0075] The specific steps of steps S1, S2, and S3 will be described below.

[0076] For step S1, in the academic paper subgraph detection method based on multi-scale features provided by this invention, the original image to be processed is first read. Then, the image is preprocessed, which includes adjusting the image size using OpenCV to match the expected size of the model input layer.

[0077] In some embodiments, the resolution of the image is adjusted to 640x640 pixels. Resizing may include operations such as scaling, cropping, or padding. For example, if the aspect ratio of the original image is inconsistent with the target size, appropriate cropping or padding operations may be required before and after scaling.

[0078] The convolutional neural network provided by this invention extracts local detail information at different scales through local branches, such as... Figure 4 The image shown is a detailed diagram of a local branch of a convolutional neural network. The local branch feature extraction module of a convolutional neural network is divided into 5 parts:

[0079] Part 1: The input image is first passed through two convolutional modules with kernel size k=3, stride s=2, and padding p=1 to achieve downsampling, and the channels of its feature map are changed to 64 and 128 respectively. Using convolution to achieve downsampling, compared with pooling layers, can enhance the network's learning ability, because pooling layers are a prior downsampling method, while the parameters of the convolution operation mentioned above are learned.

[0080] The resulting feature map is then input into the CSPGAM feature extraction module. The implementation details of the CSPGAM module are as follows: Figure 5 As shown, it mainly utilizes the idea of ​​cross-stage partial network (CSPNet) for stream extraction, combined with residual structure and efficient layer aggregation network (ELAN) concepts, and employs the channel attention and spatial attention mechanisms of global attention module (GAM) to design the CSPGAM module. Here, the main branch gradient module of the cross-stage partial network (CSP) is a BottleGAM module (a structure combining Bottleneck structure and GAM attention mechanism), also known as a residual module. The number of stacked modules is controlled by the parameter n, meaning that the value of n varies for different model sizes. In this invention, the values ​​of the parameter n for the four CSPGAM modules in the subgraph detection of this academic paper are 3, 6, 6, and 3, respectively.

[0081] The input feature map is fed into a convolution with k=1, s=1, p=0 to halve the number of channels, further reducing computation. Then, a chunking function divides the data into several blocks along a specified dimension. In this invention, the number of channels is split in two; one part is fed into n BottleGAM residual modules, and then concatenated with the other part. Finally, it is fed into a convolution with k=1, s=1, p=0 to halve the number of channels. The BottleGAM module incorporates a global attention mechanism, which can better capture the spatial and channel information of the image, extract key features, and effectively solve the problem of sub-image category similarity in academic images. The final result of GSPGAM is feature extraction of the image without changing its size.

[0082] The second, third, and fourth parts each consist of a convolutional module with k=3, s=2, and p=1 and a CSPGAM feature extraction module. The convolutional module performs downsampling, and the CSPGAM module extracts features.

[0083] The fifth part is the Spatial Pyramid Pooling-Fusion (SPPF) module. After being input into the convolutional module, it undergoes three consecutive 5x5 max-pooling layers with convolutional kernels, and the results of each step are concatenated. This can solve the multi-scale problem to some extent, effectively avoid image distortion caused by image region cropping and scaling operations, and solve the problem of repetitive feature extraction in convolutional neural networks, greatly improving the speed of candidate box generation and saving computational costs.

[0084] In some embodiments, the local feature information with multiple scale levels extracted by the convolutional neural network is obtained in the following way:

[0085] The input information of the convolutional neural network undergoes multiple convolution processes, multiple cross-level local and global attention model processes, and feature pyramid module processes to obtain the local feature information with multiple scale levels.

[0086] In some embodiments, the input information of the convolutional neural network undergoes multiple convolutional processes, multiple cross-level local and global attention model processes, and feature pyramid module processes to obtain the local feature information with multiple scale levels, specifically including:

[0087] Step S1-1-1: Perform convolution processing on the input image;

[0088] Step S1-1-2: Process the output of step S1-1-1 using the spatial and channel attention module to obtain the output information of the current scale level;

[0089] Step S1-1-3: Repeat steps S1-1-1 and S1-1-2 to obtain the output information corresponding to multiple scale levels;

[0090] Step S1-1-4: Based on the output information of the multiple scale levels, obtain local feature information of the multiple scale levels.

[0091] The specific steps of step S1-1-1 include: performing convolution processing on the input image with kernel size k=3, stride s=2, and padding p=1 to achieve downsampling operation;

[0092] Step S1-1-2 specifically includes: processing the output of step S1-1-1 using a CSPGAM attention module. The CSPGAM processing steps include: inputting the input feature map into a convolution with kernel size k=1, stride s=1, and padding margin p=0, halving the number of channels in the feature map; then dividing the channels of the feature map into two parts using a splitting function, one part of which is input into n BottleGAM modules, concatenating the outputs of all BottleGAM modules with the other part of the channels in the feature map, and then performing a convolution with kernel size k=1, stride s=1, and padding margin p=0, halving the number of channels in the feature map again, which is then used as the output information for the current scale level.

[0093] The specific steps S1-1-3 include: repeating the operations in steps S1-1-1 and S1-1-2 three times. During the repetition, the number n of BottleGAM modules in step S1-1-2 is set to 3, 6, 6, and 3 respectively, to obtain output information at multiple scale levels.

[0094] Step S1-1-4 specifically includes: taking the output information of the first three scale levels as the local feature information of the first three scale levels respectively, performing max pooling processing on the output information of the fourth scale level three times with a 5*5 kernel, and concatenating the results of each max pooling processing as the local feature information of the fourth scale level.

[0095] The Transformer global branch provided in this invention extracts long-distance dependencies at different scales. This network architecture adopts a hierarchical design, comprising four stages. Each stage reduces the resolution of the input feature map, similar to how convolutional neural networks increase the receptive field layer by layer. The detailed process is as follows:

[0096] 1) First, the image is divided into patches. For an input image, in the Swin transformer network, each patch is 4x4 in size, with an image width of W and a height of H, resulting in a total of H / 4*W / 4 patches. The input image has 3 color channels, so the feature dimension of each patch is 4x4x3 = 48. This invention employs a variable patching method, adaptively segmenting the image into patches with different positions and sizes in a data-driven manner, rather than using a predefined fixed patching method, thus avoiding the destruction of image semantic information. The specific steps are as follows:

[0097] The position and size of each patch are converted into prediction parameters based on the input content. Each patch can be viewed as a rectangular region, with its center coordinates represented as (x...). ct y ct For position, the model predicts an offset (δx, δy), and the patch can move around the center. As for size, it is represented as (S... h S w This results in a new rectangular region, whose top-left and bottom-right corner coordinates are represented as follows:

[0098]

[0099]

[0100]

[0101]

[0102] A new branch is added to predict these parameters. After inputting the feature map, the parameters of all patches are predicted first, and then the region features are embedded. The prediction method is as follows:

[0103] δx, δy = Tanh(W offest *f p (A))

[0104] S w S h =RELU(Tanh(W scale *f p (A)+b scale ))

[0105] Among them, f p () is a linear layer, b scaleInitializing to s (each patch has s*s pixels) and setting other values ​​to 0 ensures that the initial optimization is consistent with the patch embedding method in the existing ViT (Vision Transformer, a network model based on a vision-based self-attention mechanism) model.

[0106] After determining the rectangular region, the features of each patch can be extracted. Given the coordinates of the upper left and lower right corners of the rectangular region, k*k points are uniformly sampled in this region. The coordinates of the sampled points are mostly decimals, so bilinear interpolation is used when calculating the features of the sampled points. The features of the surrounding points are used to calculate the features of the current point through bilinear interpolation.

[0107] 2) Perform linear embedding (LE). Linear embedding refers to a linear transformation layer, which transforms the dimension of the vector into a pre-set value that can be accepted by the subsequent Transformer. In this invention, this hyperparameter is set as C. The value of C can be set to a specific value according to actual needs. This invention does not limit the value of C.

[0108] 3) The Swing Transformer Block further processes the data. A detailed diagram of the Swing Transformer Block is shown below. Figure 6 As shown, the linear embedding step normalizes all dimensions of features for a single sample; the window-based multi-head self-attention (W-MSA) module first divides the feature map into windows of size M*M, and then performs self-attention processing within each window individually; the shifted window multi-head self-attention (SW-MSA) module first shifts the window elements through cyclic shifting, then performs window partitioning processing, dividing the feature map into windows of size 7*7 according to window_size=7, and then performing multi-head self-attention (MSA) processing within the window. This process only updates the weights and does not change the data dimensions. Then, the data is restored to the input image size, and the previously shifted window elements are moved back to their original positions through reverse cyclic shifting operations.

[0109] 4) Patch Merging: Assuming the input to the patch merging module is a 4x4 single-channel feature map, the module divides each 2x2 adjacent pixel area into a feature map patch. Then, pixels at the same position within each feature map patch are concatenated to obtain four feature maps. These four feature maps are then concatenated along the depth direction, followed by layer normalization. Finally, a fully connected layer linearly transforms the feature map's depth, reducing the depth from C to C / 2. After the patch merging layer, the height and width of the feature map are halved, while the depth is doubled.

[0110] 5) Repeat the above steps 3 times in sequence to obtain the global feature information of stage 2, stage 3 and stage 4 respectively.

[0111] In some embodiments, the global feature information with multiple scale levels extracted by the Transformer is obtained in the following way:

[0112] The input raw image undergoes variable patch partitioning, multiple linear encoding processes, multiple SwinTransformer processes, and multiple patch merging processes to obtain global feature information with multiple scale levels.

[0113] In some embodiments, the patch partitioning includes:

[0114] Step S1-2-1, determine the prediction parameters of the patch, including: pre-dividing the image into multiple patches, and predicting the position and size of all patches.

[0115] Step S1-2-2: Based on the prediction parameters of the Patch, obtain multiple Patches after division, including: based on the position and size information of the Patch, obtain the coordinate information of the rectangular region where the Patch is located, uniformly sample m*m points in the rectangular region where each Patch is located, and calculate the features of the current point through bilinear interpolation.

[0116] Step S1-2-3: Perform linear encoding on the input information, and pre-set the dimension of the vector to a value that the Transformer can accept;

[0117] Step S1-2-4: Perform single-sample feature normalization on the output feature map of step S1-2-3. The W-MSA module first divides the feature map into M*M windows, and then encodes each window individually using a self-attention mechanism. The SW-MSA module first shifts the window elements, and then divides the window into windows of size 7. MSA operations are then performed within the windows. The window attention mechanism performs MSA operations within the 7*7 windows. This process only updates the weights and does not change the data dimensions. Then, the data is restored to the input image size, and the previously shifted windows are shifted in the reverse direction.

[0118] Step S1-2-5: Perform patch merging processing on the output feature map of step S1-2-4, including: dividing each 2x2 adjacent pixel into a patch, then concatenating the pixels at the same position in each patch to obtain 4 feature maps, concatenating these four feature maps in the depth direction, then performing normalization layer processing, and finally using a fully connected layer to perform linear transformation in the depth direction of the feature map to halve the depth of the feature map;

[0119] Step S1-2-6: Repeat steps S1-2-3, S1-2-4, and S1-2-5 three times. During the repetition, obtain the global feature information of the corresponding scale level.

[0120] In some embodiments, the window-based multi-head self-attention processing includes: dividing the input normalized feature map into windows of size M×M, and then performing self-attention processing independently within each window; the moving window multi-head self-attention processing includes cyclically shifting and adjusting the input normalized feature map, dividing the feature map into windows of size 7*7, then performing self-attention processing within each window, and then performing reverse cyclic shifting processing to restore the previously shifted window to its original position.

[0121] For step S2, based on the local feature information and global feature information, an adaptive aggregation and distribution network is used to obtain multi-scale feature fusion information;

[0122] In the multi-scale feature fusion step of the present invention, the multi-scale features are first spliced ​​together, including splicing the features extracted from the local branch of the convolutional neural network and the global branch of the Transformer at four different stages.

[0123] Then, the feature fusion network's adaptive aggregation module processes the data, such as... Figure 7The diagram shows a detailed view of the adaptive aggregation module in the adaptive aggregation-distribution fusion network. The four features of different scales, concatenated in the previous step, are input into this module. First, the four features of different scales are aligned to the distribution level. If the feature level is higher than the aligned feature level, bilinear interpolation is used for upsampling; if the feature level is lower, downsampling average pooling is used for downsampling. Considering the different contributions of features at different levels, adaptive feature weighting is finally applied to assign different spatial weights to features at different levels, enhancing the importance of key levels and reducing conflicting information from different objects, resulting in aligned features. Assuming the features after concatenation in the backbone network are p2, p3, p4, and p5, and l2, l3, l4, and l5 are the corresponding spatial weights, then the resulting adaptively fused feature is p. The adaptive feature weighting formula is shown below:

[0124] p = p2 * l2 + p3 * l3 + p4 * l4 + p5 * l5

[0125] like Figure 8 The diagram shows the details of the information fusion module. It performs convolution operations on the input features to reduce computation, and then inputs them into the RepConvBlocks module to extract and fuse richer gradient flow information.

[0126] like Figure 9 The diagram shows the details of the feature distribution module. Its input consists of the features obtained through information fusion and the adjacent and initial features extracted by the backbone network. The adjacent and initial features are directly concatenated after being aligned in size, and then input into two 1*1 convolutional modules. For the features obtained through information fusion, the module is divided into two branches. One branch performs a 1*1 convolution operation, then inputs into the sigmoid module, performs a downsampling operation, and multiplies it with the features obtained from the concatenation of the initial and adjacent features. The other branch performs a 1*1 convolution operation, then performs a downsampling operation, adds it to the features obtained from the other branch, and inputs it into the RepConvBlocks module. Finally, the features are distributed to the corresponding layers.

[0127] Then, an adaptive spatial fusion operation is performed. The features obtained by the feature distribution module should have been directly input into the detection layer for subsequent feature processing. Inspired by the Adaptive Feature Pooling Network (AFPN), an adaptive feature weighting fusion operation is performed again on features at different scales. This fusion operation can take into account the different contribution levels of features at different levels while enriching the feature information.

[0128] In some embodiments, step S2 specifically includes:

[0129] Step S2-1: Concatenate the local feature information and global feature information according to their corresponding levels;

[0130] Step S2-2: Adaptively aggregate the concatenated feature information, including:

[0131] Align the feature sizes of multiple scale levels in the spliced ​​feature information with the scale level to be distributed, and assign different spatial weights to the features of different levels to perform adaptive feature weighting, so as to obtain the weighted fused features.

[0132] Step S2-3: Perform information fusion on the adaptively clustered features;

[0133] Step S2-4: Based on the information obtained in step S2-3, the fused features are combined with the local and global feature information obtained in step S1 to perform feature distribution operations, and the results are distributed to the corresponding scale level.

[0134] Step S2-5: Perform adaptive spatial fusion operation on the features distributed to the corresponding scale level.

[0135] In some embodiments, concatenating the local feature information and the global feature information includes concatenating the local feature information at four scale levels extracted by the convolutional neural network and the global feature information at four scale levels extracted by the Transformer.

[0136] In some embodiments, aligning features of multiple scale levels in the concatenated feature information to the level to be distributed includes: if the features of the current scale level are higher than the level to be distributed, then performing an upsampling operation using bilinear interpolation; if the features of the current scale level are lower than the level to be distributed, then performing a downsampling operation using downsampling average pooling.

[0137] In some embodiments, the specific method for information fusion of adaptively aggregated features is as follows: the result of step S2-2 is convolved, and then input into the RepConvBlocks module to calculate and extract gradient flow information.

[0138] In some embodiments, the feature distribution operation based on the fused features obtained in steps S2-3 and the local and global feature information obtained in step S1 specifically includes:

[0139] Step S2-4-1: For the adjacent layer feature information after concatenating the local feature information and global feature information at a scale level, perform size alignment and concatenation, and then perform two 1*1 convolutions.

[0140] Step S2-4-2: Perform a 1*1 convolution on the fused features, then calculate the sigmoid function, then perform average pooling downsampling, and multiply the result of step S2-4-1.

[0141] Step S2-4-3: Perform a 1*1 convolution on the fused features, then perform average pooling downsampling, add the result of step S2-4-2, and then perform RepConvBlocks module calculation to obtain the distribution features of the corresponding scale level.

[0142] In some embodiments, the adaptive spatial fusion operation is implemented using an adaptive feature pooling network;

[0143] In some embodiments, before reading the original image in step S1, a pre-trained network model is loaded. This network model includes convolutional neural networks, Transformers, adaptive aggregation and distribution networks, etc., provided in this invention. The pre-trained network model is obtained through deep learning training and is pre-loaded as a pre-trained network model. The training method for deep neural networks is a conventional technique in the art, and this invention does not limit the training method for deep neural networks.

[0144] For step S3, based on the multi-scale feature fusion information, sub-graph localization information and sub-graph category information are obtained.

[0145] In this invention, the detection head adopts a decoupled head structure, separating the classification and detection heads. The localization branch first undergoes two convolution operations with k=3, s=1, p=1, followed by one convolution operation with k=1, s=1, p=0. Due to the use of Distribution Focal Loss (DFL), the number of channels in the third convolution is set to the form 4*reg_max, where the maximum regularization parameter reg_max defaults to 16. The loss function for the localization branch is set to include two parts: the CIoU function and the DFL function. Similarly, the classification branch first undergoes two convolution operations with k=3, s=1, p=1, followed by one convolution operation with k=1, s=1, p=0. The number of channels equals the number of categories to be classified. The loss function for the classification branch is set to the Binary Cross-Entropy (BCE) loss function.

[0146] The sub-image detection results will include location coordinates and classification categories.

[0147] In some embodiments, step S3 includes:

[0148] The positioning coordinates are obtained based on the positioning branch network. The positioning branch network is specifically set as follows: the positioning branch first goes through two convolution operations with k=3, s=1, p=1, and then goes through a convolution operation with k=1, s=1, p=0. The number of channels of the third convolution is set to the form of 4*reg_max, where the value of the maximum regularization parameter reg_max is 16 by default. The loss function of the positioning branch is set to include two parts: CIoU function and DFL function.

[0149] The classification branch network is used to obtain the classification categories. Specifically, the classification branch network is configured as follows: after two convolution operations with k=3, s=1, p=1, and then after one convolution operation with k=1, s=1, p=0, the number of channels is set to the number of categories to be classified, and the loss function of the classification branch is set to the binary cross-entropy function.

[0150] While the specific embodiments of the present invention depict actions or steps in a particular order, this should be understood as requiring such actions or steps to be performed in the specific order shown or in sequential order, or requiring all illustrated actions or steps to be performed to achieve the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations.

[0151] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for detecting subgraphs in academic papers based on multi-scale features, comprising the following steps: Step S1: Read the original image and obtain the local feature information and global feature information of the original image. The local feature information is the local feature information with multiple scale levels extracted by the convolutional neural network, and the global feature information is the global feature information with multiple scale levels extracted by the swin Transformer of the variable patch. Step S2: Based on the local feature information and global feature information, use an adaptive aggregation and distribution network to obtain multi-scale feature fusion information; Step S2 specifically includes: Step S2-1: Concatenate the local feature information and global feature information according to their corresponding levels; Step S2-2: Adaptively aggregate the concatenated feature information, including: Align the feature sizes of multiple scale levels in the spliced ​​feature information with the scale level to be distributed, and assign different spatial weights to the features of different levels to perform adaptive feature weighting, so as to obtain the weighted fused features. Step S2-3: Perform information fusion on the adaptively clustered features; Step S2-4-1: For the adjacent layer feature information after concatenating local and global feature information at a scale level, perform size alignment and concatenation, and then perform two concatenation operations with a size of 1. Convolution of 1; Step S2-4-2: Perform a step on the features after information fusion with a size of 1. The convolution is performed with 1, then sigmoid is calculated, then average pooling is performed and downsampled, and then multiplied with the result of step S2-4-1; Step S2-4-3: Perform a step on the features after information fusion with a size of 1. The convolution of step 1 is then performed, followed by average pooling downsampling, and then added to the result of step S2-4-2. Finally, the RepConvBlocks module is used to calculate and obtain the distribution features at the corresponding scale level. Step S2-5: Perform adaptive spatial fusion operation on the features distributed to the corresponding scale level; Step S2-1 specifically includes: concatenating the local feature information of the four scale levels extracted by the convolutional neural network and the global feature information of the four scale levels extracted by the swin Transformer of the variable patch. In step S2-2, aligning the feature sizes of multiple scale levels in the spliced ​​feature information to the scale level to be distributed includes: if the feature of the current scale level is higher than the scale level to be distributed, then upsampling is performed using bilinear interpolation; if the feature of the current scale level is lower than the scale level to be distributed, then downsampling is performed using downsampling average pooling. Step S2-3 specifically includes: performing a convolution operation on the result of step S2-2, and then processing it with the RepConvBlocks module to obtain the extracted gradient flow information; In steps S2-5, the adaptive spatial fusion operation is implemented using an adaptive feature pooling network; Step S3: Based on the multi-scale feature fusion information, obtain sub-graph localization information and sub-graph category information.

2. The academic paper subgraph detection method based on multi-scale features according to claim 1, characterized in that: The method for obtaining the local feature information with multiple scale levels extracted by the convolutional neural network is as follows: the convolutional neural network performs multiple convolution processes, multiple spatial and channel attention module processes, and feature pyramid module processes on the input original image to obtain the local feature information with multiple scale levels. The global feature information with multiple scale levels extracted by the swin Transformer of the variable patch is obtained by: the input original image is divided into variable patches, processed by multiple linear encoding processes, processed by multiple swinTransformers processes, and processed by multiple patch merging processes to obtain the global feature information with multiple scale levels.

3. The academic paper subgraph detection method based on multi-scale features according to claim 2, characterized in that, The convolutional neural network processes the input raw image through multiple convolutions, multiple spatial and channel attention modules, and a feature pyramid module to obtain the local feature information with multiple scale levels, specifically including: Step S1-1-1: Perform convolution processing on the input image; Step S1-1-2: Process the output of step S1-1-1 using the spatial and channel attention module to obtain the output information of the current scale level; Step S1-1-3: Repeat steps S1-1-1 and S1-1-2 to obtain the output information corresponding to multiple scale levels; Step S1-1-4: Based on the output information of the multiple scale levels, obtain local feature information of the multiple scale levels.

4. The academic paper subgraph detection method based on multi-scale features according to claim 3, characterized in that: The specific steps of step S1-1-1 include: performing convolution processing on the input image with kernel size k=3, stride s=2, and padding margin p=1 to achieve downsampling operation; Step S1-1-2 specifically includes: processing the output of step S1-1-1 using a CSPGAM attention module. The CSPGAM processing steps include: inputting the input feature map into a convolution with kernel size k=1, stride s=1, and padding margin p=0, halving the number of channels in the feature map; then dividing the channels of the feature map into two parts using a splitting function, one part of which is input into n BottleGAM modules, concatenating the outputs of all BottleGAM modules with the other part of the channels in the feature map, and then performing a convolution with kernel size k=1, stride s=1, and padding margin p=0, halving the number of channels in the feature map again, and using this as the output information for the current scale level. The specific steps S1-1-3 include: repeating the operations in steps S1-1-1 and S1-1-2 three times. During the repetition, the number n of BottleGAM modules in step S1-1-2 is set to 3, 6, 6, and 3 respectively, to obtain output information at multiple scale levels. Step S1-1-4 specifically includes: using the output information of the first three scale levels as local feature information of the first three scale levels respectively, and performing 3 convolution kernels on the output information of the fourth scale level. Max pooling of size 5 is performed, and the results of each max pooling process are concatenated to form the local feature information of the fourth scale level.

5. The academic paper subgraph detection method based on multi-scale features according to claim 2, characterized in that, The input original image undergoes variable patch partitioning, multiple linear encoding processes, multiple Swin Transformer processes, and multiple patch merging processes to obtain the global feature information with multiple scale levels, specifically including: Step S1-2-1: Determine the prediction parameters of the patch, including: pre-dividing the image into multiple patches, and predicting the position and size of all patches. The prediction method is as follows: in,( , ), ( , ) are the coordinates of the top left and bottom right corners of the Patch rectangle, respectively. , ) is the center coordinate of the patch, ( δx , δy ) is the offset of the Patch center coordinates, ( S h , S w ) represents the height and width of the patch. It is a linear layer, where A is the input feature map, and Tanh() and ReLU() are activation functions. For offset weights, To scale the weights, For scaling offset; Step S1-2-2: Based on the predicted parameters of the patch, obtain multiple partitioned patches, including: based on the position and size information of the patch, obtain the coordinate information of the rectangular region where the patch is located, and uniformly sample m samples within the rectangular region where each patch is located. Given m points, calculate the features of the current point using bilinear interpolation; Step S1-2-3: Perform linear encoding on the input information, and pre-set the dimension of the vector to a value that the Transformer can accept; Step S1-2-4: Perform single-sample feature normalization on the output feature map of step S1-2-3; the W-MSA module first divides the feature map into M... The system first divides the window into individual windows of size M, then encodes the self-attention mechanism within each window separately. The SW-MSA module first shifts the window elements, then divides the window into individual windows of size 7, and subsequently performs MSA operations within each window. Finally, the window attention mechanism is applied within the window of size 7. The MSA operation is performed within the 7-window. This process only updates the weights and does not change the data dimensions. Then, the data is restored to the input image size, and the previously shifted window is shifted in the reverse direction. Step S1-2-5: Perform patch merging processing on the output feature map of step S1-2-4, including: dividing each 2x2 adjacent pixel into a patch, then concatenating the pixels at the same position in each patch to obtain 4 feature maps, concatenating these 4 feature maps in the depth direction, then performing normalization layer processing, and finally using a fully connected layer to perform linear transformation in the depth direction of the feature map to halve the depth of the feature map; Step S1-2-6: Repeat steps S1-2-3, S1-2-4, and S1-2-5 three times. During the repetition, obtain the global feature information of the corresponding scale level.

6. The academic paper subgraph detection method based on multi-scale features according to claim 1, characterized in that, Step S3 specifically includes: The positioning coordinates are obtained based on a positioning branch network. Specifically, the positioning branch first undergoes two convolution operations with a size k=3, stride s=1, and padding p=1; then it undergoes another convolution operation with a size k=1, stride s=1, and padding p=0. The number of channels in the third convolution is set to 4. The form reg_max, where the maximum regularization parameter reg_max defaults to 16, sets the loss function for the localization branch to include two parts: the CIoU loss function and the DFL loss function; The classification branch network is used to obtain the classification categories. Specifically, the classification branch network is configured as follows: it performs two convolution operations with a convolution size of k=3, a stride of s=1, and a padding margin of p=1, and then performs one convolution operation with a convolution size of k=1, a stride of s=1, and a padding margin of p=0. The number of channels is set to the number of categories to be classified, and the loss function of the classification branch is set to the binary cross-entropy function.

Images