A seam cropping image detection method and system based on CNN

Abstract

The present invention discloses a seam cropping image detection method and system based on CNN. The method obtains a local feature map of the image to be detected by inputting the image to be detected into a preset convolutional neural network module; the image to be detected is input into a preset fast Fourier convolution block module to obtain a global feature map of the image to be detected; the local feature map is connected with the global feature map to obtain a connected feature image; a softmax classifier is used to perform pixel detection on the connected feature map to obtain a detection result of whether the image to be detected is cropped. The system proposes a local feature extraction module based on an attention mechanism and a global feature extraction module based on fast Fourier convolution. Compared with the existing scheme, the method can extract the global features and local features of the image to be detected, so that the detection features finally obtained by fusion are more comprehensive, and the classification and positioning are more accurate.

Description

Technical Field

[0001] The present invention relates to the technical field of image detection, and in particular to a CNN-based seam cropping image detection method and system. Background Art

[0002] With the rapid development of Internet technology and the popularity of mobile phones and computers, people's lives are becoming more and more convenient and fast. People have more opportunities to access a large number of images. At the same time, images, as a carrier of information, have been spread more quickly. In order to adapt to the display of different devices, image scaling technology is particularly important. Traditional image scaling methods often cause distortion or content loss on images of the same resolution on different devices. In order to effectively solve this problem, content-aware resizing technology has emerged. Content-aware image resizing is the process of resizing an image while retaining its important content. Among them, the most widely used content-aware image resizing technology is seam carving.

[0003] The image that has been cropped by seam cutting retains the important parts of the image well under the premise of adjusting the image size. Under the adjustment of a small ratio, it is enough to be indistinguishable from the original image and the human eye cannot distinguish whether it is the original image or the cropped image. Therefore, the detection of image cropping technology based on seam cutting is particularly important.

[0004] The existing image detection technology based on Seam carving still has the following defects:

[0005] Currently, when an image is seam cropped, the image size will be reduced and the Seam needs to be inserted again to expand the image. At this time, the distribution of pixel energy of the image will be changed, which will offset the change in energy distribution caused by the image, resulting in a decrease in detection performance. Summary of the invention

[0006] The present invention aims to at least solve the technical problems existing in the prior art. To this end, the present invention proposes a method, system, electronic device and storage medium for detecting seam cropping images based on CNN. The detection effect of cropped images based on seam cropping is enhanced by a local feature extraction module based on an attention mechanism and a global feature extraction module based on fast Fourier convolution.

[0007] A first aspect of the present invention provides a CNN-based seam cropping image detection method, comprising the following steps:

[0008] Inputting the image to be detected into a preset convolutional neural network module to obtain a local feature map of the image to be detected; wherein the convolutional neural network module includes a spatial attention mechanism and a channel attention mechanism combined with each other;

[0009] Inputting the image to be detected into a preset fast Fourier convolution block module to obtain a global feature map of the image to be detected;

[0010] Connecting the local feature map with the global feature map to obtain a connected feature image;

[0011] A softmax classifier is used to perform pixel detection on the connected feature map to obtain a detection result of whether the image to be detected is cropped.

[0012] According to the embodiments of the present invention, there are at least the following technical effects:

[0013] This method obtains the local feature map of the image to be detected by inputting the image to be detected into a preset convolutional neural network module; wherein the convolutional neural network module includes a spatial attention mechanism and a channel attention mechanism combined with each other, and inputs the image to be detected into a preset fast Fourier convolution block module to obtain the global feature map of the image to be detected; the local feature map is connected with the global feature map to obtain a connected feature image; the softmax classifier is used to perform pixel detection on the connected feature map to obtain the detection result of whether the image to be detected is cropped. This method proposes a local feature extraction module based on the attention mechanism and a global feature extraction module based on fast Fourier convolution. Compared with the existing scheme, this method extracts the global features and local features of the image to be detected, so that the final fused detection features are more comprehensive, and the classification and positioning are more accurate.

[0014] According to some embodiments of the present invention, the convolutional neural network module uses the ResNet50 network as the backbone network for extracting local features; an A_MEMM module is provided after each feature extraction stage in the ResNet50, and the A_MEMM module includes a channel attention module, a spatial attention module and a maximum entropy Markov module, the channel attention module is used to extract features of the first feature map to obtain a channel attention feature map, and the first feature map refers to the feature map output by the feature extraction stage; the spatial attention module is used to extract features of the second feature map to obtain a spatial attention feature map, and the second feature map refers to the feature map obtained by performing element-by-element multiplication operation on the channel attention feature map and the first feature map; the maximum entropy Markov module is used to increase the correlation between adjacent regions in the comprehensive feature map, and the comprehensive feature map refers to the feature map obtained by performing element-by-element multiplication operation on the spatial attention feature map and the second feature map.

[0015] According to some embodiments of the present invention, the ResNet50 network includes 5 feature extraction stages, wherein the 1st feature extraction stage, the 2nd feature extraction stage and the 3rd feature extraction stage are ordinary convolutions, the 4th feature extraction stage and the 5th feature extraction stage are composed of two residual blocks containing different numbers, and the 4th feature extraction stage and the 5th feature extraction stage are added with dilated convolutions, and the ordinary convolution is a convolution including a convolution layer, an activation layer and a pooling layer.

[0016] According to some embodiments of the present invention, the step of inputting the image to be detected into a preset convolutional neural network module to obtain a local feature map of the image to be detected includes:

[0017] The image to be detected is subjected to the first feature extraction stage to obtain a feature map F1 reduced to 1 / 2 of the image to be detected;

[0018] The feature map F1 is subjected to a second feature extraction stage to obtain a feature map F2 reduced to 1 / 4 of the image to be detected;

[0019] The feature map F2 is subjected to the third feature extraction stage to obtain a feature map F3 reduced to 1 / 8 of the image to be detected;

[0020] The feature map F3 is subjected to the fourth feature extraction stage to obtain a feature map F4 reduced to 1 / 16 of the image to be detected;

[0021] The feature map F4 is subjected to the fifth feature extraction stage to obtain a feature map F5 reduced to 1 / 32 of the image to be detected;

[0022] Perform full convolution layer operations FC6 and FC7 on the feature map F3, the feature map F4 and the feature map F5 to obtain a heat map H31, a heat map H41 and a heat map H51;

[0023] Convolving the heat map H31, the heat map H41 and the heat map H51 using a convolution function to obtain a heat map H3, a heat map H4 and a heat map H5, wherein the step size of the convolution function is 1, the kernel size is 1×1 and the number of convolution kernels is 2;

[0024] Convolve the heat map H5 using a transposed convolution with a kernel size of 4×4 and a step size of 2 to obtain a convolution result 1, and add the convolution result 1 and the heat map H4 element by element to obtain an addition result 1;

[0025] Convolving the addition result 1 with a transposed convolution with a kernel size of 4×4 and a step size of 2 to obtain a convolution result 2, and adding the convolution result 2 and the heat map H3 element by element to obtain an addition result 2;

[0026] The addition result 2 is convolved using a transposed convolution with a kernel size of 16×16 and a step size of 8 to obtain a local feature map of the image to be detected.

[0027] According to some embodiments of the present invention, the fast Fourier convolution block module includes a plurality of cascaded fast Fourier convolution blocks, and the structures of the plurality of fast Fourier convolution blocks are the same; inputting the image to be detected into a preset fast Fourier convolution block module to obtain a global feature map of the image to be detected, comprising:

[0028] Inputting the image to be detected into the first fast Fourier convolution block in the fast Fourier convolution block module to obtain a feature map output by the first fast Fourier convolution block;

[0029] Input the feature map output by the first fast Fourier convolution block to the next fast Fourier convolution block, and so on, until the last fast Fourier convolution block outputs the global feature map of the image to be detected; wherein any one of the multiple fast Fourier convolution blocks performs the following operations:

[0030] Use ordinary convolution to extract local features L1 and local features L2 of the target object; the target object refers to the feature map output by the previous fast Fourier convolution block or the image to be detected, and the ordinary convolution is a convolution including a convolution layer, an activation layer and a pooling layer;

[0031] Use ordinary convolution to extract the local feature G1 of the target object, and use channel-level fast Fourier transform to extract the global feature G2 of the target object;

[0032] Add the local feature L1 and the global feature G1 to obtain a local feature 1; add the local feature L2 and the global feature G2 to obtain a global feature 1;

[0033] The local feature 1 and the global feature 1 are batch normalized and Relu activated to obtain an activated local feature 2 and an activated global feature 2;

[0034] The activated local feature 2 and the activated global feature 2 are fused to obtain a feature map output by the fast Fourier convolution block.

[0035] According to some embodiments of the present invention, extracting the global feature G2 of the image to be detected by using channel-level fast Fourier transform includes:

[0036] Using 1×1 convolution to halve the number of channels of the image to be detected;

[0037] The image features to be detected are converted from the spatial domain to the frequency domain using fast Fourier transform to obtain the frequency domain results;

[0038] The frequency domain results are batch normalized and linearly activated using a convolution block, and restored to the spatial domain using a two-dimensional discrete Fourier inverse transform to obtain a spatial domain result;

[0039] The number of channels is restored using 1×1 convolution to obtain the global feature G2.

[0040] According to some embodiments of the present invention, after obtaining the detection result of whether the image to be detected is cropped, the method further includes:

[0041] Marking the cropped area and / or the uncropped area in the image to be detected according to the detection result;

[0042] Generate a binary image based on the marked area;

[0043] When the binary image contains a clipping area, an opening operation of first eroding and then dilating is performed on the binary image to obtain an observation result.

[0044] A second aspect of the present invention provides a CNN-based seam cropping image detection system, comprising:

[0045] A local feature module, used for inputting the image to be detected into a preset convolutional neural network module to obtain a local feature map of the image to be detected; wherein the convolutional neural network module includes a spatial attention mechanism and a channel attention mechanism combined with each other;

[0046] A global feature module, used for inputting the image to be detected into a preset fast Fourier convolution block module to obtain a global feature map of the image to be detected;

[0047] A feature connection module, used for connecting the local feature map with the global feature map to obtain a connected feature image;

[0048] The result output module is used to perform pixel detection on the connected feature map using a softmax classifier to obtain a detection result of whether the image to be detected is cropped.

[0049] This system obtains the local feature map of the image to be detected by inputting the image to be detected into the preset convolutional neural network module; wherein the convolutional neural network module includes a spatial attention mechanism and a channel attention mechanism combined with each other, inputs the image to be detected into the preset fast Fourier convolution block module, obtains the global feature map of the image to be detected; connects the local feature map with the global feature map to obtain the connected feature image; uses the softmax classifier to perform pixel detection on the connected feature image to obtain the detection result. This system proposes a local feature extraction module based on the attention mechanism and a global feature extraction module based on fast Fourier convolution. Compared with the existing scheme, this system extracts the global features and local features of the image to be detected, so that the final fused detection features are more comprehensive, and the classification and positioning are more accurate.

[0050] According to a third aspect of the present invention, there is provided an electronic device for seam cropping image detection based on CNN, comprising at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the above-mentioned seam cropping image detection method based on CNN.

[0051] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the above-mentioned CNN-based seam cropping image detection method.

[0052] It should be noted that the beneficial effects between the second to fourth aspects of the present invention and the prior art are the same as the beneficial effects between the above-mentioned CNN-based seam cropping image detection method and the prior art, and will not be described in detail here.

[0053] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0055] Figure 1 It is a flowchart of a CNN-based seam cropping image detection method provided by an embodiment of the present invention;

[0056] Figure 2 is a structural block diagram of a feature extraction module provided by an embodiment of the present invention;

[0057] Figure 3is a schematic diagram of the structure of A_MEMM provided by an embodiment of the present invention;

[0058] Figure 4 is a structural diagram of a channel attention mechanism structure diagram provided by an embodiment of the present invention;

[0059] Figure 5 is a schematic diagram of a structure of a spatial attention mechanism structure diagram provided by an embodiment of the present invention;

[0060] Figure 6 is a schematic diagram of a structure for extracting a local feature map provided by an embodiment of the present invention;

[0061] Figure 7 It is a schematic diagram of the structure of two residual blocks in the fourth feature extraction stage and the fifth feature extraction stage provided by an embodiment of the present invention;

[0062] Figure 8 is a schematic structural diagram of a fast Fourier convolution block module provided by an embodiment of the present invention;

[0063] Fig. 9 It is a structural diagram of a spectrum converter structure provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0064] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0065] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "axial", "radial", "circumferential" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0066] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0067] Before introducing the embodiment of the present invention, a brief description of image seam cropping is first given:

[0068] A seam is a pixel on the best 8-connected path from top to bottom or from left to right on a single image. The optimality is defined by an energy function. The image aspect ratio is changed by repeatedly removing (Carving) or inserting (Insert) seams in one direction. By applying this operation in two directions, the image can be repositioned to a new size. This technique is introduced by taking the removal of seams in the vertical direction as an example:

[0069] The selected Seam is an 8-connected low-energy seam from top to bottom. The energy of each pixel is calculated using the following energy function:

[0070]

[0071] For an image of size n*m, the vertical Seam is defined as:

[0072]

[0073]

[0074] In the formula, S represents the set of pixels contained in a seam; x represents the mapping from [1,2,…,n] to [1,2,…,m]. This formula indicates that the vertical Seam is an 8-connected path that runs through the entire image from top to bottom, and there is only one pixel in each row of the seam. According to formula (1), the set of pixels contained in image I can be expressed as:

[0075]

[0076] In order to achieve a visually satisfactory scaling effect, the pixel set energy contained in the seam should be as low as possible, so the optimal Seam is defined as:

[0077]

[0078] The search for the optimal Seam is implemented by a dynamic program. Taking the vertical Seam as an example, the image is traversed from the second row to the last row. The horizontal Seam is similar. For each pixel point (i, j), its cost matrix M(i, j) is calculated. The formula is defined as follows:

[0079] M(i,j)=e(i,j)+min(M(i-1,j-1),M(i-1,j+1)) (5)

[0080] The last row of the matrix M(i,j) is the energy value of various vertical seams. The element with the minimum value is locked, and the position of other elements of this seam is traced back from the bottom to the top to obtain the optimal seam.

[0081] The selection and order of Seams can protect the content of the image; they can also be applied to the removal of specific objects in the image. When performing this operation, the area of ​​the object to be removed is first marked, and then the energy value of the area is adjusted to 0. Finally, the energy value of the entire image is calculated to identify important and unimportant areas, and the Seams with the lowest energy values ​​are continuously removed until all marked areas are removed.

[0082] At present, there are mainly detection methods using deep learning and traditional detection methods using non-deep learning, but the existing solutions still have the following defects:

[0083] Currently, when an image is seam cropped, the image size will be reduced and the Seam needs to be inserted again to expand the image. At this time, the distribution of pixel energy of the image will be changed, which will offset the change in energy distribution caused by the image, resulting in a decrease in detection performance.

[0084] In order to solve the above technical defects, refer to Figure 1 to Figure 2 One embodiment of the present invention provides a CNN-based seam cropping image detection method, the method comprising the following steps:

[0085] Step S101: input the image to be detected into a preset convolutional neural network module to obtain a local feature map of the image to be detected; wherein the convolutional neural network module includes a spatial attention mechanism and a channel attention mechanism that are combined with each other.

[0086] Step S102: input the image to be detected into a preset fast Fourier convolution block module to obtain a global feature map of the image to be detected.

[0087] Step S103: connect the local feature map with the global feature map to obtain a connected feature image.

[0088] Step S104: Use a softmax classifier to perform pixel detection on the connected feature map to obtain a detection result of whether the image to be detected is cropped.

[0089] Reference Figure 3 In some embodiments, the convolutional neural network module in step S101 uses the ResNet50 network as the backbone network for extracting local features; an A_MEMM module is provided after each feature extraction stage in ResNet50, and the A_MEMM module includes a channel attention module, a spatial attention module and a maximum entropy Markov module. The channel attention module is used to extract the features of the first feature map to obtain a channel attention feature map, and the first feature map refers to the feature map output in the feature extraction stage; the spatial attention module is used to extract the features of the second feature map to obtain a spatial attention feature map, and the second feature map refers to the feature map obtained by performing element-by-element multiplication operation on the channel attention feature map and the first feature map; the maximum entropy Markov module is used to increase the correlation between adjacent regions in the comprehensive feature map, and the comprehensive feature map refers to the feature map obtained by performing element-by-element multiplication operation on the spatial attention feature map and the second feature map.

[0090] Reference Figure 4 The channel attention module compresses the first feature map in the spatial dimension, performs average pooling (Avg Pooling) and maximum pooling (Max pooling) on ​​the height and width (H×W) of the first feature map to aggregate the spatial information of the feature map, and then sends it to a shared fully connected layer to compress the spatial dimension of the first feature map and sum and merge it element by element, and finally obtains the channel attention feature map through the sigmoid nonlinear activation function.

[0091] Reference Figure 5 The spatial attention module uses the feature map output by the channel attention module as the input feature map. It first performs channel-based global maximum pooling and global average pooling operations on the input feature map, and then stacks the pooled results into a H×W×2 feature layer. It uses a convolution with a channel number of 1 and a kernel of 7×7 to reduce the dimension and obtain a H×W×1 feature layer. The sigmoid nonlinear activation function is then used to generate the spatial attention feature map.

[0092] Reference Figure 6In some embodiments, the ResNet50 network includes 5 feature extraction stages, wherein the 1st feature extraction stage, the 2nd feature extraction stage and the 3rd feature extraction stage are ordinary convolutions, the 4th feature extraction stage and the 5th feature extraction stage are composed of two residual blocks containing different numbers, and the 4th feature extraction stage and the 5th feature extraction stage are added with dilated convolutions, ordinary convolutions are convolutions including convolution layers, activation layers and pooling layers, ordinary convolutions are convolutions including convolution layers, activation layers and pooling layers, which can increase the receptive field and capture multi-scale contextual information, and the dilation rate (r) of each stage and the details of the backbone network are shown in Table 1.

[0093]

[0094] Table 1

[0095] In Table 1, stage1-stage5 refers to the first feature extraction stage to the fifth feature extraction stage, output size refers to the output image size, ResNet50+A_MEMM module refers to the ResNet50+A_MEMM module, and conv number refers to the number of convolutions.

[0096] Reference Figure 7 In some embodiments, the residual block of the 4th feature extraction stage is a convolution module for changing the dimension of the input feature map, and the residual block of the 5th feature extraction stage is a convolution module that does not change the dimension of the input feature map but can connect network layers in series.

[0097] In some embodiments, the image to be detected is input into a preset convolutional neural network module to obtain a local feature map of the image to be detected, including:

[0098] The image to be detected is subjected to the first feature extraction stage to obtain a feature map F1 reduced to 1 / 2 of the image to be detected.

[0099] The feature map F1 is subjected to the second feature extraction stage to obtain a feature map F2 reduced to 1 / 4 of the image to be detected.

[0100] The feature map F2 is subjected to the third feature extraction stage to obtain a feature map F3 reduced to 1 / 8 of the image to be detected.

[0101] The feature map F3 is subjected to the fourth feature extraction stage to obtain a feature map F4 reduced to 1 / 16 of the image to be detected.

[0102] The feature map F4 is subjected to the fifth feature extraction stage to obtain a feature map F5 reduced to 1 / 32 of the image to be detected.

[0103] The feature maps F3, F4 and F5 are subjected to full convolution layer operations FC6 and FC7 to obtain heat maps H31, H41 and H51.

[0104] The heat map H31, the heat map H41 and the heat map H51 are convolved using a convolution function to obtain heat maps H3, H4 and H5, wherein the step size of the convolution function is 1, the kernel size is 1×1 and the number of convolution kernels is 2.

[0105] The heat map H5 is convolved using a transposed convolution with a kernel size of 4×4 and a step size of 2 to obtain a convolution result 1. The convolution result 1 is added element-by-element to the heat map H4 to obtain an addition result 1.

[0106] The addition result 1 is convolved using a transposed convolution with a kernel size of 4×4 and a step size of 2 to obtain a convolution result 2. The convolution result 2 is added element by element to the heat map H3 to obtain an addition result 2.

[0107] The addition result 2 is convolved using a transposed convolution with a kernel size of 16×16 and a step size of 8 to obtain a local feature map of the image to be detected.

[0108] The present invention replaces the fully connected layers of ResNet50 with fully convolutional layers FC6 and FC7, and uses dropout to prevent overfitting, so that it can adapt to images to be detected of any size.

[0109] Reference Figure 8 In some embodiments, the fast Fourier convolution block module in step S102 includes a plurality of cascaded fast Fourier convolution blocks, and the structures of the plurality of fast Fourier convolution blocks are the same; the image to be detected is input into the preset fast Fourier convolution block module to obtain a global feature map of the image to be detected, including:

[0110] The image to be detected is input into the first fast Fourier convolution block in the fast Fourier convolution block module to obtain the feature map output by the first fast Fourier convolution block.

[0111] The feature map output by the first fast Fourier convolution block is input into the next fast Fourier convolution block, and so on, until the last fast Fourier convolution block outputs the global feature map of the image to be detected; wherein any one of the multiple fast Fourier convolution blocks performs the following operation.

[0112] Ordinary convolution is used to extract the local features L1 and L2 of the target object; the target object refers to the feature map output by the previous fast Fourier convolution block or the image to be detected, and the ordinary convolution is a convolution including a convolution layer, an activation layer, and a pooling layer.

[0113] Ordinary convolution is used to extract the local features G1 of the target object, and channel-level fast Fourier transform is used to extract the global features G2 of the target object.

[0114] Add the local feature L1 and the global feature G1 to obtain the local feature 1; add the local feature L2 and the global feature G2 to obtain the global feature 1.

[0115] The local feature 1 and the global feature 1 are batch normalized and ReLU activated to obtain the activated local feature 2 and the activated global feature 2.

[0116] The activated local feature 2 and the activated global feature 2 are fused to obtain the feature map output by the fast Fourier convolution block.

[0117] Reference Fig. 9 In some embodiments, the global feature G2 of the image to be detected is extracted using a channel-level fast Fourier transform, including:

[0118] A 1×1 convolution is used to halve the number of channels of the image to be detected.

[0119] Fast Fourier transform is used to transform the image features to be detected from the spatial domain to the frequency domain to obtain the frequency domain results.

[0120] The frequency domain results are batch normalized and linearly activated using convolution blocks, and restored to the spatial domain using two-dimensional discrete Fourier inverse transform to obtain the spatial domain results.

[0121] Use 1×1 convolution to restore the number of channels and obtain the global feature G2.

[0122] Real two-dimensional Fourier transform formula:

[0123]

[0124] Two-dimensional discrete Fourier transform formula:

[0125]

[0126] Among them, (x, y) represents the position in the spatial domain, (u, v) represents the position in the frequency domain, x, y, u, v = 0, 1, 2, ... N-1.

[0127] In some embodiments, after obtaining the detection result of whether the image to be detected is cropped, the method further includes:

[0128] The cropped area and / or uncropped area in the image to be detected is marked according to the detection result, and cropping and tampering have the same meaning.

[0129] Generate a binary image based on the marked area;

[0130] When the binary image contains a clipping area, the binary image is eroded first and then expanded to obtain the observed result.

[0131] In order to facilitate the understanding of those skilled in the art, a group of best embodiments are provided below:

[0132] 1. Obtain the local feature map of the image to be detected:

[0133] The convolutional neural network module uses the ResNet50 network as the backbone network for extracting local features; an A_MEMM module is set after each feature extraction stage in ResNet50, and the A_MEMM module includes a channel attention module, a spatial attention module and a maximum entropy Markov module. The channel attention module is used to extract the features of the first feature map to obtain a channel attention feature map. The first feature map refers to the feature map output in the feature extraction stage; the spatial attention module is used to extract the features of the second feature map to obtain a spatial attention feature map. The second feature map refers to the feature map obtained by performing element-by-element multiplication of the channel attention feature map and the first feature map; the maximum entropy Markov module is used to increase the correlation between adjacent regions in the comprehensive feature map. The comprehensive feature map refers to the feature map obtained by performing element-by-element multiplication of the spatial attention feature map and the second feature map.

[0134] The image to be detected is subjected to the first feature extraction stage to obtain a feature map F1 reduced to 1 / 2 of the image to be detected.

[0135] The feature map F1 is subjected to the second feature extraction stage to obtain a feature map F2 reduced to 1 / 4 of the image to be detected.

[0136] The feature map F2 is subjected to the third feature extraction stage to obtain a feature map F3 reduced to 1 / 8 of the image to be detected.

[0137] The feature map F3 is subjected to the fourth feature extraction stage to obtain a feature map F4 reduced to 1 / 16 of the image to be detected.

[0138] The feature map F4 is subjected to the fifth feature extraction stage to obtain a feature map F5 reduced to 1 / 32 of the image to be detected.

[0139] The feature maps F3, F4 and F5 are subjected to full convolution layer operations FC6 and FC7 to obtain heat maps H31, H41 and H51.

[0140] The heat map H31, the heat map H41 and the heat map H51 are convolved using a convolution function to obtain heat maps H3, H4 and H5, wherein the step size of the convolution function is 1, the kernel size is 1×1 and the number of convolution kernels is 2.

[0141] The heat map H5 is convolved using a transposed convolution with a kernel size of 4×4 and a step size of 2 to obtain a convolution result 1. The convolution result 1 is added element-by-element to the heat map H4 to obtain an addition result 1.

[0142] The addition result 1 is convolved using a transposed convolution with a kernel size of 4×4 and a step size of 2 to obtain a convolution result 2. The convolution result 2 is added element by element to the heat map H3 to obtain an addition result 2.

[0143] The addition result 2 is convolved using a transposed convolution with a kernel size of 16×16 and a step size of 8 to obtain a local feature map of the image to be detected.

[0144] 2. Obtain the global feature map of the image to be detected:

[0145] The image to be detected is input into the first fast Fourier convolution block in the fast Fourier convolution block module to obtain the feature map output by the first fast Fourier convolution block.

[0146] The feature map output by the first fast Fourier convolution block is input to the next fast Fourier convolution block, and so on, until the last fast Fourier convolution block outputs the global feature map of the image to be detected; wherein any one of the multiple fast Fourier convolution blocks performs the following operations:

[0147] Ordinary convolution is used to extract the local features L1 and L2 of the target object; the target object refers to the feature map output by the previous fast Fourier convolution block or the image to be detected, and the ordinary convolution is a convolution including a convolution layer, an activation layer, and a pooling layer.

[0148] Ordinary convolution is used to extract the local features G1 of the target object, and channel-level fast Fourier transform is used to extract the global features G2 of the target object.

[0149] Add the local feature L1 and the global feature G1 to obtain the local feature 1; add the local feature L2 and the global feature G2 to obtain the global feature 1.

[0150] The local feature 1 and the global feature 1 are batch normalized and ReLU activated to obtain the activated local feature 2 and the activated global feature 2.

[0151] The activated local feature 2 and the activated global feature 2 are fused to obtain the feature map output by the fast Fourier convolution block.

[0152] 3. Use the softmax classifier to perform pixel judgment on the connected feature map:

[0153] The cropped area and / or the uncropped area in the image to be detected are marked according to the detection result.

[0154] Generate a binary image based on the marked areas.

[0155] When the binary image contains a clipping area, the binary image is eroded first and then expanded to obtain the observed result.

[0156] This method obtains the local feature map of the image to be detected by inputting the image to be detected into a preset convolutional neural network module; wherein the convolutional neural network module includes a spatial attention mechanism and a channel attention mechanism combined with each other, and inputs the image to be detected into a preset fast Fourier convolution block module to obtain the global feature map of the image to be detected; the local feature map is connected with the global feature map to obtain a connected feature image; the softmax classifier is used to perform pixel detection on the connected feature map to obtain the detection result of whether the image to be detected is cropped. This method proposes a local feature extraction module based on the attention mechanism and a global feature extraction module based on fast Fourier convolution. Compared with the existing scheme, this method extracts the global features and local features of the image to be detected, so that the final fused detection features are more comprehensive, and the classification and positioning are more accurate.

[0157] An embodiment of the present invention provides a CNN-based seam cropping image detection system, including a local feature module 1100, a global feature module 1200, a feature connection module 1300 and a result output module 1400, wherein:

[0158] The local feature module 1100 is used to input the image to be detected into a preset convolutional neural network module to obtain a local feature map of the image to be detected; wherein the convolutional neural network module includes a spatial attention mechanism and a channel attention mechanism that are combined with each other.

[0159] The global feature module 1200 is used to input the image to be detected into a preset fast Fourier convolution block module to obtain a global feature map of the image to be detected.

[0160] The feature connection module 1300 is used to connect the local feature map with the global feature map to obtain a connected feature image.

[0161] The result output module 1400 is used to perform pixel detection on the connected feature map using a softmax classifier to obtain a detection result of whether the image to be detected is cropped.

[0162] This system obtains the local feature map of the image to be detected by inputting the image to be detected into the preset convolutional neural network module; wherein the convolutional neural network module includes a spatial attention mechanism and a channel attention mechanism combined with each other, inputs the image to be detected into the preset fast Fourier convolution block module, obtains the global feature map of the image to be detected; connects the local feature map with the global feature map to obtain the connected feature image; uses the softmax classifier to perform pixel detection on the connected feature map to obtain the detection result of whether the image to be detected is cropped. This system proposes a local feature extraction module based on the attention mechanism and a global feature extraction module based on fast Fourier convolution. Compared with the existing scheme, this system extracts the global features and local features of the image to be detected, so that the final fused detection features are more comprehensive, and the classification and positioning are more accurate.

[0163] It should be noted that the present system embodiment and the above-mentioned method embodiment are based on the same inventive concept, so the relevant contents of the above-mentioned method embodiment are also applicable to the present system embodiment and will not be repeated here.

[0164] The present application also provides a CNN-based seam cropping image detection electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements: the CNN-based seam cropping image detection method as described above.

[0165] The processor and the memory may be connected via a bus or other means.

[0166] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0167] The non-transient software program and instructions required to implement the CNN-based seam cropping image detection method of the above embodiment are stored in the memory. When executed by the processor, the CNN-based seam cropping image detection method of the above embodiment is executed, for example, the above-described Figure 1 The method comprises steps S101 to S104.

[0168] The present application also provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute: the CNN-based seam cropping image detection method as described above.

[0169] The computer-readable storage medium stores computer-executable instructions, which are executed by a processor or a controller, for example, by a processor in the above electronic device embodiment, so that the above processor can execute the CNN-based seam cropping image detection method in the above embodiment, for example, execute the above described Figure 1 The method comprises steps S101 to S104.

[0170] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing data (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that may be used to store desired data and may be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any data delivery media.

[0171] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0172] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A CNN-based seam cropping image detection method, characterized in that: The steps include: Inputting the image to be detected into a preset convolutional neural network module to obtain a local feature map of the image to be detected; wherein the convolutional neural network module includes a spatial attention mechanism and a channel attention mechanism combined with each other; Input the image to be detected into a preset fast Fourier convolution block module to obtain a global feature map of the image to be detected, wherein the fast Fourier convolution block module includes a plurality of cascaded fast Fourier convolution blocks, and the structures of the plurality of fast Fourier convolution blocks are the same; specifically: Inputting the image to be detected into the first fast Fourier convolution block in the fast Fourier convolution block module to obtain a feature map output by the first fast Fourier convolution block; Input the feature map output by the first fast Fourier convolution block to the next fast Fourier convolution block, and so on, until the last fast Fourier convolution block outputs the global feature map of the image to be detected; wherein any one of the multiple fast Fourier convolution blocks performs the following operations: Use ordinary convolution to extract local features L1 and local features L2 of the target object; the target object refers to the feature map output by the previous fast Fourier convolution block or the image to be detected, and the ordinary convolution is a convolution including a convolution layer, an activation layer and a pooling layer; Use ordinary convolution to extract the local feature G1 of the target object, and use channel-level fast Fourier transform to extract the global feature G2 of the target object; Add the local feature L1 and the global feature G1 to obtain a local feature 1; add the local feature L2 and the global feature G2 to obtain a global feature 1; The local feature 1 and the global feature 1 are batch normalized and Relu activated to obtain an activated local feature 2 and an activated global feature 2; The activated local feature 2 and the activated global feature 2 are merged to obtain a feature map output by the fast Fourier convolution block; Connecting the local feature map with the global feature map to obtain a connected feature map; A softmax classifier is used to perform pixel detection on the connected feature map to obtain a detection result of whether the image to be detected is cropped.

2. The CNN-based seam cropping image detection method according to claim 1, characterized in that: The convolutional neural network module uses the ResNet50 network as the backbone network for extracting local features; an A_MEMM module is set after each feature extraction stage in the ResNet50, and the A_MEMM module includes a channel attention module, a spatial attention module and a maximum entropy Markov module. The channel attention module is used to extract the features of the first feature map to obtain a channel attention feature map, and the first feature map refers to the feature map output by the feature extraction stage; the spatial attention module is used to extract the features of the second feature map to obtain a spatial attention feature map, and the second feature map refers to the feature map obtained by performing element-by-element multiplication operation on the channel attention feature map and the first feature map; the maximum entropy Markov module is used to increase the correlation between adjacent areas in the comprehensive feature map, and the comprehensive feature map refers to the feature map obtained by performing element-by-element multiplication operation on the spatial attention feature map and the second feature map.

3. The CNN-based seam cropping image detection method according to claim 2, characterized in that: The ResNet50 network includes 5 feature extraction stages, wherein the first feature extraction stage, the second feature extraction stage and the third feature extraction stage are ordinary convolutions, the fourth feature extraction stage and the fifth feature extraction stage are composed of two residual blocks containing different numbers, and the fourth feature extraction stage and the fifth feature extraction stage are added with dilated convolutions, and the ordinary convolution is a convolution including a convolution layer, an activation layer and a pooling layer.

4. The CNN-based seam cropping image detection method according to claim 3, characterized in that: The step of inputting the image to be detected into a preset convolutional neural network module to obtain a local feature map of the image to be detected includes: The image to be detected is subjected to the first feature extraction stage to obtain a feature map F1 reduced to 1 / 2 of the image to be detected; The feature map F1 is subjected to a second feature extraction stage to obtain a feature map F2 reduced to 1 / 4 of the image to be detected; The feature map F2 is subjected to the third feature extraction stage to obtain a feature map F3 reduced to 1 / 8 of the image to be detected; The feature map F3 is subjected to the fourth feature extraction stage to obtain a feature map F4 reduced to 1 / 16 of the image to be detected; The feature map F4 is subjected to the fifth feature extraction stage to obtain a feature map F5 reduced to 1 / 32 of the image to be detected; Perform full convolution layer operations FC6 and FC7 on the feature map F3, the feature map F4 and the feature map F5 to obtain a heat map H31, a heat map H41 and a heat map H51; Convolving the heat map H31, the heat map H41 and the heat map H51 using a convolution function to obtain a heat map H3, a heat map H4 and a heat map H5, wherein the step size of the convolution function is 1, the kernel size is 1×1 and the number of convolution kernels is 2; Convolve the heat map H5 using a transposed convolution with a kernel size of 4×4 and a step size of 2 to obtain a convolution result 1, and add the convolution result 1 and the heat map H4 element by element to obtain an addition result 1; Convolving the addition result 1 with a transposed convolution with a kernel size of 4×4 and a step size of 2 to obtain a convolution result 2, and adding the convolution result 2 and the heat map H3 element by element to obtain an addition result 2; The addition result 2 is convolved using a transposed convolution with a kernel size of 16×16 and a step size of 8 to obtain a local feature map of the image to be detected.

5. The CNN-based seam cropping image detection method according to claim 1, characterized in that: The extracting the global feature G2 of the image to be detected by using channel-level fast Fourier transform includes: Using 1×1 convolution to halve the number of channels of the image to be detected; The image features to be detected are converted from the spatial domain to the frequency domain using fast Fourier transform to obtain the frequency domain results; The frequency domain results are batch normalized and linearly activated using a convolution block, and restored to the spatial domain using a two-dimensional discrete Fourier inverse transform to obtain a spatial domain result; The number of channels is restored using 1×1 convolution to obtain the global feature G2.

6. The CNN-based seam cropping image detection method according to claim 1, after obtaining the detection result of whether the image to be detected is cropped, further comprising: Marking the cropped area and / or the uncropped area in the image to be detected according to the detection result; Generate a binary image based on the marked area; When the binary image contains a clipping area, an opening operation of first eroding and then dilating is performed on the binary image to obtain an observation result.

7. A CNN-based seam cropping image detection system, characterized in that: include: A local feature module, used for inputting the image to be detected into a preset convolutional neural network module to obtain a local feature map of the image to be detected; wherein the convolutional neural network module includes a spatial attention mechanism and a channel attention mechanism combined with each other; A global feature module is used to input the image to be detected into a preset fast Fourier convolution block module to obtain a global feature map of the image to be detected, wherein the fast Fourier convolution block module includes a plurality of cascaded fast Fourier convolution blocks, and the structures of the plurality of fast Fourier convolution blocks are the same; specifically: Inputting the image to be detected into the first fast Fourier convolution block in the fast Fourier convolution block module to obtain a feature map output by the first fast Fourier convolution block; Input the feature map output by the first fast Fourier convolution block to the next fast Fourier convolution block, and so on, until the last fast Fourier convolution block outputs the global feature map of the image to be detected; wherein any one of the multiple fast Fourier convolution blocks performs the following operations: Use ordinary convolution to extract local features L1 and local features L2 of the target object; the target object refers to the feature map output by the previous fast Fourier convolution block or the image to be detected, and the ordinary convolution is a convolution including a convolution layer, an activation layer and a pooling layer; Use ordinary convolution to extract the local feature G1 of the target object, and use channel-level fast Fourier transform to extract the global feature G2 of the target object; Add the local feature L1 and the global feature G1 to obtain a local feature 1; add the local feature L2 and the global feature G2 to obtain a global feature 1; The local feature 1 and the global feature 1 are batch normalized and Relu activated to obtain an activated local feature 2 and an activated global feature 2; The activated local feature 2 and the activated global feature 2 are merged to obtain a feature map output by the fast Fourier convolution block; A feature connection module, used for connecting the local feature map with the global feature map to obtain a connected feature map; The result output module is used to perform pixel detection on the connected feature map using a softmax classifier to obtain a detection result of whether the image to be detected is cropped.

8. An electronic device for detecting seam cropping images based on CNN, characterized in that: comprising at least one control processor and a memory for communicatively coupling with the at least one control processor; The memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the CNN-based seam cropping image detection method described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the CNN-based seam cropping image detection method according to any one of claims 1 to 6.

Images