System and Method for Providing Object Countdown Network Based on Deep Learning In IoT System

Abstract

The deep learning-based object countdown network system and method of providing an IoT system can count objects in an IoT system by providing a dense attention fusion network to calculate the accurate number of objects in remote sensing images, thereby improving the performance of object recognition even in images with complex backgrounds. The present invention provides a Dense Attention Fusion Network (DAFNet) for accurate object counting in remote sensing images, which has the effect of mitigating side effects that severely degrade object counting performance, such as background clutter, changes in scale, and randomness of direction.

Description

Technology Field

[0001] The present invention relates to a dense attention fusion network system, and more specifically, to a deep learning-based object countdown network system and a method for providing an IoT system that improves the performance of object recognition even in images with complex backgrounds by providing a dense attention fusion network to calculate the accurate number of objects in remote sensing images. Background Technology

[0002] The IoT is driving rapid development in the big data ecosystem, such as remote sensing technology that is always running to obtain accurate, high-quality images for subsequent image processing and content analysis on embedded devices.

[0003] Remote sensing images are often captured from the sky, so objects appear small. Due to complex backgrounds, the model cannot highlight the region of interest, causing objects to become background-cluttered. Additionally, objects in remote sensing images vary in size from a few pixels to hundreds of pixels. This variability in size is disadvantageous for predicting object counts.

[0004] Furthermore, there is a problem in that objects in remote sensing images have uncertain orientations, making remote sensing calculations more difficult than other calculation tasks.

[0005] Object counting, which aims to estimate the number of objects in captured images, is one of the most important tasks in multimedia data and wireless networks.

[0006] However, in remote sensing, there are significant inherent factors that severely degrade counting performance, such as background disturbances, changes in scale, and directional randomness. Prior art literature

[0007] Korean Registration No. 10-2448680 The problem to be solved

[0008] To solve such problems, the present invention aims to provide a deep learning-based object countdown network system and a method for providing an IoT system that improves the performance of object recognition even in images with complex backgrounds by providing a dense attention fusion network to calculate the accurate number of objects in remote sensing images. means of solving the problem

[0009] A deep learning-based object countdown network system according to the features of the present invention for achieving the above objective is,

[0010] Input unit for receiving input images;

[0011] A feature extraction unit that extracts features of an input image received from the above input unit;

[0012] An Iterative Attention Fusion (IAF) module that suppresses background confusion, which indicates a mixed and unclear image background, by emphasizing the area containing an object from the features extracted from the above-mentioned feature extraction unit; and

[0013] It includes a Dense Spatial Pyramid (DSP) module that improves counting performance by capturing reception fields using various expanded convolution layers with different expanded ratios for images received from the above iterative attention fusion module.

[0014] A method for providing a deep learning-based object countdown network according to the features of the present invention is,

[0015] As a dense attention fusion network comprising an input section, a feature extraction section, an iterative attention fusion (IAF) module, and a dense spatial pyramid (DSP) module,

[0016] The above input unit receives an input image;

[0017] The above feature extraction unit extracts features of the received input image;

[0018] The above IAF module suppresses background confusion that indicates a mixed and unclear image background by highlighting the area containing an object in the extracted features; and

[0019] The above DSP module includes a step of improving counting performance by capturing a reception field using various extended convolution layers with different extended ratios for an image received from the IAF module. Effects of the invention

[0020] By the aforementioned configuration, the present invention provides a Dense Attention Fusion Network (DAFNet) for accurate object counting in remote sensing images, thereby having the effect of mitigating side effects that severely degrade object counting performance, such as background confusion, changes in scale, and randomness of direction.

[0021] The present invention has the effect of providing excellent object recognition performance even in remote sensing images with background clusters and large-scale variations. Brief explanation of the drawing

[0022] FIG. 1 is a diagram showing the configuration of a deep learning-based object countdown network system of an IoT system according to an embodiment of the present invention. FIG. 2 is a diagram showing the framework of an MCA unit according to an embodiment of the present invention. Figure 3 is a diagram showing the flow of a DSP module according to an embodiment of the present invention. Specific details for implementing the invention

[0023] Throughout the specification, when a part is described as "including" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0024] In this invention, DAFNet (Dense Attention Fusion Network) is proposed to solve the problem using a divide-and-conquer approach.

[0025] Specifically, the present invention introduces an iterative attention fusion (IAF) module that relies primarily on a multiscale channel attention (MCA) unit to mitigate side effects caused by background clutter.

[0026] The present invention constructs a high-density spatial pyramid (DSP) module by considering hierarchical information obtained from various reception fields to overcome intrinsic scale changes.

[0027] The present invention stacks deformable convolutional layers to handle directional randomness. The synergy between the proposed IAF and DSP modules significantly enhances the efficiency of DAFNet, which can demonstrate remarkable superiority over state-of-the-art competing products in extensive experiments on remote sensing counting datasets.

[0028] FIG. 1 is a diagram showing the configuration of a deep learning-based object countdown network system of an IoT system according to an embodiment of the present invention, FIG. 2 is a diagram showing the framework of an MCA unit according to an embodiment of the present invention, and FIG. 3 is a diagram showing the flow of a DSP module according to an embodiment of the present invention.

[0029] A deep learning-based object countdown network system (100) according to an embodiment of the present invention includes an input unit (110), a feature extraction unit (120) for obtaining features of an input image, an iterative attention fusion (hereinafter referred to as 'IAF') module (130) for resolving background confusion, and a dense spatial pyramid (hereinafter referred to as 'DSP') module (140) for processing scale changes.

[0030] The deep learning-based object countdown network system (100) is a concept that includes DAFNet (Dense Attention Fusion Network).

[0031] The input unit (110) receives an input image, and the feature extraction unit (120) extracts features of the input image received from the input unit (110).

[0032] The IAF module (130) functions to suppress background confusion that indicates a mixed and unclear image background by highlighting the area containing the object from the features extracted from the feature extraction unit (120).

[0033] The Dense Spatial Pyramid (DSP) module (140) can improve counting performance by capturing reception fields using various expanded convolution layers with different expanded ratios for images received from the iterative attention fusion module (130).

[0034] The feature extraction unit (120) applies VGG-16, which is a convolutional neural network consisting of 16 layers, and the first 10 convolutional layers are used to extract low-level features. VGG-16 uses deformable convolutional layers to overcome directional randomness.

[0035] The iterative attention fusion module (130) suppresses the background of low-level features received from VGG-16 and generates a feature map.

[0036] The dense space pyramid module (140) enables the feature map received from the iterative attention fusion unit to recognize the scale through various extended convolution layers.

[0037] The Dense Attention Fusion Network System (DAFNet System) (100) can be expressed as shown in the following mathematical equation 1.

[0038]

[0039] Here, class are the prediction and low-level feature, respectively, and and is a function of the DSP module (140) and the IAF module (130), represents a deformable convolutional layer.

[0040] The IAF module (130) performs the function of suppressing background confusion and is achieved through two Multiscale Channel Attention (MCA) units (131).

[0041] The framework of the MCA unit (131) is illustrated in FIG. 2. The top branch is an input characteristic To enrich it, two 1×1 convolution layers were adopted, and the Local Attention Feature Map Prints.

[0042] X is a low-level feature, C is the original sample image matrix, and H and W are the vertical and horizontal lengths of the feature map.

[0043] Local attention feature maps can achieve local context aggregation and highlight local context information.

[0044] With the operation of the top branch, the MCA unit (131) calculates the local attention feature map by Equation 2.

[0045]

[0046] Here, is the Local Attention Feature Map, BN stands for Batch Normalization, and ReLU is the activation function, is a Low Level Feature, Conv1 represents the first convolution operation, and Conv2 represents the second convolution operation.

[0047] The intermediate branch is a Global Attention Feature Map that helps achieve global context aggregation and highlight global context information compared to the top branch. The goal is to obtain.

[0048] Global attention feature maps can achieve global context aggregation and highlight global context information.

[0049] The MCA unit (131) is a global attention feature map Calculate using mathematical formula 3.

[0050]

[0051] Here, is the Global Attention Feature Map, GAP is the Global Average Pooling Operation, BN stands for Batch Normalization, and ReLU is the activation function, is a Low Level Feature, Conv1 represents the first convolution operation, and Conv2 represents the second convolution operation.

[0052] Dimensional local attention map this Dimensional global attention map It has a different dimension from...

[0053] This is because the GAP operation compresses the input feature map into 1×1 in the spatial dimension.

[0054] The MCA unit (131) collects internal identification information by aggregating the local attention map and the global attention map through a sum operation. Then, the fused features are activated by a Sigmoid function in the spatial dimension without changing the channel dimension.

[0055] The MCA unit (131) aggregates the features of the local attention map and the global attention map to generate a high-quality density map containing local context information and global context information.

[0056] Simply put, the feature map refined by the MCA unit (131) is defined by the following mathematical formula 4. A refined feature map is equivalent to a high-quality dense map.

[0057]

[0058] Here, class represents the input feature map and the refined feature map, respectively, and are local attention features and global attention features, Sigmoid is the sigmoid function, and is element-wise multiplication, and represents the element-wise sum operation.

[0059] The present invention can better distinguish individual regions by enhancing features through the MCA unit (131).

[0060] Due to the advantages of the MCA unit (131), the present invention can construct an IAF module (130) by assembling it. As shown in FIG. 2, the IAF module (130) fuses two input feature maps, namely X and Y, obtained from convolution layers with various kernel sizes.

[0061] As shown in Figure 2, X is an image of Conv3×3 performed on the output of VGG-16, and Y is an image of Conv5×5 performed on the output of VGG-16.

[0062] This strategy differs from traditional fusion methods, and most of them take only one feature as input to the fusion process and adopt multiple branches to obtain features of various levels. First, the IAF module (130) fuses X and Y into a feature map through element-wise summation.

[0063] The IAF module (130) inputs the fused feature map into the MCA unit (131) and outputs two feature maps. One is weighted to the initial feature X. One is obtained by multiplying by , and the other is weighted to the initial feature Y (1 - It is obtained by multiplying by ) (indicated by the red line in Figure 2).

[0064] To fully utilize context information, selecting high-quality inputs helps generate useful fusion weights. Therefore, an intuitive approach is to adopt another attention unit to fuse input features. Since this is an iterative process, the IAF module (130) can be defined by the following mathematical formula 5.

[0065]

[0066] Here, is a fused feature map, and X and Y are two input feature maps obtained from convolution layers with various kernel sizes, and represents the feature fusion process of the first stage of attention, and It can be defined by the following mathematical formula 6.

[0067]

[0068] Here, represents element-wise multiplication, and represents the element-wise sum operation, is a function of the MCA unit (131), of the aforementioned mathematical formula 2 , of mathematical formula 3 It may be a concept that includes.

[0069] In remote sensing images, objects often struggle with size variations in dense regions. To address this problem, a Dense Spatial Pyramid (hereinafter referred to as 'DSP') module is introduced.

[0070] The framework of the DSP module (140) is shown in FIG. 2.

[0071] The DSP module (140) is mainly composed of densely connected standard convolution layers and extended convolution layers. Extended convolution layers with different extension ratios can expand the acceptance field while storing parameters.

[0072] A convolution layer with a kernel size of 1×1 is added before each extended convolution layer to reduce the number of channels.

[0073] A 3×3 convolution layer is used to integrate previously generated features.

[0074] Additionally, the DSP module (140) can use the convolution layer for channel reduction as the number of channels increases fourfold due to dense connection. When such a structure is arranged, another important issue is selecting an expansion rate suitable for the expanded convolution layer.

[0075] Considering that objects in remote sensing images often appear small, selecting a large scaling factor can lead to a loss of semantic information. This is disadvantageous for capturing detailed features. Additionally, a dense sampling range is required because changes in object size within the image occur continuously.

[0076] To this end, the expansion ratios are set to 1, 2, and 3, respectively. The DSP module (140) can maintain the spatial information of the object as much as possible through parameter configuration.

[0077] As can be seen in FIG. 3, the DSP module (140) can handle all pixel information of the input feature map. Meanwhile, because the selected expansion ratio is small, the module can prevent capturing irrelevant information.

[0078] The process of the DSP module (140) can be expressed by mathematical formula 7.

[0079]

[0080] Here, is a convolution layer with a kernel size of 1×1, and is a convolutional layer with a kernel size of 3×3, D and X are the features after the expanded convolutional layer and the input features, respectively, and i represents the dilated rate. i can be represented as dr. The black blocks in Fig. 3 represent the sources of information.

[0081]

[0082] Here, O is the output of the DSP module (140), and DC represents a dense connection.

[0083] The extended convolution layer of the DSP module (140) is closely connected to other layers. In this case, each layer can obtain information from the previous layer and transmit that information to the next layer. The DSP module (140) can increase scale diversity to handle scale changes.

[0084] The control unit (not shown) uses a loss function to measure the discrepancy between the estimated density map and the actual density map. Utilize the distance.

[0085] The loss function calculates the error between the actual value and the estimated value when training a deep learning model, and takes the actual value and the estimated value as input to create a real-valued score; the higher this score, the worse the state of the model.

[0086] A control unit (not shown) can measure the discrepancy between the estimated density map and the actual density map using Equation 9. The control unit can count objects in the corresponding image based on the match between the estimated density map and the actual density map.

[0087]

[0088] Here, N is the number of test images, and represents the estimated value and the actual value, respectively.

[0089] Ground Truth Generation refers to the activities necessary to reliably acquire training data through the synthetic generation of the relevant data.

[0090] The control unit (not shown) is the Ground Truth Map is defined by the following mathematical formula 10.

[0091]

[0092] Ground Truth Map It is generated by adopting a Gaussian kernel that convolves the delta function.

[0093] Here, is a Ground Truth Map, and indicates the number of header comments and is a location pixel, and is the variance of the Gaussian kernel, and represents the target head.

[0094] The purpose of object counting is to estimate the number of objects in a given image.

[0095] The IAF module is designed to resolve background clutter by highlighting the area containing the object. A core component of the IAF module is an MCA device built to merge local and global features.

[0096] To solve the problem of scale fluctuation in dense regions, a DSP module is constructed that adopts various expanded convolution layers with small expansion rates (1, 2, and 3) and captures a large acceptance field to improve object performance.

[0097] Finally, a deformable convolution layer is introduced to handle directional randomness.

[0098] Although embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements by those skilled in the art using the basic concept of the present invention as defined in the following claims also fall within the scope of the present invention. Explanation of the symbols

[0099] 100: Deep Learning-Based Object Countdown Network System 110: Input section 120: Feature extraction unit 130: Recurrent Attention Fusion Module, IAF Module 140: Dense Space Pyramid Module, DSP Module

Claims

Claim 1 A dense attention fusion network system comprises: an input unit for receiving an input image; a feature extraction unit for extracting features of the input image received from the input unit; an iterative attention fusion (IAF) module for suppressing background confusion that indicates a mixed and unclear image background by emphasizing the region containing an object among the features extracted from the feature extraction unit; and a dense spatial pyramid (DSP) module for improving counting performance by capturing receptive fields using various expanded convolution layers with different expansion ratios for the image received from the iterative attention fusion module, wherein the IAF module suppresses background confusion through at least one multiscale channel attention (MCA) unit, and the MCA unit calculates a local attention feature map that achieves local context aggregation and emphasizes local context information by the following Equation 2, [Equation 2] Here, is the Local Attention Feature Map, BN stands for Batch Normalization, and ReLU is the activation function, is a Low Level Feature, Conv1 is the first convolution operation, Conv2 is the second convolution operation, and the above MCA unit achieves global context aggregation and calculates a global attention feature map that emphasizes global context information by the following Equation 3, [Equation 3] Here, is the Global Attention Feature Map, GAP is the Global Average Pooling Operation, BN stands for Batch Normalization, and ReLU is the activation function, is a low-level feature, Conv1 is the first convolution operation, Conv2 is the second convolution operation, and the MCA unit aggregates the features of the local attention feature map and the global attention map to produce a refined feature map containing the local context information and the global context information. is calculated by the following mathematical formula 4, [Mathematical Formula 4] Here, class represents the input feature map and the refined feature map, respectively, and are local attention features and global attention features, Sigmoid is the sigmoid function, and is element-wise multiplication, and A deep learning-based object countdown network system in which the IAF module fuses two input feature maps (X and Y) obtained from convolution layers with various kernel sizes through elemental summation, calculates the fused feature map according to the following Equations 5 and 6, inputs the fused feature map to the MCA unit, and the MCA unit outputs a first feature map obtained by multiplying the input feature map X by a first weight and a second feature map obtained by multiplying the input feature map Y by (1 - the first weight). [Equation 5] Here, is a fused feature map, and X and Y are two input feature maps obtained from convolution layers with various kernel sizes, and represents the feature fusion process of the first stage of attention, and is defined by the following mathematical formula 6.[Mathematical Formula 6] Here, represents element-wise multiplication, and represents the element-wise sum operation, is a function of the MCA unit, of the aforementioned mathematical formula 2 , of mathematical formula 3 It is a concept that includes. Claim 2 The deep learning-based object countdown network system according to claim 1, wherein the feature extraction unit is VGG-16, a convolutional neural network composed of 16 layers, and extracts low-level features using the first 10 convolutional layers. Claim 3 In claim 1, the dense attention fusion network system is a deep learning-based entity countdown network system represented by the following mathematical formula 1. [Mathematical Formula 1] Here, class are the prediction and low-level feature, respectively, and and is a function of the above DSP module and the above IAF module, and is a deformable convolutional layer. Claim 4 delete Claim 5 delete Claim 6 delete Claim 7 delete Claim 8 A deep learning-based object countdown network system according to claim 1, wherein the DSP module is composed of densely connected standard convolution layers and extended convolution layers to handle scale changes, defined by the following Equations 7 and 8.[Equation 7] Here, is a convolution layer with a kernel size of 1×1, and is a convolutional layer with a kernel size of 3×3, D and X are the features after the expanded convolutional layer and the input features, respectively, and i is the dilated rate.[Equation 8] Here, O is the output of the DSP module, and DC is a dense connection. Claim 9 A dense attention fusion network comprising an input unit, a feature extraction unit, an iterative attention fusion (IAF) module, and a dense spatial pyramid (DSP) module, wherein the input unit receives an input image; the feature extraction unit extracts features of the received input image; the IAF module suppresses background disturbance, which indicates a mixed and unclear image background, by emphasizing the area containing an object in the extracted features; and the DSP module improves counting performance by capturing a receptive field using various expanded convolution layers with different expansion ratios for the image received from the IAF module, and wherein the step of suppressing background disturbance comprises the IAF module suppressing background disturbance through at least one Multiscale Channel Attention (MCA) unit. and the above MCA unit further includes a step of calculating a Local Attention Feature Map that achieves local context aggregation and highlights local context information by the following Equation 2, [Equation 2] Here, is the Local Attention Feature Map, BN stands for Batch Normalization, and ReLU is the activation function, is a Low Level Feature, Conv1 is the first convolution operation, Conv2 is the second convolution operation, and the above MCA unit further includes a step of calculating a global attention feature map that achieves global context aggregation and emphasizes global context information by the following Equation 3, [Equation 3] Here, is the Global Attention Feature Map, GAP is the Global Average Pooling Operation, BN stands for Batch Normalization, and ReLU is the activation function, is a low-level feature, Conv1 is the first convolution operation, Conv2 is the second convolution operation, and the MCA unit aggregates the features of the local attention feature map and the global attention map to produce a refined feature map containing the local context information and the global context information. Includes a step calculated by the following mathematical formula 4, [Mathematical Formula 4] Here, class represents the input feature map and the refined feature map, respectively, and are local attention features and global attention features, Sigmoid is the sigmoid function, and is element-wise multiplication, and A method for providing a deep learning-based object countdown network, comprising: a step in which the IAF module fuses two input feature maps (X and Y) obtained from a convolution layer having various kernel sizes through elemental summation; a step in which the IAF module calculates the fused feature map according to the following mathematical formulas 5 and 6 and inputs the fused feature map to the MCA unit; and a step in which the MCA unit outputs a first feature map obtained by multiplying the input feature map X by a first weight and a second feature map obtained by multiplying the input feature map Y by (1 - the first weight). [Mathematical Formula 5] Here, is a fused feature map, and X and Y are two input feature maps obtained from convolution layers with various kernel sizes, and represents the feature fusion process of the first stage of attention, and is defined by the following mathematical formula 6.[Mathematical Formula 6] Here, represents element-wise multiplication, and represents the element-wise sum operation, is a function of the MCA unit, of the aforementioned mathematical formula 2 , of mathematical formula 3 It is a concept that includes. Claim 10 A method for providing a deep learning-based object countdown network according to claim 9, wherein the step of extracting features of the received input image comprises: the feature extraction unit being VGG-16, a convolutional neural network composed of 16 layers; and further comprising the step of extracting low-level features using a plurality of convolutional layers. Claim 11 The method of claim 9, wherein the dense attention fusion network system comprises a step represented by the following mathematical formula 1, providing a deep learning-based object countdown network. [Mathematical Formula 1] Here, class are the prediction and low-level feature, respectively, and and is a function of the above DSP module and the above IAF module, and is a deformable convolutional layer. Claim 12 delete Claim 13 delete Claim 14 delete Claim 15 delete Claim 16 A method for providing a deep learning-based object countdown network according to claim 9, wherein the step of improving the counting performance further comprises the step defined by the following Equations 7 and 8, wherein the DSP module is configured with densely connected standard convolution layers and extended convolution layers to handle scale changes. [Equation 7] Here, is a convolution layer with a kernel size of 1×1, and is a convolutional layer with a kernel size of 3×3, D and X are the features after the expanded convolutional layer and the input features, respectively, and i is the dilated rate.[Equation 8] Here, O is the output of the DSP module, and DC is a dense connection.

Images