Spatial attention-based contextual feature learning in smart homes

Abstract

The present invention discloses a context feature learning method based on spatial attention in a smart home, and belongs to the field of smart homes. The present invention discloses a context feature learning method based on spatial attention in a smart home. The scheme learns the context relationship between different regions through an attention mechanism, and fuses these context relationships, so as to find unique context features for different regions, and improves the target detection performance in complex scenes. Through a practical visual geometry group and a residual network, the features of an input image can be extracted to obtain a convolution feature map. An object pair includes two objects, each object includes a spatial position and a semantic feature, and every two objects are set as an object pair. With the help of a three-layer perceptron and a linear rectifier function, the context relationship between the object pairs is obtained with the help of the spatial position and the semantic feature of each object.

Description

Technical Field

[0001] The present invention relates to the field of smart home, and more specifically, to a context feature learning method based on spatial attention in a smart home. Background Art

[0002] In the real world, there is usually a coexistence relationship between "target and scene" and "target and target". Utilizing this relationship will help improve the detection performance of difficult targets with unclear visual features. Before deep learning, studies have shown that the target detection performance can be improved by properly modeling the context, especially for smaller targets, targets with large occlusions, or targets with unclear appearance features due to poor image quality. With the widespread application of deep neural networks, some studies have also attempted to integrate the context around the target into the deep neural network, and have achieved certain results. The following will briefly introduce and analyze the background technology related to the present disclosure, namely, the target detection method based on implicit context feature learning.

[0003] Implicit context features refer to the background features around the target area or the global scene features. In fact, the convolution operation in the convolutional neural network has already considered the implicit context features around the target area to a certain extent. In order to make full use of the context information, some methods try to integrate the global context information into the target detection model.

[0004] As complementary information to target features, contextual features are extremely important for target detection. Most existing methods extract local contextual features by setting a context window around the target area or directly extract the features of the entire image as global contextual features. However, these two methods are not sufficient and accurate in the use of context. The former may lose some important contextual information, while the latter may introduce some background interference noise. Therefore, it is urgent to design a contextual feature learning method based on spatial attention in smart homes. The contextual relationships between different regions are learned through the attention mechanism, and these contextual relationships are fused to find unique contextual features for different regions. Summary of the invention

[0005] 1. Technical problems to be solved

[0006] In view of the problems existing in the prior art, the purpose of the present invention is to provide a context feature learning method based on spatial attention in smart homes, which learns the contextual relationships between different regions through the attention mechanism and integrates these contextual relationships, so as to find unique contextual features for different regions and improve the target detection performance in complex scenarios.

[0007] 2. Technical solution

[0008] To solve the above problems, the present invention adopts the following technical solutions.

[0009] The context feature learning method based on spatial attention in smart home includes the following steps:

[0010] S1, convolution feature extraction: extract features from the input image through the feature extraction network to obtain a convolution feature map;

[0011] S2. Local context learning: The convolution feature map is divided into multiple units of the same size. Each unit in the feature map is equivalent to each object. The object can be a real object and a background, or a combination of a real object and a background. This design allows greater flexibility in the learning process. In order to learn the spatial and semantic contextual relationships between all pairs of objects, the perceptron is used and then followed by the linear rectifier function ReLU. The object group is input into the perceptron to learn the contextual relationship between the object pairs.

[0012] S3, global context learning: Based on S2, a two-layer convolutional neural network is added. For each object, it is integrated with other objects to learn the relationship between it and the entire scene. Since different objects in the scene have different influences on the category judgment of the target object, different objects are adaptively weighted through learnable convolution operations to achieve the selection of context features.

[0013] S4, feature fusion: The context feature map in S3 is fused with the original convolution feature map to obtain a convolution feature map with rich context information.

[0014] Furthermore, the feature extraction network in S1 includes a visual geometry group and a residual network. By using the visual geometry group and the residual network, the features of the input image can be extracted to obtain a convolutional feature map.

[0015] Furthermore, an object pair includes two objects, each of which includes a spatial position and semantic features. Every two objects are set as an object pair, and with the help of a three-layer perceptron and a linear rectifier function, as well as the spatial position and semantic features of each object, the contextual relationship between the object pairs is obtained.

[0016] Furthermore, the perceptron in S2 is set to have three layers, and the number of neurons in each layer of the perceptron is 512.

[0017] Furthermore, the feature map fusion in S4 includes pixel-by-pixel multiplication, pixel-by-pixel addition and 1×1 convolution operations of the feature maps.

[0018] 3. Beneficial effects

[0019] Compared with the prior art, the advantages of the present invention are:

[0020] (1) This scheme learns the contextual relationships between different regions through the attention mechanism and fuses these contextual relationships to find unique contextual features for different regions and improve the object detection performance in complex scenarios.

[0021] (2) The feature extraction network in S1 includes a visual geometry group and a residual network. By using the visual geometry group and the residual network, the features of the input image can be extracted to obtain a convolutional feature map.

[0022] (3) An object pair consists of two objects, each of which contains a spatial position and semantic features. Every two objects are called an object pair, and with the help of a three-layer perceptron and a linear rectifier function, as well as the spatial position and semantic features of each object, the contextual relationship between the object pairs is obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A flow chart of the whole of the present invention;

[0024] Figure 2 A flowchart of the local context learning part of the present invention;

[0025] Figure 3 Flowchart of the global context learning part of the present invention. DETAILED DESCRIPTION

[0026] The following will combine the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all the embodiments. All other embodiments obtained by ordinary technicians in this field without creative work based on the embodiments of the present invention belong to the scope of protection of the present invention.

[0027] In the description of the present invention, it should be noted that the terms "upper", "lower", "inner", "outer", "top / bottom" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific position, be constructed and operated in a specific position, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0028] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "provided with", "mounted / connected", "connected", etc. should be understood in a broad sense. For example, "connected" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0029] Example:

[0030] See also Figure 1-3 ,The context feature learning method based on spatial attention in smart home, includes the following steps:

[0031] S1, convolution feature extraction: extract features from the input image through the feature extraction network to obtain a convolution feature map;

[0032] S2. Local context learning: The convolution feature map is divided into multiple units of the same size. Each unit in the feature map is equivalent to each object. The object can be a real object and a background, or a combination of a real object and a background. This design allows greater flexibility in the learning process. In order to learn the spatial and semantic contextual relationships between all pairs of objects, the perceptron is used and then followed by the linear rectifier function ReLU. The object group is input into the perceptron to learn the contextual relationship between the object pairs.

[0033] S3, global context learning: Based on S2, a two-layer convolutional neural network is added. For each object, it is integrated with other objects to learn the relationship between it and the entire scene. Since different objects in the scene have different influences on the category judgment of the target object, different objects are adaptively weighted through learnable convolution operations to achieve the selection of context features.

[0034] S4, feature fusion: The context feature map in S3 is fused with the original convolution feature map to obtain a convolution feature map with rich context information.

[0035] See also Figure 2-3The feature extraction network in S1 includes a visual geometry group and a residual network. By using the visual geometry group and the residual network, the features of the input image can be extracted to obtain a convolutional feature map. An object pair includes two objects, each of which contains a spatial position and semantic features. Every two objects are set as an object pair, and with the help of a three-layer perceptron and a linear rectifier function, the spatial position and semantic features of each object are used to obtain the contextual relationship between the object pairs. The perceptron in S2 is set to three layers, and the number of neurons in each layer of the perceptron is 512. The feature map fusion in S4 includes pixel-by-pixel dot multiplication, pixel-by-pixel addition, and 1×1 convolution operations of the feature map.

[0036] The method designed in this disclosure mainly consists of a three-layer perceptron and a two-layer convolutional neural network. The three-layer perceptron learns the semantic and spatial contextual relationships between different regions, while the two-layer convolutional neural network adaptively selects valuable contexts by feature fusion. The whole process can be expressed as follows:

[0037]

[0038]

[0039] Among them, formula (1) is used to generate the context feature map, and formula (2) is used to fuse the convolution feature map and the context feature map. o , F c and Represent the convolutional feature map, the contextual feature map and the enhanced feature map respectively. i and j They represent the i-th and j-th elements in the input convolution feature map, corresponding to different regions in the original image. θ and l p They represent the local context learning function and the global context learning function respectively, and θ and p represent the parameters in the two functions respectively. is the feature fusion function.

[0040] The above is only a preferred specific implementation manner of the present invention; but the protection scope of the present invention is not limited thereto; any technician familiar with the technical field within the technical scope disclosed by the present invention; any equivalent replacement or change based on the technical solution and improved concept of the present invention shall be covered by the protection scope of the present invention.

Claims

1. A context feature learning method based on spatial attention in smart home, characterized by: The following steps are involved: S1, convolution feature extraction: extract features from the input image through the feature extraction network to obtain a convolution feature map; S2. Local context learning: The convolution feature map is divided into multiple units of the same size. Each unit in the feature map is equivalent to each object. The perceptron is used and then followed by the linear rectifier function ReLU. The object group is input into the perceptron to learn the contextual relationship between object pairs. An object pair includes two objects, each of which contains a spatial position and semantic features. Every two objects are called an object pair. The contextual relationship between object pairs is obtained by using a three-layer perceptron and a linear rectifier function, as well as the spatial position and semantic features of each object. S3, global context learning: Based on S2, a two-layer convolutional neural network is added to fuse each object with other objects to learn the relationship between it and the entire scene; S4, feature fusion: The context feature map in S3 is fused with the original convolution feature map to obtain a convolution feature map with rich context information.

2. The context feature learning method based on spatial attention in smart home according to claim 1, characterized in that: The feature extraction network in S1 includes a visual geometry group and a residual network.

3. The context feature learning method based on spatial attention in smart home according to claim 1, characterized in that: The perceptron in S2 is set to three layers, and the number of neurons in each layer of the perceptron is 512.

4. The context feature learning method based on spatial attention in smart home according to claim 1, characterized in that: The feature map fusion in S4 includes pixel-by-pixel multiplication, pixel-by-pixel addition and 1×1 convolution operations of the feature maps.

Images