A social relationship recognition method based on a multi-layer self-attention network

Abstract

The application provides a social relationship recognition method based on a multi-layer self-attention network, and mainly relates to the problem of extracting and reasoning related features through a multi-layer self-attention network and recognizing social relationships in deep learning. First, relevant features are extracted through a feature extraction module. Then, the internal relationship between different features is reasoned by using an internal relationship reasoning module, and a learnable additional embedding block is introduced to integrate global information of a person pair. Finally, the logical relationship between different relationships in the same scene is mined by using a logical relationship reasoning module, and the output is finally classified in a social relationship. The application fully considers feature extraction related to social relationship recognition, uses the internal relationship reasoning module and the logical relationship reasoning module to mine the internal relationship between different features and the logical relationship between relationships, respectively, and solves the problems of insufficient feature extraction and too rough feature fusion mode in social relationship recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the social relationship recognition problem in the field of deep learning, and in particular to a social relationship recognition method based on a multi-layer self-attention network. BACKGROUND

[0002] In the field of computer vision, social relationship recognition is an important task to study the social relationship between people, which provides important clues for understanding human interaction behavior. At present, the existing researches mostly base on facial features, body region features, scene features and the like to perform social relationship recognition, and certain effects have been achieved. In recent years, with the powerful reasoning ability for information, the network based on the Transformer structure has made great breakthroughs in the field of natural language processing and the field of computer vision, and thus has shown certain potential in the task of social relationship recognition under the field of computer vision. At present, social relationship recognition plays an important role in the fields of photo classification, group division and crowd activity analysis.

[0003] As an important research task in the field of computer vision, social relationship recognition has been widely concerned by relevant researchers at home and abroad. At present, the existing methods often adopt a traditional convolutional neural network as a feature extraction backbone network, thereby paying more attention to local information and failing to effectively extract the interactive information of the person pair hidden in the global information. In addition, most of the methods only adopt a simple splicing fusion manner for the extracted feature vectors, or directly splice them as nodes of a graph neural network, and thus cannot fully mine the internal relationship between the person pair features and the logical relationship hidden in the relationship. Therefore, the present application firstly extracts the relative position features of the person pair, the features of each person in the person pair, the features of the common region of the person pair and the scene features of the whole image through one full connection layer, two weight-shared visual Transformer networks, one parameter-independent visual Transformer network and one ResNet-50 network; then reasons the internal relationship between the person pair-related features and the scene features through a Transformer encoder network, and introduces an additional learnable embedding block to integrate the global person pair information reflecting the social relationship as the output of the network; subsequently introduces another Transformer encoder network to reason the logical relationship between different social relationships based on the person pair in the same RGB image; and finally classifies the global information of the person pair after reasoning, so as to improve the rationality and recognition accuracy of the social relationship. SUMMARY

[0004] The purpose of the present application is to provide a social relationship recognition method based on a multi-layer self-attention network, which first fully extracts the features of the person pair related to social relationship recognition by using a visual Transformer network and a fully connected network, extracts scene features by using a ResNet-50 network, then infers the internal relationship between the person pair related features and the scene between the person pair related features by a layer of Transformer encoder network and introduces additional learnable embedding blocks to integrate the global information reflecting the social relationship of the person pair, then another layer of Transformer encoder network infers the logical relationship between different relationships in the same scene, and finally classifies the global information of the person pair after reasoning, effectively solving the problems of insufficient feature extraction, too rough feature fusion method and unreasonable reasoning and classification in social relationship recognition.

[0005] For the convenience of explanation, first introduce the following concepts:

[0006] Pre-trained model: the training of neural network requires a large amount of data, time and sufficient computing resources, in order to avoid repeated training of network, the model parameters of other researchers with good effect are migrated to the model in specific task, and are fine-tuned to adapt to the needs of the task.

[0007] Transformer: a kind of deep learning network, which extracts input data in parallel mode by using self-attention mechanism, generally composed of decoder and encoder.

[0008] Vision Transformer (ViT): a Transformer network application expanded from natural language processing field to computer vision field, which divides the picture into blocks, compresses it to one dimension as the input of Transformer encoder, and extracts features globally by using the self-attention mechanism of Transformer itself.

[0009] Graph: refers to the graph in graph theory, which is a graph in non-euclidean space composed of nodes and edges connecting the nodes.

[0010] Graph Neural Network (GNN): a neural network structure directly calculated on the graph, which learns the expression of the node by message passing, updates the information of the current node with the adjacent nodes until the whole graph converges to a stable state.

[0011] Gated Graph Neural Network (GGNN): To solve the application limitation of traditional graph neural network based on Banach's Fixed Point Theorem, a new graph neural network is introduced by using the update method of Gated Recurrent Unit (GRU).

[0012] Residual Network (ResNets): A kind of deep learning network, which solves the side effects caused by network depth by introducing residual blocks; according to the number of network layers, it is divided into ResNet-18, ResNet-34, ResNet-50, ResNet-101 and ResNet-152.

[0013] The specific technical scheme of the application is as follows:

[0014] A social relationship recognition method based on a multi-layer self-attention network, characterized in that:

[0015] a. Extract the character features and scene features associated with social relationship recognition through full connection network, deep self-attention network visual Transformer and convolutional neural network;

[0016] b. Use the Transformer encoder network based on self-attention mechanism to infer the relationship between the character features associated with social relationship recognition and the scene features;

[0017] c. Introduce another layer of Transformer encoder network to infer the logical relationship between the social relationships based on character pairs through self-attention mechanism;

[0018] The method mainly includes the following steps:

[0019] (1) Data processing and enhancement: The bounding box area of the two characters and the joint area of the character pair as input are uniformly cropped to 224x224 size, the whole picture is cropped to 448x448 size, and the cropped picture is normalized and randomly horizontally flipped; In addition, the position information and area information of the bounding box of the two characters are normalized and used as one-way input;

[0020] (2) Feature extraction: the relative position features of the person pair, the features of each person in the person pair, the features of the common region of the person pair and the scene features of the whole image are extracted in turn through a full connection layer in the model, two weight-shared and pre-trained deep self-attention network visual Transformers, a parameter-independent and pre-trained deep self-attention network visual Transformer and a pre-trained ResNet-50 network.

[0021] (3) Intrinsic relationship reasoning between features: the four person features and one scene feature in the feature extraction module are reasoned through a Transformer encoder network based on a self-attention mechanism, and the intrinsic relationship between the person features and the scene features is mined; it should be noted that an additional parameter-learnable embedding block is introduced in the network, and the global person pair social information is integrated as the output by using the self-attention mechanism.

[0022] (4) Logical relationship reasoning between relationships: the global information of the person pair in step (3) is divided according to the different RGB images, and the global information of the person pair in the same RGB image is taken as the input of another Transformer encoder network, and the logical relationship between different relationships is reasoned by the self-attention mechanism.

[0023] (5) Social relationship classification: the global information of the person pair reasoned by the Transformer encoder network in step (4) is taken as the input of the classification layer, and a full connection layer is used to classify the global information of the person pair.

[0024] The beneficial effects of the present application are:

[0025] (1) The pre-trained visual Transformer network model is fully utilized for feature extraction, which saves a lot of training time and improves the social relationship recognition effect.

[0026] (2) The intrinsic relationship between the person pair related feature information and the scene is extracted through a layer of Transformer encoder network, and an additional learnable embedding block is introduced to integrate the global information of the person pair reflecting the social relationship.

[0027] (3) The logical relationship between different social relationships in the same scene is reasoned through another layer of Transformer encoder network, and the information between relationships is effectively mined. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 It is a schematic diagram of the intrinsic relationship reasoning module between features.

[0029] Figure 2 This is a schematic diagram of the logical connection reasoning module between relationships.

[0030] Figure 3 This is a diagram of the overall model framework. Detailed Implementation

[0031] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the following embodiments are only used to further illustrate the present invention and should not be construed as limiting the scope of protection of the present invention. Those skilled in the art can make some non-essential improvements and adjustments to the present invention based on the above-described invention, and these improvements and adjustments should still fall within the scope of protection of the present invention.

[0032] The social relationship recognition method based on multi-layer self-attention networks specifically includes the following steps:

[0033] (1) Data processing and enhancement

[0034] The bounding box regions of the two figures and the joint region of the figure pair, which are used as input, are uniformly cropped to a size of 224×224. The entire image is then cropped to a size of 448×448. The cropped images are then normalized and randomly horizontally flipped. In addition, the position and area information of the bounding boxes of the two figures are normalized and used as one input, which is specifically represented as follows:

[0035]

[0036] Where, x min_A y min_A x max_A y max_A They represent Figure 1 The coordinates of the top left and bottom right corners of the bounding box of person A, area A This represents the area of ​​the bounding box of person A. Similarly, x min_B ,y min_B ,x max_B ,y max_B area B The coordinates and area of ​​the bounding box corresponding to person B. When this vector is used as network input, its coordinates and area are normalized to [-1 to 1].

[0037] (2) Feature extraction

[0038] pass Figure 3two weight-shared and pre-trained visual Transformer networks in the feature extraction module in the image, one parameter-independent and pre-trained visual Transformer network in the image, one fully connected layer, and one pre-trained ResNet-50 network in the Places365-Standard dataset sequentially extract the features of each person in the pair, the features of the common region of the pair, the relative position features of the pair, and the scene features of the whole image; it should be noted that the last classification layer in the visual Transformer network and the ResNet-50 network is removed because it only serves as a feature extraction network.

[0039] (3) Inter-feature relationship reasoning

[0040] As shown in Figure 1 , the extracted features of each person in the pair, the features of the common region of the pair, the relative position features, and the scene features of the whole image are taken as five feature vectors, which are jointly input into a learnable additional residual embedding block as the input of the Transformer encoder network. The self-attention mechanism of the kernel of the Transformer encoder network mines the internal relationship between the pair-related features and the relationship between the pair-related features and the scene features. At the same time, the residual embedding block also obtains the global pair-related information, which is taken as the output of the network and has a dimension of 2048.

[0041] In addition, the five feature vectors and the learnable additional embedding block are regarded as six inputs of a sentence sequence, and are artificially given position information embedding (PE); among them, the additional embedding block is position 0, the first person region feature, the second person region feature, the pair joint region feature, the pair relative position feature, and the scene feature extracted from the whole image are given position 1, position 2, position 3, position 4, and position 5, respectively.

[0042] (4) Inter-relationship logical connection reasoning

[0043] As shown in Figure 2 , the global information of the pair output in step (3) is regarded as the corresponding social relationship, and another layer of Transformer encoder network is introduced to reason the logical connection between different social relationships in the same scene; as shown in Figure 2 , there are four people in the input image, so they are grouped into six pairs to form six global information blocks of the pair, i.e., M equals 6.

[0044] In addition, similar to the network in step (3), for different social relationships in the same RGB image, they are regarded as several inputs of a sentence sequence, and are also artificially given position information, such as Figure 2 As shown in the figure, since there are six pairs of people, the global information block is artificially and randomly given positions 0 to 5 when input. It should be noted that the position relationship is only added by the network input requirement, and has no effect on the final reasoning result. In addition, the output of this layer network is 2048 dimensions.

[0045] (5) Social relationship classification

[0046] After fully reasoning the logical connection between the relationships in step (4), the global information of the pairs of people containing the logical connection between the relationships is classified through a fully connected layer.

Claims

1. A social relationship recognition method based on a multi-layer self-attention network, characterized in that: a. Extracting character and scene features associated with social relationship recognition using fully connected networks, deep self-attention network visual Transformers, and convolutional neural networks; b. Using a Transformer encoder network based on a self-attention mechanism to infer and identify the relationships between character features and between character features and scene features related to social relationship recognition; c. Introduce another layer of Transformer encoder network to infer the logical connections between social relationships based on person pairs through a self-attention mechanism; This method mainly includes the following steps: (1) Data processing and enhancement: The bounding box regions of the two characters and the joint region of the character are uniformly cropped to a size of 224×224. The entire image is cropped to a size of 448×448. The cropped image is normalized and randomly horizontally flipped. In addition, the position and area information of the bounding boxes of the two characters are normalized and used as one input. (2) Feature extraction: The relative position features of the person pair, the features of each person in the person pair, the features of the common area of ​​the person pair, and the scene features of the whole image are extracted sequentially through a fully connected network, a deep self-attention visual Transformer network, a fully connected layer in a convolutional neural network, two pre-trained visual Transformer networks with shared weights, a pre-trained visual Transformer network with independent parameters, and a pre-trained ResNet-50 network. (3) Inference of intrinsic relationships between features: The four character features and one scene feature in the feature extraction module are inferred by the Transformer encoder network based on the self-attention mechanism to explore the intrinsic relationships between character features and between character features and scene features. It should be noted that an additional learnable embedding block is introduced into the network to integrate global character-to-social information as output using the self-attention mechanism. (4) Reasoning about logical connections between relationships: The global information of the characters output in step (3) is distinguished according to the different RGB images in which they are located. The global information of the characters in the same RGB image is used as the input of another Transformer encoder network, and the logical connections between different relationships are reasoned through the self-attention mechanism. (5) Social relationship classification: The global information of the person after reasoning through the Transformer encoder network in step (4) is used as the input of the classification layer, and a fully connected layer is used to classify the global information of the person.

2. The social relationship recognition method based on a multi-layer self-attention network as described in claim 1, characterized in that... In step (3), the relationships between the person-related features extracted in step (2) and between the person-related features and the scene features are inferred through the Transformer encoder network. An additional parameter-learnable empty embedding block is introduced at the input end. The size of the embedding block is consistent with the person-related features and the scene features. Then, the different features of the person-pair and the scene features are integrated through the self-attention mechanism to output the global information of the person-pair reflecting social relationships.

3. The social relationship recognition method based on a multi-layer self-attention network as described in claim 1, characterized in that... In step (4), the global information of people reflecting social relationships output in step (3) is used as the input of this step, and the logical connections between different social relationships within the same RGB image are mined through another Transformer encoder network.

Images