Social relationship identification method, device, equipment and storage medium

Abstract

The present application relates to a social relationship recognition method, device, equipment and storage medium, and specifically relates to the field of image recognition. The method includes: obtaining target image features of a target image, and extracting each target person feature from the target image features; constructing a person feature relationship graph based on each target person feature and target image feature; generating each target query vector based on each node in the person feature relationship graph; performing encoding processing on the target image feature after dimensional transformation, and obtaining a target encoding vector; performing decoding processing on the target query vector and the target encoding vector through a decoder with a self-attention layer, and obtaining the social relationship between each target person. The above scheme can pay more attention to the features related to the person in the overall features of the target image, thereby improving the accuracy of the acquired social relationship.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a social relationship recognition method, apparatus, device and storage medium. Background Art

[0002] Image recognition refers to the technology of using computers to process, analyze and understand images to identify targets and objects of various different patterns. It is a practical application of deep learning algorithms.

[0003] In the task of identifying social relationship types, given an image and the coordinates of multiple human frames within it, the model needs to predict the type of social relationship between each pair of human frames. Common types of social relationships include friends, family, couples, and strangers. This task requires the model to perform advanced semantic understanding of the image, such as understanding the scene, objects, and interactions between people. In the field of social relationship type prediction, traditional models using attention mechanisms first detect objects using object detection algorithms, then apply attention mechanisms to these detected objects to determine the relationships between them.

[0004] In the above scheme, the target is first detected by the target detection algorithm, and then the attention mechanism is used on the detected target. The influence of the scene cannot be taken into account when determining the social relationship, and the accuracy of social relationship recognition is low. Summary of the Invention

[0005] The present application provides a social relationship identification method, apparatus, device and storage medium, which improves the accuracy of social relationship classification. The technical solution is as follows.

[0006] In one aspect, a method for identifying social relationships is provided, the method comprising:

[0007] Acquire the target image;

[0008] Extract features from the target image using a feature extraction model to obtain target image features, and extract features of each target person from the target image features;

[0009] Constructing a character feature relationship graph based on the target character features and the target image features; each node in the character feature relationship graph is used to indicate the image features of each target character; and the edges between the nodes are used to indicate the feature relationships between the target characters;

[0010] Generate target query vectors according to the nodes in the character feature relationship graph; the target query vectors are used to indicate the relationship between any two nodes in the character feature relationship graph;

[0011] After the target image features are dimensionally transformed, encoding is performed through an encoder to obtain a target encoding vector;

[0012] According to each target query vector and target encoding vector, decoding processing is performed through a decoder with a self-attention layer to obtain the social relationship between each of the target persons.

[0013] In another aspect, a social relationship identification device is provided, the device comprising:

[0014] A target image acquisition module is used to acquire a target image;

[0015] A feature extraction module is used to extract features of the target image using a feature extraction model to obtain target image features and extract features of each target person from the target image features;

[0016] A relationship graph construction module is used to construct a character feature relationship graph based on the characteristics of each target person and the characteristics of the target image; each node in the character feature relationship graph is used to indicate the image characteristics of each target person; and the edges between the nodes are used to indicate the characteristic relationships between the target persons;

[0017] A query vector generation module, configured to generate target query vectors based on the nodes in the character feature relationship graph; the target query vectors are used to indicate the relationship between any two nodes in the character feature relationship graph;

[0018] A coding vector generation module is used to perform dimensionality transformation on the target image features and then perform encoding processing through an encoder to obtain a target coding vector;

[0019] The social relationship reasoning module is used to obtain the social relationship between each target person by decoding each target query vector and target encoding vector through a decoder with a self-attention layer.

[0020] On the other hand, a computer device is provided, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the above-mentioned social relationship identification method.

[0021] On the other hand, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement the above-mentioned social relationship identification method.

[0022] In another aspect, a computer program product or a computer program is provided, wherein the computer program product or the computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-described social relationship identification method.

[0023] The technical solution provided by this application may have the following beneficial effects:

[0024] When social relationship recognition is required, the computer device can first extract features of the target image through a feature extraction model, and then extract the features of each target person from the target image features; the computer device can then construct a character relationship feature graph based on the target person features corresponding to each target person and the target image features corresponding to the overall image containing each target person. At this time, the character relationship feature graph construction process simultaneously considers the individual image features of each person and the overall features of the environment in which the person is located. The computer device then generates a target query vector based on the relationship between the nodes in the character relationship feature graph; the computer device then performs dimensionality change and encoding operations on the target image features to obtain a coding vector. At this time, the coding vector contains the overall features of the target image. The target query vector and the target coding vector are decoded by a decoder with a self-attention layer, so that the decoder can pay more attention to the features related to the person in the overall features of the target image based on the features between the people contained in the target query vector, thereby improving the accuracy of the acquired social relationships. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the specific implementation methods or the description of the prior art. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0026] Figure 1 The figure is a structural diagram of a social relationship identification system according to an exemplary embodiment.

[0027] Figure 2 The figure is a flowchart of a method for identifying a social relationship according to an exemplary embodiment.

[0028] Figure 3 The figure is a flowchart of a method for identifying a social relationship according to an exemplary embodiment.

[0029] Figure 4A schematic diagram of a process for identifying social relationships using a machine learning model according to an embodiment of the present application is shown.

[0030] Figure 5 It is a structural block diagram of a social relationship identification device according to an exemplary embodiment.

[0031] Figure 6 It is a schematic diagram of a computer device provided according to an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0032] The following will clearly and completely describe the technical solution of this application in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0033] It should be understood that the "indication" mentioned in the embodiments of this application can be a direct indication, an indirect indication, or an indication of an association. For example, "A indicates B" can mean that A directly indicates B, for example, B can be obtained through A; it can also mean that A indirectly indicates B, for example, A indicates C, and B can be obtained through C; it can also mean that there is an association between A and B.

[0034] In the description of the embodiments of the present application, the term "corresponding" may indicate a direct or indirect correspondence between the two, or an association relationship between the two, or a relationship between indication and being indicated, configuration and being configured, etc.

[0035] In an embodiment of the present application, "predefinition" can be achieved by pre-saving corresponding codes, tables or other methods that can be used to indicate relevant information in a device (for example, including a terminal device and a network device). This application does not limit its specific implementation method.

[0036] The social relationship recognition method provided in the embodiments of the present application can be applied to computer devices with strong data processing capabilities. The social relationship recognition method also includes a method for training a social relationship recognition model, which can process an input image to obtain social relationships between the various people in the input image. In one possible implementation, the social relationship recognition method provided in the embodiments of the present application can be applied to a personal computer, workstation, or server, that is, the social relationship recognition model is trained by at least one of the personal computer, workstation, and server.

[0037] Figure 11 is a schematic diagram illustrating the structure of a social relationship identification system according to an exemplary embodiment. The social relationship identification system includes a server 110 and a terminal 120. The terminal 120 and the server 110 communicate data via a communication network, which can be a wired network or a wireless network.

[0038] Optionally, an application with image processing function is installed in the terminal 120. The application may be a professional image processing application, a social application, a virtual reality application, or an AI application with image processing function, which is not limited in the embodiment of the present application.

[0039] Optionally, the terminal 120 can be a terminal device having an image acquisition component, which is used to acquire images and store them in a data storage module in the terminal 120; the terminal 120 can also be a terminal device having a data transmission interface, which is used to receive image data captured by an image acquisition device having an image acquisition component.

[0040] Optionally, the terminal 120 can be a mobile terminal such as a smart phone, a tablet computer, a laptop computer, or a similar mobile terminal, or a terminal such as a desktop computer, a projection computer, or a smart terminal with a data processing component, and the embodiment of the present application does not impose any restrictions on this.

[0041] The server 110 may be implemented as a single server or a server cluster consisting of a group of servers. The server 110 may be a physical server or a cloud server. In one possible implementation, the server 110 is a background server of an application in the terminal 120 .

[0042] In one possible implementation of the present application, server 110 trains the social relationship recognition model using a pre-set training sample set (including various training sample images). The training sample set may include training sample images of different categories, each of which has its own category annotation information (i.e., the social relationships between various people). After server 110 completes training of the social relationship recognition model, it transmits the trained social relationship recognition model to terminal 120 via a wired or wireless network.

[0043] Terminal 120 receives the trained social relationship recognition model and sends the data information corresponding to the social relationship recognition model (such as model structure and weight information) to an application with image recognition function, so that when the user uses the application, the input image can be recognized to obtain the social relationship between the characters in the image.

[0044] Optionally, the above-mentioned server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms and other technical operations and computing services.

[0045] Optionally, the system may further include a management device for managing the system (such as managing the connection status between each module and the server, etc.), and the management device and the server are connected via a communication network. Optionally, the communication network is a wired network or a wireless network.

[0046] Optionally, the above-mentioned wireless network or wired network uses standard communication technology and / or protocol. The network is typically the Internet, but may also be any other network, including but not limited to any combination of a local area network, a metropolitan area network, a wide area network, a mobile, a limited or wireless network, a private network, or a virtual private network. In some embodiments, the data exchanged over the network is represented using technologies and / or formats including hypertext markup language, extensible markup language, etc. In addition, conventional encryption technologies such as secure socket layer, transport layer security, virtual private network, internet protocol security, etc. may be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies may also be used to replace or supplement the above-mentioned data communication technologies.

[0047] Figure 2 is a flowchart of a method for identifying a social relationship according to an exemplary embodiment. The method is executed by a computer device, which may be a computer device such as Figure 1 The server or terminal in the social relationship identification system shown in FIG. Figure 2 As shown, the social relationship identification method may include the following steps:

[0048] Step 201: Acquire a target image.

[0049] In a possible implementation of an embodiment of the present application, the target image may be image data containing images of various characters, and the computer device may perform data processing on the target image as shown in the embodiment of the present application to extract the social relationships between the various characters detected in the target image.

[0050] Step 202 : extracting features from the target image using a feature extraction model to obtain target image features, and extracting features of each target person from the target image features.

[0051] In one possible implementation of an embodiment of the present application, the feature extraction model may be a deep neural network model, which converts the target image into a data format corresponding to the feature extraction model, that is, the target image may be processed by the feature extraction model to extract the target image features in the target image (that is, the feature map of the target image, also called Featuremap).

[0052] After the feature map is extracted, the features of each target person in the feature map can be obtained using the coordinates of the given target person through the region of interest pooling (ROI Pooling) method.

[0053] Optionally, the feature extraction model may be pre-trained using sample images.

[0054] Step 203 constructs a character feature relationship graph based on the characteristics of each target person and the characteristics of the target image.

[0055] Among them, each node in the character feature relationship graph is used to indicate the image feature of each target person; and the edges between the nodes are used to indicate the feature relationship between the target persons.

[0056] In a possible implementation, after acquiring the target person features and the target image features, the computer device may construct nodes based on the target person features and construct edges between the nodes based on the target image features.

[0057] For example, when each area where a target person exists is detected from the target image features (such as through an object detection algorithm), and the features in each area where a target person exists are obtained as the features of each target person, the features of each target person and the target image features can also be input into a pre-trained machine learning model to further extract the features of each target person, thereby constructing a node for each target person.

[0058] For example, the target person features and the pooled target image features can be spliced ​​together respectively, and the spliced ​​features can be input into the machine learning model. At this time, the features extracted by the machine learning model are features extracted after taking into account both the target person and the environment; or the machine learning model can be a two-branch machine learning model, that is, the features of the node corresponding to the target person are obtained by splicing the target person features and the target image features after feature extraction through different convolutional layers.

[0059] In the character feature relationship diagram processed in the above manner, the image features of the target person indicated by each node are image features obtained after taking into account the overall environment in which the target person is located. Therefore, compared with directly extracting the target person features, it is easier to take into account the impact of the overall environment on the target person.

[0060] Step 204 : generating target query vectors according to the nodes in the character feature relationship graph.

[0061] Since the character feature relationship graph is constructed based on both the target character features and the target image features, each node in the character feature relationship graph should also be obtained by taking into account the overall target image features. The features indicated by each node are features generated after taking into account both the target character and the overall target image.

[0062] At this time, in a possible implementation method, the computer device can extract the nodes in the character feature relationship graph in pairs, and calculate through the features between the two nodes to generate target query vectors of the two nodes to indicate the relationship between the two nodes in the character feature graph.

[0063] Step 205 : After the target image features are dimensionally transformed, encoding is performed through an encoder to obtain a target encoding vector.

[0064] In the embodiment of the present application, while constructing a query vector based on the target image features, a target encoding vector can also be obtained based on the target image features.

[0065] In one possible implementation of an embodiment of the present application, the encoder may be the encoder part of a neural network model having an encoder-decoder structure. In the neural network model having an encoder-decoder structure, the neural network model may encode the input data through the encoder, and then decode the encoded data through the decoder, thereby obtaining the result of analyzing the input data.

[0066] In step 206 , based on the target query vector and the target encoding vector, decoding processing is performed through a decoder with a self-attention layer to obtain the social relationships between the target persons.

[0067] In a possible implementation of an embodiment of the present application, the decoder may be the decoder part of a neural network model having an encoder-decoder structure. In the neural network model, both the encoder and the decoder include self-attention layers. In the encoder, the self-attention layer may process the various vector sequences and position codes into which the target image features are split. In the decoder, the self-attention layer may process the target query vector and the target encoding vector to obtain the weights of each position in the target encoding vector, thereby determining some features that require more attention during the decoding process.

[0068] Since the target query vector is a node in the character feature relationship graph generated by fully considering the influence of the overall scene, and is a vector obtained by calculating the features between the nodes, the weight determined based on the target query vector can make it easier to notice the relevant features between the characters corresponding to each node (that is, each target character) during the decoding process, thereby improving the accuracy of the social relationships between the target characters finally obtained.

[0069] To sum up, when social relationship recognition is required, the computer device can first extract features of the target image through a feature extraction model, and then extract the features of each target person from the target image features; the computer device can then construct a character relationship feature graph based on the target person features corresponding to each target person and the target image features corresponding to the overall image containing each target person. At this time, the character relationship feature graph construction process simultaneously considers the individual image features of each person and the overall features of the environment in which the person is located. The computer device then generates a target query vector based on the relationship between the nodes in the character relationship feature graph; the computer device then performs dimensionality change and encoding operations on the target image features to obtain a coding vector. At this time, the coding vector contains the overall features of the target image. The target query vector and the target coding vector are decoded by a decoder with a self-attention layer, so that the decoder can pay more attention to the features related to the person in the overall features of the target image based on the features between the people contained in the target query vector, thereby improving the accuracy of the acquired social relationships.

[0070] Figure 3 is a flowchart of a method for identifying a social relationship according to an exemplary embodiment. The method is executed by a computer device, which may be a computer device such as Figure 1 The server or terminal in the social relationship identification system shown in FIG. Figure 3 As shown, the social relationship identification method may include the following steps:

[0071] Step 301: Acquire a target image.

[0072] Step 302 : extracting features from the target image using a feature extraction model to obtain target image features, and extracting features of each target person from the target image features.

[0073] Please refer to Figure 4 , which shows a flow chart of a machine learning model used in an embodiment of the present application to identify social relationships. In an embodiment of the present application, a computer device can use Figure 4The machine learning model shown processes the input image to obtain the social relationships between the characters in the input image. The Feature Extraction Module on the left is the feature extraction module, the Graph-based Query Module in the lower right corner is the graph convolution query module, and the Transformer Reasoning Module in the upper right corner is the relationship reasoning module.

[0074] In the embodiment of the present application, when it is necessary to identify the social relationships between the characters in the target image, it is necessary to first use a feature extraction model (i.e. Figure 4 The feature extraction module in the target image extracts features from the target image, obtains a feature map in the target image, and detects the target person features corresponding to each person in the feature map. The feature extraction model can be a backbone network such as the residual convolutional network (ResNet) or the Swin Transformer.

[0075] For example, Figure 4 As shown, in the embodiment of the present application, the computer device can use the Swin Transformer pre-trained on the ImageNet dataset as the backbone network. Given a picture with a dimension of 3*H0*W0, the backbone network Backbone transforms it into a feature map Feature with a dimension of C*H*W. Among them, H=H0 / 32, W=W0 / 32, and C=1024 are the dimensions of the feature. In the embodiment of the present application, GAP can also be used to extract the feature vector of the entire picture from the feature map, which is recorded as x I , use ROIP to extract the character's feature vector from the feature map, denoted as x i .

[0076] In a possible implementation, the target image features are processed by a region of interest pooling layer (ROIP) to obtain various regions of interest.

[0077] The image features in each region of interest are pooled to obtain features of each target person.

[0078] That is, the extraction of the above-mentioned target person features can be obtained by performing data processing on the target image features through the region of interest pooling layer (ROIP, region of proposals pooling).

[0079] ROIs are regions of interest (ROIs) in the original image, which can be understood as candidate boxes for object detection, or region of interest proposals. When we perform feature extraction on the original image, we extract the corresponding feature map. The corresponding ROI is then mapped on the feature map. This mapping process is part of ROIpooling. Generally, the ROI step is followed by max pooling to obtain the required feature map, which is then fed into subsequent calculations.

[0080] According to the candidate box coordinates provided by the above process, they are mapped to FeatureMap, and then the max-pooling operation is performed to obtain the target person features corresponding to each person. The feature is a vector with a dimension of C. C depends on the feature extraction model used. For example, if it is Resnet101, then C = 2048, and if it is Swin Transformer, then C = 1024.

[0081] Step 303: construct a character feature relationship graph based on the target character features and the target image features.

[0082] In a possible implementation method of an embodiment of the present application, each target person feature is used as the initial feature of each node; the target image feature is processed through a global average pooling layer, and the processing result is used as the initial feature of the edge between each node to construct a person feature relationship graph.

[0083] At this point, the computer device can directly construct the acquired target person features as the initial features of each node in the person feature relationship graph, and perform data processing on the target image features through an average pooling layer, thereby using the overall features of the target image as the relationship between the target person features. Optionally, in subsequent operations, the computer device can fuse the features of each node with the features of the edges between nodes, so that the node can take into account the features of other nodes and the environment in which it is located, thereby increasing the information content of the features in the node.

[0084] Step 304: performing an iterative graph convolution operation on the character feature relationship graph to update the features of each node in the character feature relationship graph.

[0085] In a possible implementation method of an embodiment of the present application, after the computer device obtains the target person features, it can construct the target person features as nodes, and then construct edges between the nodes based on the target image features (i.e., the overall image features), generate a person feature relationship graph, and use the person feature relationship graph as a graph convolution model to perform iterative graph convolution operations. At this time, during the iterative process of the graph convolution model, it is possible to integrate the relationship between the various characters with the features of the overall image.

[0086] In one possible implementation, each target person feature is used as the initial feature of each node; the target image feature is processed through a global average pooling layer, and the processing result is used as the initial feature of the edge between each node.

[0087] First, for each picture, we can construct a graph. This graph has N nodes, each node represents a person in the picture. There are edges between the nodes, that is, the graph is a fully connected graph. For example, in Figure 4 There are three people in the recognition picture shown. The fully connected graph constructed at this time has three nodes and three edges. Each node and each edge has a feature. The initial feature of the i-th node is The initial features of the edge connecting the i-th node and the j-th node are

[0088] Therefore, in the embodiment of the present application, when constructing a character feature relationship graph as a graph convolution model (i.e., by Figure 4 In the Graph Construction operation shown in the figure, the graph convolution module is generated. The acquired features of each target person can be used as the initial features of the nodes corresponding to each person. The target image features are then processed through the global average pooling layer GAP to obtain the overall image features of the corresponding size as the initial features of the edges between each node.

[0089] After the graph convolution model is constructed, during the iteration of the graph convolution model, each node will update itself according to the characteristics of other nodes and the characteristics of the edges connecting the node and other nodes. That is to say, during the node update process (Nodeupdate), the characteristics of the node will gradually approach the characteristics of other nodes and the overall image characteristics.

[0090] During the iteration process of the graph convolution model, each edge will be updated (Edgeupdate) according to the features of the two nodes connected by the edge. In other words, each edge will pay more attention to the features of the connected nodes based on the overall image features.

[0091] In one possible implementation, a calculation is performed based on the features of the first node in the Tth round, the features of the second node in the Tth round, and the features of the edge between the first node and the second node in the Tth round to obtain the features of the edge between the first node and the second node in the T+1th round; the first node and the second node are any two nodes in the character feature relationship graph;

[0092] The features of the first node in round T, the features of each target node connected to the first node in round T, and the features of the edges between each target node and the first node in round T are calculated to obtain the features of the first node in round T+1.

[0093] For example, the specific operation process of the graph convolution operation can be shown as follows:

[0094] The constructed graph undergoes T rounds of graph convolution operations. Each round of graph convolution operations consists of two steps, namely updating edge features and updating node features. Specifically, in the tth round of graph convolution operations, the update formulas for edge features and node features are as follows:

[0095]

[0096]

[0097] where N i represents the number of neighbor nodes of the i-th node, W t and is the learnable parameter in the tth round of updates, σ represents the ReLU nonlinear activation function, and ⊙ is the element-wise product. After T rounds of graph convolution operations, the features of each person and the image in the image are fully integrated.

[0098] Step 305 : perform vector concatenation between two nodes in the updated character feature relationship graph to obtain each target query vector.

[0099] In an embodiment of the present application, after the iterative process of the graph convolution model is completed, that is, after the iterative graph convolution operation of the character feature relationship graph is completed, each node in the updated character feature relationship graph can be selected, and the features between each node can be spliced ​​two by two and constructed into a query vector.

[0100] For example, when a computer device needs to query the social relationship between a first person and a second person, it can select the first node corresponding to the first person and the second node corresponding to the second person in the iterated graph convolution model, and concatenate the feature vectors of the first node and the second node to construct a query vector for subsequent identification of the social relationship between the first person and the second person.

[0101] That is, after T rounds of graph convolution, the features of each person and the image are fully integrated. We concatenate the features of each two nodes together as the query vector. Specifically, the query vector for person i and person j can be expressed as:

[0102]

[0103] Step 306 : After the target image features are dimensionally transformed, encoding is performed through an encoder to obtain a target encoding vector.

[0104] The social relationship recognition solution in this application embodiment utilizes the attention mechanism in machine learning. For example, when a person views a painting, they can see the entire painting, but when they look closely, their eyes focus on a small area. At this time, the brain primarily focuses on this small area. In other words, the brain's attention to the entire image is not balanced, but rather weighted. This is the core idea of ​​the Attention Model in deep learning.

[0105] That is to say, on the one hand, the computer device constructs the target image features and the target person features contained in the target image features into a graph convolution model through the above steps to obtain the query vector, so as to assist the computer device in calculating the corresponding weights in the process of social relationship recognition, so that it is easier to pay attention to relevant features during feature analysis.

[0106] On the other hand, the computer device can further transform the extracted target image features into a feature map of a specified dimension, and split them into individual vectors for encoding through an encoder to obtain target coding vectors. At this time, each target coding vector is the feature at different positions of the target image feature.

[0107] In one possible implementation, a dimensionality reduction operation is performed on the target image feature and split into vector sequences with target dimensions; the target encoding vector is obtained by splicing the first position feature with each vector sequence and processing the vector through an encoder.

[0108] Since the vector flattening operation loses the position feature after the feature map of the specified dimension is split into individual vectors, the first position feature can be spliced ​​with each vector sequence at this time to compensate for the position feature of each vector sequence to a certain extent. At this time, each target encoding vector obtained by the encoder processing will still retain certain position features.

[0109] Optionally, the first position feature may be a parameter in the encoder, which may be trained through training samples to learn a more appropriate value.

[0110] In one possible implementation, Figure 4 As shown, in the embodiment of the present application, the computer device can process the extracted image features and query vectors through a relationship reasoning module to obtain the social relationships between the various characters.

[0111] The relational reasoning module is mainly composed of a Transformer containing an encoder and a decoder structure. The encoder is responsible for encoding the feature map extracted from the backbone network. We perform dimensionality reduction on the feature map, changing its dimension from C*H*W to d*H*W. Then, the computer performs the flattening operation Flattened Feature, turning it into a vector sequence of HW dimensions of d, and inputs the sequence into the encoder TransformerEncoder. At the same time, since the vector flattening operation loses the position information, in the embodiment of the present application, a learnable position encoding (Position Embedding) is added to the encoder.

[0112] Step 307: Concatenate the target query vector with the second position feature to obtain a position query vector.

[0113] Optionally, the second position feature may be a parameter in a decoder, which may be trained through training samples to learn a more appropriate value.

[0114] Step 308: Determine each attention coefficient based on each position query vector and each target encoding vector.

[0115] In the attention layer with an attention mechanism, the attention coefficient of each target coding vector can be obtained through the position query vector and each target coding vector. At this time, the attention coefficient can be used as the weight of each target coding formed by image feature decomposition.

[0116] Step 309: Based on the attention coefficient, the target coding vector is decoded by the decoder to obtain the probability distribution of each target person; each probability value in the probability distribution is used to indicate the probability of each social relationship between the target persons.

[0117] After obtaining the attention coefficient, the target coding vector can be decoded and weighted by the attention coefficient during the decoding process. The probability distribution of the social relationship types between the characters obtained at this time fully considers the relationship between the characters in the query vector. Therefore, the decoder can pay more attention to the features related to the characters in the overall features of the target image based on the features between the characters contained in the target query vector. The probability distribution between the characters obtained by the above scheme has a higher accuracy rate for the social relationships obtained.

[0118] In a possible implementation, in the embodiment of the present application, the computer device can be configured as follows: Figure 4 The decoder shown performs relationship type inference, and the input of the decoder consists of two parts, namely the encoding vector from the encoder and the query vector from the graph convolution module. In an embodiment of the present application, a learnable position encoding is also added to the input of the decoder to assist the decoder in locating different relationship pairs in the image. The output of the decoder passes through a multi-layer fully connected network and outputs the probability distribution of various relationship types. The global self-attention mechanism of the decoder can automatically learn the impact of different image positions on social relationship classification from the image feature map.

[0119] And in Figure 4 In the machine learning model shown, the entire model consisting of the feature extraction module, the graph convolution module and the relationship reasoning module can be trained end-to-end, that is, the machine learning model is directly trained through the input sample images and the social relationship annotations between the characters in the sample images, and each module is back-propagated and updated through each loss function, thereby improving the social relationship recognition ability of the updated machine learning model.

[0120] To sum up, when social relationship recognition is required, the computer device can first extract features of the target image through a feature extraction model, and then extract the features of each target person from the target image features; construct a graph convolution model through the target person features and the overall image features and iterate, so that the person features are fully integrated with the overall image features, and then generate a query vector based on the iterated graph convolution model; the computer device then performs dimensionality change and encoding operations on the target image features to obtain a target encoding vector. At this time, the target encoding vector contains the overall features of the target image. The target query vector and the target encoding vector are decoded by a decoder with a self-attention layer, so that the decoder can pay more attention to the features related to the person in the overall features of the target image based on the features between the people contained in the target query vector, thereby improving the accuracy of the acquired social relationships.

[0121] Figure 5It is a structural block diagram of a social relationship identification device according to an exemplary embodiment.

[0122] The device comprises:

[0123] The target image acquisition module 501 is used to acquire the target image;

[0124] A feature extraction module 502 is configured to extract features from a target image using a feature extraction model to obtain target image features and extract features of each target person from the target image features;

[0125] The relationship graph construction module 503 is used to construct a character feature relationship graph based on the characteristics of each target person and the characteristics of the target image; each node in the character feature relationship graph is used to indicate the image characteristics of each target person; and the edges between the nodes are used to indicate the characteristic relationships between the target persons;

[0126] A query vector generating module 504 is configured to generate target query vectors based on the nodes in the character feature relationship graph; the target query vectors are used to indicate the relationship between any two nodes in the character feature relationship graph;

[0127] The coding vector generating module 505 is configured to perform dimensionality transformation on the target image features and then perform coding processing through an encoder to obtain a target coding vector;

[0128] The social relationship reasoning module 506 is used to obtain the social relationship between each target person by decoding each target query vector and target encoding vector through a decoder with a self-attention layer.

[0129] In a possible implementation, the feature extraction module is further configured to:

[0130] Performing data processing on the target image features through the region of interest pooling layer ROIP to obtain various regions of interest;

[0131] The image features in each region of interest are pooled to obtain features of each target person.

[0132] In a possible implementation, the graph convolution module is further configured to:

[0133] Using the target person features as initial features of the nodes;

[0134] The target image features are processed by a global average pooling layer, and the processing results are used as the initial features of the edges between the nodes.

[0135] In a possible implementation, the apparatus further includes:

[0136] A relationship graph iteration module, configured to perform an iterative graph convolution operation on the character feature relationship graph to update the features of each node in the character feature relationship graph;

[0137] The query vector generation module is further configured to:

[0138] In the updated character feature relationship graph, vector concatenation is performed between every two nodes to obtain each target query vector.

[0139] In a possible implementation, the relationship graph iteration module is further configured to:

[0140] performing a calculation based on the feature of the first node in the Tth round, the feature of the second node in the Tth round, and the feature of the edge between the first node and the second node in the Tth round to obtain the feature of the edge between the first node and the second node in the T+1th round; the first node and the second node are any two nodes in the character feature relationship graph;

[0141] The features of the first node in the Tth round, the features of each target node adjacent to the first node in the Tth round, and the features of the edges between each target node and the first node in the Tth round are calculated to obtain the features of the first node in the T+1th round.

[0142] In a possible implementation, the encoding vector generation module is further configured to:

[0143] Performing a dimensionality reduction operation on the target image features and splitting them into vector sequences with target dimensions;

[0144] The target encoding vector is obtained by splicing the first position feature with each of the vector sequences respectively and processing them through an encoder; the first position feature is a parameter in the encoder.

[0145] In a possible implementation, the social relationship reasoning module is used to

[0146] Concatenate the target query vector with a second position feature to obtain a position query vector; the second position feature is a parameter in the decoder;

[0147] Determining an attention coefficient of each target encoding vector according to the position query vector and each target encoding vector;

[0148] Based on the various attention coefficients, the decoder performs a decoding operation on the target coding vector to obtain the probability distribution of the various target persons; the various probability values ​​in the probability distribution are used to indicate the probability of each social relationship between the target persons.

[0149] To sum up, when social relationship recognition is required, the computer device can first extract features of the target image through a feature extraction model, and then detect the features of each target person in the target image features; construct a graph convolution model through the target person features and the overall image features and iterate, so that the person features are fully integrated with the overall image features, and then generate a query vector based on the iterated graph convolution model; the computer device then performs dimensionality change and encoding operations on the target image features to obtain a coding vector. At this time, the coding vector contains the overall features of the target image. The target query vector and the target coding vector are decoded by a decoder with a self-attention layer, so that the decoder can pay more attention to the features related to the characters in the overall features of the target image based on the features between the characters contained in the target query vector, thereby improving the accuracy of the acquired social relationships.

[0150] See also Figure 6 , which is a schematic diagram of a computer device provided according to an exemplary embodiment of the present application, the computer device includes a memory and a processor, the memory is used to store a computer program, and when the computer program is executed by the processor, the above method is implemented.

[0151] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0152] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the methods described in the embodiments of the present invention. The processor executes the non-transitory software programs, instructions, and modules stored in the memory to perform various processor functions and data processing, thereby implementing the methods described in the aforementioned method embodiments.

[0153] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include a memory remotely located relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0154] In an exemplary embodiment, a computer-readable storage medium is further provided, storing at least one computer program, which is loaded and executed by a processor to implement all or part of the steps of the above method. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, or an optical data storage device.

[0155] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0156] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A social relationship identification method, characterized in that: The method comprises: Acquire the target image; Extract features from the target image using a feature extraction model to obtain target image features, and extract features of each target person from the target image features; Constructing a character feature relationship graph based on the target character features and the target image features; each node in the character feature relationship graph is used to indicate the image feature relationship of each target character, wherein each target character feature is constructed as a node, and the target image feature is constructed as an edge between the nodes. During the iterative process of the graph convolution model, the relationship between the characters is integrated with the features of the overall image; Generate target query vectors according to the nodes in the character feature relationship graph; the target query vectors are used to indicate the relationship between any two nodes in the character feature relationship graph; After the target image features are dimensionally transformed, encoding is performed through an encoder to obtain a target encoding vector; After the target image features are dimensionally transformed, encoding is performed through an encoder to obtain a target encoding vector, including: performing a dimensionality reduction operation on the target image features and splitting them into vector sequences with target dimensions; splicing the first position features with each of the vector sequences, and processing them through an encoder to obtain the target encoding vector; the first position features are parameters in the encoder; According to the target query vectors and the target encoding vectors, decoding is performed by a decoder with a self-attention layer to obtain the social relationships between the target persons; The method is based on each target query vector and the target encoding vector, and decoding processing is performed by a decoder with a self-attention layer to obtain the social relationship between each target person, including: splicing each target query vector with a second position feature to obtain each position query vector; the second position feature is a parameter in the decoder; according to each position query vector and each target encoding vector, each attention coefficient is determined; based on the each attention coefficient, the target encoding vector is decoded by the decoder to obtain the probability distribution of each target person; each probability value in the probability distribution is used to indicate the probability of each social relationship between the target persons.

2. The method according to claim 1, characterized in that Extracting each target person's features from the target image features includes: Performing data processing on the target image features through the region of interest pooling layer ROIP to obtain various regions of interest; The image features in each region of interest are pooled to obtain features of each target person.

3. The method according to claim 1, characterized in that The step of constructing a character feature relationship graph based on the target character features and the target image features includes: Using the target person features as initial features of each node; The target image features are processed by a global average pooling layer, and the processing results are used as the initial features of the edges between the nodes.

4. The method according to claim 3, characterized in that Before generating each target query vector according to each node in the character feature relationship graph, the method further includes: Performing an iterative graph convolution operation on the character feature relationship graph to update the features of each node in the character feature relationship graph; Generating target query vectors according to node features between each pair of nodes in the character feature relationship graph includes: In the updated character feature relationship graph, vector concatenation is performed between every two nodes to obtain each target query vector.

5. The method according to claim 4, characterized in that The iterative graph convolution operation is performed on the character feature relationship graph, comprising: performing a calculation based on the feature of the first node in the Tth round, the feature of the second node in the Tth round, and the feature of the edge between the first node and the second node in the Tth round to obtain the feature of the edge between the first node and the second node in the T+1th round; the first node and the second node are any two nodes in the character feature relationship graph; The features of the first node in the Tth round, the features of each target node adjacent to the first node in the Tth round, and the features of the edges between each target node and the first node in the Tth round are calculated to obtain the features of the first node in the T+1th round.

6. A social relationship recognition device, characterized in that: The device comprises: A target image acquisition module is used to acquire a target image; A feature extraction module is used to extract features of the target image using a feature extraction model to obtain target image features and extract features of each target person from the target image features; A relationship graph construction module is configured to construct a character feature relationship graph based on the target character features and the target image features; each node in the character feature relationship graph is used to indicate the image feature of each target character; and the edges between the nodes are used to indicate the feature relationship between the target characters, wherein each target character feature is constructed as a node, and the target image feature is constructed as an edge between the nodes. During the iteration process of the graph convolution model, the relationship between the characters is integrated with the features of the overall image. A query vector generation module, configured to generate target query vectors based on the nodes in the character feature relationship graph; the target query vectors are used to indicate the relationship between any two nodes in the character feature relationship graph; A coding vector generation module is used to perform dimensionality transformation on the target image features and then perform encoding processing through an encoder to obtain a target coding vector; After the target image features are dimensionally transformed, encoding is performed through an encoder to obtain a target encoding vector, including: performing a dimensionality reduction operation on the target image features and splitting them into vector sequences with target dimensions; splicing the first position features with each of the vector sequences, and processing them through an encoder to obtain the target encoding vector; the first position features are parameters in the encoder; A social relationship inference module is configured to decode the target query vectors and target encoding vectors using a decoder with a self-attention layer to obtain the social relationships between the target persons. The method is based on each target query vector and the target encoding vector, and decoding processing is performed by a decoder with a self-attention layer to obtain the social relationship between each target person, including: splicing each target query vector with a second position feature to obtain each position query vector; the second position feature is a parameter in the decoder; according to each position query vector and each target encoding vector, each attention coefficient is determined; based on the each attention coefficient, the target encoding vector is decoded by the decoder to obtain the probability distribution of each target person; each probability value in the probability distribution is used to indicate the probability of each social relationship between the target persons.

7. A computer device, characterized in that: The computer device includes a processor and a memory, and the memory stores at least one instruction, at least one program, code set or instruction set. The at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the social relationship identification method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The storage medium stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the social relationship identification method according to any one of claims 1 to 5.

Images