Life photo person recognition model training method, person recognition method and electronic device

Abstract

The invention provides a method for training a life photo character recognition model, a character recognition method and an electronic device, which include: constructing an initial life photo character recognition model based on a deep residual network; obtaining a life photo sample set containing a person to be detected; obtaining a character feature map, a background feature map, a fused character feature, a character relationship within a group, a character relationship between groups, a final character feature map and an adjacency matrix in the sample set through the initial life photo character recognition model; updating the parameters of the initial life photo character recognition model according to the loss values ​​calculated by the multiple loss functions used in the final character feature map and the adjacency matrix until the model converges to obtain a life photo character recognition model. The invention also provides a life photo character recognition method and an electronic device. The invention improves the accuracy of character identity recognition by extracting multiple feature information contained in the life photo and modeling the character relationship.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and image processing, and in particular to a life photo character recognition model training method, a character recognition method and electronic equipment. Background Art

[0002] With the rapid increase in the number of personal smart mobile devices and the growing enthusiasm of people to record their lives with photos, a large number of life photos are stored on everyone's mobile devices and Internet platforms. Most of the life photos are people. Identifying the people in life photos is conducive to applications such as rapid retrieval and classification.

[0003] The conventional approach to identifying a person's identity is to identify the person's identity through the face. Due to the rapid development of deep learning, face recognition technology has made significant progress in recent years. However, since face recognition is highly sensitive to the facial environment, many unfavorable conditions will seriously affect the recognition accuracy. In real scenes, it is often difficult to ensure that there is a high-resolution and well-lit frontal face to distinguish the identity of the person. Therefore, other modal information other than the face is needed as auxiliary recognition. Early methods mainly tried to use additional visual clues, such as clothing, timestamps, scenes, etc., but the improvement in accuracy was limited. Subsequently, the emergence of deep learning revolutionized this field. Methods based on deep learning have the following two advantages: first, deep features generally have strong recognition capabilities; second, deep learning provides better tools for extracting character features. At the same time, because deep learning uses graphics cards to accelerate calculations, multimodal clues can be effectively utilized, but some special situations occur, such as: the person is partially blocked, the resolution is too low due to the perspective, and extreme lighting; in these cases, the information of other areas of the face and body of the person becomes insufficient to identify the person's identity.

[0004] Therefore, it is necessary to propose a life photo character recognition model training method, a character recognition method and an electronic device to solve the technical problems existing in the prior art of low accuracy in character recognition and inability to recognize character identity when the image quality is not high. Summary of the invention

[0005] In view of this, it is necessary to provide a life photo character recognition model training method, a character recognition method and a device to solve the technical problems existing in the prior art of low character recognition accuracy and inability to recognize character identity when the image quality is not high and the character identity information in the image is insufficient.

[0006] In one aspect, the present invention provides a method for training a life photo character recognition model, comprising:

[0007] Construct an initial life photo character recognition model based on a deep residual network;

[0008] Acquire a sample set of life photos of the person to be tested, wherein the sample set includes the image to be tested and a control image;

[0009] Inputting the image to be tested and the control image into the initial life photo character recognition model, and processing to obtain a feature map of the person to be tested, a feature map of the control person, a feature map of the background to be tested, and a feature map of the control background;

[0010] Based on a preset fusion neural network model, the test person feature map and the test background feature map are fused to obtain a test fused person feature, and the control person feature map and the control background feature map are fused to obtain a control fused person feature;

[0011] Taking the fused character features to be tested and the fused character features to be controlled as nodes, determining the character relationships within the tested group between the fused character features to be tested, the character relationships within the control group between the fused character features to be controlled, and the character relationships between the fused character features to be tested and the fused character features to be controlled based on the connection relationships between different nodes;

[0012] Constructing an adjacency matrix based on the intra-group character relationships to be tested and the inter-group character relationships, and performing dimension splicing on the intra-group character relationships to be tested and the inter-group character relationships to obtain a final character feature map;

[0013] A variety of loss functions are used to calculate the losses of the final character feature graph and the adjacency matrix, and the model parameters of the initial life photo character recognition model are updated according to the calculated loss values ​​until the model converges to obtain the life photo character recognition model.

[0014] In some possible implementations, the step of inputting the image to be tested and the reference image into the initial life photo character recognition model to obtain a feature map of the person to be tested and a feature map of the reference person includes:

[0015] The image of the person in the marked person frame of the image to be tested and the control image is divided into P block areas, and L pooling layers are used to replace the average pooling layer of the last layer of the initial life photo person recognition model based on different body areas of the person, wherein P and L are both integers greater than 1;

[0016] Perform feature extraction on P blocks of body regions of the person in the image to be tested and the control image, respectively, to obtain feature vectors of S different body regions, and then concatenate the feature vectors of the S different body regions to obtain a feature map of the person to be tested and a feature map of the control person, respectively, wherein S is an integer greater than 1;

[0017] Among them, the image to be tested and the reference image both contain K persons with marked person frames, the number of the feature maps of the person to be tested and the number of the feature maps of the reference person are both R, and K and R are both integers greater than 1.

[0018] In some possible implementations, the step of inputting the image to be tested and the reference image into the initial life photo character recognition model to obtain the background feature map to be tested and the reference background feature map includes:

[0019] Erasing the pixels in the marked character frames of the image to be tested and the reference image to obtain the background image to be tested and the reference background image containing only background information;

[0020] The fully connected layer and the normalized layer in the initial life photo character recognition model are removed, and feature extraction is performed on the background image to be tested and the control background image respectively to obtain a background feature map to be tested and a background feature map to be controlled.

[0021] In some possible implementations, the method of fusing the subject feature map and the background feature map to be tested based on a preset fusion neural network model to obtain a fused subject feature, and fusing the control subject feature map and the control background feature map to obtain a control fused subject feature, includes:

[0022] Based on a preset fusion neural network model, the R to-be-tested person feature maps are fused with the to-be-tested background feature map to obtain O to-be-tested fused person features, and the R control person feature maps are fused with the control background feature map to obtain Q control fused person features, wherein O and Q are both integers greater than 1;

[0023] The to-be-tested fused character features include the to-be-tested character features and the to-be-tested background features, and the ....

[0024] In some possible implementations, the method of using the fused character features to be tested and the fused character features to be controlled as nodes, and determining the character relationships within the test group between the fused character features to be tested, the character relationships within the control group between the fused character features to be controlled, and the character relationships between the fused character features to be tested and the control fused character features based on the connection relationships between different nodes, includes:

[0025] The O fused character features to be tested are used as nodes to perform composition modeling inside the image to be tested, each node represents a group member, and different members in the image group to be tested are connected by edges, and different nodes are connected in pairs. At the same time, the dependency relationship between members is modeled to construct a character relationship graph in the image group to be tested;

[0026] The Q reference fusion character features are used as nodes to perform composition modeling inside the reference image, where each node represents a group member, and different members in the reference image group are connected by edges, and different nodes are connected in pairs. At the same time, the dependency relationship between members is modeled to construct a relationship graph within the reference image group;

[0027] The O fused character features to be tested and the Q fused character features to be controlled are used as nodes, a composition model is constructed between the image to be tested and the control image, the nodes of the image to be tested and the control image are connected with edges, and the nodes between different images are connected in pairs to construct an inter-group relationship graph.

[0028] In some possible implementations, constructing an adjacency matrix based on the intra-group character relationships to be tested and the inter-group character relationships, and dimensionally splicing the intra-group character relationships to be tested and the inter-group character relationships to obtain a final character feature map includes:

[0029] Constructing an adjacency matrix through the intra-group character relationships to be tested and the inter-group character relationships, wherein the rows and columns of the matrix respectively represent all members included in the intra-group character relationships to be tested and all members included in the inter-group character relationships;

[0030] The final character feature map is obtained by dimensionally splicing the character relationship within the group to be tested and the character relationship between groups, and the final character feature map contains character features, background features, and information on the relationship between members within the group and between members.

[0031] In some possible implementations, the final character feature graph and the adjacency matrix are subjected to a plurality of loss functions to calculate the loss, and the model parameters of the initial life photo character recognition model are updated according to the calculated loss values ​​until the model converges to obtain the life photo character recognition model, including:

[0032] A cross entropy loss function and a triplet loss function are used to calculate the matching degree between the person to be identified in the final person feature graph and the identities of other person instances in the reference set, wherein the value of the matching degree is a linear accumulation of the loss values ​​calculated by the cross entropy loss function and the triplet loss function;

[0033] A pairwise loss function is used to calculate the similarity of features of all members of the group where the target to be identified is located with the corresponding members of the group in the reference set, and the value of the similarity is the loss value calculated by the pairwise loss function;

[0034] Linearly add the loss values ​​calculated by the cross entropy loss function, the triplet loss function, and the pairwise loss function to obtain a target loss value;

[0035] The target loss value is used to update the parameters of the initial life photo character recognition model until the model converges to obtain the life photo character recognition model.

[0036] The target loss value is ultimately the linear accumulation of the loss values ​​calculated for the P-block body regions respectively, and the reference set is the life photo images of the marked character frames in the sample set except the image to be tested.

[0037] On the other hand, the present invention also provides a method for identifying people in daily life photos, comprising:

[0038] Obtaining a test life photo and a control life photo containing a person to be identified;

[0039] Inputting the life photo to be tested and the life photo to be controlled containing the person to be identified into the life photo person identification model, and outputting the person identification result;

[0040] Among them, the life photo character recognition model is trained by using any training method in the above-mentioned implementation manner.

[0041] On the other hand, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the life photo character recognition model training method described above and / or the life photo character recognition method described above are implemented.

[0042] On the other hand, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned life photo character recognition model training method and / or the above-mentioned life photo character recognition method.

[0043] The beneficial effect of adopting the above-mentioned embodiments is as follows: the present invention provides a life photo character recognition training method and a character recognition method, which fully considers the appearance characteristics of the characters in the life photos, the characteristics of the scenes in which the characters are located, the relationships between characters, and the relationships between groups of people, constructs and integrates the visual information contained in the photos from a multimodal perspective, and models the character relationships through a neural network, thereby improving the accuracy of character identity recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 A schematic flow chart of an embodiment of a method for training a life photo character recognition model provided by the present invention;

[0046] Figure 2 The present invention provides Figure 1 A method flow chart of steps S103 to S105 in an embodiment;

[0047] Figure 3 The present invention provides Figure 1 A method flow chart of an embodiment of step S105;

[0048] Figure 4 A schematic diagram of the structure of an embodiment of the adjacency matrix provided by the present invention;

[0049] Figure 5 The present invention provides Figure 1 A schematic diagram of a flow chart of an embodiment of step S107;

[0050] Figure 6 A schematic flow chart of another embodiment of the method for training a life photo character recognition model provided by the present invention;

[0051] Figure 7 A schematic flow chart of an embodiment of a method for identifying people in life photos provided by the present invention;

[0052] Figure 8 A schematic diagram of the structure of an embodiment of a person recognition model training device provided by the present invention;

[0053] Fig. 9 A schematic diagram of the structure of an embodiment of a person recognition device provided by the present invention;

[0054] Fig.10 A schematic structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0056] It should be understood that the schematic drawings are not drawn to scale. The flow chart used in the present invention shows the operations implemented according to some embodiments of the present invention. It should be understood that the operations of the flow chart can be implemented out of order, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flow chart under the guidance of the content of the present invention, and can also remove one or more operations from the flow chart.

[0057] Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor systems and / or microcontroller systems.

[0058] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0059] The embodiments of the present invention provide a method for training a life photo character recognition model, a method and a device for character recognition, which are described below respectively.

[0060] On the one hand, if Figure 1 As shown, Figure 1 A flow chart of an embodiment of a method for training a life photo character recognition model provided by the present invention, the method for training a life photo character recognition model comprises:

[0061] S101. Constructing an initial life photo character recognition model based on a deep residual network;

[0062] S102, obtaining a sample set of life photos of the person to be tested, wherein the sample set includes the image to be tested and a control image;

[0063] S103, inputting the image to be tested and the control image into the initial life photo character recognition model, and processing to obtain a feature map of the person to be tested, a feature map of the control person, a feature map of the background to be tested, and a feature map of the control background;

[0064] S104, based on a preset fusion neural network model, the test person feature map and the test background feature map are fused to obtain a test fused person feature, and the control person feature map and the control background feature map are fused to obtain a control fused person feature;

[0065] S105, using the to-be-tested fused character features and the control fused character features as nodes, determining, based on the connection relationships between different nodes, the to-be-tested group character relationships between the to-be-tested fused character features, the control group character relationships between the control fused character features, and the inter-group character relationships between the to-be-tested fused character features and the control fused character features;

[0066] S106, constructing an adjacency matrix based on the intra-group character relationships to be tested and the inter-group character relationships, and performing dimension splicing on the intra-group character relationships to be tested and the inter-group character relationships to obtain a final character feature map;

[0067] S107, using a variety of loss functions to calculate the loss of the final character feature graph and the adjacency matrix, and updating the model parameters of the initial life photo character recognition model according to the calculated loss values ​​until the model converges to obtain the life photo character recognition model.

[0068] Compared with the prior art, the life photo character recognition model training method provided by the embodiment of the present invention fully utilizes the various modal information contained in the life photos by extracting various feature information contained in the life photos, fusing the various feature information through a graph neural network, and modeling the character relationships. The loss values ​​calculated by multiple loss functions are used to update the model parameters of the initial life photo character recognition model, thereby improving the efficiency of model training and the accuracy of recognition.

[0069] In some embodiments of the present invention, the initial life photo character recognition model constructed based on the deep residual network in step S101 adopts a residual network with a depth of 50 layers as the backbone network, uses the deep network to extract various feature information contained in the life photos, and uses residual blocks to improve the gradient disappearance and network degradation problems.

[0070] In some embodiments of the present invention, the sample set of life photos of the person to be tested obtained in step S102 includes 3839 images in 1558 groups, of which 859 groups of image pairs including the images to be tested and the control images are used as training sets, and 699 groups of image pairs including the images to be tested and the control images are used as test sets. During the test, the images in the test set are selected in sequence as the query set, and all the remaining images are regarded as the reference set. The images in the query set are matched one by one in the reference set, and the matching results are evaluated according to the evaluation index. In this way, there is no viewpoint overlap between the query set images and the reference set images.

[0071] As a preferred embodiment, Figure 2 Come and see, Figure 2 The present invention provides Figure 1 The method flow chart of steps S103 to S105 of an embodiment, wherein:

[0072] Step S103 includes: dividing the image to be tested and the person images in the person frames marked in the control image into P block areas, and replacing the average pooling layer of the last layer of the initial life photo person recognition model with L pooling layers based on different body areas of the person, wherein P and L are both integers greater than 1;

[0073] Performing feature extraction on P blocks of body regions of the person in the test image and the control image respectively, obtaining feature vectors of S different body regions respectively, and then concatenating the feature vectors of the S different body regions to obtain a feature map of the person to be tested and a feature map of the control person respectively, wherein S is an integer greater than 1;

[0074] Among them, the image to be tested and the reference image both contain K persons with marked person frames, the number of the feature maps of the person to be tested and the number of the feature maps of the reference person are both R, and K and R are both integers greater than 1.

[0075] Erasing the pixels in the marked character frames of the image to be tested and the reference image to obtain the background image to be tested and the reference background image containing only background information;

[0076] The fully connected layer and the normalized layer of the initial life photo character recognition model are removed, and feature extraction is performed on the background image to be tested and the control background image respectively to obtain a background feature map to be tested and a background feature map to be controlled.

[0077] In the specific embodiments of the present invention, please refer to Figure 2 , the characters in the K character frames that have been annotated in the sample set are horizontally segmented and divided into P block areas. The deep residual network is modified, and L pooling layers based on different body regions of the characters are used to replace the average pooling layer of the last layer of the traditional residual network, so that the network can extract more detailed features of the character's appearance. The modified network is named backbone_P, and the character images divided into P blocks are respectively input into backbone_P to extract features of the P block body regions of the character, and feature vectors of S different body regions are obtained respectively. The feature vectors of S different body regions are then spliced ​​to obtain the character feature map of the person to be tested and the control character feature map of the person. Since the image to be tested and the control image both contain K characters with annotated character frames, the number of the character feature map to be tested and the control character feature map of the person is R, and the values ​​of P, L, and S are all 4, and the values ​​of K and R are the number of people contained in the image to be tested and the control image.

[0078] It should be noted that in the original sample set, there will be a variable number of person annotation frames in each picture. When the number of person frames in the picture is greater than K, K person frames are taken. When the number of person frames is less than K, the existing characters are copied to complete the K person frames.

[0079] The specific implementation details are as follows: Assume that the sample set contains M life photos, each containing X m For the input life photos I mFirst, extract the person part in the image. In order to fit the proportion of the person in the photo, scale it to 256×128. Then input the cropped person image into backbone_P. After a series of convolution and pooling operations, the feature vector of each body region is obtained as O(f mxp )∈R 2×1×2048 ; Finally, the feature vectors of the P blocks of body regions are concatenated, and the obtained character feature map is represented as O(f mx )∈R 8×4×2048 . Character traits are calculated according to the following formula:

[0080] f mx =backbone_P(I mx ), x=1, ..., X m

[0081] where f mx =[f mx1 , ..., f msP ] means that each character feature in the photo is composed of P blocks of body region features, [·] means vector splicing, f mxp (p=1, ..., P) represents the p-th body region feature of the x-th person in the photo.

[0082] The embodiment of the present invention divides the body area of ​​a person into multiple parts, extracts the features of different body areas respectively, and finally splices the feature vectors of different body areas. The obtained person feature map makes full use of the various feature information contained in the body area of ​​the person in the life photo, thereby improving the accuracy of the final person recognition.

[0083] In the specific embodiments of the present invention, please refer to Figure 2 , the pixels of the person area in the input life photos are erased. Different from random erasing, this method fixes the erased area and sets the pixels of the area where the person is located in the life photos to 0. The pixels of the scene parts other than the person remain unchanged. The deep residual network is modified, and the residual network with the last fully connected layer and normalization layer removed is used as the scene feature extraction network in the life photos, named bacbone_S. The image containing only the scene information after the pixels of the person area have been erased is input into bacbone_S to extract the background features of the image, and the background feature map to be tested and the control background feature map are obtained respectively.

[0084] The specific implementation details are: all pixels in the area where the character appears are set to 0, the formula is as follows:

[0085] A m =area(x0,y0,x1-x0,y1-y0)

[0086] Im (A m )=0

[0087] Among them A m Represents the character frame area, which can be calculated by the lower left corner coordinates (x0, y0) and the upper right corner coordinates (x1, y1). m (A m ) indicates the area where the character frame appears in the input life photo, and the pixels in the corresponding area in the photo are set to 0.

[0088] Life Photos I m The background characteristics are calculated according to the following formula:

[0089] f sm =backbone_S(I m ), m=1, ..., M

[0090] where f sm Represents the background features of the photo, and its dimension is O(f sm )∈R 1×1×2048 .

[0091] The embodiment of the present invention obtains background features by extracting features from life photos that only contain background information, and can fully utilize the background information contained in the life photos to achieve the purpose of improving recognition accuracy.

[0092] Further, step S104 includes: based on a preset fusion neural network model, the R to-be-tested person feature maps are respectively fused with the to-be-tested background feature map to obtain O to-be-tested fused person features, and the R control person feature maps are respectively fused with the control background feature map to obtain Q control fused person features, wherein O and Q are both integers greater than 1;

[0093] The to-be-tested fused character features include the to-be-tested character features and the to-be-tested background features, and the ....

[0094] It should be noted that the values ​​of O and Q are the same as those of R, which are the number of people included in the test image and the reference image.

[0095] In the specific embodiments of the present invention, please refer to Figure 2 , the character feature map is fused with the background feature map to obtain the fused character feature containing the character features and the background features. The formula is as follows:

[0096] f mi =[f mxp , f sm ],p=1,...,P

[0097] Since the dimensions of character features and background features are inconsistent, the background features need to be copied first in the fusion process, and then fused with different body regions of the character to obtain the new character features f mi .

[0098] It should be noted that the fusion neural network model used in the embodiment of the present invention adopts a graph neural network model (GNN), in which the role of the graph neural network is to process graph structured data by modeling and composing within the image and / or between images, using the character features of different characters as nodes, and connecting different nodes through edges, so as to model the characters and character relationships contained in the image as graph structured data.

[0099] Further, step S105 includes: using the O fused character features to be tested as nodes to perform composition modeling inside the image to be tested, each node represents a group member, connecting different members in the image group to be tested with edges, connecting different nodes in pairs, and modeling the dependency relationship between members, and constructing a character relationship graph in the image group to be tested;

[0100] The Q reference fusion character features are used as nodes to perform composition modeling inside the reference image, where each node represents a group member, and different members in the reference image group are connected by edges, and different nodes are connected in pairs. At the same time, the dependency relationship between members is modeled to construct a relationship graph within the reference image group;

[0101] The O fused character features to be tested and the Q fused character features to be controlled are used as nodes, a composition model is constructed between the image to be tested and the control image, the nodes of the image to be tested and the control image are connected with edges, and the nodes between different images are connected in pairs to construct an inter-group relationship graph.

[0102] In the specific embodiments of the present invention, please refer to Figure 2 The process of constructing the intra-group relationship graph is as follows: composition modeling is performed inside the life photo. The graph is represented by a set of nodes and edges connecting the nodes. The fused character features are used as nodes. Each node in the graph represents a group member. Edges are used to connect different members in the group in the life photo. At the same time, the dependency relationship between members is modeled to construct the intra-group relationship graph. The relationship between the characters in the group is calculated according to the following formula:

[0103]

[0104] in is the mean of the identity characteristics of other people in the same group as the target to be identified, It is a weight matrix that converts input features into deeper features. The specific method is to accumulate the features of the members in the group, calculate the mean, and then update the features, for a total of T rounds of cycles.

[0105] Further, in the specific embodiments of the present invention, please refer to Figure 3 , Figure 3 The present invention provides Figure 1 In the method flow chart of step S105 of an embodiment, the process of constructing an inter-group relationship graph is as follows: composition modeling is performed between the image to be tested and the control image, the graph is represented by two groups of nodes and edges connecting the nodes, the fused character features of the image to be tested and the control image are used as nodes, each node in the graph represents an inter-group member, and different members in the image to be tested and the control image are connected by edges, and the dependency relationship between the members is modeled to construct an inter-group relationship graph.

[0106] It should be noted that the feature fusion between groups is different from the feature fusion within a group. Since each person instance to be detected must be compared with the person instances in all photos in the reference set, and the similarity with each group member in the reference set is different, it is necessary to use edges with different weights to connect the target to be identified and all group members in the other group. The degree of association between members of two groups is calculated according to the following formula:

[0107]

[0108] in It represents the importance of another group of members in the test image and the control image to the target member, and the importance is proportional to the similarity. Then normalize it:

[0109]

[0110]

[0111] in It is the weighted sum of the features of all members in the group in the test image and the control image of the person instance to be detected, and is used to indicate the degree of association between group members.

[0112] The embodiment of the present invention fuses the character feature map and the background feature map to obtain fused character features through a graph neural network, and constructs an intra-group relationship map and an inter-group relationship map with the fused character features as nodes. By fully considering the appearance features of the characters in the life photos, the features of the scenes where the characters are located, the relationships between characters, and the relationships between groups of people, the visual information contained in the photos is constructed and fused from a multimodal perspective, thereby achieving the purpose of improving the accuracy of character identity recognition.

[0113] Further, step S106 includes: constructing an adjacency matrix through the intra-group character relationships to be tested and the inter-group character relationships, wherein the rows and columns of the matrix represent all members included in the intra-group character relationships to be tested and all members included in the inter-group character relationships, respectively;

[0114] The final character feature map obtained by dimensionally splicing the character relationship within the group to be tested and the character relationship between groups includes the background feature image, the member relationship within the image group to be tested, and the member relationship between groups.

[0115] In the specific embodiment of the present invention, please refer to Figure 4 , Figure 4 This is a schematic diagram of an embodiment of the adjacency matrix provided by the present invention. The graph G = {V, E} can be approximated as a group of nodes and edges connecting the nodes. The node V represents the members of the group in the image to be tested, and the edge E represents the relationship between the members. The graph consists of the query (test) set of photos. Character Examples and reference set photos Character Examples Specifically, the cropped person image is input into the backbone network and the output features are used as graph nodes, and different nodes are connected to each other through edges. Using the adjacency matrix Representative Figure G:

[0116]

[0117] The group level in the photo is determined by the co-occurrence of characters. Each character instance node has the same weight, so the edge connecting the nodes is set to 1. That is, when the group membership in two life photos is the same, the corresponding position value in the adjacency matrix A is 1.

[0118] Furthermore, the final character features after integrating the background feature image, the relationship between members in the image group to be tested, and the relationship between members in the group are calculated according to the following formula:

[0119]

[0120] where f r represents the final appearance features of the characters including their social relationships, and [·] represents the concatenation on the dimensions.

[0121] For further information, see Figure 5 , Figure 5 The present invention provides Figure 1 The flowchart of step S107 in the embodiment of FIG. 1 is as follows, step S107 includes:

[0122] S501, using a cross entropy loss function and a triplet loss function to calculate the matching degree between the person to be identified in the final person feature graph and the identities of other person instances in the reference set, wherein the value of the matching degree is the linear accumulation of the loss values ​​calculated by the cross entropy loss function and the triplet loss function;

[0123] S502, using a pairwise loss function to calculate the similarity of features of all members of the group where the target to be identified is located with the corresponding members in the group in the reference set, wherein the value of the similarity is the loss value calculated by the pairwise loss function;

[0124] S503, linearly adding the loss values ​​calculated by the cross entropy loss function, the triple loss function and the pairwise loss function to obtain a target loss value;

[0125] S504: Use the target loss value to update the parameters of the initial life photo character recognition model until the model converges to obtain the life photo character recognition model.

[0126] The target loss value is ultimately the linear accumulation of the loss values ​​calculated for the P-block body regions respectively, and the reference set is the life photo images of the marked character frames in the sample set except the image under test.

[0127] In a specific embodiment of the present invention, the cross entropy loss and the triplet loss are combined to generate the matching degree between the target to be identified in different photos and the identity of other person instances in the reference set. The value is the linear accumulation of the two loss functions. The cross entropy loss L cre and triplet loss L tri Calculated according to the following formula:

[0128]

[0129]

[0130] in Represents the predicted label value of the pth body region of the i-th person instance in the photo, which can be 0 or 1; represents the true label value of the p-th body region of the i-th person instance in the photo, and its value is 0 or 1; x ip Represents the feature vector of the pth body region of the i-th person instance in the photo; α represents the interval parameter, and its value is fixed to 0.3.

[0131] The pairwise loss is used to generate the matching degree between the group where the target to be identified belongs and other groups between different photos. Its value is the sum of the feature similarity calculation values ​​of all members of the group where the target to be identified belongs and the group members in the control group in the reference set. Calculated according to the following formula:

[0132]

[0133] x ip The feature vector representing the pth body region of the i-th person instance in the photo; Represents the predicted label value of the p-th body region on the i-th person instance and the j-th person instance in the photo, and its value is 0 or 1. The distance between features is measured using the Euclidean distance. + means that when the value in [·] is greater than zero, the value is taken as the loss, and when it is less than zero, the loss is zero; β represents the interval parameter, and its value is fixed at 0.3.

[0134] The loss values ​​calculated by the cross entropy loss function, the triple loss function and the pairwise loss function are linearly accumulated to obtain the target loss value, the target loss L person Calculated according to the following formula, its value is the linear accumulation of all the above loss functions:

[0135]

[0136] The calculated target loss value is used to update the parameters of the initial life photo character recognition model until the model converges to obtain the life photo character recognition model.

[0137] The embodiment of the present invention updates the model parameters through the target loss values ​​calculated by three loss functions, so as to achieve the purpose of constraining model training and improving the model convergence efficiency.

[0138] In order to more intuitively demonstrate the life photo character recognition model training method provided by the present invention, please refer to Figure 6 , Figure 6 A schematic flow chart of another embodiment of the method for training a life photo character recognition model provided by the present invention.

[0139] The method for training a life photo character recognition model provided by an embodiment of the present invention extracts features of the characters and backgrounds in the life photos respectively, and uses a graph neural network to use fused character features obtained by fusing the extracted character features and background features as nodes to perform composition modeling within the image and / or between images, thereby obtaining an intra-group relationship graph, an inter-group relationship graph adjacency matrix, and final character features. Finally, the model parameters are updated by target loss values ​​calculated by three loss functions, so that various modal information contained in the life photos is fully utilized, including character appearance features, character scene features, relationships between characters, and relationships between groups of people. The model parameters are updated by the calculated target loss values, thereby achieving the purpose of improving model training efficiency and character identity recognition accuracy.

[0140] On the other hand, the embodiment of the present invention also provides a method for identifying people in life photos, such as Figure 7 As shown, Figure 7 A schematic flow chart of an embodiment of a method for identifying people in life photos provided by the present invention, the method for identifying people in life photos comprises:

[0141] S701, obtaining a test life photo and a control life photo containing a person to be identified;

[0142] S702, inputting the life photo to be tested and the control life photo containing the person to be identified into the life photo person identification model, and outputting the person identification result;

[0143] Among them, the life photo character recognition model is trained by using the training method in any of the above embodiments, so it will not be described in detail.

[0144] Specifically, in some embodiments of the present invention, in order to verify the recognition accuracy of the life photo character recognition method provided by the embodiment of the present invention, please refer to Table 1, which compares the recognition results of the life photo character recognition method provided by the embodiment of the present invention with the recognition results of other character identity recognition networks:

[0145] Table 1 Comparison of person recognition accuracy

[0146]

[0147] Table 1 compares the recognition results of the life photo character recognition method provided by the embodiment of the present invention with the recognition results of other character identification networks. These methods are mainly divided into two categories: 1. Traditional character identification methods without other modal information, such as Vanilla GNN, Strong-Baseline and PCB; 2. Identification methods that explore multimodal relationship features, such as OSNet, MGR, CG and MACG. The last row MMCC is the life photo character recognition method provided by the embodiment of the present invention.

[0148] The rank-k indicator reflects the degree to which all correct images in the database are ranked at the front of the sorted list, which can more comprehensively measure the performance of the ReID algorithm. The defect of rank-k is that it only considers the probability of the first k hits. However, there are actually multiple correct answers, and the hits of some later samples cannot be evaluated by rank-k. Therefore, it is necessary to supplement the mAP indicator to consider all samples.

[0149] It can be seen from Table 1 above that the recognition accuracy of the life photo character recognition method provided by the embodiment of the present invention is significantly improved compared with the recognition accuracy of other character identity recognition networks, indicating that the recognition accuracy of the life photo character recognition method proposed by the embodiment of the present invention is relatively high.

[0150] A method for identifying people in life photos provided by an embodiment of the present invention constructs and fuses the visual information contained in the photos from a multimodal perspective by fully considering the appearance features of the people in the life photos, the features of the scenes in which the people are located, the relationships between people, and the relationships between groups of people. The method also models the relationships between people through a neural network to improve the accuracy of person identification.

[0151] In order to better implement the life photo character recognition model training method in the embodiment of the present invention, based on the life photo character recognition model training method, the embodiment of the present invention provides a life photo character recognition model training device 800, such as Figure 8 As shown, including:

[0152] An acquisition module 801 is used to acquire a sample set of life photos containing the person to be tested;

[0153] The acquisition module 801 is also used to acquire, according to the life photo sample set, a character feature map to be tested, a character feature map to be controlled, a background feature map to be tested, a background feature map to be controlled, fused character features to be tested, fused character features to be controlled, member relationships within the test group, member relationships within the control group, character relationships between groups, a final character feature map and an adjacency matrix;

[0154] The training module 802 is used to calculate the loss value of the final character feature map and the adjacency matrix using multiple loss functions, accumulate the calculated loss values ​​to obtain the target loss value, and use the target loss value to update the model parameters of the initial life photo character recognition model until the model converges to obtain the life photo character recognition model.

[0155] The life photo character recognition model training device 800 provided in the above embodiment can implement the technical solution described in the above life photo character recognition model training method embodiment. The specific implementation principles of the above modules or units can refer to the corresponding contents in the above life photo character recognition model training method embodiment, which will not be repeated here.

[0156] In order to better implement the method for identifying people in life photos in the embodiment of the present invention, based on the method for identifying people in life photos, the embodiment of the present invention provides a device for identifying people in life photos 900, such as Fig. 9 As shown, including:

[0157] A real-time acquisition module 901 is used to acquire a life photo to be tested and a control life photo containing a person to be identified;

[0158] The real-time acquisition module 901 is also used to obtain a final character feature map and an adjacency matrix through the life photo character recognition model based on the test life photos and the control life photos containing the person to be recognized.

[0159] The determination module is used to determine the character recognition result according to the final character feature map and the adjacency matrix.

[0160] The device 900 for identifying people in life photos provided in the above embodiment can implement the technical solution described in the above embodiment of the method for identifying people in life photos. The specific implementation principles of the above modules or units can refer to the corresponding contents in the above embodiment of the method for identifying people in life photos, which will not be repeated here.

[0161] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the method for training a model for recognizing people in life photos as described above and / or the method for recognizing people in life photos as described above are implemented.

[0162] Generally speaking, the computer instructions for implementing the method of the present invention can be carried by any combination of one or more computer-readable storage media. Non-transitory computer-readable storage media can include any computer-readable media except for the signal itself that is temporarily propagating.

[0163] Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or device.

[0164] Computer program code for performing the operation of the present invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages, in particular, Python suitable for neural network computing and platform frameworks based on TensorFlow, PyTorch, etc. can be used. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0165] The present invention also provides an electronic device. Fig.10 , Fig.10 This is a structural diagram of an embodiment of an electronic device provided by the present invention. The electronic device 1000 includes a processor 1001, a memory 1002, and a computer program stored in the memory 1002 and executable on the processor 1001. When the processor 1001 executes the program, the life photo character recognition model training method described above and / or the life photo character recognition method described above are implemented.

[0166] As a preferred embodiment, the electronic device 1000 further includes a display 1003 for displaying that the processor 1001 executes the above-mentioned life photo character recognition model training method and / or the above-mentioned life photo character recognition method.

[0167] The electronic device 1000 may be a desktop computer, a notebook, a PDA or a smart phone with an adjustable camera module.

[0168] The processor 1001 may be an integrated circuit chip with signal processing capability. The processor 1001 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; or a digital signal processor.

[0169] (Digital Signal Processor, DSP), Application Specific Integrated Circuit

[0170] (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present invention can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0171] The memory 1002 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc. The memory 1002 is used to store programs, and the processor 1001 executes the program after receiving an execution instruction. The process definition method disclosed in any of the embodiments of the present invention may be applied to the processor 1001 or implemented by the processor 1001.

[0172] The display 1003 may be an LCD display screen or an LED display screen, for example, a display screen on a mobile phone.

[0173] Understandably, Fig.10 The structure shown is only a schematic diagram of the structure of the electronic device 1000. The electronic device 1000 may also include Fig.10 More or fewer components as shown. Fig.10 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0174] The computer-readable storage medium and electronic device provided according to the above embodiments of the present invention can be implemented with reference to the content specifically described in the life photo character recognition model training method and / or the life photo character recognition method according to the present invention, and have similar beneficial effects as the life photo character recognition model training method and / or the life photo character recognition method, which will not be repeated here.

[0175] The above is a detailed introduction to a life photo character recognition model training method, a character recognition method and an electronic device provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for training a life photo character recognition model, characterized in that: include: Construct an initial life photo character recognition model based on a deep residual network; Acquire a sample set of life photos of the person to be tested, wherein the sample set includes the image to be tested and a control image; Inputting the image to be tested and the control image into the initial life photo character recognition model, and processing to obtain a feature map of the person to be tested, a feature map of the control person, a feature map of the background to be tested, and a feature map of the control background; Based on a preset fusion neural network model, the test person feature map and the test background feature map are fused to obtain a test fused person feature, and the control person feature map and the control background feature map are fused to obtain a control fused person feature; Taking the fused character features to be tested and the fused character features to be controlled as nodes, determining the character relationships within the tested group between the fused character features to be tested, the character relationships within the control group between the fused character features to be controlled, and the character relationships between the fused character features to be tested and the fused character features to be controlled based on the connection relationships between different nodes; Constructing an adjacency matrix based on the intra-group character relationships to be tested and the inter-group character relationships, and performing dimension splicing on the intra-group character relationships to be tested and the inter-group character relationships to obtain a final character feature map; A variety of loss functions are used to calculate the losses of the final character feature graph and the adjacency matrix, and the model parameters of the initial life photo character recognition model are updated according to the calculated loss values ​​until the model converges to obtain the life photo character recognition model.

2. The method for training a life photo character recognition model according to claim 1, characterized in that: The step of inputting the image to be tested and the reference image into the initial life photo character recognition model to obtain a feature map of the person to be tested and a feature map of the reference person includes: The image of the person in the marked person frame of the image to be tested and the control image is divided into P block areas, and L pooling layers are used to replace the average pooling layer of the last layer of the initial life photo person recognition model based on different body areas of the person, wherein P and L are both integers greater than 1; Perform feature extraction on P blocks of body regions of the person in the test image and the control image respectively, obtain feature vectors of S different body regions respectively, and then splice the feature vectors of the S different body regions to obtain a feature map of the person to be tested and a feature map of the control person respectively, wherein S is an integer greater than 1; Among them, the image to be tested and the reference image both contain K persons with marked person frames, the number of the feature maps of the person to be tested and the number of the feature maps of the reference person are both R, and K and R are both integers greater than 1.

3. The method for training a life photo character recognition model according to claim 1, characterized in that: The step of inputting the image to be tested and the reference image into the initial life photo character recognition model and processing to obtain the background feature map to be tested and the reference background feature map comprises: Erasing the pixels in the marked character frames of the image to be tested and the reference image to obtain the background image to be tested and the reference background image containing only background information; The fully connected layer and the normalized layer in the initial life photo character recognition model are removed, and feature extraction is performed on the background image to be tested and the control background image respectively to obtain a background feature map to be tested and a background feature map to be controlled.

4. The method for training a life photo character recognition model according to claim 1, characterized in that: The method of fusing the subject feature map and the background feature map to be tested based on a preset fusion neural network model to obtain a fused subject feature, and fusing the control subject feature map and the control background feature map to obtain a control fused subject feature, includes: Based on a preset fusion neural network model, R to-be-tested person feature maps are fused with to-be-tested background feature maps to obtain O to-be-tested fused person features, and the R control person feature maps are fused with control background feature maps to obtain Q control fused person features, wherein O and Q are both integers greater than 1; The to-be-tested fused character features include the to-be-tested character features and the to-be-tested background features, and the ....

5. The method for training a life photo character recognition model according to claim 4, characterized in that: Taking the fused character features to be tested and the fused character features to be controlled as nodes, determining the character relationships within the tested group between the fused character features to be tested, the character relationships within the control group between the fused character features to be controlled, and the character relationships between the fused character features to be tested and the control fused character features based on the connection relationships between different nodes, including: The O fused character features to be tested are used as nodes to perform composition modeling inside the image to be tested, each node represents a group member, and different members in the image group to be tested are connected by edges, and different nodes are connected in pairs. At the same time, the dependency relationship between members is modeled to construct a character relationship graph in the image group to be tested; The Q reference fusion character features are used as nodes to perform composition modeling inside the reference image, where each node represents a group member, and different members in the reference image group are connected by edges, and different nodes are connected in pairs. At the same time, the dependency relationship between members is modeled to construct a relationship graph within the reference image group; The O fused character features to be tested and the Q fused character features to be controlled are used as nodes, a composition model is constructed between the image to be tested and the control image, the nodes of the image to be tested and the control image are connected with edges, and the nodes between different images are connected in pairs to construct an inter-group relationship graph.

6. The method for training a life photo character recognition model according to claim 1, characterized in that: The step of constructing an adjacency matrix based on the intra-group character relationships to be tested and the inter-group character relationships, and performing dimension splicing on the intra-group character relationships to be tested and the inter-group character relationships to obtain a final character feature graph includes: Constructing an adjacency matrix through the intra-group character relationships to be tested and the inter-group character relationships, wherein the rows and columns of the matrix respectively represent all members included in the intra-group character relationships to be tested and all members included in the inter-group character relationships; The final character feature map is obtained by dimensionally splicing the character relationship within the group to be tested and the character relationship between groups, and the final character feature map contains character features, background features, and information on the relationship between members within the group and between members.

7. The method for training a life photo character recognition model according to claim 1, characterized in that: The method uses a plurality of loss functions to calculate the loss of the final character feature graph and the adjacency matrix, and updates the model parameters of the initial life photo character recognition model according to the calculated loss value until the model converges to obtain the life photo character recognition model, including: A cross entropy loss function and a triplet loss function are used to calculate the matching degree between the person to be identified in the final person feature graph and the identities of other person instances in the reference set, wherein the value of the matching degree is a linear accumulation of the loss values ​​calculated by the cross entropy loss function and the triplet loss function; A pairwise loss function is used to calculate the similarity of features of all members of the group where the target to be identified is located with the corresponding members of the group in the reference set, and the value of the similarity is the loss value calculated by the pairwise loss function; Linearly add the loss values ​​calculated by the cross entropy loss function, the triplet loss function, and the pairwise loss function to obtain a target loss value; Using the target loss value, the parameters of the initial life photo character recognition model are updated until the model converges to obtain a life photo character recognition model; The target loss value is ultimately the linear accumulation of the loss values ​​calculated for the P-block body regions respectively, and the reference set is the life photo images of the marked character frames in the sample set except the image to be tested.

8. A method for identifying people in daily life photos, characterized in that: include: Obtaining a test life photo and a control life photo containing a person to be identified; Inputting the life photo to be tested and the life photo to be controlled containing the person to be identified into the life photo person identification model, and outputting the person identification result; Wherein, the life photo character recognition model is obtained by training using any one of the training methods of claims 1 to 7 above.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the life photo character recognition model training method according to any one of claims 1 to 7 and the life photo character recognition method according to claim 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for training a model for recognizing a person in a life photo according to any one of claims 1 to 7 and the method for recognizing a person in a life photo according to claim 8 are implemented.

Images