Character relationship identification method and device

Abstract

The invention discloses a character relationship recognition method and device, and the method comprises the steps: obtaining the character features of all character identifiers on a to-be-processed image needing to be processed; performing scene feature extraction on the to-be-processed image to obtain image scene features; and determining an initial character relationship between the character identifiers based on the image scene features and the character features of the character identifiers. According to the method, the image scene feature of the to-be-processed image and the character feature of each character identifier are acquired, and character relationship recognition is performed by means of richer information such as the character features and the scene, so that the recognition accuracy of the character relationship can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and specifically to a method and device for identifying person relationships. Background Art

[0002] With the development of deep learning, especially the popularization of face recognition algorithms, image understanding and album management based on face information have become an indispensable and important function in mobile phones. Existing human relationship recognition technology extracts features based on clues such as facial attributes, and estimates intimacy and relationships based on rule judgment. This method does not take into account information such as the scene in the image, resulting in a low recognition accuracy of human relationships.

[0003] That is, the recognition accuracy of character relationships in the prior art is low. Summary of the invention

[0004] The embodiments of the present application provide a method and device for identifying relationships between people, which can improve the accuracy of identifying relationships between people.

[0005] In a first aspect, the present application provides a method for identifying a relationship between characters, including:

[0006] Obtaining the character features of each character identification on the image to be processed;

[0007] Extracting scene features from the image to be processed to obtain image scene features;

[0008] An initial character relationship between the character identifiers is determined based on the image scene feature and the character features of the character identifiers.

[0009] In a second aspect, the present application provides a device for identifying a person relationship, including:

[0010] An acquisition module is used to acquire the character features of each character identification on the image to be processed;

[0011] A feature extraction module is used to extract scene features from the image to be processed to obtain image scene features;

[0012] The relationship determination module is used to determine the initial character relationship between each character identification based on the image scene feature and the character feature of each character identification.

[0013] On the third aspect, the electronic device provided in the present application includes a memory and a processor, the memory stores a computer program, and the processor is used to run the computer program in the memory to implement the steps in the method for identifying character relationships provided in the present application.

[0014] In a fourth aspect, the computer-readable storage medium provided in the present application stores a plurality of instructions, which are suitable for loading by a processor to implement the steps in the method for identifying character relationships provided in the present application.

[0015] In a fifth aspect, the computer program product provided in the present application includes a computer program or instructions, which, when executed by a processor, implement the steps in the method for identifying character relationships provided in the present application.

[0016] Compared with the related art, in this application, the character features of each character identification on the image to be processed are obtained; scene features of the image to be processed are extracted to obtain image scene features; and the initial character relationships between the character identifications are determined based on the image scene features and the character features of each character identification. This application obtains the image scene features of the image to be processed and the character features of each character identification, and then uses richer information such as character features and scenes to perform character relationship recognition, which can improve the recognition accuracy of character relationships. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 It is a scene schematic diagram of a system for identifying character relationships provided in an embodiment of the present application;

[0019] Figure 2 It is a flowchart of an embodiment of a method for identifying a person relationship provided in an embodiment of the present application;

[0020] Figure 3 It is a schematic diagram of a flow chart of obtaining the character features of each character identification on an image to be processed that needs to be processed in an embodiment of a method for identifying character relationships provided in an embodiment of the present application;

[0021] Figure 4 It is a schematic diagram of an information processing process of obtaining the character features of each character identification on an image to be processed in an embodiment of a method for identifying character relationships provided by an embodiment of the present application;

[0022] Figure 5 It is a flow chart of an embodiment of a method for identifying character relationships provided by an embodiment of the present application, in which a character relationship extraction model is used to predict character relationships based on image scene features and character features of each character identifier to obtain an initial character relationship between each character identifier;

[0023] Figure 6 It is a flowchart of obtaining a pre-trained character relationship extraction model in one embodiment of a character relationship recognition method provided in an embodiment of the present application;

[0024] Figure 7 is a flowchart of another embodiment of the method for identifying character relationships provided in an embodiment of the present application;

[0025] Figure 8 It is a schematic diagram of the framework of the recognition model of the character relationship of this application;

[0026] Fig. 9 It is a structural schematic diagram of a device for identifying a person relationship provided in an embodiment of the present application;

[0027] Fig.10 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] It should be noted that the principles of the present application are illustrated by implementing them in an appropriate computing environment. The following description is based on the illustrated specific embodiments of the present application and should not be considered as limiting other specific embodiments of the present application that are not described in detail herein.

[0029] In the following description of the present application, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0030] In the following description of the present application, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0032] In order to improve the effect of character relationship recognition, the embodiments of the present application provide a character relationship recognition method, a character relationship recognition device, an electronic device, a computer-readable storage medium, and a computer program product. The character relationship recognition method can be executed by the character relationship recognition device, or by an electronic device integrated with the character relationship recognition device.

[0033] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0034] Please refer to Figure 1 , the present application also provides a character relationship recognition system, such as Figure 1 As shown, the character relationship recognition system includes an electronic device 100, in which the character relationship recognition device provided by the present application is integrated.

[0035] Among them, the electronic device 100 can be any device equipped with a processor and has processing capabilities, such as mobile electronic devices with processors such as smart phones, tablet computers, PDAs, laptops, smart speakers, or fixed electronic devices with processors such as desktop computers, televisions, servers, industrial equipment, etc.

[0036] In addition, if Figure 1 As shown, the character relationship recognition system may further include a memory 200 for storing original data, intermediate data, and result data.

[0037] In the embodiment of the present application, the memory 200 can be a cloud storage. Cloud storage is a new concept extended and developed from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as the storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of different types of storage devices (storage devices are also called storage nodes) in the network through application software or application interfaces to work together and provide external data storage and business access functions.

[0038] At present, the storage method of the storage system is: create a logical volume, and when creating a logical volume, allocate physical storage space for each logical volume. The physical storage space may be composed of disks of a storage device or several storage devices. The client stores data on a logical volume, that is, stores the data on the file system. The file system divides the data into many parts, each of which is an object. The object contains not only data but also additional information such as data identification (ID, ID entity). The file system writes each object into the physical storage space of the logical volume, and the file system records the storage location information of each object, so that when the client requests to access the data, the file system can allow the client to access the data according to the storage location information of each object.

[0039] The process of the storage system allocating physical storage space to a logical volume is as follows: based on the estimated capacity of the objects stored in the logical volume (this estimate often has a large margin relative to the actual capacity of the objects to be stored) and the grouping of independent redundant disk arrays (RAID, Redundant Array of Independent Disks), the physical storage space is pre-divided into stripes. A logical volume can be understood as a stripe, thereby allocating physical storage space to the logical volume.

[0040] It should be noted that Figure 1 The scenario diagram of the character relationship recognition system shown is merely an example. The character relationship recognition system and scenario described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. A person of ordinary skill in the art can appreciate that with the evolution of the character relationship recognition system and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is equally applicable to similar technical problems.

[0041] It should be noted that the serial numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

[0042] Please refer to Figure 2 , Figure 2 FIG. 1 is a flow chart of an embodiment of a method for identifying a person relationship provided in an embodiment of the present application. Figure 2 As shown, the process of the method for identifying character relationships provided by this application is as follows:

[0043] 201. Obtaining the character features of each character identification on the to-be-processed image to be processed.

[0044] The image to be processed may be an image uploaded by the user, or an image extracted from the user's photo album with the user's authorization. The image to be processed may also be a frame image obtained by parsing a video uploaded by the user.

[0045] The person identification is the identification of the same person on the image to be processed, which may be a name or other forms of identification such as ID (Identity document). For example, there are three person identifications on the image to be processed, namely ID1, ID2 and ID3.

[0046] The character features may include at least one of a face detection frame and a human body detection frame. The face detection frame and the human body detection frame may be manually annotated in advance or may be predicted by various detection models, which are not limited here.

[0047] 202. Perform scene feature extraction on the image to be processed to obtain image scene features.

[0048] In a specific embodiment, the scene features of the image to be processed may be extracted by using an existing feature extraction model to obtain the image scene features.

[0049] In another specific embodiment, scene feature extraction is performed on the image to be processed to obtain image scene features, including: obtaining a pre-trained character relationship extraction model; and using the character relationship extraction model to perform scene feature extraction on the image to be processed to obtain image scene features. The character relationship extraction model can complete feature extraction and prediction.

[0050] In a specific embodiment, the character relationship extraction model includes a second convolution module, which uses the second convolution module to extract features of the first image to obtain second image features; and pools the second image features to obtain image scene features. The second convolution module can be a CNN or other types of convolutional neural networks. Specifically, the character relationship extraction model includes a global average pooling layer, which uses the global average pooling layer to pool the second image features to obtain image scene features.

[0051] It should be noted that artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0052] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes machine learning (ML) technology, among which deep learning (DL) is a new research direction in machine learning. It is introduced into machine learning to make it closer to the original goal, namely artificial intelligence. At present, deep learning is mainly used in the fields of machine vision and natural speech processing.

[0053] Deep learning is the process of learning the inherent laws and representation levels of sample data. The information obtained in the learning process is very helpful for interpreting data such as text, images, and sound. Using deep learning technology and corresponding sample sets, network models that can achieve different functions can be trained.

[0054] 203. Determine an initial character relationship between the character identifiers based on the image scene feature and the character features of the character identifiers.

[0055] In a specific implementation, a character relationship extraction model is used to predict character relationships based on image scene features and character features of each character identifier, and the initial character relationships between each character identifier are obtained. Of course, in other implementations, other neural network models such as CNN and RNN can also be used to determine the initial character relationships between each character identifier based on image scene features and character features of each character identifier.

[0056] For example, there are three character identifiers on the image to be processed, and the initial character relationship between each character identifier can be: the initial character relationship between character identifier ID1 and character identifier ID2 is father and son; the initial character relationship between character identifier ID1 and character identifier ID3 is mother and son; the initial character relationship between character identifier ID2 and character identifier ID3 is husband and wife. Of course, the initial character relationship between each character identifier can also be: the initial character relationship between character identifier ID1 and character identifier ID2 is father and son, and character identifier ID1 is the son of character identifier ID2, and character identifier ID2 is the father of character identifier ID1. The specific method of the initial character relationship can be based on the specific setting and is not limited here.

[0057] Furthermore, in order to improve the accuracy of character relationship recognition, the character relationship recognition method may also include: obtaining the initial character relationships between each character identification on multiple images to be processed; calculating multiple different initial character relationships between two character identifications based on the initial character relationships between each character identification on multiple images to be processed; determining the initial character relationship with the highest frequency among the multiple different initial character relationships between the two character identifications as the target character relationship between the two character identifications, and obtaining the target character relationship between each character identification.

[0058] Since the faces of the same person identification in different images may be different, in order to improve the accuracy of relationship recognition, multiple person features on multiple images to be processed are clustered to obtain multiple person feature clustering clusters, and the person identifications of the face features in the same person feature clustering cluster are determined to be the same person identification. Specifically, multiple face detection frames on multiple images to be processed are clustered to obtain multiple face clustering clusters, and the person identifications in the same face clustering cluster are determined to be the same person identification.

[0059] Among them, the multiple images to be processed can be multiple images uploaded by users, or multiple images obtained by parsing videos uploaded by users. For each image to be processed, the initial character relationship between each character identification on the image to be processed can be obtained according to steps 201-203. Since each image to be processed will output the initial character relationship between two character identifications, multiple images to be processed will output multiple initial character relationships between two character identifications. The initial character relationship with the highest frequency of occurrence of multiple initial character relationships between two character identifications in multiple images to be processed has the highest probability of being the true relationship between the two character identifications. The initial character relationship with the highest frequency of occurrence of multiple initial character relationships between two character identifications in multiple images to be processed is determined as the target initial character relationship of the two character identifications, thereby obtaining the target character relationship between each character identification. By voting on the initial character relationships on multiple images to be processed, the accuracy of character relationship determination can be improved.

[0060] See also Figure 3 and Figure 4 In order to automatically and accurately obtain a face detection frame and a corresponding human body detection frame, in a specific embodiment, obtaining the character features of each character identification on the image to be processed may include:

[0061] 301. Perform face detection on the image to be processed to obtain face detection frames of various character identifiers.

[0062] In the embodiment of the present application, a pre-trained face detection model is used to perform face detection on the image to be processed to obtain a face detection frame for each character identification.

[0063] like Figure 4 As shown, the pre-trained face detection model is used to perform face detection on the image to be processed, and two face detection frames of two person identifiers are obtained, namely, the face detection frame of person identifier ID1 and the face detection frame of person identifier ID2.

[0064] 302. Perform human body detection on the image to be processed to obtain human body detection frames of various person identifications.

[0065] In the embodiment of the present application, a pre-trained human body detection model is used to perform human body detection on the image to be processed to obtain a human body detection frame of each person identification.

[0066] like Figure 4 As shown, a pre-trained human detection model is used to perform human detection on the image to be processed, and two human detection frames of two person identifiers are obtained, namely, a human detection frame BBox1 of the person identifier ID1 and a human detection frame BBox2 of the person identifier ID2.

[0067] 303. Match the face detection frame of each character identification with the human body detection frame of each character identification to obtain the face detection frame of each character identification and the corresponding human body detection frame.

[0068] In a specific embodiment, matching the face detection frame of each character identification with the human body detection frame of each character identification to obtain the face detection frame of each character identification and the corresponding human body detection frame includes:

[0069] (1) Calculate the overlap index and / or upper frame distance index of the face detection frame and the body detection frame.

[0070] Among them, the overlap index is the ratio of the overlapping area of ​​the face detection frame and the body detection frame to the area of ​​the face detection frame. Of course, the overlap index can also be other indicators such as the intersection of union (IoU). In the object detection task, the intersection of union (IoU) is usually used as a measurement indicator to measure the relationship between two rectangular frames. This concept is generally used to describe the overlap between two frames. The two frames can be regarded as a set of two pixels, and their intersection of union (IoU) is equal to the area of ​​the overlapping part of the two frames divided by their combined area.

[0071] The upper border distance indicator is the distance between the center of the upper border of the face detection frame and the center of the upper border of the body detection frame. Of course, the upper border distance indicator can also be the vertical distance between the upper border of the face detection frame and the upper border of the body detection frame, which can be set according to the specific situation.

[0072] (2) Determine the degree of match between the face detection frame and the body detection frame based on the overlap index and / or the upper border distance index, and obtain the degree of match between each face detection frame and the body detection frame.

[0073] In a specific embodiment, the ratio of the overlap index to the upper border distance index is determined as the matching degree between the face detection frame and the body detection frame, and the matching degree between each face detection frame and the body detection frame is obtained.

[0074] (3) The human body detection frame with the highest matching degree with the face detection frame is determined as the human body detection frame corresponding to the face detection frame.

[0075] Since the human body detection frame of the same person identification must surround the face detection frame, and its face detection frame is often close to the top edge center of the human body detection frame, maximizing the matching degree will mean that the larger the overlap index and the smaller the upper border distance index, the more a set of face detection frames and human body detection frames should be bound to each other.

[0076] like Figure 4As shown, the face detection frame of person ID1 is matched with the human body detection frame BBox1 of person ID1 and the human body detection frame BBox2 of person ID2, and the face detection frame of person ID2 is matched with the human body detection frame BBox1 of person ID1 and the human body detection frame BBox2 of person ID2. Finally, by comparing the matching degree, the face detection frame of person ID1 is matched with the human body detection frame BBox1 of person ID1, and the face detection frame of person ID2 is matched with the human body detection frame BBox2 of person ID2.

[0077] For further information, see Figure 5 , Figure 5 This is a flow chart of an embodiment of a method for identifying character relationships provided by an embodiment of the present application, in which a character relationship extraction model is used to predict character relationships based on image scene features and character features of each character identifier to obtain an initial character relationship between each character identifier. In a specific embodiment, the character relationship extraction model includes a first convolution module, and a character relationship extraction model is used to predict character relationships based on image scene features and character features of each character identifier to obtain an initial character relationship between each character identifier, including the following steps:

[0078] 401. Use a first convolution module to perform feature extraction on the image to be processed to obtain a first image feature.

[0079] The first convolution module may be a CNN or other types of convolutional neural networks.

[0080] In a specific embodiment, in order to improve the feature extraction effect, the character relationship extraction model includes a multi-scale feature extraction module, and uses a first convolution module to perform feature extraction on the processed image to obtain a first image feature, including: using the multi-scale feature extraction module to perform multi-scale feature extraction on the processed image to obtain multi-scale features, and using the first convolution module to perform feature extraction on the multi-scale features to obtain the first image feature.

[0081] Multi-scale feature maps are a technique used in computer vision and image processing tasks that can extract useful feature information from different image scales. These scales usually refer to the size or distance of objects in the image. Multi-scale feature maps are used for several reasons: For different objects in the image, their sizes may be different. When using traditional convolutional neural networks, different sizes of convolution operations are required for objects of different sizes. Using multi-scale feature maps, features can be extracted at different scales to process objects of different sizes. For some tasks, it is necessary to pay attention to features of different scales in the image at the same time. For example, in object detection, objects of different sizes need to be detected at the same time. Using multi-scale feature maps, features can be extracted at different scales and combined for classification and detection. For some tasks, images need to be processed at different scales. For example, in image segmentation, images at different scales need to be segmented. Using multi-scale feature maps, features can be extracted at different scales and combined to achieve image segmentation.

[0082] In a specific embodiment, a multi-scale feature extraction module is used to perform multiple convolutions and multiple downsamplings on the image to be processed to obtain multiple feature maps of different sizes, and multiple feature maps of different sizes are fused to obtain multi-scale features. For example, the image to be processed is convolved and downsampled by 2 times to obtain a 256*256 feature map, and the image to be processed is convolved and downsampled by 4 times to obtain a 128*128 feature map. The 256*256 feature map and the 128*128 feature map are fused to obtain multi-scale features. Multi-scale features include features of multiple scales of the image to be processed.

[0083] Among them, the multi-scale feature extraction module can be an image pyramid, a U-Net network, etc., which can be selected according to the specific situation. The image pyramid scales the image at different scales to obtain an image pyramid, and then extracts features of different scales for each layer of the image to obtain a feature map. The pyramid of an image is a set of images arranged in a pyramid shape with gradually decreasing resolutions and derived from the same original image. It is obtained by stepwise down sampling until a certain termination condition is reached. We compare the layers of images to a pyramid. The higher the level, the smaller the image and the lower the resolution.

[0084] 402. Resample the first image features based on the character features of each character identification to obtain resampled image features of each character identification.

[0085] In the embodiment of the present application, the character features include a face detection frame and a body detection frame of the character identification.

[0086] Specifically, the first image feature is resampled for the part located in the face detection frame and the body detection frame of the person identification to obtain the resampled image feature. The resampled image feature includes the resampled feature of the feature in the face detection frame and the resampled feature of the feature in the body detection frame.

[0087] 403. Initialize the graph based on the resampled image features and image scene features of each character identifier to obtain an initialized graph.

[0088] In a specific embodiment, the graph is initialized based on the resampled image features and image scene features of each character identifier to obtain an initialized graph, including: determining each node feature corresponding to each character identifier based on the resampled image features of each character identifier; determining two character identifiers as a character identifier group; and determining the edge features of each character identifier group based on the image scene features. The graph is initialized based on each node feature and each edge feature to obtain an initialized graph.

[0089] Specifically, the number of character identifiers is N, where N is a positive integer. N node features corresponding to the N character identifiers are obtained. The node features include features for resampling features within the face detection frame and features for resampling features within the body detection frame. Two character identifiers are determined as a character identifier group, that is, a character identifier group contains two character identifiers, and the character relationship between the two character identifiers is an edge feature between the two character identifiers, that is, the edge feature of the character identifier group represents the character relationship between the two character identifiers in the character identifier group. N character identifiers are combined in pairs to obtain N*N character identifier groups, each character identifier group corresponds to an edge feature, and thus N*N edge features representing the character relationship between the N character identifiers are obtained. For example, the character identifier group contains ID1 and ID2, and the character relationship between ID1 and ID2 in the character identifier group is "father and son", then the edge feature corresponding to the character identifier group is "father and son". The image scene feature is a 1*N vector, and the 1*N image scene features are respectively determined as the edge features of each character identifier group, that is, the edge features of each character identifier group are 1*N image scene features. The initialization graph is a graph consisting of multiple nodes and multiple edges. The node features of each node of the initialization graph are the resampled image features of each character identifier; the edge features of each edge of the initialization graph are the edge features of each character identifier group. For example, if the number of character identifiers is N, then N node features and N*N edge features are obtained.

[0090] 404. Perform character relationship prediction based on the initialization graph to obtain initial character relationships between various character identifiers.

[0091] In a specific embodiment, the initialization graph is input into a preset graph convolutional neural network for graph convolution to obtain the initial character relationships between the various character identifiers.

[0092] Graph Convolution Neural Networks (GCNNs) methods are divided into two categories, spectral domain-based methods and spatial domain-based methods. Spectral domain-based methods define graph convolution by introducing filters from the perspective of graph signal processing, where the graph convolution operation is interpreted as removing noise from the graph signal. Spatial domain-based methods represent graph convolution as aggregating feature information from neighbors.

[0093] In another specific embodiment, the character relationship extraction model includes a semantic feature encoding module and a semantic feature decoding module. Among them, the semantic feature encoding module and the semantic feature decoding module can be the encoder and decoder of Transformer, or the encoder and decoder of other models, which are not limited here. Transformer is a typical Seq2Seq model. The core idea of ​​the Seq2Seq model is to encode a sequence (such as text) into a vector representation of a fixed size, and then decode it into another sequence. This model usually includes an encoder and a decoder, which are responsible for encoding the input sequence into a hidden representation and decoding the hidden representation into an output sequence, respectively.

[0094] Among them, the semantic feature decoding module is a semantic feature decoding module based on the attention mechanism. The attention mechanism can be described as a process of mapping a set of query vectors Q, key vectors K, and value vectors V to an output, where the query vectors Q, key vectors K, value vectors V, and the output are all vectors. The output is calculated as the weighted sum of the value vectors V, where the weight assigned to each value is calculated by a function of the query vector Q and the corresponding key vector K.

[0095] Correspondingly, the graph is initialized based on the resampled image features and image scene features of each character identifier to obtain an initialized graph, including:

[0096] (1) Use the semantic feature encoding module to encode the image to be processed to obtain semantic encoding features.

[0097] (2) Perform graph convolution on the initialization graph to obtain the output matrix.

[0098] Specifically, a preset graph convolution neural network is used to perform graph convolution on the initialization graph to obtain an output matrix. For example, the initialization graph includes N node features and N*N edge features, and the output matrix includes N node features and N*N edge features.

[0099] (3) The output matrix and semantic encoding features are input into the semantic feature decoding module to obtain the initial character relationships between the various character identifiers.

[0100] In a specific embodiment, after transforming the output matrix using the first weight matrix WQ, the obtained vector is input into the semantic feature decoding module as the Q vector of the semantic feature decoding module; after transforming the semantic coding feature using the second weight matrix WK, the obtained vector is input into the semantic feature decoding module as the K vector of the semantic feature decoding module; after transforming the semantic coding feature using the third weight matrix WV, the obtained vector is input into the semantic feature decoding module as the V vector of the semantic feature decoding module; and the initial character relationship between each character identification output by the semantic feature decoding module is obtained. For example, the output matrix includes N node features and N*N edge features, and the output matrix and the semantic coding features are input into the semantic feature decoding module to obtain new N*N edge features, and the new N*N edge features represent the initial character relationship between each character identification, thereby obtaining the initial character relationship between each character identification.

[0101] Among them, when the model is trained for the first time, the first weight matrix WQ, the second weight matrix WK and the third weight matrix WV are randomly generated and updated as the model is trained.

[0102] For further information, see Figure 6 , Figure 6 The present invention is a flowchart of obtaining a pre-trained character relationship extraction model in an embodiment of the character relationship recognition method provided in an embodiment of the present application. In order to improve the prediction accuracy of the character relationship extraction model, the pre-trained character relationship extraction model is obtained, including:

[0103] 501. Obtain sample images and corresponding annotation relationship labels.

[0104] Specifically, the annotated relationship labels are the annotated character relationships of various character identifiers on the sample images, which are pre-annotated manually or by a machine.

[0105] 502. Input the sample image into the person relationship extraction model to obtain the predicted value of each relationship category on the sample image.

[0106] Among them, the predicted value of each relationship category on the sample image is the probability that each character identifier in the sample image belongs to each relationship category.

[0107] 503. Perform weighted summation of the annotated relationship labels of the sample image and the predicted values ​​of each relationship category on the sample image based on the first preset weight and the second preset weight to obtain a first weighted summation value of each relationship category on the sample image.

[0108] In a specific embodiment, the sum of the first preset weight and the second preset weight is 1, for example, the first preset weight is β, and the second preset weight is 1-β. The calculation formula of the first weighted sum value is as follows:

[0109] βtk +(1-β)q k

[0110] Among them, t k is the annotation relation label of the sample image, q k is the predicted value of the kth relationship category on the sample image, where k represents the relationship category.

[0111] 504. Determine a target loss value based on a first weighted sum value of each relationship category on the sample image.

[0112] In a specific embodiment, the calculation formula of the target loss value L(q,t) is as follows:

[0113]

[0114] Among them, t k is the annotation relation label of the sample image, q k is the predicted value of the kth relationship category on the sample image, k represents the relationship category, β is the first preset weight, and 1-β is the second preset weight.

[0115] By weighting the annotation relationship label of the sample image and the predicted value of the sample image by the first preset weight β and the second preset weight 1-β, noise data in the annotation can be smoothed to avoid overfitting.

[0116] In another specific embodiment, a target loss value is determined based on a first weighted sum value of each relationship category on a sample image, including: weighting the first weighted sum value of each relationship category based on a third preset weight of each relationship category to obtain a second weighted sum value of each relationship category, wherein the third preset weight of each relationship category decreases as the proportion of annotated relationship labels corresponding to each category increases; and determining the target loss value based on the second weighted sum value of each relationship category.

[0117] Specifically, the calculation formula for the second weighted sum value is as follows:

[0118] α k [βt k +(1-β)q k ](1-β)

[0119] Among them, t k is the annotation relation label of the sample image, q k is the predicted value of the kth relationship category on the sample image, k represents the relationship category, α k is the third preset weight of the kth relationship category.

[0120] Specifically, the third preset weight of each relationship category is the inverse of the proportion of the annotated relationship tags of each relationship category. For example, there are 10 annotated relationship tags on the sample image, of which 3 are parent-child relationship tags, and the proportion of the annotated relationship tags of the parent-child relationship category is 0.3.

[0121] The calculation formula of the target loss value L(q,t) is as follows:

[0122]

[0123] Among them, t k is the annotation relation label of the sample image, q k is the predicted value of the kth relationship category on the sample image, k represents the relationship category, α k is the third preset weight of the kth relationship category.

[0124] By weighting the data of each category using the inverse of the proportion of annotated relationship labels of each relationship category, category balance can be achieved and overfitting can be avoided.

[0125] 505. Iteratively train the character relationship extraction model based on the target loss value until a preset stopping condition is met.

[0126] Among them, the preset stopping condition can be that the number of iterations reaches a preset number or the target loss value is less than a preset value, etc., which can be set according to the specific situation.

[0127] See also Figure 7 and Figure 8 , Figure 7 is a flow chart of another embodiment of the method for identifying character relationships provided in an embodiment of the present application, Figure 8 : is a schematic diagram of the framework of the character relationship recognition model of the present application. The character relationship recognition method provided in the embodiment of the present application includes the following steps:

[0128] 601. Obtain an image to be processed.

[0129] The image to be processed may be an image uploaded by the user, or an image extracted from the user's photo album with the user's authorization. The image to be processed may also be a frame image obtained by parsing a video uploaded by the user.

[0130] 602. Perform face detection on the image to be processed to obtain face detection frames for each person identification.

[0131] like Figure 8 As shown, the image to be processed is input into the face and human body detection and matching module for face detection to obtain face detection frames of N person identifications.

[0132] 603. Perform human body detection on the image to be processed to obtain human body detection frames of various person identifiers.

[0133] like Figure 8 As shown, the image to be processed is input into the face and body detection and matching module for body detection to obtain body detection frames of N person identifiers.

[0134] 604 . Match the face detection frame of each character identification with the human body detection frame of each character identification to obtain the face detection frame of each character identification and the corresponding human body detection frame.

[0135] In a specific embodiment, matching the face detection frame of each character identification with the human body detection frame of each character identification to obtain the face detection frame of each character identification and the corresponding human body detection frame includes:

[0136] (1) Calculate the overlap index and upper frame distance index of the face detection frame and the body detection frame.

[0137] Among them, the overlap index is the ratio of the overlapping area of ​​the face detection frame and the body detection frame to the area of ​​the face detection frame. Of course, the overlap index can also be other indicators such as the intersection of union (IoU). In the object detection task, the intersection of union (IoU) is usually used as a measurement indicator to measure the relationship between two rectangular frames. This concept is generally used to describe the overlap between two frames. The two frames can be regarded as a set of two pixels, and their intersection of union (IoU) is equal to the area of ​​the overlapping part of the two frames divided by their combined area.

[0138] The upper border distance indicator is the distance between the center of the upper border of the face detection frame and the center of the upper border of the body detection frame. Of course, the upper border distance indicator can also be the vertical distance between the upper border of the face detection frame and the upper border of the body detection frame, which can be set according to the specific situation.

[0139] (2) Based on the overlap index and the upper border distance index, the matching degree between the face detection frame and the human body detection frame is determined to obtain the matching degree between each face detection frame and the human body detection frame.

[0140] In a specific embodiment, the ratio of the overlap index to the upper border distance index is determined as the matching degree between the face detection frame and the body detection frame, and the matching degree between each face detection frame and the body detection frame is obtained.

[0141] (3) The human body detection frame with the highest matching degree with the face detection frame is determined as the human body detection frame corresponding to the face detection frame.

[0142] like Figure 8As shown, the human body detection frame with the highest matching degree with the face detection frame is determined as the human body detection frame corresponding to the face detection frame, and the face detection frames of N person labels and the human body detection frames of N person labels are obtained. The face detection frames of the N person labels and the human body detection frames of the N person labels correspond one to one.

[0143] 605. Perform multi-scale feature extraction on the image to be processed to obtain multi-scale features.

[0144] like Figure 8 As shown, the image to be processed is input into the multi-scale feature extraction module, and the multi-scale feature extraction module is used to extract multi-scale features from the image to be processed to obtain multi-scale features.

[0145] 606. Use the first convolution module to extract the multi-scale features to obtain the first image features.

[0146] The first convolution module may be a CNN or other types of convolutional neural networks.

[0147] like Figure 8 As shown, the multi-scale features are input into the first convolution module of the character relationship extraction model to obtain the first image features.

[0148] 607 . Perform feature resampling on the portion of the first image feature located within the face detection frame and the body detection frame of each character identifier to obtain a resampled image feature of each character identifier.

[0149] like Figure 8 As shown, the resampling module of the character relationship extraction model is used to resample the first image feature located in the face detection frame and the human body detection frame of the N character identifiers to obtain the resampled image features of the N character identifiers. The N resampled image features include the features resampled from the features in the N face detection frames and the features resampled from the features in the N human body detection frames.

[0150] 608. Use a second convolution module to extract features of the first image to obtain second image features.

[0151] Among them, the character relationship extraction model includes a second convolution module, and the second convolution module can be CNN or other types of convolutional neural networks.

[0152] like Figure 8 As shown, the first image feature is input into the second convolution module of the character relationship extraction model for feature extraction to obtain the second image feature.

[0153] 609. Pool the second image features to obtain image scene features.

[0154] like Figure 8As shown, the character relationship extraction model includes a global average pooling layer, which uses the global average pooling layer to pool the second image feature to obtain the image scene feature. The image scene feature is a 1*N vector.

[0155] 610. Initialize the graph based on the resampled image features and image scene features of each character identifier to obtain an initialized graph.

[0156] like Figure 8 As shown, the graph is initialized based on the resampled image features and image scene features of each character identifier to obtain an initialized graph, including:

[0157] (1) Determine each node feature of each character identifier based on the resampled image features of each character identifier.

[0158] Specifically, N resampled image features are used as N node features.

[0159] (2) The two character identifiers are determined as a character identifier group.

[0160] (3) Determine the edge features of each character identification group based on the image scene features.

[0161] Specifically, the number of character identifications is N. The image scene feature is a 1*N vector, and the 1*N image scene features are respectively determined as the edge features of each character identification group, that is, the edge features of each character identification group are all 1*N image scene features.

[0162] (4) Initialize the graph based on the features of each node and each edge to obtain an initialized graph.

[0163] The initialization graph is a graph consisting of multiple nodes and multiple edges. The node features of each node of the initialization graph are the resampled image features of each character identifier; the edge features of each edge of the initialization graph are the edge features of each character identifier group. For example, if the number of character identifiers is N, then N node features and N*N edge features are obtained.

[0164] 611. Encode the second image feature using a semantic feature encoding module to obtain a semantic encoding feature.

[0165] Among them, the semantic feature encoding module can be the encoder of Transformer

[0166] 612. Use the preset graph convolutional neural network to perform graph convolution on the initialized graph to obtain the output matrix.

[0167] Specifically, the initialization graph includes N node features and N*N edge features, and the output matrix includes N node features and N*N edge features.

[0168] 613. Input the output matrix and the semantic encoding features into a semantic feature decoding module to obtain the initial character relationships between the various character identifiers.

[0169] Among them, the semantic feature decoding module is a semantic feature decoding module based on the attention mechanism.

[0170] In a specific embodiment, after transforming the output matrix using the first weight matrix WQ, the obtained vector is input into the semantic feature decoding module as the Q vector of the semantic feature decoding module; after transforming the semantic coding feature using the second weight matrix WK, the obtained vector is input into the semantic feature decoding module as the K vector of the semantic feature decoding module; after transforming the semantic coding feature using the third weight matrix WV, the obtained vector is input into the semantic feature decoding module as the V vector of the semantic feature decoding module; and the initial character relationship between each character identification output by the semantic feature decoding module is obtained. For example, the output matrix includes N node features and N*N edge features, and the output matrix and the semantic coding features are input into the semantic feature decoding module to obtain new N*N edge features, and the new N*N edge features represent the initial character relationship between each character identification, thereby obtaining the initial character relationship between each character identification.

[0171] 614. Obtain initial character relationships between character identifiers of multiple images to be processed.

[0172] Specifically, a plurality of images to be processed are respectively input into the character relationship extraction model to obtain the initial character relationship between each character identification of each image to be processed.

[0173] 615. Calculate multiple different initial character relationships between two character identifiers according to the initial character relationships between the character identifiers on the multiple images to be processed.

[0174] For example, the multiple images to be processed are three images to be processed, in which the initial character relationship between character identifier ID1 and character identifier ID2 in two of the images to be processed is father and son, and in the other image to be processed, the initial character relationship between character identifier ID1 and character identifier ID2 is brothers.

[0175] 616. Determine the initial character relationship with the highest occurrence frequency among the multiple different initial character relationships of the two character identifiers as the target character relationship between the two character identifiers, and obtain the target character relationship between the various character identifiers.

[0176] like Figure 8 As shown, specifically, the initial character relationships between the character identifiers of the plurality of images to be processed are input into the voting module to obtain the target character relationships between the character identifiers.

[0177] For example, the multiple images to be processed are three images to be processed, in which the initial character relationship between character identifier ID1 and character identifier ID2 on two of the images to be processed is father and son, and in the other image to be processed, the initial character relationship between character identifier ID1 and character identifier ID2 is brothers. If the initial character relationship between character identifier ID1 and character identifier ID2 is father and son, which occurs most frequently, then the initial character relationship between character identifier ID1 and character identifier ID2 is father and son.

[0178] In order to facilitate better implementation of the character relationship recognition method provided in the embodiment of the present application, the embodiment of the present application also provides a character relationship recognition device based on the above character relationship recognition method. The meaning of the nouns is the same as that in the above character relationship recognition method. For specific implementation details, please refer to the description in the above method embodiment.

[0179] Please refer to Fig. 9 , Fig. 9 The schematic diagram of the structure of the device for identifying the relationship between characters provided in the embodiment of the present application is as follows. The device for identifying the relationship between characters may include an acquisition module 701, a feature extraction module 702, and a relationship determination module 703, wherein:

[0180] An acquisition module 701 is used to acquire the character features of each character identification on the image to be processed;

[0181] A feature extraction module 702 is used to extract scene features from the image to be processed to obtain image scene features;

[0182] The relationship determination module 703 is used to determine the initial character relationship between the character identifiers based on the image scene feature and the character features of the character identifiers.

[0183] The specific implementation of each of the above modules can be found in the previous embodiments and will not be described in detail here.

[0184] An embodiment of the present application also provides an electronic device, including a memory and a processor, wherein the processor is used to execute the steps in the method for identifying character relationships provided in this embodiment by calling a computer program stored in the memory.

[0185] Please refer to Fig.10 , Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0186] The electronic device may include components such as a processor 101 with one or more processing cores, a memory 102 with one or more computer-readable storage media, a power supply 103, and an input unit 104. Those skilled in the art will appreciate that the electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:

[0187] The processor 101 is the control center of the electronic device, which uses various interfaces and lines to connect various parts of the entire electronic device, and executes various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 102, and calling data stored in the memory 102. Optionally, the processor 101 may include one or more processing cores; optionally, the processor 101 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 101.

[0188] The memory 102 can be used to store software programs and modules. The processor 101 executes various functional applications and data processing by running the software programs and modules stored in the memory 102. The memory 102 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 102 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 102 may also include a memory controller to provide the processor 101 with access to the memory 102.

[0189] The electronic device also includes a power supply 103 for supplying power to each component. Optionally, the power supply 103 can be logically connected to the processor 101 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 103 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0190] The electronic device may further include an input unit 104, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0191] Although not shown, the electronic device may also include a display unit, an image acquisition component, etc., which will not be described in detail here. Specifically in this embodiment, the processor 101 in the electronic device will load the executable code corresponding to one or more computer programs into the memory 102 according to the following instructions, and the processor 101 will execute the steps in the method for identifying the relationship between people provided in this application, such as:

[0192] Obtaining the character features of each character identification on the image to be processed;

[0193] Perform scene feature extraction on the image to be processed to obtain image scene features;

[0194] The initial character relationship between the character identifiers is determined based on the image scene feature and the character features of the character identifiers.

[0195] It should be noted that the electronic device provided in the embodiment of the present application belongs to the same concept as the method for identifying character relationships and / or the method for identifying character relationships in the above embodiments. The specific implementation process is detailed in the above related embodiments and will not be repeated here.

[0196] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program stored therein is executed on a processor of an electronic device provided in an embodiment of the present application, the processor of the electronic device executes the method for identifying a person relationship and / or the steps in the method for identifying a person relationship provided in the present application. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0197] The present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above-mentioned method for identifying a relationship between characters and / or various optional implementations of the method for identifying a relationship between characters.

[0198] The above is a detailed introduction to a method and device for identifying character relationships provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, based on the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0199] It should be noted that when the above embodiments of the present application are applied to specific products or technologies, the relevant data of the user is involved, and the user's permission or consent is required, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

Claims

1. A method for identifying character relationships, characterized in that: include: Obtaining the character features of each character identification on the image to be processed; Extracting scene features from the image to be processed to obtain image scene features; An initial character relationship between the character identifiers is determined based on the image scene feature and the character features of the character identifiers.

2. The method for identifying character relationships according to claim 1, characterized in that: The step of extracting scene features from the image to be processed to obtain image scene features includes: Get the pre-trained character relationship extraction model; Extracting scene features of the image to be processed by using the character relationship extraction model to obtain the image scene features; The determining the initial character relationship between the character identifiers based on the image scene feature and the character features of the character identifiers includes: The character relationship extraction model is used to predict character relationships based on the image scene features and the character features of each character identifier, so as to obtain the initial character relationships between each character identifier.

3. The method for identifying character relationships according to claim 2, characterized in that: The character relationship extraction model includes a first convolution module, and the character relationship extraction model is used to predict the character relationship based on the image scene features and the character features of each character identifier to obtain the initial character relationship between each character identifier, including: Using the first convolution module to extract features from the image to be processed to obtain first image features; Resampling the first image features based on the character features of each character identifier to obtain resampled image features of each character identifier; Initializing a graph based on the resampled image features of each of the character identifiers and the image scene features to obtain an initialized graph; Character relationship prediction is performed based on the initialization graph to obtain the initial character relationship between each of the character identifiers.

4. The method for identifying character relationships according to claim 3, characterized in that: The initialization of the graph based on the resampled image features of each of the character identifiers and the image scene features to obtain an initialized graph includes: Determine each node feature corresponding to each character identifier based on the resampled image features of each character identifier; Determine the two character identifiers as a character identifier group; Determining edge features of each of the character identification groups based on the image scene features; The graph is initialized based on each of the node features and each of the edge features to obtain an initialized graph.

5. The method for identifying character relationships according to claim 4, characterized in that: The character relationship extraction model includes a semantic feature encoding module and a semantic feature decoding module. The character relationship extraction model is used to predict the character relationship based on the image scene features and the character features of each character identifier to obtain the initial character relationship between each character identifier, including: Encoding the image to be processed by using the semantic feature encoding module to obtain semantic encoding features; Performing graph convolution on the initialization graph to obtain an output matrix; The output matrix and the semantic coding features are input into the semantic feature decoding module to obtain the initial character relationships between the various character identifiers.

6. The method for identifying character relationships according to claim 2, characterized in that: The character relationship extraction model includes a second convolution module, and the scene feature extraction of the image to be processed using the character relationship extraction model to obtain the image scene feature includes: Using the second convolution module to perform feature extraction on the image to be processed to obtain second image features; The second image features are pooled to obtain the image scene features.

7. The method for identifying character relationships according to claim 1, characterized in that: The character features include a face detection frame and a corresponding human body detection frame of each character identifier, and the obtaining of the character features of each character identifier on the to-be-processed image to be processed includes: Performing face detection on the image to be processed to obtain a face detection frame for each of the character identifiers; Performing human body detection on the image to be processed to obtain human body detection frames of the respective human characters; The face detection frame of each character identification is matched with the human body detection frame of each character identification to obtain the face detection frame of each character identification and the corresponding human body detection frame.

8. The method for identifying character relationships according to claim 7, characterized in that: The step of matching the face detection frame of each character identification with the human body detection frame of each character identification to obtain the face detection frame of each character identification and the corresponding human body detection frame includes: Calculating an overlap index and / or an upper frame distance index between the face detection frame and the body detection frame; Determine the matching degree between the face detection frame and the body detection frame based on the overlap index and / or the upper frame distance index, and obtain the matching degree between each face detection frame and each body detection frame; The human body detection frame having the highest matching degree with the face detection frame is determined as the human body detection frame corresponding to the face detection frame.

9. The method for identifying character relationships according to claim 1, characterized in that: The method for identifying character relationships includes: Acquire initial character relationships between the character identifiers on a plurality of images to be processed; Calculating a plurality of different initial character relationships between two character identifiers according to the initial character relationships between the character identifiers on the plurality of images to be processed; The initial character relationship with the highest occurrence frequency among a plurality of different initial character relationships between the two character identifiers is determined as the target character relationship between the two character identifiers, and the target character relationship between the character identifiers is obtained.

10. The method for identifying character relationships according to claim 2, characterized in that: The obtaining of a pre-trained character relationship extraction model comprises: Get sample images and corresponding annotation relationship labels; Inputting the sample image into the character relationship extraction model to obtain the predicted value of each relationship category on the sample image; Based on the first preset weight and the second preset weight, weighted summation is performed on the annotated relationship label of the sample image and the predicted value of each relationship category on the sample image to obtain a first weighted summation value of each relationship category on the sample image; Determine a target loss value based on a first weighted sum value of each relationship category on the sample image; The character relationship extraction model is iteratively trained based on the target loss value until a preset stopping condition is met.

11. The method for identifying character relationships according to claim 10, characterized in that: The determining of the target loss value based on the first weighted sum value of each relationship category on the sample image comprises: The first weighted sum values ​​of each relationship category are weighted based on the third preset weight of each relationship category to obtain the second weighted sum value of each relationship category, wherein the third preset weight of each relationship category decreases as the proportion of the annotated relationship labels corresponding to each category increases; The target loss value is determined based on the second weighted sum values ​​of the respective relationship categories.

12. A device for identifying a person relationship, characterized in that: include: An acquisition module is used to acquire the character features of each character identification on the image to be processed; A feature extraction module is used to extract scene features from the image to be processed to obtain image scene features; The relationship determination module is used to determine the initial character relationship between each character identification based on the image scene feature and the character feature of each character identification.

13. An electronic device, characterized in that: It comprises a memory and a processor, wherein the memory stores a computer program, and the processor is used to run the computer program in the memory to execute the steps in the method for identifying character relationships as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, which are suitable for being loaded by a processor to execute the steps in the method for identifying character relationships as described in any one of claims 1 to 11.

15. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps in the method for identifying character relationships described in any one of claims 1 to 11 are implemented.

Images