Social relationship identification method, device, electronic device and storage medium

By acquiring and processing the captured image sequence, combining space-time features, extracting and fusing social relationship features, the problem of low accuracy of single image recognition is solved, and the accuracy of social relationship recognition is significantly improved.

CN112668520BActive Publication Date: 2024-11-19SHENZHEN INTELLIFUSION TECHNOLOGIES CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202011639956.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-12-31
Publication Date
2024-11-19
Estimated Expiration
2040-12-31

AI Technical Summary

Technical Problem

The existing social relationship recognition methods based on single images have low accuracy and cannot effectively reflect the relationship between people, especially when the distance is far or dynamically changing.

Method used

By obtaining the captured image sequence, the trained network model extracts the social relationship characteristics of the target personnel, and sequentially fuses these characteristics, and combines the spatial and temporal features to identify them to improve the accuracy of the recognition.

Benefits of technology

Through the combination of image sequence and spatiotemporal features, the accuracy of social relationship recognition is significantly improved, and visual and activity features can be used more comprehensively to accurately identify the relationship between people.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112668520B_ABST
    Figure CN112668520B_ABST
Patent Text Reader

Abstract

The embodiment of the present invention provides a social relationship recognition method, device, electronic device and storage medium, the method comprising: obtaining a captured image sequence, the captured image sequence including a first target person and a second target person; extracting the social relationship features of the first target person and the second target person in the current captured image through a trained first network model; inputting the social relationship features corresponding to all captured images into the trained second network model in sequence for sequence fusion to obtain target fusion features; obtaining the spatiotemporal features of the first target person and the second target person, inputting the fusion features and the spatiotemporal features into a trained third network model for social relationship recognition, and obtaining the social relationship between the first target person and the second target person. By comprehensively utilizing the visual features and spatiotemporal activity features of the target person, the accuracy of social relationship recognition is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image relationship recognition, and in particular to a video-based social relationship recognition method, device, electronic device and storage medium. Background Art

[0002] Social relations (SR) are the foundation of human society, so mastering the social relations of the entire society is extremely important for both national governance and scientific research. Identifying social relations between people in images can enable intelligent agents to better understand human behavior or emotions. The task of image-based social relation recognition is to classify a pair of people in an image into one of the predefined relationship types, such as family, friends, strangers, etc. It has many important applications, such as personal image collection and mining and social event understanding.

[0003] Image-based sociological analysis has been a hot topic in the past decade. What is different is that in recent years, with the release of large-scale datasets, image-based social relationship recognition has attracted researchers. From traditional manual features to recent deep learning-based algorithms, the performance of methods has been significantly improved. However, existing methods are mainly based on a single image to judge the social relationship between two people. This method has the following problems: the information in a single image is very limited and cannot accurately reflect the relationship between the two people. For example, an old couple goes out for a walk, usually at a certain distance, and sometimes people cannot judge the relationship between the two from a single image. Therefore, the existing method only uses a single image to judge social relationships, which leads to a relatively low recognition accuracy. Summary of the invention

[0004] The embodiment of the present invention provides a social relationship identification method, which can improve the accuracy of social relationship identification.

[0005] In a first aspect, an embodiment of the present invention provides a social relationship identification method, comprising the following steps:

[0006] Acquire a captured image sequence, wherein the captured image sequence includes a first target person and a second target person;

[0007] Extracting social relationship features between the first target person and the second target person in the current captured image through the trained first network model;

[0008] The social relationship features corresponding to all captured images are sequentially input into the trained second network model for sequence fusion to obtain the target fusion features;

[0009] The spatiotemporal features of the first target person and the second target person are obtained, and the fusion features and the spatiotemporal features are input into a trained third network model to identify social relations, so as to obtain the social relations between the first target person and the second target person.

[0010] Optionally, extracting the social relationship features of the first target person and the second target person in the current captured image by using the trained first network model includes:

[0011] Inputting the current captured image into a target detection network, and outputting a first human body frame of the first target person, a second human body frame of the second target person, and a human body joint frame of the first target person and the second target person corresponding to the current captured image through the target detection network;

[0012] Based on the first human body frame, the second human body frame and the human body joint frame, the social relationship features of the first target person and the second target person are extracted from the current captured image through the pre-trained first network model.

[0013] Optionally, the first network model includes a first extraction network, a second extraction network, and a third extraction network, and the extracting the social relationship features of the first target person and the second target person from the current captured image based on the first human frame, the second human frame, and the human joint frame through the pre-trained first network model includes:

[0014] Acquire a corresponding human body joint image from the current captured image according to the human body joint frame;

[0015] Extracting layer features and third human features from the human joint image through the first extraction network;

[0016] Acquire a first human body feature of the first human body frame and a second human body feature of the second human body frame from the layer features through the second extraction network;

[0017] Based on the first human feature, the second human feature and the third human feature, the social relationship features of the first target person and the second target person are obtained through the third extraction network.

[0018] Optionally, obtaining the social relationship features of the first target person and the second target person through the third extraction network based on the first human feature, the second human feature and the third human feature includes:

[0019] Performing feature splicing on the first human feature, the second human feature and the third human feature to obtain a spliced ​​feature;

[0020] The social relationship features between the first target person and the second target person are obtained from the spliced ​​features through the third extraction network.

[0021] Optionally, the second network model includes a feature fusion network, and the social relationship features corresponding to all captured images are sequentially input into the pre-trained second network model for sequence fusion to obtain target fusion features, including:

[0022] Obtain the social relationship features corresponding to all captured images;

[0023] The social relationship features of all the captured images are sequentially input into the feature fusion network for feature fusion to obtain target fusion features.

[0024] Optionally, the step of sequentially inputting the social relationship features of all the captured images into the feature fusion network for feature fusion to obtain target fusion features includes:

[0025] Acquiring a current fusion feature of the current captured image based on the social relationship feature of the current captured image;

[0026] Inputting the current fusion feature and the social relationship feature of the next captured image into the feature fusion network for feature fusion to obtain the next fusion feature;

[0027] Repeat the above steps until the last frame of the captured image sequence is obtained to obtain the target fusion feature.

[0028] Optionally, the spatiotemporal features include features of the first target person and the second target person traveling together in different time periods and locations.

[0029] In a second aspect, an embodiment of the present invention provides a social relationship identification device, including:

[0030] An acquisition module, used for acquiring a captured image sequence, wherein the captured image sequence includes a first target person and a second target person;

[0031] An extraction module, used to extract the social relationship features between the first target person and the second target person in the current captured image through the trained first network model;

[0032] A fusion module, used to sequentially input the social relationship features corresponding to all captured images into the trained second network model for sequence fusion to obtain target fusion features;

[0033] An identification module is used to obtain the spatiotemporal features of the first target person and the second target person, input the fusion features and the spatiotemporal features into a trained third network model to identify social relationships, and obtain the social relationship between the first target person and the second target person.

[0034] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the social relationship identification method provided in the embodiment of the present invention when executing the computer program.

[0035] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the social relationship identification method provided in the embodiment of the invention are implemented.

[0036] In an embodiment of the present invention, the social relationship features of the first target person and the second target person in all captured images in the captured image sequence are extracted through a first network model, and then the social relationship features of all captured images are sequenced and fused through a second network model to obtain a high-dimensional target fusion feature of the entire captured image sequence and input it into a third network model together with the spatiotemporal features of the target person's activities for social relationship recognition, thereby obtaining the social relationship of the target person. The social relationship is recognized based on the image fusion of the captured image sequence and combined with the spatiotemporal features of the target person, which can fully utilize the visual features and spatiotemporal activity features of the target person, thereby improving the accuracy of social relationship recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 is a flow chart of a social relationship identification method provided by an embodiment of the present invention;

[0039] Figure 2 is a flow chart of a social relationship feature extraction method provided by an embodiment of the present invention;

[0040] Figure 3 is a flow chart of another social relationship feature extraction method provided by an embodiment of the present invention;

[0041] Figure 4is a flow chart of another social relationship feature extraction method provided by an embodiment of the present invention;

[0042] Figure 5 is a flow chart of a feature fusion method provided by an embodiment of the present invention;

[0043] Figure 6 is a structural schematic diagram of a social relationship identification device provided by an embodiment of the present invention;

[0044] Figure 7 is a structural schematic diagram of a social relationship feature extraction device provided by an embodiment of the present invention;

[0045] Figure 8 is a structural schematic diagram of an extraction module provided by an embodiment of the present invention;

[0046] Fig. 9 is a structural schematic diagram of a second extraction submodule provided by an embodiment of the present invention;

[0047] Fig.10 is a structural schematic diagram of a feature fusion device provided by an embodiment of the present invention;

[0048] Fig.11 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0050] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] See also Figure 1 , Figure 1 is a flow chart of a social relationship identification method provided by an embodiment of the present invention. Figure 1 As shown, the following steps are included:

[0053] 101. Obtain a captured image sequence.

[0054] The captured image sequence includes a first target person and a second target person.

[0055] In an embodiment of the present invention, the above-mentioned social relationship identification method can be used in the personnel relationship identification scenario under video surveillance. The electronic device on which the above-mentioned social relationship identification method is running can obtain the above-mentioned captured image sequence and transmit data through a wired connection or a wireless connection. It should be pointed out that the above-mentioned wireless connection method may include but is not limited to 3G / 4G connection, WiFi (Wireless-Fidelity) connection, Bluetooth connection, WiMA× (Worldwide Interoperability for Microwave Access) connection, Zigbee (low-power LAN protocol, also known as Zigbee protocol) connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0056] The above-mentioned captured image sequence can be captured in real time by an image acquisition device, or can be manually uploaded from a terminal, etc., and then the captured personnel are archived or filed, and stored in the personal file data on the server accordingly. The above-mentioned image acquisition device may include a camera and an electronic device equipped with a camera and capable of image acquisition. The above-mentioned captured image sequence may be an image sequence that includes two target persons while requiring personnel relationship analysis, may be a multi-frame image captured at a certain interval, may be a multi-frame image captured in a continuous time, or may be a multi-frame image of two target persons traveling together obtained at different times and / or different locations, and then the target persons in the image sequence are identified and the images are stored in the files of the corresponding persons respectively. At least 2 people may be included in the above-mentioned captured image, namely the first target person and the second target person. In addition, some interference objects may also be included, such as buildings, other irrelevant persons, trees, etc. The above-mentioned two target persons may refer to persons who need to be identified in social relationships, and social relationships are roughly divided into close relationships, general relationships, and unknown relationships.

[0057] 102. Extract social relationship features between the first target person and the second target person in the current captured image through the trained first network model.

[0058] In an embodiment of the present invention, the first network model can extract features of the target person in the image based on a single-frame image (such as the current captured image) and perform preliminary social relationship classification on the features (such as marking social relationship labels), that is, the social relationship features of the target person can be obtained. The current captured image can be any frame of the captured image sequence obtained above that contains two target persons at the same time, and social relationship recognition is being performed. The first target person and the second target person in the current captured image can be detected by the target detection network, and the corresponding human body frame (that is, the coordinate frame of the target person in the image) is output. Then the human body frames of the first target person and the second target person and the current captured image can be input into the first network model to extract social relationship features, and high-dimensional social relationship features with social relationship labels including the two target persons can be obtained.

[0059] Optional, see Figure 2 , Figure 2 is a flow chart of a social relationship feature extraction method provided by an embodiment of the present invention. Figure 2 As shown, the social relationship features of the first target person and the second target person in the current captured image are extracted by the trained first network model, including:

[0060] 201. Input the current captured image into a target detection network, and output a first human body frame of a first target person, a second human body frame of a second target person, and a joint human body frame of the first target person and the second target person corresponding to the current captured image through the target detection network.

[0061] In an embodiment of the present invention, the target detection network may be a trained neural network model capable of target recognition and positioning, such as Faster R-CNN, YOLO, etc. The target detection network detects the first target person and the second target person from the current captured image, and obtains corresponding human rectangular frames (coordinates in the current frame image), that is, obtains the first human frame of the first target person and the second human frame of the second target person. Furthermore, the minimum rectangular frame including the first target person and the second target person may be obtained, and then the coordinates of the minimum rectangular frame in the current captured image are obtained as the human joint frame.

[0062] 202. Based on the first human body frame, the second human body frame and the human body joint frame, extract the social relationship features of the first target person and the second target person from the current captured image through the trained first network model.

[0063] In an embodiment of the present invention, the above-mentioned first network model can extract the characteristics of the target person from the images corresponding to the above-mentioned first human body frame, the second human body frame and the human body joint frame and perform preliminary social relationship classification on the characteristics (such as marking social relationship labels), that is, the above-mentioned high-dimensional social relationship features with social relationship labels including the two target persons can be obtained.

[0064] Optionally, the first network model includes a first extraction network, a second extraction network and a third extraction network, see Figure 3 , Figure 3 is a flow chart of another social relationship feature extraction method provided by an embodiment of the present invention. Figure 3 As shown, the above-mentioned social relationship features of the first target person and the second target person are extracted from the current captured image through the trained first network model based on the first human frame, the second human frame and the human joint frame, including:

[0065] 301. Obtain a corresponding human body joint image from the current captured image according to the human body joint frame.

[0066] In the embodiment of the present invention, through the above step 201, a human body joint frame including the above first target person and the second target person can be obtained, and then the above human body joint image can be obtained from the above current captured image according to the coordinates of the above human body joint frame.

[0067] 302. Extract layer features and third human features from the human joint image through a first extraction network.

[0068] In an embodiment of the present invention, the first extraction network may be a resent101 residual neural network. The human body joint image is first resized into a tensor of size [448, 448] and then input into the resent101 residual neural network to extract a 2048-dimensional third human body feature and a [14, 14, 1024]-dimensional layer feature.

[0069] 303. Obtain a first human body feature of the first human body frame and a second human body feature of the second human body frame from the layer features through a second extraction network.

[0070] In an embodiment of the present invention, the second extraction network may be a ROI Pooling neural network layer, and the layer features may be further extracted through the ROIPooling neural network layer. Specifically, the coordinates of the first human body frame and the second human body frame are transformed to the layer features to obtain the transformed coordinates relative to the layer features. The ROIPooling neural network layer then locates the first target person and the second target person in the layer features according to the transformed coordinates, and extracts two 1024-dimensional human features after convolution and pooling, namely the first human body feature and the second human body feature.

[0071] 304. Based on the first human feature, the second human feature, and the third human feature, obtain social relationship features between the first target person and the second target person through a third extraction network.

[0072] Optional, see Figure 4 , Figure 4 is a flow chart of another social relationship feature extraction method provided by an embodiment of the present invention. Figure 4 As shown, the above step 304 includes:

[0073] 3041. Concatenate the first human feature, the second human feature, and the third human feature to obtain a concatenated feature;

[0074] In an embodiment of the present invention, the above-mentioned 2048-dimensional third human body feature, 1024-dimensional first feature and 1024-dimensional second feature may be spliced ​​together to obtain a 4096-dimensional spliced ​​feature.

[0075] 3042. Obtain social relationship features between the first target person and the second target person from the concatenated features through a third extraction network.

[0076] In an embodiment of the present invention, the third extraction network may be a two-layer fully connected layer. By performing an inner product operation on the first fully connected layer of the third extraction network and the 4096-dimensional concatenated features, the concatenated features may be linearly integrated. Then, the high-dimensional features of the two target persons are extracted through the second fully connected layer and the high-dimensional features are preliminarily classified into social relationships (such as labeling social relationships, with 2 representing close relationships, 1 representing general relationships, and 0 representing unfamiliarity, etc.), so that the social relationship features of the first target person and the second target person may be obtained. The social relationship features include high-dimensional features with relationship labels of the two target persons. The two fully connected layers of the third extraction network can achieve high integration and high-dimensional abstraction of features without increasing parameters and computational complexity, thereby achieving better feature extraction effects.

[0077] 103. The social relationship features corresponding to all captured images are sequentially input into the trained second network model for sequence fusion to obtain target fusion features.

[0078] In an embodiment of the present invention, each frame of the captured image sequence can obtain a high-dimensional social relationship feature with social relationship labels including two target persons after the above steps, and then the high-dimensional social relationship feature corresponding to each frame of the captured image is input into the trained second network model in turn, and all features are fused in time series to obtain the above-mentioned target fusion feature, so that the social relationship of the target person can be identified as a whole based on the above-mentioned target fusion feature.

[0079] Optionally, the second network model includes a feature fusion network, see Figure 5 , Figure 5 is a flow chart of a feature fusion method provided by an embodiment of the present invention. Figure 5 As shown, the social relationship features of all captured images are sequentially input into the feature fusion network for feature fusion to obtain target fusion features, including:

[0080] 501. Acquire a current fusion feature of the current captured image based on the social relationship feature of the current captured image;

[0081] 502. Input the current fusion feature and the social relationship feature of the next captured image into a feature fusion network for feature fusion to obtain the next fusion feature;

[0082] 503. Repeat the above steps until the last frame of the captured image sequence is obtained to obtain the target fusion feature.

[0083] In an embodiment of the present invention, the feature fusion network of the second network model may be a neural network that can perform feature fusion on a sequence of captured images in order, such as a recurrent neural network RNN, a long short-term memory network LSTM, etc. Specifically, the fusion features of the last captured image obtained after the last captured image passes through the feature fusion network of the second network model can be obtained through the above steps, and then the social relationship features of the current captured image and the fusion features of the last captured image can be further fused. For example, the social relationship features of the current captured image and the input weights of the RNN can be calculated, the fusion features of the last captured image and the hidden layer shared weights of the RNN can be calculated, and then the results of the two calculations can be added together to obtain the current fusion features of the current captured image; similarly, the current fusion features and the social relationship features of the next captured image can be input into the feature fusion network to perform the feature fusion calculation, and the next fusion features can be obtained.

[0084] It should be noted that each frame of the captured image sequence can be the current captured image, that is, from the first frame to the last frame of the captured image sequence, and they all contain two target persons whose social relationships are to be identified, and the fusion features of all the captured images before the current captured image can be obtained by repeating the above steps, until the last frame of the captured image sequence is used as the current frame to perform the above-mentioned corresponding feature fusion calculation, that is, to obtain the above-mentioned target fusion features.

[0085] 104. Obtain the spatiotemporal features of the first target person and the second target person, input the target fusion features and the spatiotemporal features into the trained third network model to identify the social relationship, and obtain the social relationship between the first target person and the second target person.

[0086] In an embodiment of the present invention, the spatiotemporal characteristics of the first target person and the second target person include the characteristics of the first target person and the second target person traveling together at different time periods and different locations. The spatiotemporal characteristics can be obtained from the archival data of each target person. The archival data of each target person can be obtained by collecting and filing pedestrian data from a community monitoring system or a public system, which includes the travel data of the target person at different time periods and different locations. For example, pedestrian data in the community can be captured by a surveillance camera in the community, and the captured image can be used to perform face recognition and then file the face. N captured faces can obtain N captured face thumbnails, and then the time, location, original image and other data of the captured time are correspondingly classified into the personal archives of the community or public system according to each person's face thumbnails. Each archive represents a person, which stores the spatiotemporal characteristic data of each person's travel at different time periods and different locations, including the spatiotemporal characteristics of traveling with other target persons. The captured image sequence can be a plurality of captured images of the two target persons in the archive when they travel together at different times and different locations.

[0087] The third network model includes a feature splicing layer and a fully connected layer. Specifically, first, an image containing the second target person can be obtained from the archive data of the first target person through a search method such as face matching, and the corresponding time and place of travel can be obtained and processed together as a feature vector as the spatiotemporal features of the first target person and the second target person, and then the spatiotemporal features and the target fusion features obtained in step 103 are input into the feature splicing layer of the trained third network model to splice and integrate the two features, and obtain splicing features containing high-dimensional human features and historical spatiotemporal features of the first target person and the second target person at the same time, and finally the splicing features are input into the fully connected layer of the third network model to classify and identify social relationships, thereby obtaining the social relationship between the first target person and the second target person, that is, one of a close relationship, a general relationship, and an unfamiliar relationship.

[0088] It should be noted that the training of the first network model, the second network model and the third network model can be performed in an end-to-end manner, that is, the three network models are combined into one, and the joint training is performed through the same training data set to obtain the corresponding trained models, that is, the trained first network model, the trained second network model and the trained third network model. The training data set can be obtained by collecting images from the archival data of the sample personnel and annotating the social relationships of the peers in the images (that is, marking the peers in the images with social relationship labels of 0, 1 or 2).

[0089] To summarize, in an embodiment of the present invention, the social relationship features of the first target person and the second target person in all captured images in the captured image sequence are extracted through the first network model, and then the social relationship features of all captured images are sequenced and fused through the second network model to obtain the high-dimensional target fusion features of the entire captured image sequence and input them together with the spatiotemporal features of the target person's activities into the third network model for social relationship recognition, thereby obtaining the social relationship of the target person. The social relationship is recognized based on the image fusion of the captured image sequence and combined with the spatiotemporal features of the target person, which can fully utilize the visual features and spatiotemporal activity features of the target person, thereby improving the accuracy of social relationship recognition.

[0090] It should be noted that the social relationship identification method provided in the embodiment of the present invention can be applied to devices such as mobile phones, monitors, computers, servers, etc. that can perform social relationship identification.

[0091] See also Figure 6 , Figure 6 is a schematic diagram of the structure of a social relationship identification device provided by an embodiment of the present invention. Figure 6 As shown, the device 600 includes:

[0092] An acquisition module 601 is used to capture a sequence of images, where the sequence of images includes a first target person and a second target person;

[0093] An extraction module 602 is used to extract social relationship features between a first target person and a second target person in a current captured image through a trained first network model;

[0094] Fusion module 603, used to sequentially input the social relationship features corresponding to all captured images into the trained second network model for sequence fusion to obtain target fusion features;

[0095] The identification module 604 is used to obtain the spatiotemporal features of the first target person and the second target person, input the fusion features and the spatiotemporal features into the trained third network model to identify the social relationship, and obtain the social relationship between the first target person and the second target person.

[0096] Optional, such as Figure 7 As shown, Figure 7 is a schematic diagram of the structure of a social relationship feature extraction device provided by an embodiment of the present invention. The social relationship feature extraction device 700 includes:

[0097] An output module 701 is used to input the current captured image into a target detection network, and output a first human body frame of a first target person, a second human body frame of a second target person, and a human body joint frame of the first target person and the second target person corresponding to the current captured image through the target detection network;

[0098] The extraction module 702 is used to extract the social relationship features of the first target person and the second target person from the current captured image based on the first human frame, the second human frame and the human joint frame through a pre-trained first network model.

[0099] Optionally, the first network model includes a first extraction network, a second extraction network and a third extraction network, such as Figure 8 As shown, Figure 8 702 is a schematic diagram of the structure of an extraction module provided by an embodiment of the present invention. The extraction module 702 includes:

[0100] The first acquisition submodule 7021 is used to acquire a corresponding human body joint image from the current captured image according to the human body joint frame;

[0101] A first extraction submodule 7022 is used to extract layer features and third human features from the human joint image through a first extraction network;

[0102] The second acquisition submodule 7023 is used to acquire the first human body feature of the first human body frame and the second human body feature of the second human body frame from the layer features through the second extraction network.

[0103] The second extraction submodule 7024 is used to obtain the social relationship characteristics of the first target person and the second target person through a third extraction network based on the first human feature, the second human feature and the third human feature.

[0104] Optional, see Fig. 9 , Fig. 9 is a schematic diagram of the structure of a second extraction submodule provided by an embodiment of the present invention, such as Fig. 9 As shown, the second extraction submodule 7024 includes:

[0105] A splicing unit 70241 is used to splice the first human feature, the second human feature and the third human feature to obtain a spliced ​​feature;

[0106] The extraction unit 70242 is used to obtain the social relationship features of the first target person and the second target person from the spliced ​​features through a third extraction network.

[0107] Optional, see Fig.10 , Fig.10 is a structural schematic diagram of a feature fusion device provided by an embodiment of the present invention, such as Fig.10As shown, the feature fusion device 1000 includes:

[0108] An acquisition module 1001 is used to acquire a current fusion feature of a current captured image based on a social relationship feature of the current captured image;

[0109] Fusion module 1002, used for inputting the current fusion feature and the social relationship feature of the next captured image into a feature fusion network for feature fusion to obtain the next fusion feature;

[0110] The repetition module 1003 is used to repeat the above steps until the last frame image of the captured image sequence is obtained to obtain the target fusion feature.

[0111] It should be noted that the social relationship identification device provided in the embodiment of the present invention can be applied to devices such as mobile phones, monitors, computers, servers, etc. that can perform social relationship identification.

[0112] The social relationship identification device provided in the embodiment of the present invention can implement each process implemented by the social relationship identification method in the above method embodiment, and can achieve the same beneficial effects. To avoid repetition, it will not be described here.

[0113] See also Fig.11 , Fig.11 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Fig.11 As shown, it includes: a memory 1102, a processor 1101, and a computer program stored in the memory 1102 and executable on the processor 1101, wherein:

[0114] The processor 1101 is used to call the computer program stored in the memory 1102 and execute the following steps:

[0115] Acquire a captured image sequence, wherein the captured image sequence includes a first target person and a second target person;

[0116] Extracting social relationship features between the first target person and the second target person in the current captured image through the trained first network model;

[0117] The social relationship features corresponding to all captured images are sequentially input into the trained second network model for sequence fusion to obtain the target fusion features;

[0118] The spatiotemporal features of the first target person and the second target person are obtained, and the fusion features and the spatiotemporal features are input into a trained third network model to identify social relations, so as to obtain the social relations between the first target person and the second target person.

[0119] Optionally, the processor 1101 extracts the social relationship features of the first target person and the second target person in the current captured image through the trained first network model, including:

[0120] Inputting the current captured image into a target detection network, and outputting a first human body frame of the first target person, a second human body frame of the second target person, and a human body joint frame of the first target person and the second target person corresponding to the current captured image through the target detection network;

[0121] Based on the first human body frame, the second human body frame and the human body joint frame, the social relationship features of the first target person and the second target person are extracted from the current captured image through the trained first network model.

[0122] Optionally, the first network model includes a first extraction network, a second extraction network, and a third extraction network, and the processor 1101 performs the extraction of the social relationship features of the first target person and the second target person from the current captured image based on the first human frame, the second human frame, and the human joint frame through the trained first network model, including:

[0123] Acquire a corresponding human body joint image from the current captured image according to the human body joint frame;

[0124] Extracting layer features and third human features from the human joint image through the first extraction network;

[0125] Acquire a first human body feature of the first human body frame and a second human body feature of the second human body frame from the layer features through the second extraction network;

[0126] Based on the first human feature, the second human feature and the third human feature, the social relationship features of the first target person and the second target person are obtained through the third extraction network.

[0127] Optionally, the processor 1101 performs the obtaining of the social relationship feature between the first target person and the second target person through the third extraction network based on the first human feature, the second human feature, and the third human feature, including:

[0128] Performing feature splicing on the first human feature, the second human feature and the third human feature to obtain a spliced ​​feature;

[0129] The social relationship features between the first target person and the second target person are obtained from the spliced ​​features through the third extraction network.

[0130] Optionally, the second network model includes a feature fusion network, and the processor 1101 performs the step of sequentially inputting the social relationship features corresponding to all captured images into the trained second network model for sequence fusion to obtain target fusion features, including:

[0131] Obtain the social relationship features corresponding to all captured images;

[0132] The social relationship features of all the captured images are sequentially input into the feature fusion network for feature fusion to obtain target fusion features.

[0133] Optionally, the processor 1101 sequentially inputs the social relationship features of all the captured images into the feature fusion network for feature fusion to obtain target fusion features, including:

[0134] Acquiring a current fusion feature of the current captured image based on the social relationship feature of the current captured image;

[0135] Inputting the current fusion feature and the social relationship feature of the next captured image into the feature fusion network for feature fusion to obtain the next fusion feature;

[0136] Repeat the above steps until the last frame of the captured image sequence is obtained to obtain the target fusion feature.

[0137] Optionally, in the steps executed by processor 1101, the spatiotemporal features include features of the first target person and the second target person traveling together in different time periods and at different locations.

[0138] It should be noted that the above-mentioned electronic devices may be mobile phones, monitors, computers, servers and other devices that can be used for social relationship identification.

[0139] The electronic device provided by the embodiment of the present invention can implement each process implemented by the social relationship identification method in the above method embodiment, and can achieve the same beneficial effects. To avoid repetition, it will not be described here.

[0140] The memory 1102 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 1102 can be an internal storage unit of the electronic device 1100, such as a hard disk or memory of the electronic device 1100. In other embodiments, the memory 1102 can also be an external storage device of the electronic device 1100, such as a plug-in hard disk equipped on the electronic device 1100, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 1102 can also include both the internal storage unit of the electronic device 1100 and its external storage device. In this embodiment, the memory 1102 is generally used to store an operating system and various application software installed on the electronic device 1100, such as a program code of a social relationship recognition method, etc. In addition, the memory 1102 can also be used to temporarily store various types of data that have been output or are to be output.

[0141] In some embodiments, the processor 1101 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 1101 is generally used to control the overall operation of the electronic device 1100. In this embodiment, the processor 1101 is used to run the program code stored in the memory 1102 or process data, such as running a program code of a social relationship recognition method.

[0142] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the social relationship identification method provided by the embodiment of the present invention are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0143] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0144] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A social relationship identification method, It is characterized in that The following steps are involved: Acquire a captured image sequence, wherein the captured image sequence includes a first target person and a second target person; Extracting social relationship features between the first target person and the second target person in the current captured image through the trained first network model; The social relationship features corresponding to all captured images are sequentially input into the trained second network model for sequence fusion to obtain the target fusion features; The spatiotemporal features of the first target person and the second target person are obtained, and the fused features and the spatiotemporal features are input into a trained third network model to identify social relations, so as to obtain the social relations between the first target person and the second target person; the spatiotemporal features of the first target person and the second target person include features of the first target person and the second target person traveling together in different time periods and at different locations.

2. The method according to claim 1, It is characterized in that The step of extracting the social relationship features between the first target person and the second target person in the current captured image by using the trained first network model includes: Inputting the current captured image into a target detection network, and outputting a first human body frame of the first target person, a second human body frame of the second target person, and a human body joint frame of the first target person and the second target person corresponding to the current captured image through the target detection network; Based on the first human body frame, the second human body frame and the human body joint frame, the social relationship features of the first target person and the second target person are extracted from the current captured image through the trained first network model.

3. The method according to claim 2, It is characterized in that The first network model includes a first extraction network, a second extraction network, and a third extraction network. The social relationship features of the first target person and the second target person are extracted from the current captured image based on the first human frame, the second human frame, and the human joint frame through the trained first network model, including: Acquire a corresponding human body joint image from the current captured image according to the human body joint frame; Extracting layer features and third human features from the human joint image through the first extraction network; Acquire a first human body feature of the first human body frame and a second human body feature of the second human body frame from the layer features through the second extraction network; Based on the first human feature, the second human feature and the third human feature, the social relationship features of the first target person and the second target person are obtained through the third extraction network.

4. The method according to claim 3, It is characterized in that The step of obtaining the social relationship feature between the first target person and the second target person through the third extraction network based on the first human feature, the second human feature and the third human feature includes: Performing feature splicing on the first human feature, the second human feature and the third human feature to obtain a spliced ​​feature; The social relationship features between the first target person and the second target person are obtained from the spliced ​​features through the third extraction network.

5. The method according to claim 4, It is characterized in that The second network model includes a feature fusion network, and the social relationship features corresponding to all captured images are sequentially input into the trained second network model for sequence fusion to obtain target fusion features, including: Obtain the social relationship features corresponding to all captured images; The social relationship features of all the captured images are sequentially input into the feature fusion network for feature fusion to obtain target fusion features.

6. The method according to claim 5, It is characterized in that The step of sequentially inputting the social relationship features of all the captured images into the feature fusion network for feature fusion to obtain target fusion features includes: Acquiring a current fusion feature of the current captured image based on the social relationship feature of the current captured image; Inputting the current fusion feature and the social relationship feature of the next captured image into the feature fusion network for feature fusion to obtain the next fusion feature; Repeat the above steps until the last frame of the captured image sequence is obtained to obtain the target fusion feature.

7. The method according to claim 6, It is characterized in that The spatiotemporal features include features of the first target person and the second target person traveling together in different time periods and locations.

8. A social relationship recognition device, It is characterized in that include: An acquisition module, used for acquiring a captured image sequence, wherein the captured image sequence includes a first target person and a second target person; An extraction module, used to extract the social relationship features between the first target person and the second target person in the current captured image through the trained first network model; A fusion module, used to sequentially input the social relationship features corresponding to all captured images into the trained second network model for sequence fusion to obtain target fusion features; An identification module is used to obtain the spatiotemporal features of the first target person and the second target person, input the fused features and the spatiotemporal features into a trained third network model to identify social relations, and obtain the social relations between the first target person and the second target person; the spatiotemporal features of the first target person and the second target person include features of the first target person and the second target person traveling together in different time periods and different locations.

9. An electronic device, It is characterized in that include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the social relationship identification method as claimed in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the social relationship identification method according to any one of claims 1 to 7 are implemented.