Target search method, device and computer readable storage medium
By combining facial and biological characteristics analysis, the target correlation is calculated using a fully connected neural network, the problem of inaccurate target search caused by visual attribute dependence in the existing technology is solved, and the high-accurate target search in diverse scenarios is achieved .
Patent Information
- Application Number
- CN201910669148.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-07-24
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2039-07-24
AI Technical Summary
Existing target search techniques rely too much on the visual attributes of the face and body, resulting in lower accuracy in searching targets in videos when the character is facing away from the camera or posture changes.
By utilizing the facial features of the target portrait and the facial and biological characteristics of the candidate frame sequence, combined with a fully connected neural network, the facial similarity and identity correlation are determined, and the target correlation is calculated, and the accuracy between the target portrait and the candidate frame sequence is achieved. Related.
It improves the accuracy of searching for targets in videos, and can accurately determine the correlation between target portraits and frame sequences in a variety of target activities, which enhances the reliability of target search in videos.
Smart Images

Figure CN111783507B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of multimedia technology, and in particular to a target search method, device and computer-readable storage medium. Background Art
[0002] Multimedia video analysis technology is increasingly being used in daily life, public safety, criminal investigation, and other fields. It can be seen that searching for content in videos plays an increasingly important role in combating crime and maintaining social security. There are a variety of targets in videos, such as people, animals, etc., among which people are the main active subjects in videos.
[0003] Target search is a technology that can find video frame sequences with the same identity in a video based on only one portrait of the target. In daily life, this technology can identify the target video frame sequence that the audience is interested in in the video, making our lives more efficient and convenient. In public security and criminal investigation, this technology can assist video investigators to quickly and accurately find pedestrians in the picture, which is of great significance to the public security department to improve the crime solving rate and protect the lives and property of the people. Summary of the invention
[0004] The inventors have found that related target search technologies rely too much on the visual attributes of the face and body. For example, face recognition technology can provide satisfactory discrimination performance under the premise that the face area in the picture can be obtained. However, in actual applications, people are constantly moving, and it is not guaranteed that every video frame contains a standard face. If the face cannot be detected in the picture with the back of the person to the camera, the face recognition technology will be directly invalid. This results in low accuracy in searching for targets in the video.
[0005] A technical problem solved by the present disclosure is how to use the target portrait to more accurately search for the target video in the video.
[0006] According to one aspect of an embodiment of the present disclosure, a target search method is provided, comprising: using facial features of a target portrait and facial features of each candidate frame sequence to determine facial similarities between the target portrait and each candidate frame sequence, respectively; wherein each candidate frame in each candidate frame sequence corresponds to the same candidate target; using facial features and other biometric features of each candidate frame sequence to determine identity correlations between each candidate frame sequence; using facial similarities and identity correlations to respectively determine target correlations between the target portrait and each candidate frame sequence.
[0007] In some embodiments, using the facial features and other biometric features of each candidate frame sequence to determine the identity correlation between each candidate frame sequence includes: cascading the facial features and other biometric features of each candidate frame sequence and inputting them into a pre-trained fully connected neural network to obtain the weights of the facial features and other biometric features of each candidate frame sequence; using the weights, the facial features and other biometric features of each candidate frame sequence to determine the identity features of each candidate frame sequence; using the identity features of each candidate frame sequence to determine the identity correlation between each candidate frame sequence.
[0008] In some embodiments, using weights, facial features and other biometric features of each candidate frame sequence to determine the identity features of each candidate frame sequence includes: using the weight of any candidate frame sequence to weight the facial features and other biometric features of any candidate frame sequence; cascading the weighted results to obtain the identity features of any candidate frame sequence.
[0009] In some embodiments, it also includes: constructing a fully connected neural network; constructing a loss function using the Euclidean distance between the identity features of the first candidate frame sequence and the identity features of the second candidate frame sequence; the first candidate frame sequence and the second candidate frame sequence are any candidate frame sequences; wherein, when the first candidate frame sequence and the second candidate frame sequence correspond to the same candidate target, the value of the loss function is positively correlated with the Euclidean distance; when the first candidate frame sequence and the second candidate frame sequence correspond to different candidate targets, the value of the loss function is positively correlated with the difference between the Euclidean distance and a preset value; adjusting the parameters of the fully connected neural network to reduce the value of the loss function.
[0010] In some embodiments, the loss function is:
[0011]
[0012] Wherein, L represents the loss function, N represents the number of identity feature pairs consisting of the identity features of the first candidate frame sequence and the identity features of the second candidate frame sequence, Y=1 represents that the first candidate frame sequence and the second candidate frame sequence correspond to the same candidate target, and Y=0 represents that the first candidate frame sequence and the second candidate frame sequence correspond to different candidate targets; d represents the Euclidean distance; δ represents the preset value.
[0013] In some embodiments, using the identity features of each candidate frame sequence to determine the identity correlation between each candidate frame sequence includes: determining the sum of cosine similarities between the identity features of the third candidate frame sequence and the identity features of other candidate frame sequences; wherein the third candidate frame sequence is any candidate frame sequence; using the cosine similarity between the identity features of the third candidate frame sequence and the identity features of the fourth candidate frame sequence, divided by the sum of cosine similarities, to obtain the identity correlation between the third candidate frame sequence and the fourth candidate frame sequence; wherein the fourth candidate frame sequence is any other candidate frame sequence.
[0014] In some embodiments, using facial similarity and identity correlation to respectively determine the target correlation between the target portrait and each candidate frame sequence includes: accumulating the identity correlations between any candidate frame sequence and each candidate frame sequence to obtain the total identity correlation of any candidate frame sequence; dividing the identity correlations between any candidate frame sequence and each candidate frame sequence by the total identity correlation to obtain the normalized identity correlations between any candidate frame sequence and each candidate frame sequence; using the normalized identity correlations, weighting the facial similarities between any candidate frame sequence and each corresponding candidate frame sequence to obtain the target correlation between the target portrait and any candidate frame sequence.
[0015] In some embodiments, using the facial features of the target portrait and the facial features of each candidate frame sequence to determine the facial similarity between the target portrait and each candidate frame sequence includes: using the cosine similarity between the facial features of the target portrait and the facial features of each candidate frame sequence as the facial similarity between the target portrait and each candidate frame sequence.
[0016] In some embodiments, the other biometric features include at least one of body features and voice features, and the weights of the other biometric features include the weights of each of the other biometric features.
[0017] In some embodiments, it also includes: using a facial detection neural network to extract facial features of each candidate frame in any candidate frame sequence; averaging the facial features of each candidate frame in any candidate frame sequence to obtain facial features of any candidate frame sequence; using a body detection neural network to extract body features of each candidate frame in any candidate frame sequence; averaging the body features of each candidate frame in any candidate frame sequence to obtain body features of any candidate frame sequence; extracting audio from any candidate frame sequence; using a speech recognition neural network to extract speech features from the audio to obtain speech features of any candidate frame sequence.
[0018] According to another aspect of an embodiment of the present disclosure, a target search device is provided, including: a facial similarity determination module, configured to determine the facial similarities between the target portrait and each candidate frame sequence respectively, using the facial features of the target portrait and the facial features of each candidate frame sequence; wherein each candidate frame in each candidate frame sequence corresponds to the same candidate target; an identity correlation determination module, configured to determine the identity correlation between each candidate frame sequence using the facial features and other biometric features of each candidate frame sequence; and a target correlation determination module, configured to determine the target correlation between the target portrait and each candidate frame sequence respectively, using the facial similarity and the identity correlation.
[0019] In some embodiments, the identity correlation determination module is configured to: cascade the facial features and other biometric features of each candidate frame sequence and input them into a pre-trained fully connected neural network to obtain the weights of the facial features and other biometric features of each candidate frame sequence; use the weights, the facial features and other biometric features of each candidate frame sequence to determine the identity features of each candidate frame sequence; use the identity features of each candidate frame sequence to determine the identity correlation between each candidate frame sequence.
[0020] In some embodiments, the identity relevance determination module is configured to: weight the facial features and other biometric features of any candidate frame sequence respectively using the weight of any candidate frame sequence; and cascade the weighted results to obtain the identity features of any candidate frame sequence.
[0021] In some embodiments, the identity relevance determination module is configured to: construct a fully connected neural network; construct a loss function using the Euclidean distance between the identity features of the first candidate frame sequence and the identity features of the second candidate frame sequence; the first candidate frame sequence and the second candidate frame sequence are any candidate frame sequences; wherein, when the first candidate frame sequence and the second candidate frame sequence correspond to the same candidate target, the value of the loss function is positively correlated with the Euclidean distance; when the first candidate frame sequence and the second candidate frame sequence correspond to different candidate targets, the value of the loss function is positively correlated with the difference between the Euclidean distance and a preset value; and adjust the parameters of the fully connected neural network to reduce the value of the loss function.
[0022] In some embodiments, the loss function is:
[0023]
[0024] Wherein, L represents the loss function, N represents the number of identity feature pairs consisting of the identity features of the first candidate frame sequence and the identity features of the second candidate frame sequence, Y=1 represents that the first candidate frame sequence and the second candidate frame sequence correspond to the same candidate target, and Y=0 represents that the first candidate frame sequence and the second candidate frame sequence correspond to different candidate targets; d represents the Euclidean distance; δ represents the preset value.
[0025] In some embodiments, the identity correlation determination module is configured to: determine the sum of cosine similarities between the identity features of the third candidate frame sequence and the identity features of other candidate frame sequences; wherein the third candidate frame sequence is any candidate frame sequence; use the cosine similarity between the identity features of the third candidate frame sequence and the identity features of the fourth candidate frame sequence, divided by the sum of cosine similarities, to obtain the identity correlation between the third candidate frame sequence and the fourth candidate frame sequence; wherein the fourth candidate frame sequence is any other candidate frame sequence.
[0026] In some embodiments, the target correlation determination module is configured to: accumulate the identity correlations between any candidate frame sequence and each candidate frame sequence respectively to obtain the total identity correlation of any candidate frame sequence; divide the identity correlations between any candidate frame sequence and each candidate frame sequence respectively by the total identity correlation to obtain the normalized identity correlations between any candidate frame sequence and each candidate frame sequence respectively; use the normalized identity correlations to weight the facial similarities between any candidate frame sequence and each corresponding candidate frame sequence respectively to obtain the target correlation between the target portrait and any candidate frame sequence.
[0027] In some embodiments, the facial similarity determination module is configured to use the cosine similarities between the facial features of the target portrait and the facial features of each candidate frame sequence as the facial similarities between the target portrait and each candidate frame sequence.
[0028] In some embodiments, the other biometric features include at least one of body features and voice features, and the weights of the other biometric features include the weights of each of the other biometric features.
[0029] In some embodiments, it also includes a feature extraction module, which is configured to: use a facial detection neural network to extract facial features of each candidate frame in any candidate frame sequence; average the facial features of each candidate frame in any candidate frame sequence to obtain the facial features of any candidate frame sequence; use a body detection neural network to extract body features of each candidate frame in any candidate frame sequence; average the body features of each candidate frame in any candidate frame sequence to obtain the body features of any candidate frame sequence; extract audio from any candidate frame sequence; use a speech recognition neural network to extract speech features from the audio to obtain speech features of any candidate frame sequence.
[0030] According to another aspect of an embodiment of the present disclosure, a target search device is provided, including: a memory; and a processor coupled to the memory, wherein the processor is configured to execute the aforementioned target search method based on instructions stored in the memory.
[0031] According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and when the instructions are executed by a processor, the aforementioned target search method is implemented.
[0032] The present disclosure can more accurately determine the target correlation between the target portrait and each candidate frame sequence, so that the target video can be more accurately searched in the video using the target portrait.
[0033] Other features and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0035] Figure 1 A flowchart of a target search method according to some embodiments of the present disclosure is shown.
[0036] Figure 2 A flowchart of an embodiment of training a fully connected neural network is shown.
[0037] Figure 3 A schematic diagram of the structure of a fully connected neural network is shown.
[0038] Figure 4 A schematic diagram of a flow chart of an embodiment of determining identity correlations between candidate frame sequences is shown.
[0039] Figure 5 A schematic structural diagram of a target search device according to some embodiments of the present disclosure is shown.
[0040] Figure 6 A schematic structural diagram of a target search device according to some other embodiments of the present disclosure is shown. DETAILED DESCRIPTION
[0041] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present disclosure and its application or use. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.
[0042] First combine Figure 1 Some embodiments of the target search method disclosed herein are described.
[0043] Figure 1 FIG. 1 is a flow chart showing a target search method according to some embodiments of the present disclosure. Figure 1 As shown, this embodiment includes steps S101 to S103.
[0044] In step S101, facial features of the target portrait and facial features of each candidate frame sequence are used to determine facial similarities between the target portrait and each candidate frame sequence.
[0045] Each candidate frame sequence includes multiple candidate frames, each candidate frame in each candidate frame sequence corresponds to the same candidate target, and each candidate frame in different candidate frame sequences corresponds to the same or different candidate targets.
[0046] Facial features include key features such as facial features. For the target portrait, the facial features can be extracted using a facial detection neural network. For example, the face detection sub-FaceBoxes detector is first used to obtain the face area in the target portrait, which is normalized to a size of 120*120. Then, the facial detection neural network ResNet101 trained in the large-scale face recognition data MS-Celeb-1M-v1c is used to extract 512-dimensional facial features.
[0047] For each candidate frame sequence, the above-mentioned face detection neural network can be used to extract the facial features of each candidate frame in any candidate frame sequence. Then, the facial features of each candidate frame in any candidate frame sequence are averaged to obtain the facial features of the any candidate frame sequence.
[0048] After extracting the facial features of the target portrait and the facial features of each candidate frame sequence, the cosine similarity between the facial features of the target portrait and the facial features of each candidate frame sequence can be used as the facial similarity between the target portrait and each candidate frame sequence.
[0049] For example, the facial similarity between the target portrait and the candidate frame sequence is calculated by formula (1):
[0050]
[0051] Among them, s(q,k) represents the similarity between the target portrait q and the candidate frame sequence k, f q represents the facial features of the target portrait, f k represents the facial features of candidate sequence k, and ||·|| represents the binary norm operation. The greater the facial similarity, the more likely it is that the target portrait and the candidate frame sequence contain the same target; the smaller the facial similarity, the more likely it is that the target portrait and the candidate frame sequence contain different targets. The candidate frame sequences can be sorted in descending order of facial similarity with the target portrait.
[0052] In step S102, the identity correlation between the candidate frame sequences is determined using the facial features and other biological features of the candidate frame sequences.
[0053] Other biometric features may include physical features, voice features, etc. Identity features mostly reflect the appearance attributes of the target. Voice features are a type of biometric feature with good robustness, covering the target's gender, age, nationality and other attributes, and can maintain identity consistency in long-term activities.
[0054] When extracting the body features of any candidate frame sequence, Faster RCNN can be used as a pedestrian detector to obtain the pedestrian region in the candidate frame. The pedestrian region is normalized to a size of 384*168, and the body detection neural network ResNet50 pre-trained in the pedestrian search data set is used to extract the body features of each candidate frame in the any candidate frame sequence. The dimension of the body feature can be, for example, 2048 dimensions. Then, the body features of each candidate frame in the any candidate frame sequence are averaged to obtain the body features of the any candidate frame sequence.
[0055] When extracting the speech features of any candidate frame sequence, the sound wave extraction technology can be used to extract the audio in the video frame sequence at a sampling rate of 16000 Hz. Then, the speech recognition neural network 3DCNN pre-trained in the speech recognition dataset VoxCeleb2 is used to extract the speech features of any candidate frame sequence. The dimension of the speech feature can be, for example, 128 dimensions.
[0056] After extracting the facial features and other biometric features of each candidate frame sequence, the identity correlation between each candidate frame sequence can be determined according to the cosine similarity between the facial features and other biometric features of each candidate frame sequence. Taking facial features, body features, and voice features as examples, the facial similarity between two candidate frame sequences is first calculated using the facial features of each candidate frame sequence. The body similarity and voice similarity between the two candidate frame sequences are calculated using a similar calculation method. Then the facial similarity, body similarity, and voice similarity between the two candidate frame sequences are averaged to obtain the identity correlation between the two candidate frame sequences. The greater the identity correlation, the more similar the target identities contained in the two candidate frame sequences are. For candidate frame sequences with large cosine similarities (for example, greater than 0.5) on the three channels of facial features, body features, and voice features, identity indexes can be established between each other to indicate that they contain the same target identity. When the target portrait is used to search for the target video in the video later, all candidate frame sequences containing the target can be obtained.
[0057] In step S103, the target relevance between the target portrait and each candidate frame sequence is determined respectively by using the facial similarity and the identity relevance.
[0058] When determining the target correlation between the target portrait and each candidate frame sequence, firstly, the identity correlation between any candidate frame sequence and each candidate frame sequence is accumulated to obtain the total identity correlation of any candidate frame sequence. Then, the identity correlation between any candidate frame sequence and each candidate frame sequence is divided by the total identity correlation to obtain the normalized identity correlation between any candidate frame sequence and each candidate frame sequence. The calculation method of the normalized identity correlation is shown in formula (2):
[0059]
[0060] Among them, i represents any candidate frame sequence among all candidate frame sequences, ω jk represents the identity correlation between candidate frame sequence j and candidate frame sequence k, η jk Represents the normalized identity correlation between candidate frame sequence j and candidate frame sequence k.
[0061] Finally, the normalized identity correlation is used to weight the facial similarities between any candidate frame sequence and the corresponding candidate frame sequences, and the target correlation between the target portrait and any candidate frame sequence is obtained. The target correlation calculation method is shown in formula (3):
[0062]
[0063] Wherein, j represents any candidate frame sequence among all candidate frame sequences, and s'(q,k) represents the target correlation between the target portrait q and any candidate frame sequence k. Those skilled in the art should understand that in order to make the calculation result more accurate, formula (3) can be used only in n jk When the value is greater than the preset value, weighted calculation is performed.
[0064] Since the target only contains facial information, if the search is performed directly in the candidate frame sequence without constructing an index between the candidate frame sequences, only the candidate frame sequences in which the face appears can be searched. However, the target varies in the video and may have different postures. The target may be facing away from the camera or sideways to the camera. In this embodiment, multiple biometric features are used to accurately associate candidate frame sequences with the same target identity. When calculating the target correlation between the target portrait and the candidate frame sequence, the target correlation between the target portrait and multiple associated candidate frame sequences is also considered, thereby reducing interference caused by other factors such as viewing angle, and more accurately determining the target correlation between the target portrait and each candidate frame sequence, so that the target portrait can be used to more accurately search for the target video in the video.
[0065] Next, we will describe in detail how to determine the identity correlation between each candidate frame sequence.
[0066] A fully connected neural network is needed to determine the identity correlation between each candidate frame sequence, so it is necessary to train the fully connected neural network in advance. Figure 2 Describes how to train a fully connected neural network.
[0067] Figure 2 FIG. 1 shows a flow chart of an embodiment of training a fully connected neural network. Figure 2 As shown, this embodiment includes steps S201 to S203.
[0068] In step S201, a fully connected neural network is constructed.
[0069] Figure 3 Figure 2 shows a schematic diagram of the structure of a fully connected neural network. Figure 3 As shown, the fully connected neural network includes an input layer, a hidden layer, and an output layer. The input layer may include, for example, 2688 neurons for processing the input feature vector. The output layer may include, for example, 3 neurons for outputting facial feature weight α, identity feature weight β, and speech feature weight λ, respectively.
[0070] In step S202, a loss function is constructed using the Euclidean distance between the identity feature of the first candidate frame sequence and the identity feature of the second candidate frame sequence, wherein the first candidate frame sequence and the second candidate frame sequence are any candidate frame sequences.
[0071] The Euclidean distance between the identity feature of the first candidate frame sequence and the identity feature of the second candidate frame sequence can be calculated as shown in formula (4):
[0072] d=||(α m f m1 ,β m f m2 ,λ m f m3 )-(α n f n1 ,β n f n2 ,λ n f n3 )|| (4)
[0073] Where d represents the above Euclidean distance, m represents the first candidate frame sequence, n represents the second candidate frame sequence, and α m , β m , m represents the weights of the first candidate frame sequence on the three channels, α n , β n , n represents the weights of the second candidate frame sequence on the three channels, f m1 、f m2 、f m3 represents the identity features of the first candidate frame sequence on three channels, f n1 、f n2 、f n3 Indicates the identity features of the second candidate frame sequence on three channels. When the first candidate frame sequence and the second candidate frame sequence correspond to the same candidate target, the value of the loss function is positively correlated with the above Euclidean distance; when the first candidate frame sequence and the second candidate frame sequence correspond to different candidate targets, the value of the loss function is positively correlated with the difference between the above Euclidean distance and the preset value.
[0074] For example, the loss function can be specifically shown as formula (5):
[0075]
[0076] Wherein, L represents the loss function, N represents the number of identity feature pairs consisting of the identity features of the first candidate frame sequence and the identity features of the second candidate frame sequence; Y can take the value of 0 or 1, Y=1 indicates that the first candidate frame sequence and the second candidate frame sequence correspond to the same candidate target, and Y=0 indicates that the first candidate frame sequence and the second candidate frame sequence correspond to different candidate targets; δ represents a preset value, for example, it can be set to 1.0.
[0077] In step S203, the parameters of the fully connected neural network are adjusted to reduce the value of the loss function.
[0078] By adjusting the parameters of the fully connected neural network, the fully connected neural network can weight the various biometric features of the candidate frame sequences, output similar identity features for the same candidate frame sequences, and output identity features that are far apart for different candidate frame sequences.
[0079] Combine the following Figure 4 Describes how to use a fully connected neural network to determine the identity association between candidate frame sequences.
[0080] Figure 4 FIG. 2 is a flow chart showing an embodiment of determining the identity correlation between candidate frame sequences. Figure 4 As shown, this embodiment includes steps S4021 to S4023.
[0081] In step S4021, the facial features and other biometric features of each candidate frame sequence are cascaded and input into a pre-trained fully connected neural network to obtain the weights of the facial features and other biometric features of each candidate frame sequence.
[0082] The weights of other biometric features include the weights of each of the other biometric features. For example, the weights of other biometric features may include the weights of body features and the weights of voice features.
[0083] In step S4022, the identity features of each candidate frame sequence are determined respectively using the above weights, facial features of each candidate frame sequence, and other biological features.
[0084] For example, the facial features and other biometric features (identity features, voice features) of any candidate frame sequence are weighted using the facial feature weight α, identity feature weight β, and voice feature weight λ of any candidate frame sequence. Then, the weighted results are cascaded to obtain the identity features of any candidate frame sequence. The dimension of the identity feature can be, for example, 2688 dimensions.
[0085] In some embodiments, the facial features and other biometric features of any candidate frame sequence may be passed through a fully connected layer to obtain a new feature expression, and then the new feature expression of any candidate frame sequence may be weighted using the above weights, and the weighted results may be cascaded.
[0086] In step S4023, the identity correlation between the candidate frame sequences is determined using the identity features of the candidate frame sequences.
[0087] For example, first determine the sum of cosine similarities between the identity features of the third candidate frame sequence and the identity features of other candidate frame sequences. The third candidate frame sequence is any candidate frame sequence. Then, divide the cosine similarity between the identity features of the third candidate frame sequence and the identity features of the fourth candidate frame sequence by the sum of cosine similarities to obtain the identity correlation between the third candidate frame sequence and the fourth candidate frame sequence. The fourth candidate frame sequence is any other candidate frame sequence.
[0088] The inventors have taken into account that some frame sequences in the candidate frame sequence may not contain facial features, and the voices in some frame sequences may contain background music or the voices of other targets. These interference situations will result in a low accuracy in searching for targets in the video. To this end, the weights of different biometric features in the identity features of the candidate frame sequence are calculated through a fully connected neural network, which can measure the importance of different biometric features to the current target, thereby highlighting important biometric features and weakening non-important biometric features in the multimodal identity features. Therefore, this embodiment can further accurately determine the target correlation between the target portrait and each candidate frame sequence, is suitable for scenes with variable target activities, and improves the accuracy of searching for targets in videos.
[0089] Combine the following Figure 5 Some embodiments of the target search device disclosed herein are described.
[0090] Figure 5 FIG. 1 is a schematic diagram showing the structure of a target search device according to some embodiments of the present disclosure. Figure 5 As shown, the target search device 50 of this embodiment includes:
[0091] The facial similarity determination module 501 is configured to use the facial features of the target portrait and the facial features of each candidate frame sequence to determine the facial similarities between the target portrait and each candidate frame sequence, wherein each candidate frame in each candidate frame sequence corresponds to the same candidate target; the identity correlation determination module 502 is configured to use the facial features and other biometric features of each candidate frame sequence to determine the identity correlation between each candidate frame sequence; the target correlation determination module 503 is configured to use the facial similarity and the identity correlation to determine the target correlation between the target portrait and each candidate frame sequence, respectively.
[0092] In some embodiments, the identity correlation determination module 502 is configured to: cascade the facial features and other biometric features of each candidate frame sequence and input them into a pre-trained fully connected neural network to obtain the weights of the facial features and other biometric features of each candidate frame sequence; use the weights, the facial features and other biometric features of each candidate frame sequence to determine the identity features of each candidate frame sequence; use the identity features of each candidate frame sequence to determine the identity correlation between each candidate frame sequence.
[0093] In some embodiments, the identity relevance determination module 502 is configured to: weight the facial features and other biometric features of any candidate frame sequence respectively using the weight of any candidate frame sequence; and cascade the weighted results to obtain the identity features of any candidate frame sequence.
[0094] In some embodiments, the identity relevance determination module 502 is configured to: construct a fully connected neural network; construct a loss function using the Euclidean distance between the identity features of the first candidate frame sequence and the identity features of the second candidate frame sequence; the first candidate frame sequence and the second candidate frame sequence are any candidate frame sequences; wherein, when the first candidate frame sequence and the second candidate frame sequence correspond to the same candidate target, the value of the loss function is positively correlated with the Euclidean distance; when the first candidate frame sequence and the second candidate frame sequence correspond to different candidate targets, the value of the loss function is positively correlated with the difference between the Euclidean distance and a preset value; and adjust the parameters of the fully connected neural network to reduce the value of the loss function.
[0095] In some embodiments, the loss function is:
[0096]
[0097] Wherein, L represents the loss function, N represents the number of identity feature pairs consisting of the identity features of the first candidate frame sequence and the identity features of the second candidate frame sequence, Y=1 represents that the first candidate frame sequence and the second candidate frame sequence correspond to the same candidate target, and Y=0 represents that the first candidate frame sequence and the second candidate frame sequence correspond to different candidate targets; d represents the Euclidean distance; δ represents the preset value.
[0098] In some embodiments, the identity correlation determination module 502 is configured to: determine the sum of cosine similarities between the identity features of the third candidate frame sequence and the identity features of other candidate frame sequences; wherein the third candidate frame sequence is any candidate frame sequence; use the cosine similarity between the identity features of the third candidate frame sequence and the identity features of the fourth candidate frame sequence, divided by the sum of cosine similarities, to obtain the identity correlation between the third candidate frame sequence and the fourth candidate frame sequence; wherein the fourth candidate frame sequence is any other candidate frame sequence.
[0099] In some embodiments, the target correlation determination module 503 is configured to: accumulate the identity correlations between any candidate frame sequence and each candidate frame sequence respectively to obtain the total identity correlation of any candidate frame sequence; divide the identity correlations between any candidate frame sequence and each candidate frame sequence respectively by the total identity correlation to obtain the normalized identity correlations between any candidate frame sequence and each candidate frame sequence respectively; use the normalized identity correlations to weight the facial similarities between any candidate frame sequence and each corresponding candidate frame sequence respectively to obtain the target correlation between the target portrait and any candidate frame sequence.
[0100] In some embodiments, the facial similarity determination module 501 is configured to use the cosine similarities between the facial features of the target portrait and the facial features of each candidate frame sequence as the facial similarities between the target portrait and each candidate frame sequence.
[0101] In some embodiments, the other biometric features include at least one of body features and voice features, and the weights of the other biometric features include the weights of each of the other biometric features.
[0102] In some embodiments, it also includes a feature extraction module 500, which is configured to: use a facial detection neural network to extract facial features of each candidate frame in any candidate frame sequence; average the facial features of each candidate frame in any candidate frame sequence to obtain facial features of any candidate frame sequence; use a body detection neural network to extract body features of each candidate frame in any candidate frame sequence; average the body features of each candidate frame in any candidate frame sequence to obtain body features of any candidate frame sequence; extract audio from any candidate frame sequence; use a speech recognition neural network to extract speech features from the audio to obtain speech features of any candidate frame sequence.
[0103] Combine the following Figure 6 Some embodiments of the target search device disclosed herein are described.
[0104] Figure 6 FIG. 2 is a schematic diagram showing the structure of a target search device in some other embodiments of the present disclosure. Figure 6 As shown, the target search device 60 of this embodiment includes: a memory 610 and a processor 620 coupled to the memory 610, and the processor 620 is configured to execute the target search method in any of the aforementioned embodiments based on the instructions stored in the memory 610.
[0105] The memory 610 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, and other programs.
[0106] The target search device 60 may also include an input / output interface 630, a network interface 640, a storage interface 650, etc. These interfaces 630, 640, 650, the memory 610, and the processor 620 may be connected, for example, via a bus 660. The input / output interface 630 provides a connection interface for input / output devices such as a display, a mouse, a keyboard, and a touch screen. The network interface 640 provides a connection interface for various networked devices. The storage interface 650 provides a connection interface for external storage devices such as SD cards and USB flash drives.
[0107] The present disclosure also includes a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the target search method in any of the aforementioned embodiments.
[0108] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0109] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0110] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0111] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.
Claims
1. A target search method, comprising: Determine the facial similarity between the target portrait and each candidate frame sequence respectively by using the facial features of the target portrait and the facial features of each candidate frame sequence; wherein each candidate frame in each candidate frame sequence corresponds to the same candidate target; Using the facial features and other biometric features of each candidate frame sequence, the identity correlation between each candidate frame sequence is determined, including: The facial features and other biometric features of each candidate frame sequence are concatenated and input into a pre-trained fully connected neural network to obtain the weights of the facial features and other biometric features of each candidate frame sequence; Determine the identity features of each candidate frame sequence respectively by using the weights, facial features of each candidate frame sequence and other biometric features; Determine the identity correlation between the candidate frame sequences by using the identity features of the candidate frame sequences; The target relevance between the target portrait and each candidate frame sequence is determined respectively by using the facial similarity and the identity relevance.
2. The target search method according to claim 1, wherein: The step of using the weights, facial features of each candidate frame sequence, and other biological features to respectively determine the identity features of each candidate frame sequence comprises: Using the weight of any candidate frame sequence, weighting the facial features and other biometric features of any candidate frame sequence respectively; The weighted results are cascaded to obtain the identity features of any candidate frame sequence.
3. The target search method according to claim 2, further comprising: Build a fully connected neural network; A loss function is constructed using the Euclidean distance between the identity feature of the first candidate frame sequence and the identity feature of the second candidate frame sequence; the first candidate frame sequence and the second candidate frame sequence are any candidate frame sequences; Wherein, when the first candidate frame sequence and the second candidate frame sequence correspond to the same candidate target, the value of the loss function is positively correlated with the Euclidean distance; when the first candidate frame sequence and the second candidate frame sequence correspond to different candidate targets, the value of the loss function is positively correlated with the difference between the Euclidean distance and a preset value; The parameters of the fully connected neural network are adjusted to reduce the value of the loss function.
4. The target search method according to claim 3, wherein: The loss function is: Wherein, L represents the loss function, N represents the number of identity feature pairs formed by the identity features of the first candidate frame sequence and the identity features of the second candidate frame sequence, Indicates that the first candidate frame sequence and the second candidate frame sequence correspond to the same candidate target, indicates that the first candidate frame sequence and the second candidate frame sequence correspond to different candidate targets; d indicates the Euclidean distance; Indicates the preset value.
5. The target search method according to claim 1, wherein: Determining the identity correlation between the candidate frame sequences by using the identity features of the candidate frame sequences includes: Determine the sum of cosine similarities between the identity features of the third candidate frame sequence and the identity features of other candidate frame sequences; wherein the third candidate frame sequence is any candidate frame sequence; The identity correlation between the third candidate frame sequence and the fourth candidate frame sequence is obtained by dividing the cosine similarity between the identity features of the third candidate frame sequence and the identity features of the fourth candidate frame sequence by the sum of the cosine similarities; wherein the fourth candidate frame sequence is any of the other candidate frame sequences.
6. The target search method according to claim 1, wherein: The step of using the facial similarity and the identity correlation to respectively determine the target correlation between the target portrait and each candidate frame sequence comprises: Accumulating the identity correlations between any candidate frame sequence and each candidate frame sequence respectively to obtain the total identity correlation of any candidate frame sequence; Dividing the identity correlation between any candidate frame sequence and each candidate frame sequence by the total identity correlation to obtain a normalized identity correlation between any candidate frame sequence and each candidate frame sequence; The normalized identity correlation is used to weight the facial similarities between any candidate frame sequence and corresponding candidate frame sequences, respectively, to obtain a target correlation between the target portrait and any candidate frame sequence.
7. The target search method according to claim 1, wherein: Determining the facial similarities between the target portrait and each candidate frame sequence using the facial features of the target portrait and the facial features of each candidate frame sequence includes: The cosine similarities between the facial features of the target portrait and the facial features of each candidate frame sequence are used as the facial similarities between the target portrait and each candidate frame sequence.
8. The target search method according to any one of claims 1 to 7, wherein: The other biometric features include at least one of a body feature and a voice feature, and the weights of the other biometric features include the weights of each of the other biometric features.
9. The target search method according to claim 8, further comprising: Using a face detection neural network, facial features of each candidate frame in any candidate frame sequence are extracted respectively; An average operation is performed on the facial features of each candidate frame in any candidate frame sequence to obtain the facial features of any candidate frame sequence; Using a body detection neural network, respectively extracting body features of each candidate frame in any candidate frame sequence; averaging the body features of each candidate frame in any candidate frame sequence to obtain the body features of any candidate frame sequence; Extracting audio from any of the candidate frame sequences; extracting speech features from the audio using a speech recognition neural network to obtain speech features of any of the candidate frame sequences.
10. A target search device, comprising: A facial similarity determination module is configured to determine facial similarities between the target portrait and each candidate frame sequence using facial features of the target portrait and facial features of each candidate frame sequence; wherein each candidate frame in each candidate frame sequence corresponds to the same candidate target; The identity correlation determination module is configured to use the facial features and other biometric features of each candidate frame sequence to determine the identity correlation between each candidate frame sequence, including: cascading the facial features and other biometric features of each candidate frame sequence and inputting them into a pre-trained fully connected neural network to obtain the weights of the facial features and other biometric features of each candidate frame sequence, respectively, using the weights, the facial features and other biometric features of each candidate frame sequence, respectively determining the identity features of each candidate frame sequence, and using the identity features of each candidate frame sequence to determine the identity correlation between each candidate frame sequence; The target relevance determination module is configured to use the facial similarity and the identity relevance to respectively determine the target relevance between the target portrait and each candidate frame sequence.
11. A target search device, comprising: Memory; as well as A processor coupled to the memory, wherein the processor is configured to execute the target search method according to any one of claims 1 to 9 based on instructions stored in the memory.
12. A computer-readable storage medium, wherein: The computer-readable storage medium stores computer instructions, and when the instructions are executed by a processor, the target search method according to any one of claims 1 to 9 is implemented.
Citation Information
Patent Citations
Method and apparatus for searching for target person in video
CN106127106A
Surveillance video character identity identification system and method thereof fusing facial multi-angle feature
CN106503687A
Intelligent detection tracking and spatial-temporal trajectory generation method for specific object
CN107016374A
A method for identifying a video character
CN109359570A
Video recognition method and device, computer device and storage medium
CN109376603A
