Personnel identification method, device, equipment, pedestrian monitoring system and storage medium

Abstract

The present invention relates to the field of artificial intelligence and computer vision processing technology, and discloses a personnel identification method, device, equipment, pedestrian monitoring system and storage medium. That is, after obtaining an image of a person to be inspected, the image of the person to be inspected and a given target person image can be respectively obtained by a feature fusion method based on the fusion of global features and key point features to obtain corresponding feature vectors, and then in the image similarity difference comparison process based on the obtained feature vectors, the vector inner product method is calculated by matrix multiplication to achieve similarity comparison, and finally the person identification result is obtained. Not only can the robustness of feature description in complex scenes such as image distortion, person posture change or occlusion be improved, ensuring that the occlusion scene and person posture change can be effectively dealt with, the accuracy and effectiveness of personnel identification can be improved, but also the calculation complexity can be reduced, and the identification result can be obtained quickly.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and computer vision processing technology, and specifically relates to a personnel recognition method, device, equipment, pedestrian monitoring system and storage medium. Background Art

[0002] With the rapid development of artificial intelligence and the increasing demand for intelligent services in society, as an important transportation carrier and key channel in public places of residential buildings, the demand for intelligent video surveillance of elevators is also increasing day by day. Pedestrians are one of the important monitoring objects in security systems. Analyzing the rich information in elevator monitoring videos based on the characteristics of pedestrians and using machine vision has important application value and significance for building intelligent monitoring systems.

[0003] The video images in the elevator car are often distorted due to reasons such as the camera's viewing angle and position, which makes it difficult to conduct research on target detection and attribute recognition in the later stage. Intelligent pedestrian recognition and screening refers to the process of re-identifying a pedestrian from a large number of images captured in surveillance videos or other paths given a pedestrian image to be found (i.e., target image). It has been a research hotspot in the field of computer vision in recent years and belongs to the category of image retrieval problems. It is widely used in the fields of intelligent video surveillance and intelligent security. The key to identifying and retrieving target pedestrians in cross-temporal and spatial fields is to select appropriate pedestrian feature extraction methods and metric learning methods for feature comparison to explore the similarity between two images.

[0004] Traditional solutions are mainly implemented through the method of "manually designed features + distance metrics", in which feature extraction and representation methods can be roughly divided into three categories: low-level feature expression, mid- and high-level feature learning, and hybrid model feature expression. Metric learning methods include Mahalanobis distance, local adaptive decision function, and saliency weighted metric learning. Due to complex challenges such as scene occlusion or posture changes, the above traditional low-level visual feature extraction methods are difficult to obtain strong robust features, which in turn affects the accuracy and effectiveness of metric learning methods based on such features.

[0005] In recent years, machine learning technologies such as neural networks have been widely used in the field of digital image processing. Pedestrian feature extraction methods based on deep learning can be roughly divided into local region-based methods, video sequence-based methods, and generative adversarial neural network-based methods. Among them, local region-based methods mainly include methods that use grid features, methods that cut images into regular blocks such as horizontal stripes, and methods that divide local regions according to posture information / semantic segmentation. Although the differences between partitions can be effectively observed, there are still great challenges for problems such as occlusion and posture misalignment, and the ability to adapt to large changes in pedestrian image posture is not strong. Most methods based on generative adversarial networks are used to enhance data sets or achieve style transfer between different cameras. They are generally combined with other types of methods to complete the recognition process. Therefore, there is no improvement effect when facing problems such as occlusion and posture misalignment.

[0006] Therefore, for the video images in the elevator car, how to solve the problem of passenger identification and retrieval in the case of image distortion or scene changes, and effectively deal with situations such as occluded scenes and changes in passenger postures, and improve the accuracy of passenger identification, is a topic that technical personnel in this field urgently need to study. Summary of the invention

[0007] In order to solve the problem that the existing pedestrian intelligent recognition and screening technology has poor accuracy and effectiveness in personnel recognition under conditions of image distortion or scene changes, the purpose of the present invention is to provide a personnel recognition method, device, computer equipment, pedestrian monitoring system and computer-readable storage medium, which can not only improve the robustness of feature description in complex scenes such as image distortion, personnel posture changes or occlusion, ensure that it can effectively cope with occlusion scenes and personnel posture changes, and improve the accuracy and effectiveness of personnel recognition, but also reduce the computational complexity and quickly obtain recognition results, which is beneficial to improve the intuitiveness of the expression of recognition results of multiple personnel images compared with the existing technology of comparing features one by one, and meet the requirements of fast retrieval.

[0008] In a first aspect, the present invention provides a method for identifying a person, comprising:

[0009] Acquire an image of the person to be inspected;

[0010] For the image of the person to be inspected and the given target person image, a corresponding global feature map is acquired based on a convolutional neural network;

[0011] For the image of the person to be inspected and the image of the target person, based on a human body key point model and a corresponding global feature map, a corresponding plurality of local features are acquired;

[0012] For the image of the person to be inspected and the image of the target person, the corresponding multiple local features are concatenated with the global features in the corresponding global feature map after average pooling processing to obtain a corresponding feature row vector containing multi-dimensional feature values;

[0013] Combining the characteristic row vectors of the image of the person to be examined and the image of the target person into a characteristic matrix, wherein each row element in the characteristic matrix is ​​a characteristic row vector of the image of the person to be examined or a characteristic row vector of the image of the target person;

[0014] Normalizing the feature matrix to obtain a normalized feature matrix;

[0015] Calculate the inner product of the normalized feature matrix and its transposed matrix to obtain a similar cosine square matrix;

[0016] According to the row positions of the feature row vectors of the image of the person to be inspected and the feature row vectors of the target person image in the feature matrix, elements of corresponding row and column positions are found in the similar cosine matrix, and the elements are used as the person similarity between the image of the person to be inspected and the target person image.

[0017] Based on the above invention content, a personnel identification scheme for realizing the fusion of global features and key point features is provided, that is, after obtaining the image of the person to be inspected, the corresponding feature vectors can be obtained respectively by feature fusion means based on the fusion of global features and key point features for the image of the person to be inspected and the given target person image, and then in the process of image similarity difference comparison based on the obtained feature vectors, the similarity comparison is realized by calculating the vector inner product method by matrix multiplication, and finally the person identification result is obtained, which can not only improve the robustness of feature description in complex scenes such as image distortion, person posture change or occlusion, ensure that it can effectively deal with occlusion scenes and person posture changes, and improve the accuracy and effectiveness of person identification, but also reduce the calculation complexity, and quickly obtain the identification result, which is beneficial to improve the intuitiveness of the expression of the identification results of multiple person images compared with the existing technology of comparing features one by one, and meet the requirements of fast retrieval. In addition, since the global feature map extracted by the backbone network and the local features based on the human key point model are combined in the feature vector extraction process, the robustness of the features is enhanced compared with the method of simply extracting global features; and since cosine similarity is used as the basis for judging the relative differences between images, it is simple to operate and is not affected by the absolute differences between the eigenvalues, compared with the method of using Euclidean distance renormalization, it is easy to apply and promote in practice.

[0018] In a possible design, if multiple images of the person to be inspected are obtained, after obtaining the similar cosine matrix, the method further includes:

[0019] According to the row position of the characteristic row vector of the target person image in the characteristic matrix, searching for a plurality of elements of corresponding row positions in the similar cosine square matrix;

[0020] Sort all elements in the plurality of elements by value from large to small to obtain an element queue;

[0021] According to the order of the elements corresponding to the images of the persons to be inspected in the element queue, all the images of the persons to be inspected among the multiple images of the persons to be inspected are ordered in a consistent manner to obtain a queue of images of the persons to be inspected;

[0022] The images in the queue of images of the person to be examined are adjusted to images of uniform size, and are displayed simultaneously by horizontal stacking.

[0023] Based on the above possible designs, the similarity comparison can be realized by calculating the vector inner product through matrix multiplication, and the images of the persons to be inspected with the highest similarity can be screened and sorted to display, so that when the database to be inspected is large, the calculation redundancy can be effectively reduced, the time spent can be shortened, and the practicality of intelligent identification and screening of pedestrians can be enhanced. In addition, compared with the existing method of comparing features one by one, the expression is intuitive and the calculation complexity is greatly reduced, which can better meet the requirements of fast retrieval.

[0024] In a possible design, obtaining an image of a person to be inspected includes:

[0025] Acquire surveillance video images;

[0026] Calling a target detection algorithm to perform personnel detection processing on the surveillance video image, and obtaining a position of a personnel detection result mark box in the surveillance video image;

[0027] According to the position of the person detection result mark box, the image in the box is cut out from the monitoring video image as the image of the person to be detected.

[0028] In a possible design, for the image of the person to be inspected and the given target person image, a corresponding global feature map is obtained based on a convolutional neural network, including:

[0029] The person image is input into a convolutional neural network, and the input image is encoded into a global feature map through multi-layer convolution, pooling and sampling processing of the convolutional neural network, wherein the person image is the image of the person to be inspected or a given target person image, and the step size of the last spatial downsampling of the convolutional neural network is set to one.

[0030] In a possible design, for the image of the person to be inspected and the image of the target person, a plurality of corresponding local features are obtained based on a human key point model and a corresponding global feature map, including:

[0031] Acquire multiple local heat maps of a person image based on a human key point model, wherein the person image is the image of the person to be inspected or the image of the target person;

[0032] For each of the multiple local heat maps, the global feature map of the person image and the corresponding heat map are first multiplied at the pixel level, and then average pooled to obtain the corresponding local features.

[0033] In a possible design, the feature matrix is ​​normalized to obtain a normalized feature matrix, including:

[0034] The feature matrix is ​​normalized according to the mean and standard deviation of the ImageNet project to obtain a normalized feature matrix.

[0035] In a second aspect, the present invention provides a personnel identification device, comprising an image acquisition module, a global feature extraction module, a local feature extraction module, a feature vector splicing module, a feature matrix combination module, a normalization processing module, a square matrix calculation module and a personnel similarity determination module;

[0036] The image acquisition module is used to acquire the image of the person to be examined;

[0037] The global feature extraction module is communicatively connected to the image acquisition module, and is used to acquire corresponding global feature maps based on a convolutional neural network for the image of the person to be inspected and a given target person image;

[0038] The local feature extraction module is communicatively connected to the image acquisition module, and is used to acquire corresponding multiple local features for the image of the person to be inspected and the image of the target person based on the human body key point model and the corresponding global feature map;

[0039] The feature vector splicing module is respectively connected to the global feature extraction module and the local feature extraction module for performing series splicing of the corresponding multiple local features with the global features in the corresponding global feature map after average pooling processing for the image of the person to be inspected and the image of the target person, so as to obtain a corresponding feature row vector containing multi-dimensional feature values;

[0040] The feature matrix combination module is communicatively connected to the feature vector splicing module, and is used to combine the feature row vectors of the image of the person to be examined and the image of the target person into a feature matrix, wherein each row element in the feature matrix is ​​a feature row vector of the image of the person to be examined or a feature row vector of the image of the target person;

[0041] The normalization processing module is communicatively connected to the feature matrix combination module and is used to perform normalization processing on the feature matrix to obtain a normalized feature matrix;

[0042] The square matrix calculation module is communicatively connected to the normalization processing module and is used to calculate the inner product of the normalized feature matrix and its transposed matrix to obtain a similar cosine square matrix;

[0043] The personnel similarity determination module is communicatively connected to the feature matrix combination module and the square matrix calculation module, and is used to find the elements of the corresponding row and column positions in the similar cosine square matrix according to the row positions of the feature row vectors of the image of the person to be tested and the feature row vectors of the target person image in the feature matrix, and use the elements as the personnel similarity between the image of the person to be tested and the target person image.

[0044] In a possible design, if the image acquisition module acquires multiple images of the person to be examined, it also includes a square matrix element search module, an element sorting module, an image sorting module and an image display module;

[0045] The square matrix element search module is respectively connected to the feature matrix combination module and the square matrix calculation module for searching for a plurality of elements at corresponding row positions in the similar cosine square matrix according to the row positions of the feature row vectors of the target person image in the feature matrix;

[0046] The element sorting module is communicatively connected to the square matrix element searching module and is used to sort all the elements of the multiple elements from large to small values ​​to obtain an element queue;

[0047] The image sorting module is respectively connected to the image acquisition module and the element sorting module for performing consistency sorting on all the images of the persons to be inspected among the multiple images of the persons to be inspected according to the sorting of the elements corresponding to the images of the persons to be inspected in the element queue, so as to obtain a queue of images of the persons to be inspected;

[0048] The image display module is communicatively connected to the image sorting module, and is used to adjust the images in the queue of images of the person to be examined into images of a uniform size, and display them simultaneously in a horizontal stacking manner.

[0049] In a third aspect, the present invention provides a computer device comprising a memory and a processor that are communicatively connected, wherein the memory is used to store a computer program, and the processor is used to read the computer program to execute the personnel identification method as described in the first aspect or any possible design of the first aspect.

[0050] In a fourth aspect, the present invention provides a pedestrian monitoring system, comprising a camera, a video server and a personal computer PC which are sequentially connected in communication;

[0051] The camera is used to be arranged in a pedestrian monitoring location and collect video signals of the pedestrian monitoring location;

[0052] The video server is used to digitally convert the collected video signal to obtain a video stream in a unified video format;

[0053] The personal computer PC is used to execute the personnel identification method as described in the first aspect or any possible design of the first aspect after receiving the video stream, and determine whether the target person appears in the pedestrian monitoring place according to the personnel identification result.

[0054] In a fifth aspect, the present invention provides a computer-readable storage medium having instructions stored thereon. When the instructions are run on a computer, the personnel identification method as described in the first aspect or any possible design of the first aspect is executed.

[0055] In a sixth aspect, the present invention provides a computer program product comprising instructions, which, when executed on a computer, causes the computer to execute the personnel identification method as described in the first aspect or any possible design of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0057] Figure 1 It is a flow chart of the personnel identification method provided by the present invention.

[0058] Figure 2 It is a flowchart of the work flow of the target detection algorithm used in the personnel recognition method provided by the present invention.

[0059] Figure 3 It is a flowchart of the workflow for acquiring feature vectors in the personnel identification method provided by the present invention.

[0060] Figure 4 It is a flowchart of the workflow for sorting and displaying multiple personnel images in the personnel identification method provided by the present invention.

[0061] Figure 5 It is a structural schematic diagram of the personnel identification device provided by the present invention.

[0062] Figure 6 It is a structural schematic diagram of the computer device provided by the present invention.

[0063] Figure 7 It is a structural schematic diagram of the pedestrian monitoring system provided by the present invention.

[0064] In the above drawings: 1-camera; 2-video server; 3-personal computer PC; 4-elevator car; 5-pedestrian. DETAILED DESCRIPTION

[0065] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments. It should be noted that although the description of these embodiments is used to help understand the present invention, it does not constitute a limitation of the present invention. The specific structures and functional details disclosed herein are only used to describe the embodiments of the present invention. However, the present invention can be embodied in many alternative forms, and it should not be understood that the present invention is limited to the embodiments set forth herein.

[0066] It should be understood that although the terms first and second, etc. may be used herein to describe various objects, these objects should not be limited by these terms. These terms are only used to distinguish one object from another object. For example, a first object can be referred to as a second object, and similarly, a second object can be referred to as a first object without departing from the scope of the exemplary embodiments of the present invention.

[0067] It should be understood that the term "and / or" that may appear in this article is merely a description of the association relationship between associated objects, indicating that there may be three relationships. For example, A and / or B can represent three situations: A exists alone, B exists alone, or A and B exist at the same time; the term " / and" that may appear in this article describes another type of association object relationship, indicating that there may be two relationships. For example, A / and B can represent two situations: A exists alone or A and B exist at the same time; in addition, the character " / " that may appear in this article generally indicates that the previous and next associated objects are in an "or" relationship.

[0068] like Figure 1As shown, the personnel identification method provided in the first aspect of this embodiment can be, but is not limited to, executed by a computer device with certain computing resources, such as a personal computer (PC, which refers to a multi-purpose computer with a size, price and performance suitable for personal use; desktops, laptops, small laptops, tablets and ultrabooks are all personal computers), smart phones, personal digital assistants (PDAs), etc. The process is executed by an electronic device such as a digital assistant (PAD) or a wearable device, so that after obtaining the image of the person to be inspected, the corresponding feature vectors can be obtained for the image of the person to be inspected and the given target person image respectively by means of feature fusion based on the fusion of global features and key point features. Then, in the process of image similarity difference comparison based on the obtained feature vectors, the vector inner product method is used to calculate the similarity by matrix multiplication, and finally the person recognition result is obtained. This can not only improve the robustness of feature description in complex scenes such as image distortion, person posture change or occlusion, ensure that it can effectively deal with occlusion scenes and person posture changes, and improve the accuracy and effectiveness of person recognition, but also reduce the computational complexity and quickly obtain the recognition result. Compared with the existing technology of comparing features one by one, it is beneficial to improve the intuitiveness of the expression of the recognition results of multiple person images and meet the requirements of fast retrieval. Figures 1 to 3 As shown, the personnel identification method may include, but is not limited to, the following steps S1 to S8.

[0069] S1. Obtain an image of the person to be inspected.

[0070] In step S1, the image of the person to be inspected is a person image captured from a surveillance video or captured in other ways during the process of intelligent identification and screening of pedestrians, so as to compare the person with a given target person image to determine whether the person to be inspected in the image of the person to be inspected is the target person. In addition, the image of the person to be inspected can be, but is not limited to, read from a storage unit or transmitted by a transmission unit.

[0071] In the step S1, specifically, obtaining an image of the person to be examined includes but is not limited to the following steps S11 to S13.

[0072] S11. Obtain surveillance video images.

[0073] In step S11, the monitoring video image may be, but is not limited to, a real-time image or a historical image captured by a monitoring camera located in the elevator car, which may be obtained through conventional transmission, digital conversion, decoding, and the like.

[0074] S12. Calling a target detection algorithm to perform personnel detection processing on the surveillance video image, and obtaining the position of the personnel detection result mark box in the surveillance video image.

[0075] In the step S12, the target detection algorithm is an existing artificial intelligence recognition algorithm for identifying objects in a picture and marking the position of the objects. Specifically, it can be but not limited to Faster R-CNN (Faster Regions with Convolutional Neural Networks features, a target detection algorithm proposed by He Kaiming et al. in 2015, which won multiple first places in the 2015 ILSVRV and COCO competitions) target detection algorithm, SSD (Single Shot MultiBox Detector, a target detection algorithm proposed by Wei Liu at ECCV 2016, which is one of the currently popular main detection frameworks) target detection algorithm or YOLO (You only look once, currently the latest version has been developed to V4, and is also widely used in the industry. Its basic principle is: first, the input image is divided into 7x7 grids, and 2 borders are predicted for each grid. Then, according to the threshold, the target window with a relatively low possibility is removed, and finally, the redundant window is removed by border merging to obtain the detection result) target detection algorithm, etc., wherein the Faster The R-CNN target detection algorithm is a two-stage detection method based on candidate regions, and the SSD target detection algorithm and the YOLO target detection algorithm are single-stage detection methods that directly perform classification and regression based on feature extraction. The workflow diagram of the target detection algorithm can be shown as follows: Figure 2 As shown, the image to be tested is firstly gridded, and multiple prediction values ​​are generated for the bounding box corresponding to each grid, then the score of the bounding box is calculated according to the confidence, and then the target bounding box with low score is removed by threshold comparison, and then the intersection and union ratio of the target bounding box and the real bounding box is calculated, and the bounding boxes with high overlap but relatively inaccurate predicted positions are quickly filtered out by non-maximum suppression (NMS, as the name implies, suppressing elements that are not maximum values, which can be understood as local maximum search), and finally the detection result is output. Therefore, through the conventional detection processing method, the position, category, and prediction probability of the identified person can be detected from the surveillance video image, and then the identification information such as a rectangular box and a label can be drawn for the identified person, and the person detection result mark box (that is, the bounding box of the area where the identified person is located in the surveillance video image) and its position in the surveillance video image are obtained.

[0076] S13. According to the position of the person detection result mark box, the image in the box is cut out from the monitoring video image as the image of the person to be detected.

[0077] After step S13, the image in the frame captured from the surveillance video image may be saved to a designated path so as to construct a database to be detected for comparison with the target person image.

[0078] S2. For the image of the person to be inspected and the given target person image, a corresponding global feature map is obtained based on a convolutional neural network.

[0079] In step S2, the convolutional neural network (CNN) is a type of feedforward neural network (CNN) that includes convolution calculation and has a deep structure. It is one of the representative algorithms of deep learning. It is used as a backbone network for feature vector extraction, and can be, but not limited to, a basic convolutional network structure such as a residual network ResNet50. Specifically, for the image of the person to be inspected and the given target person image, the corresponding global feature map is obtained based on the convolutional neural network, including but not limited to: inputting the person image into the convolutional neural network, and encoding the input image into a global feature map through multi-layer convolution, pooling and sampling processing of the convolutional neural network, wherein the person image is the image of the person to be inspected or the given target person image, and the step size of the last spatial downsampling of the convolutional neural network is set to one. Since the step size of the last spatial downsampling of the convolutional neural network is set to one, it can neither increase the number of parameters nor change the model structure, but also expand the size of the feature map, extract more fine-grained features, thereby improving the model performance and the resolution of the global feature map.

[0080] S3. For the image of the person to be inspected and the image of the target person, a corresponding plurality of local features are acquired based on a human key point model and a corresponding global feature map.

[0081] In step S3, the human key point model is a commonly used algorithm model in existing human key point detection research, and can be, but not limited to, the algorithm model in the OpenPose software of the human posture recognition project or the HRNet neural network model (which is a new human posture estimation model released by USTC and Microsoft Research Asia, with a unique parallel structure, which can maintain high-resolution representation at any time). The local features can be obtained through key point heat maps and uniform pooling processing, that is, for each local area, in order to obtain local features of several groups of key point areas, the global feature map extracted by the backbone network and the local heat map (i.e., key point heat map) extracted by the human key point model can be multiplied at the pixel level. Specifically, for the image of the person to be inspected and the image of the target person, based on the human key point model and the corresponding global feature map, a plurality of corresponding local features are obtained, including but not limited to the following steps S31 to S32: S31. Based on the human key point model, a plurality of local heat maps of the person image are obtained, wherein the person image is the image of the person to be inspected or the image of the target person; S32. For each of the plurality of local heat maps, the global feature map of the person image and the corresponding heat map are first subjected to pixel-level product processing, and then average pooling processing is performed to obtain the corresponding local features. The purpose of the average pooling processing is to reduce the data dimension, reduce the training parameters, enhance the generalization ability of the model, and thus prevent overfitting while retaining the features. Compared with the maximum pooling processing method, the average pooling processing can better represent the background information of the image. In addition, this embodiment does not limit the execution order of the step S3 and the step S2, and the step S3 can be executed before, after or simultaneously with the step S2.

[0082] S4. For the image of the person to be inspected and the image of the target person, the corresponding multiple local features are concatenated with the global features in the corresponding global feature map after average pooling processing to obtain a corresponding feature row vector containing multi-dimensional feature values.

[0083] In step S4, the serial splicing method is a common method in existing feature splicing and fusion technology. The dimension of the feature row vector can be represented by a positive integer m, which depends on the convolution kernel size of each convolution layer in the selected convolutional neural network.

[0084] S5. Combining the characteristic row vectors of the image of the person to be examined and the image of the target person into a characteristic matrix, wherein each row element in the characteristic matrix is ​​a characteristic row vector of the image of the person to be examined or a characteristic row vector of the image of the target person.

[0085] In step S5, for example, the first row element in the feature matrix is ​​the feature row vector of the target person image, and the second row element in the feature matrix is ​​the feature row vector of the image of the person to be inspected. In addition, there may be multiple images of the person to be inspected, and in this case, the feature matrix may have (n+1)×m elements, where n is a positive integer and represents the total number of images of the person to be inspected.

[0086] S6. Normalize the feature matrix to obtain a normalized feature matrix.

[0087] In the step S6, since there is a situation that the value range of the elements of each dimension in the characteristic row vector is inconsistent, it is necessary to perform normalization before performing inner product calculation. The normalization can be performed according to the value range of each dimensional element. Conventional normalization calculation can also be performed according to the mean and standard deviation of the ImageNet project (it is a large-scale visualization database for visual object recognition software research, wherein more than 14 million image URLs are manually annotated by ImageNet to indicate the object in the picture, and in at least one million images, a bounding box is also provided), the feature matrix is ​​normalized to obtain a normalized feature matrix, that is, the normalization calculation formula can be as follows:

[0088]

[0089] In the formula, featurea represents the element value after normalization, features represents the element value before normalization, mean represents the mean of the ImageNet project, and std represents the standard deviation of the ImageNet project.

[0090] S7. Calculate the inner product of the normalized feature matrix and its transposed matrix to obtain a similar cosine square matrix.

[0091] In step S7, the transposed matrix is ​​the transposed matrix of the normalized feature matrix, so the inner product can be calculated by conventional linear algebra to obtain the similar cosine square matrix with (n+1)×(n+1) elements.

[0092] S8. According to the row positions of the characteristic row vectors of the image of the person to be inspected and the characteristic row vectors of the target person image in the characteristic matrix, respectively, find the elements of the corresponding row and column positions in the similar cosine matrix, and use the elements as the person similarity between the image of the person to be inspected and the target person image.

[0093] In step S8, since the elements in the similar cosine matrix represent the inner product of the corresponding feature row vectors of the two images and their transposed column vectors, the cosine distance of the two vectors can be expressed under the condition of feature normalization. Therefore, the element can be used to measure the similarity distance of the corresponding images, that is, to measure the similarity between them by comparing the cosine values ​​of the angles between the vectors; specifically, the closer the element value (i.e., the cosine value) is to 1, the closer the angle between the corresponding feature row vectors of the two images is to 0 degrees, and the more similar the two feature row vectors are. Then, the elements in the similar cosine matrix corresponding to the image of the person to be inspected and the image of the target person can be used as the similarity of the persons in the two images. When the similarity exceeds a preset threshold, the person to be inspected in the image of the person to be inspected can be determined to be the target person, thereby achieving the purpose of intelligent identification and screening of pedestrians. For example, if the row position of the feature row vector of the image of the person to be inspected in the feature matrix is ​​the i-th row, and the row position of the feature row vector of the target person image in the feature matrix is ​​the j-th row, then the i-th row and j-th column element or the i-th column and j-th row element can be found in the similar cosine matrix as the person similarity between the image of the person to be inspected and the target person image, where i and j are positive integers respectively.

[0094] Therefore, based on the personnel identification method described in the aforementioned steps S1 to S8, a personnel identification scheme for realizing the fusion of global features and key point features is provided, that is, after obtaining the image of the person to be inspected, the corresponding feature vectors can be obtained respectively by feature fusion means based on the fusion of global features and key point features for the image of the person to be inspected and the given target person image, and then in the process of image similarity difference comparison based on the obtained feature vectors, the similarity comparison is realized by calculating the vector inner product method by matrix multiplication, and finally the person identification result is obtained, which can not only improve the robustness of feature description in complex scenes such as image distortion, person posture change or occlusion, ensure that it can effectively deal with occlusion scenes and person posture changes, and improve the accuracy and effectiveness of person identification, but also reduce the computational complexity and quickly obtain the identification result, which is beneficial to improve the intuitiveness of the expression of the identification results of multiple person images compared with the existing technology of comparing features one by one, and meet the requirements of fast retrieval. In addition, since the global feature map extracted by the backbone network and the local features based on the human key point model are combined in the feature vector extraction process, the robustness of the features is enhanced compared with the method of simply extracting global features; and since cosine similarity is used as the basis for judging the relative differences between images, it is simple to operate and is not affected by the absolute differences between the eigenvalues, compared with the method of using Euclidean distance renormalization, it is easy to apply and promote in practice.

[0095] Based on the technical solution of the first aspect, this embodiment further provides a possible design for screening and sorting multiple images of persons to be examined. Figure 4 As shown, if multiple images of persons to be inspected are acquired, after obtaining the similar cosine matrix, the method further includes but is not limited to the following steps S91 to S94.

[0096] S91. According to the row position of the characteristic row vector of the target person image in the characteristic matrix, find a plurality of elements corresponding to the row position in the similar cosine matrix.

[0097] In the step S91, if the row position of the feature row vector of the target person image in the feature matrix is ​​the first row, multiple elements of the first row can be found in the similar cosine matrix.

[0098] S92. Sort all elements in the multiple elements by numerical value from large to small to obtain an element queue.

[0099] S93. According to the order of the elements corresponding to the images of the persons to be examined in the element queue, all the images of the persons to be examined in the plurality of images of the persons to be examined are ordered in a consistent manner to obtain a queue of images of the persons to be examined.

[0100] In step S93, the consistency sorting means that the serial number of the image of the person to be examined in the queue of images of the person to be examined and the serial number of the element corresponding to the image of the person to be examined in the element queue are the same, thereby ensuring that the image of the person to be examined with a higher similarity is arranged at the front in the queue of images of the person to be examined for priority display.

[0101] S94. Adjust the images in the queue of images of the person to be examined to images of a uniform size, and display them simultaneously by horizontal stacking.

[0102] Based on the possible design 1 described in the aforementioned steps S91 to S94, the vector inner product can be calculated by matrix multiplication to achieve the comparison of similarity, and the images of the persons to be inspected with the highest similarity can be screened, sorted and displayed, so that when the database to be inspected is large, the calculation redundancy can be effectively reduced, the time spent can be shortened, and the practicality of intelligent identification and screening of pedestrians can be enhanced. In addition, compared with the existing method of comparing features one by one, the expression is intuitive and the calculation complexity is greatly reduced, which can better meet the requirements of fast retrieval.

[0103] like Figure 5 As shown, the second aspect of this embodiment provides a virtual device for implementing the personnel identification method described in the first aspect or any possible design of the first aspect, including an image acquisition module, a global feature extraction module, a local feature extraction module, a feature vector splicing module, a feature matrix combination module, a normalization processing module, a square matrix calculation module and a personnel similarity determination module;

[0104] The image acquisition module is used to acquire the image of the person to be examined;

[0105] The global feature extraction module is communicatively connected to the image acquisition module, and is used to acquire corresponding global feature maps based on a convolutional neural network for the image of the person to be inspected and a given target person image;

[0106] The local feature extraction module is communicatively connected to the image acquisition module, and is used to acquire corresponding multiple local features for the image of the person to be inspected and the image of the target person based on the human body key point model and the corresponding global feature map;

[0107] The feature vector splicing module is respectively connected to the global feature extraction module and the local feature extraction module for performing series splicing of the corresponding multiple local features with the global features in the corresponding global feature map after average pooling processing for the image of the person to be inspected and the image of the target person, so as to obtain a corresponding feature row vector containing multi-dimensional feature values;

[0108] The feature matrix combination module is communicatively connected to the feature vector splicing module, and is used to combine the feature row vectors of the image of the person to be examined and the image of the target person into a feature matrix, wherein each row element in the feature matrix is ​​a feature row vector of the image of the person to be examined or a feature row vector of the image of the target person;

[0109] The normalization processing module is communicatively connected to the feature matrix combination module and is used to perform normalization processing on the feature matrix to obtain a normalized feature matrix;

[0110] The square matrix calculation module is communicatively connected to the normalization processing module and is used to calculate the inner product of the normalized feature matrix and its transposed matrix to obtain a similar cosine square matrix;

[0111] The personnel similarity determination module is communicatively connected to the feature matrix combination module and the square matrix calculation module, and is used to find the elements of the corresponding row and column positions in the similar cosine square matrix according to the row positions of the feature row vectors of the image of the person to be tested and the feature row vectors of the target person image in the feature matrix, and use the elements as the personnel similarity between the image of the person to be tested and the target person image.

[0112] In a possible design, if the image acquisition module acquires multiple images of the person to be examined, it also includes a square matrix element search module, an element sorting module, an image sorting module and an image display module;

[0113] The square matrix element search module is respectively connected to the feature matrix combination module and the square matrix calculation module for searching for a plurality of elements at corresponding row positions in the similar cosine square matrix according to the row positions of the feature row vectors of the target person image in the feature matrix;

[0114] The element sorting module is communicatively connected to the square matrix element searching module and is used to sort all the elements of the multiple elements from large to small values ​​to obtain an element queue;

[0115] The image sorting module is respectively connected to the image acquisition module and the element sorting module for performing consistency sorting on all the images of the persons to be inspected among the multiple images of the persons to be inspected according to the sorting of the elements corresponding to the images of the persons to be inspected in the element queue, so as to obtain a queue of images of the persons to be inspected;

[0116] The image display module is communicatively connected to the image sorting module, and is used to adjust the images in the queue of images of the person to be examined into images of a uniform size, and display them simultaneously in a horizontal stacking manner.

[0117] The working process, working details and technical effects of the aforementioned device provided in the second aspect of this embodiment can be referred to the personnel identification method described in the first aspect or any possible design of the first aspect, and will not be repeated here.

[0118] like Figure 6 As shown, the third aspect of this embodiment provides a computer device for executing the personnel identification method described in the first aspect or any possible design of the first aspect, including a memory and a processor connected in communication, wherein the memory is used to store a computer program, and the processor is used to read the computer program and execute the personnel identification method described in the first aspect or any possible design of the first aspect. Specifically, the memory may include, but is not limited to, a random access memory (Random-Access Memory, RAM), a read-only memory (Read-Only Memory, ROM), a flash memory (Flash Memory), a first-in-first-out memory (First Input First Output, FIFO) and / or a first-in-last-out memory (First Input Last Output, FILO), etc.; the processor may include, but is not limited to, a microprocessor of the STM32F105 series. In addition, the computer device may also include, but is not limited to, a power module, a display screen, and other necessary components.

[0119] The working process, working details and technical effects of the aforementioned computer device provided in the third aspect of this embodiment can be referred to the personnel identification method described in the first aspect or any possible design of the first aspect, and will not be repeated here.

[0120] like Figure 7 As shown, the fourth aspect of this embodiment provides a pedestrian monitoring system that adopts the personnel identification method as described in the first aspect or any possible design of the first aspect, including a camera 1, a video server 2 and a personal computer PC terminal 3 that are sequentially communicatively connected; the camera 1 is used to be arranged in a pedestrian monitoring place, and to collect and obtain video signals of the pedestrian monitoring place; the video server 2 is used to digitally convert the collected video signals to obtain a video stream in a unified video format; the personal computer PC terminal 3 is used to execute the personnel identification method as described in the first aspect or any possible design of the first aspect after receiving the video stream, and judge whether a target person appears in the pedestrian monitoring place according to the personnel identification result. Figure 7 As shown, the pedestrian monitoring location is exemplified as an elevator car 4. The video server 2 can also store the obtained video stream file locally so that the operator can manage and view the video data, and encrypt the video stream to effectively protect the privacy information of residents in the residential building. The personal computer PC terminal 3 can be designed as an elevator car intelligent monitoring system software that realizes passenger attribute recognition and screening retrieval, so that after selecting and loading the elevator monitoring video from the local, in addition to enabling it to be played smoothly, it can also be intelligently analyzed by adopting the personnel recognition method described in the first aspect or any possible design of the first aspect to achieve the purpose of intelligent recognition and screening of pedestrians.

[0121] The working process, working details and technical effects of the aforementioned pedestrian monitoring system provided in the fourth aspect of this embodiment can be referred to the personnel identification method described in the first aspect or any possible design of the first aspect, and will not be repeated here.

[0122] In a fifth aspect of this embodiment, there is provided a computer-readable storage medium storing instructions including the first aspect or any possible design of the personnel identification method in the first aspect, that is, the computer-readable storage medium stores instructions, and when the instructions are executed on a computer, the personnel identification method as described in the first aspect or any possible design of the first aspect is executed. The computer-readable storage medium refers to a carrier for storing data, which may include but is not limited to computer-readable storage media such as floppy disks, optical disks, hard disks, flash memories, USB flash drives, and / or memory sticks, and the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0123] The working process, working details and technical effects of the aforementioned computer-readable storage medium provided in the fifth aspect of this embodiment can be referred to the personnel identification method described in the first aspect or any possible design of the first aspect, and will not be repeated here.

[0124] In a sixth aspect of the present embodiment, a computer program product comprising instructions is provided, and when the instructions are executed on a computer, the computer is caused to execute the personnel identification method as described in the first aspect or any possible design of the first aspect. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0125] Finally, it should be noted that the present invention is not limited to the above optional implementations, and anyone can derive other various forms of products under the enlightenment of the present invention. The above specific implementations should not be understood as limiting the scope of protection of the present invention. The scope of protection of the present invention should be based on the definition in the claims, and the description can be used to interpret the claims.

Claims

1. A method for identifying a person, characterized in that: include: Acquire an image of the person to be inspected; For the image of the person to be inspected and the given target person image, a corresponding global feature map is acquired based on a convolutional neural network; For the image of the person to be inspected and the image of the target person, based on the human key point model and the corresponding global feature map, a corresponding plurality of local features are obtained, specifically including: obtaining a plurality of local heat maps of the person image based on the human key point model, wherein the person image is the image of the person to be inspected or the image of the target person; for each of the plurality of local heat maps, first performing pixel-level multiplication processing on the global feature map of the person image and the corresponding heat map, and then performing average pooling processing to obtain the corresponding local features; For the image of the person to be inspected and the image of the target person, the corresponding multiple local features are concatenated with the global features in the corresponding global feature map after average pooling processing to obtain a corresponding feature row vector containing multi-dimensional feature values; Combining the characteristic row vectors of the image of the person to be examined and the image of the target person into a characteristic matrix, wherein each row element in the characteristic matrix is ​​a characteristic row vector of the image of the person to be examined or a characteristic row vector of the image of the target person; Normalizing the feature matrix to obtain a normalized feature matrix; Calculate the inner product of the normalized feature matrix and its transposed matrix to obtain a similar cosine square matrix; According to the row positions of the feature row vectors of the image of the person to be inspected and the feature row vectors of the target person image in the feature matrix, elements of corresponding row and column positions are found in the similar cosine matrix, and the elements are used as the person similarity between the image of the person to be inspected and the target person image.

2. The method for identifying a person according to claim 1, wherein: If multiple images of the person to be inspected are obtained, after obtaining the similar cosine matrix, the method further includes: According to the row position of the characteristic row vector of the target person image in the characteristic matrix, searching for a plurality of elements of corresponding row positions in the similar cosine square matrix; Sort all elements in the plurality of elements by value from large to small to obtain an element queue; According to the order of the elements corresponding to the images of the persons to be inspected in the element queue, all the images of the persons to be inspected among the multiple images of the persons to be inspected are ordered in a consistent manner to obtain a queue of images of the persons to be inspected; The images in the queue of images of the person to be examined are adjusted to images of uniform size, and are displayed simultaneously by horizontal stacking.

3. The method for identifying a person according to claim 1, wherein: Obtain images of the person to be inspected, including: Acquire surveillance video images; Calling a target detection algorithm to perform personnel detection processing on the surveillance video image, and obtaining a position of a personnel detection result mark box in the surveillance video image; According to the position of the person detection result mark box, the image in the box is cut out from the monitoring video image as the image of the person to be detected.

4. The method for identifying a person according to claim 1, wherein: For the image of the person to be inspected and the given target person image, a corresponding global feature map is obtained based on a convolutional neural network, including: The person image is input into a convolutional neural network, and the input image is encoded into a global feature map through multi-layer convolution, pooling and sampling processing of the convolutional neural network, wherein the person image is the image of the person to be inspected or a given target person image, and the step size of the last spatial downsampling of the convolutional neural network is set to one.

5. The method for identifying a person according to claim 1, wherein: The feature matrix is ​​normalized to obtain a normalized feature matrix, including: The feature matrix is ​​normalized according to the mean and standard deviation of the ImageNet project to obtain a normalized feature matrix.

6. A person identification device, characterized in that: It includes an image acquisition module, a global feature extraction module, a local feature extraction module, a feature vector splicing module, a feature matrix combination module, a normalization processing module, a square matrix calculation module and a personnel similarity determination module; The image acquisition module is used to acquire the image of the person to be examined; The global feature extraction module is communicatively connected to the image acquisition module, and is used to acquire corresponding global feature maps based on a convolutional neural network for the image of the person to be inspected and a given target person image; The local feature extraction module is communicatively connected to the image acquisition module, and is used to obtain a plurality of corresponding local features for the image of the person to be inspected and the image of the target person based on the human key point model and the corresponding global feature map, specifically comprising: obtaining a plurality of local heat maps of the person image based on the human key point model, wherein the person image is the image of the person to be inspected or the image of the target person; for each of the plurality of local heat maps, first performing pixel-level product processing on the global feature map of the person image and the corresponding heat map, and then performing average pooling processing to obtain the corresponding local features; The feature vector splicing module is respectively connected to the global feature extraction module and the local feature extraction module for performing series splicing of the corresponding multiple local features with the global features in the corresponding global feature map after average pooling processing for the image of the person to be inspected and the image of the target person, so as to obtain a corresponding feature row vector containing multi-dimensional feature values; The feature matrix combination module is communicatively connected to the feature vector splicing module, and is used to combine the feature row vectors of the image of the person to be examined and the image of the target person into a feature matrix, wherein each row element in the feature matrix is ​​a feature row vector of the image of the person to be examined or a feature row vector of the image of the target person; The normalization processing module is communicatively connected to the feature matrix combination module and is used to perform normalization processing on the feature matrix to obtain a normalized feature matrix; The square matrix calculation module is communicatively connected to the normalization processing module and is used to calculate the inner product of the normalized feature matrix and its transposed matrix to obtain a similar cosine square matrix; The personnel similarity determination module is communicatively connected to the feature matrix combination module and the square matrix calculation module, and is used to find the elements of the corresponding row and column positions in the similar cosine square matrix according to the row positions of the feature row vectors of the image of the person to be tested and the feature row vectors of the target person image in the feature matrix, and use the elements as the personnel similarity between the image of the person to be tested and the target person image.

7. A computer device, characterized in that: It comprises a memory, a processor and a transceiver which are communicatively connected in sequence, wherein the memory is used to store a computer program, the transceiver is used to send and receive data, and the processor is used to read the computer program to execute the personnel identification method as described in any one of claims 1 to 5.

8. A pedestrian monitoring system, characterized in that: It comprises a camera (1), a video server (2) and a personal computer (PC) (3) which are sequentially connected in communication; The camera (1) is used to be arranged in a pedestrian monitoring location and to collect and acquire video signals of the pedestrian monitoring location; The video server (2) is used to digitally convert the collected video signal to obtain a video stream in a unified video format; The personal computer PC (3) is used to execute the personnel identification method according to any one of claims 1 to 5 after receiving the video stream, and judge whether a target person appears in the pedestrian monitoring location according to the personnel identification result.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores instructions, and when the instructions are executed on the computer, the personnel identification method as described in any one of claims 1 to 5 is executed.

Images