A method and system for extracting appearance features from a monocular vertical perspective

Abstract

The present invention discloses a method and system for extracting appearance features under a monocular vertical viewing angle, which comprises the following steps: obtaining an original picture taken under a monocular vertical viewing angle as an input picture; performing pedestrian detection on the input picture and cropping the picture of the pedestrian according to the position; inputting the picture of the pedestrian into a human body deconstruction network to obtain segmentation results of different parts of the pedestrian's body; extracting different features of the obtained pedestrians respectively; aggregating the extracted results into the appearance features of the pedestrian; and constructing the appearance features of the pedestrian by first identifying the pedestrian and then separately identifying different features of the pedestrian, thereby obtaining more detailed and accurate appearance feature results.

Description

Technical Field

[0001] The present invention relates to the technical field of appearance feature extraction, and in particular to an appearance feature extraction method and system under a monocular vertical viewing angle. Background Art

[0002] Traditionally, some security personnel are tasked with watching surveillance cameras in real time to guard against abnormal behavior or situations of people appearing in the surveillance area, or for post-investigation, they need to review hours or days of video footage to extract effective information to describe the appearance characteristics of potential key persons. However, the time and resources required to manually review the massive amount of available videos cannot truly utilize all the key information that video data can provide, and staff are prone to errors, which makes it more difficult to process surveillance camera data efficiently or effectively. Based on this situation, there is an urgent need for a more automated and effective method to extract pedestrian appearance features. Summary of the invention

[0003] In view of this, the present invention provides a method for extracting appearance features under a monocular vertical perspective, the purpose of which is to (1) automatically extract pedestrians under a monocular vertical perspective based on a deep learning network, thereby establishing an independent appearance feature library for each pedestrian; and (2) after detecting a pedestrian, automatically extract features of different parts of the pedestrian, and integrate these features to form the appearance features of the pedestrian.

[0004] To achieve the above object, the present invention provides a method for extracting appearance features under a monocular vertical viewing angle, comprising the following steps:

[0005] S1: Obtain the original image taken by a monocular vertical perspective as the input image;

[0006] S2: Detect pedestrians in the input image and crop the image of pedestrians according to their positions;

[0007] S3: Input each pedestrian image into a human body deconstruction network to obtain a segmentation result of the pedestrian body; wherein the human body deconstruction network includes a first part and a second part;

[0008] Optionally, the human body deconstruction network outputs two parts of prediction results. The first part predicts the information of each joint of the human body, which is used to determine the position of different parts of the human body. The second part of the prediction results will further segment various types of information on the pedestrian's body, which is conducive to the subsequent judgment of the pedestrian's facial features and clothing. The two parts work together to obtain more accurate prediction results.

[0009] S4: extract different features of the pedestrians respectively;

[0010] S5: Summarize the extracted results into the appearance features of the pedestrian.

[0011] Furthermore, the step S2 includes: inputting the image acquired by the camera into an adapted pedestrian detection network based on YoloV3 to obtain a detection frame marking each pedestrian, and the parameters of the detection frame are determined by the following parameters:

[0012] (b x , b y， b w , b h , p)

[0013] Among them, b x , b y Indicates the horizontal and vertical coordinates of the center of the detection frame; b w , b h Represents the width and height of the detection box; p represents the probability that the detection box detects a pedestrian.

[0014] Furthermore, the original YoloV3 loss function needs to be modified accordingly according to the characteristics of the pedestrian detection task and the camera pictures:

[0015]

[0016] Among them, S represents the number of grids defined in YoloV3; Indicates that if there is a pedestrian in the detection box, its value is 1, otherwise it is 0; If there is no pedestrian in the detection box, its value is 1, otherwise it is 0; b′ x , b′ y , b′ w , b′ h Indicates the horizontal and vertical coordinates, width and height of the center position of the detection frame that actually contains the pedestrian; b x , b y , b w , b h , p represents the horizontal and vertical coordinates, width and height of the detection box predicted by the network, and the probability of a pedestrian in the box; λ coord ,λ noperson , γ represents the weights of different indicators.

[0017] Furthermore, in the step S3, the pedestrian image is input into the human body deconstruction network to obtain the pedestrian body segmentation result, including:

[0018] According to the result of pedestrian detection frame, each pedestrian is cut out from the surveillance image separately and sent to the human body deconstruction network to obtain the appearance information of different parts of the pedestrian's body; the loss function of the first part of the human body deconstruction network to predict the human body joint point result is:

[0019]

[0020] Where N represents the number of joints that can be detected for the pedestrian; l′ i，x , l′ i，y The horizontal and vertical coordinates representing the real position of the i-th joint point of the pedestrian; l i，x , l i，y Represents the horizontal and vertical coordinates of the predicted position of the i-th joint point of the pedestrian.

[0021] Furthermore, the loss function of the second part predicting various information results on the pedestrian's body is:

[0022]

[0023] Wherein, W and H represent the width and height of the pedestrian image; C represents the type of pedestrian part that needs to be identified in advance; If the pixel is of type k, its value is 1, otherwise it is 0; h k (i, j) represents the probability that the network predicts that the pixel is of category k; m(i, j) means that if the pixel is a pedestrian face pixel, its value is 1, otherwise it is 0; δ represents the weight.

[0024] Furthermore, the loss functions of the first and second parts together constitute the loss function of the human deconstruction network:

[0025] L st =L jt *L ol .

[0026] Furthermore, in step S4, different features of the pedestrians are extracted respectively, including:

[0027] S41, setting the type to be identified;

[0028] S42, cropping the pedestrian image according to the outer bounding boxes of different parts identified and inputting it into the classification network to obtain specific information of the type;

[0029] S43, different types of classification network structures are consistent. According to the different fully connected layers of the input image and the final output category, the output contains specific category information and color information; the loss functions of different classification networks can be summarized as:

[0030]

[0031] Among them, B represents the number of classes that need to be identified in this category; D represents the number of classes of different colors contained in this category; 1 i Indicates that if the image contains the i-th category, its value is 1, otherwise it is 0; 1 j Indicates that if the image contains the jth color, its value is 1, otherwise it is 0; r iIndicates the probability that the network predicts that the image contains the i-th category; r′ j It indicates that the network predicts the probability that the image contains the jth color.

[0032] Furthermore, in step S5, the extracted results are summarized as the appearance features of the pedestrian, including:

[0033] According to the results of the human body deconstruction network, the facial image is extracted separately as the facial features of the pedestrian, and the hairstyle and hair color, upper and lower clothing and clothing colors of the pedestrian are used as its external features to form the appearance features of the pedestrian.

[0034] In order to solve the above problems, the present invention provides an appearance feature extraction system under a monocular vertical viewing angle, the system comprising:

[0035] Image acquisition module: used to obtain the original image taken by a monocular vertical perspective as an input image;

[0036] Image processing module: used to detect pedestrians in the input image and crop the image of pedestrians according to their positions;

[0037] Segmentation module: used for inputting the pedestrian image into the human body deconstruction network to obtain the segmentation result of the pedestrian body; wherein the human body deconstruction network includes a first part and a second part;

[0038] Feature extraction module: extract different features of pedestrians respectively;

[0039] Summarization module: summarizes the extracted results into the appearance features of the pedestrian.

[0040] The beneficial effects of the present invention are:

[0041] The original image taken by a monocular vertical perspective is obtained as the input image; pedestrian detection is performed on the input image and the image of each pedestrian is obtained by cropping according to the position; each pedestrian image is input into the human body deconstruction network to obtain the segmentation results of different parts of the pedestrian's body; features are extracted from different parts of the pedestrian respectively; the extracted results are summarized as the appearance features of the pedestrian; by first identifying the pedestrian and then separately identifying the features of different parts of the pedestrian to construct the appearance features of the pedestrian, a more detailed and accurate appearance feature result can be obtained. After obtaining the appearance feature result, the pedestrian can be quickly retrieved by inputting the corresponding appearance feature. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram of a flow chart of a method for extracting appearance features under a monocular vertical viewing angle according to an embodiment of the present invention;

[0043] Figure 2A schematic diagram of a human body deconstruction network according to an embodiment of the present invention;

[0044] Figure 3 A schematic diagram of a classification network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The present invention is further described below in conjunction with the accompanying drawings, but the present invention is not limited in any way. Any changes or substitutions made based on the teachings of the present invention belong to the protection scope of the present invention.

[0046] In order to achieve the above object, the present invention provides a method for extracting appearance features under a monocular vertical viewing angle, comprising the following steps:

[0047] S1: Obtain the original image taken by a monocular vertical perspective as the input image;

[0048] Optionally, the original image captured by the monocular vertical perspective is used as the input image. The camera used to obtain the monocular vertical perspective image can directly use the surveillance cameras distributed in shopping malls or public areas to extract the appearance features of pedestrians;

[0049] S2: Detect pedestrians in the input image and crop the image of pedestrians according to their positions;

[0050] Optionally, the image captured by the camera is input into an adapted pedestrian detection network based on YoloV3 to obtain a detection frame marking each pedestrian. The parameters of the detection frame are determined by the following parameters:

[0051] (b x , b y , b w , b h , p)

[0052] Among them, b x , b y Indicates the horizontal and vertical coordinates of the center of the detection frame; b w , b h Represents the width and height of the detection box; p represents the probability that the detection box detects a pedestrian.

[0053] Furthermore, the original YoloV3 loss function needs to be modified accordingly according to the characteristics of the pedestrian detection task and the camera pictures:

[0054]

[0055] Wherein, S represents the number of grids defined in YoloV3, which is 26×26 in this embodiment; Indicates that if there is a pedestrian in the detection box, its value is 1, otherwise it is 0; If there is no pedestrian in the detection box, its value is 1, otherwise it is 0; b′ x , b′ y , b′ w , b′ h Indicates the horizontal and vertical coordinates, width and height of the center position of the detection frame that actually contains the pedestrian; b x , b y , b w , b h , p represents the horizontal and vertical coordinates, width and height of the detection box predicted by the network, and the probability of a pedestrian in the box; λ coord and λ noperson The weights of different loss items are controlled, which are 5 and 0.5 respectively in this embodiment; γ controls the weight of the biased shape of the detection box, which is 2 in this embodiment; the present invention fully considers the aspect ratio of pedestrians in the picture under vertical viewing angle and incorporates this prior knowledge into the loss function. The Adam optimizer is used to train the entire network according to Lp. The Adam optimizer is one of the most common optimizers in deep learning and can adaptively optimize the parameters of the network.

[0056] S3: Input the pedestrian image into the human body deconstruction network to obtain the pedestrian body segmentation result;

[0057] According to the result of pedestrian detection frame, each pedestrian is cut out from the surveillance image separately and sent to the human body deconstruction network to obtain the appearance information of different parts of the pedestrian body, such as Figure 2 As shown, the detailed structures of the backbone network, joint extraction subnetwork and deconstruction subnetwork are listed in Tables 1, 2 and 3. The results of the backbone network take ResNet-34 as an example. The human body deconstruction network outputs two parts of prediction results. The first part predicts the information of each joint point of the human body, which is used to determine the positions of different parts of the human body. In this embodiment, the predictable joint positions are head, upper body, lower body, left arm, right arm, left leg, right leg, left foot and right foot, a total of 9 nodes. The second part of the prediction results will further segment various types of information on the pedestrian's body, which is conducive to the subsequent judgment of the pedestrian's facial features and dressing. The appearance features that need to be identified by the pedestrians defined in this embodiment include face, hair, hat, top, pants and shoes, a total of 6 categories. The two parts work together to obtain more accurate prediction results. The loss function of the first part of the human body deconstruction network predicting the results of human body joints is:

[0058]

[0059] Where N represents the number of joints that can be detected for the pedestrian. Since there may be occlusions in the obtained image, N ranges from 1 to 9. i，x , l′ i，y The horizontal and vertical coordinates representing the real position of the i-th joint point of the pedestrian; l i，X , li，y Represents the horizontal and vertical coordinates of the predicted position of the i-th joint point of the pedestrian.

[0060] Table 1 Backbone network structure

[0061]

[0062]

[0063] Table 2 Joint extraction sub-network structure

[0064]

[0065] Table 3 Deconstruction of sub-network structure

[0066]

[0067]

[0068] Furthermore, the loss function of the second part predicting various information results on the pedestrian's body is:

[0069]

[0070] Wherein, W and H represent the width and height of the input single pedestrian image, which depends on the width and height of the detection frame obtained in s2; C represents the type of pedestrian part that needs to be identified, which is 6 in this embodiment; If the pixel is of type k, its value is 1, otherwise it is 0; h k (i, j) represents the probability that the network predicts that the pixel is of category k; m(i, j) represents that if the pixel is a pedestrian face pixel, its value is 1, otherwise it is 0; δ represents the weight used to control the importance of facial recognition, which is 4 in this embodiment.

[0071] Furthermore, the loss functions of the first and second parts together constitute the loss function of the human deconstruction network:

[0072] L st =L jt *L ol

[0073] Use Adam optimizer to optimize L st Train the entire network.

[0074] S4: Extract different features of the pedestrians, including:

[0075] Set the specific types that need to be identified in each type of appearance feature, such as Figure 3For example, for the appearance feature of tops, 10 categories can be set to be recognized, including shirts, cardigans, hoodies, sweaters, sleeveless shirts, sweaters, T-shirts, suits, vests and sweatshirts. At the same time, the color information of the corresponding appearance features also needs to be recognized, and the colors to be recognized can be set to 10 common colors, including black, white, gray, red, orange, yellow, green, cyan, blue and purple.

[0076] The outer bounding box is calculated as follows:

[0077] box=[min(coord k，x ), max(coord k，x ), min(coord k，y ), max(coord k，y )]

[0078] Among them, the min and max functions respectively represent obtaining the minimum or maximum value of the internal variable; coord k，x Represents the horizontal coordinates of all pixels belonging to the kth class; coord k，y Represents the ordinate of all pixels belonging to the kth class.

[0079] Pedestrian images are cropped based on the outer bounding boxes of the identified different appearance features and input into the classification network.

[0080] Get the specific information of the category. The classification network structure of different appearance features is the same, only the input image and the final output

[0081] The fully connected layers of different categories have different outputs, which contain specific category information and color information. The loss functions of different classification networks can be summarized as follows:

[0082]

[0083] Wherein, B represents the number of categories required to be identified contained in the appearance feature, which is 10 in this embodiment, taking the top as an example; D represents the number of categories of different colors contained in the appearance feature, which is 10 in this embodiment; 1 i Indicates that if the image contains the i-th category, its value is 1, otherwise it is 0; 1 j Indicates that if the image contains the jth color, its value is 1, otherwise it is 0; r i Indicates the probability that the network predicts that the image contains the i-th category; r′ j Indicates the probability that the network predicts that the image contains the jth color. Using the Adam optimizer according to L c Train the entire network.

[0084] S5: Summarize the extracted results into the appearance features of the pedestrian, including:

[0085] According to the results of the human body deconstruction network, the facial image is extracted separately as the facial features of the pedestrian, and the hairstyle and hair color, upper and lower clothing and clothing colors of the pedestrian are used as its external features to form the appearance features of the pedestrian.

[0086] The present invention also provides a system for extracting appearance features under a monocular vertical viewing angle, comprising:

[0087] Image acquisition module: used to obtain the original image taken by a monocular vertical perspective as an input image;

[0088] Image processing module: used to detect pedestrians in the input image and crop the image of pedestrians according to their positions;

[0089] Segmentation module: used for inputting the pedestrian image into the human body deconstruction network to obtain the segmentation result of the pedestrian body; wherein the human body deconstruction network includes a first part and a second part;

[0090] Feature extraction module: used to extract different features of pedestrians;

[0091] Summarization module: used to summarize the extracted results into the appearance features of the pedestrian.

[0092] It should be noted that the serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. And the terms "including", "comprising" or any other variants thereof in this article are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "including a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0093] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for extracting appearance features from a monocular vertical viewing angle, characterized in that: The following steps are involved: S1: Obtain the original image taken by a monocular vertical perspective as the input image; S2: Detect pedestrians in the input image and crop the image of pedestrians according to their positions; The S2 step includes: The image captured by the camera is input into the adapted pedestrian detection network based on YoloV3 to obtain the detection frame marking each pedestrian. The parameters of the detection frame are determined by the following parameters: (b x ,b y ,b w ,b h ,p) Among them, b x , b y Indicates the horizontal and vertical coordinates of the center of the detection frame; b w , b h represents the width and height of the detection box; p represents the probability that the detection box detects a pedestrian; Also included: The original YoloV3 loss function needs to be modified accordingly according to the characteristics of the pedestrian detection task and the camera pictures: Among them, S represents the number of grids defined in YoloV3; Indicates that if there is a pedestrian in the detection box, its value is 1, otherwise it is 0; If there is no pedestrian in the detection box, its value is 1, otherwise it is 0; b′ x , b′ y , b′ w , b′ h Indicates the horizontal and vertical coordinates, width and height of the center position of the detection frame that actually contains the pedestrian; b x , b y , b w , b h , p represents the horizontal and vertical coordinates, width and height of the detection box predicted by the network, as well as the probability of a pedestrian in the box; λ coord ,λ noperson , γ represents the weights of different indicators; S3: Inputting the pedestrian image into a human body deconstruction network to obtain segmentation results of different parts of the pedestrian's body; wherein the human body deconstruction network includes a first part and a second part; The S3 step includes: According to the result of pedestrian detection frame, each pedestrian is cut out from the monitoring image separately and sent to the human body deconstruction network to obtain the appearance information of different parts of the pedestrian's body; the loss function of the first part of the human body deconstruction network to predict the human body joint point result is: Where N represents the number of joints that can be detected for the pedestrian; l′ i，x , l i，y The horizontal and vertical coordinates representing the real position of the i-th joint point of the pedestrian; l i，x , l i，y Represents the horizontal and vertical coordinates of the predicted position of the i-th joint point of the pedestrian; The loss function of the second part predicting various information results on the pedestrian's body is: Wherein, W and H represent the width and height of the pedestrian image; C represents the type of pedestrian part that needs to be identified in advance; If the pixel is of type k, its value is 1, otherwise it is 0; h k (i, j) represents the probability that the network predicts that the pixel is of category k; m(i, j) means that if the pixel is a pedestrian face pixel, its value is 1, otherwise it is 0; δ represents the weight; S4: extract different features of the pedestrians respectively; S5: Summarize the extracted results into the appearance features of the pedestrian.

2. The appearance feature extraction method under a monocular vertical viewing angle as claimed in claim 1, characterized in that: The loss functions of the first part and the second part together constitute the loss function of the human body deconstruction network: L st =L jt *L ol .

3. The appearance feature extraction method under a monocular vertical viewing angle as claimed in claim 1, characterized in that: In the step S5, the extracted results are summarized as the appearance features of the pedestrian, including: based on the results of the human body deconstruction network, the facial image is extracted separately as the facial features of the pedestrian, and the hairstyle and hair color, upper and lower clothing and clothing color features of the pedestrian are used as its external features to jointly constitute the appearance features of the pedestrian.

4. A system for extracting appearance features under a monocular vertical viewing angle as claimed in any one of claims 1 to 3, characterized in that: include: Image acquisition module: used to obtain the original image taken by a monocular vertical perspective as an input image; Image processing module: used to detect pedestrians in the input image and crop the image of pedestrians according to their positions; Segmentation module: used for inputting the pedestrian image into the human body deconstruction network to obtain the segmentation result of the pedestrian body; wherein the human body deconstruction network includes a first part and a second part; Feature extraction module: used to extract different features of pedestrians; Summarization module: used to summarize the extracted results into the appearance features of the pedestrian.

Images