Crowd positioning method and device, electronic device and storage medium

Abstract

The present disclosure relates to a crowd positioning method and device, an electronic device and a storage medium, the method comprising: extracting features from a crowd image to obtain at least one first feature map; detecting human key points on the at least one first feature map to determine a positioning probability map corresponding to the crowd image, wherein the positioning probability map is used to indicate the probability that each pixel point in the crowd image is a target human point; performing weighted processing on the at least one first feature map according to the positioning probability map to obtain at least one second feature map; performing human frame detection on the at least one second feature map to determine a target human frame detection result corresponding to the crowd image, wherein the target human frame detection result includes a target human frame corresponding to each human body in the crowd image. The embodiments of the present disclosure can improve the accuracy of crowd positioning.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a crowd positioning method and device, an electronic device, and a storage medium. Background Art

[0002] With the growth of population and the acceleration of urbanization, the number of large crowd gatherings is increasing and the scale is getting larger. Crowd analysis is of great significance to public safety and urban planning. Common crowd analysis tasks include crowd counting, group behavior analysis, crowd positioning, etc. Among them, crowd positioning is the basis of other crowd analysis tasks. Crowd positioning refers to estimating the position of the human body included in the image or video through computer vision algorithms, determining the coordinates of the human body included in the image or video, and providing data basis for subsequent crowd analysis tasks such as crowd counting and group behavior analysis. The accuracy of crowd positioning directly affects the accuracy of crowd counting and the results of crowd behavior analysis. Therefore, a crowd positioning method with high accuracy is urgently needed. Summary of the invention

[0003] The present disclosure proposes a technical solution of a crowd positioning method and device, an electronic device and a storage medium.

[0004] According to one aspect of the present disclosure, a crowd positioning method is provided, comprising: performing feature extraction on a crowd image to obtain at least one first feature map; performing human key point detection on the at least one first feature map to determine a positioning probability map corresponding to the crowd image, wherein the positioning probability map is used to indicate the probability that each pixel point in the crowd image is a target human point; performing weighted processing on the at least one first feature map according to the positioning probability map to obtain at least one second feature map; performing human body frame detection on the at least one second feature map to determine a target human body frame detection result corresponding to the crowd image, wherein the target human body frame detection result includes a target human body frame corresponding to each human body in the crowd image.

[0005] In a possible implementation, performing human key point detection on the at least one first feature map to determine the positioning probability map corresponding to the crowd image includes: performing convolution processing on the target first feature map to obtain a third feature map, wherein the target first feature map is one of the at least one first feature map; performing transposed convolution processing on the third feature map to obtain a fourth feature map, wherein the fourth feature map and the crowd image have the same size; and performing convolution processing on the fourth feature map to obtain the positioning probability map.

[0006] In a possible implementation, performing weighted processing on the at least one first feature map according to the positioning probability map to obtain at least one second feature map includes: performing pixel-level weighted processing on each of the first feature maps according to the positioning probability map to obtain the second feature map corresponding to each of the first feature maps.

[0007] In a possible implementation manner, performing human frame detection on the at least one second feature map to determine a target human frame detection result corresponding to the crowd image includes: performing human frame detection on the at least one second feature map to obtain at least one human frame prediction result; and fusing the at least one human frame prediction result to obtain the target human frame detection result.

[0008] In a possible implementation manner, the fusing the at least one human frame prediction result to obtain the target human frame detection result includes: performing non-maximum suppression processing on the at least one human frame prediction result to obtain the target human frame detection result.

[0009] In a possible implementation, the method further includes: performing crowd analysis on the crowd image according to the positioning probability map to obtain a first analysis result corresponding to the crowd image; performing crowd analysis on the crowd image according to the target human body frame detection result to obtain a second analysis result corresponding to the crowd image; and fusing the first analysis result and the second analysis result to obtain a target analysis result corresponding to the crowd image.

[0010] In a possible implementation, the crowd positioning method is implemented by a crowd positioning neural network, and the training samples of the crowd positioning neural network include: a crowd sample image, a real positioning map corresponding to the crowd sample image, and a real human frame detection result corresponding to the crowd sample image; the training method of the crowd positioning neural network is as follows: through the crowd positioning neural network, a predicted positioning probability map corresponding to the crowd sample image and a predicted human frame detection result corresponding to the crowd sample image are determined, wherein the predicted positioning probability map is used to indicate the probability that each pixel point in the crowd sample image is a target human point, and the predicted human frame detection result includes a predicted human frame corresponding to each human body in the crowd sample image; based on the predicted positioning probability map and the real positioning map, a first positioning loss is determined; based on the predicted human frame detection result and the real human frame detection result, a second positioning loss is determined; based on the first positioning loss and the second positioning loss, the crowd positioning neural network is optimized.

[0011] In a possible implementation, the optimizing the crowd positioning neural network based on the first positioning loss and the second positioning loss includes: summing the first positioning loss and the second positioning loss to obtain a target positioning loss; and optimizing the crowd positioning neural network based on the target positioning loss.

[0012] In a possible implementation, the real positioning map includes the position of the target human body point corresponding to each human body in the crowd sample image, wherein one human body corresponds to one target human body point; the method further includes: performing face detection on the crowd sample image to determine the predicted face frame corresponding to the crowd sample image; determining the real human body frame corresponding to each human body in the crowd sample image through a linear interpolation algorithm based on the position of the target human body point corresponding to each human body in the crowd sample image and the predicted face frame corresponding to the crowd sample image; determining the real human body frame detection result based on the real human body frame corresponding to each human body in the crowd sample image.

[0013] According to one aspect of the present disclosure, a crowd positioning device is provided, comprising: a feature extraction module, configured to perform feature extraction on a crowd image to obtain at least one first feature map; a human key point detection module, configured to perform human key point detection on the at least one first feature map to determine a positioning probability map corresponding to the crowd image, wherein the positioning probability map is used to indicate the probability that each pixel point in the crowd image is a target human point; a feature enhancement module, configured to perform weighted processing on the at least one first feature map according to the positioning probability map to obtain at least one second feature map; and a human frame detection module, configured to perform human frame detection on the at least one second feature map to determine a target human frame detection result corresponding to the crowd image, wherein the target human frame detection result includes a target human frame corresponding to each human body in the crowd image.

[0014] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0015] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor.

[0016] In the disclosed embodiment, feature extraction is performed on a crowd image to obtain at least one first feature map; human key point detection is performed on at least one first feature map to determine a positioning probability map corresponding to the crowd image, wherein the positioning probability map is used to indicate the probability that each pixel point in the crowd image is a target human point; based on the positioning probability map, weighted processing is performed on at least one first feature map to obtain at least one second feature map; human frame detection is performed on at least one second feature map to determine a target human frame detection result corresponding to the crowd image, wherein the target human frame detection result includes a target human frame corresponding to each human body in the crowd image. In the process of crowd positioning, comprehensive consideration of human key point detection and human frame detection can effectively improve the accuracy of crowd positioning.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only and do not limit the present disclosure. Other features and aspects of the present disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0019] Figure 1 A flow chart showing a crowd positioning method according to an embodiment of the present disclosure is shown;

[0020] Figure 2 A schematic diagram showing a crowd positioning neural network according to an embodiment of the present disclosure;

[0021] Figure 3 A block diagram of a crowd positioning device according to an embodiment of the present disclosure is shown;

[0022] Figure 4 A block diagram of an electronic device according to an embodiment of the present disclosure is shown;

[0023] Figure 5 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0024] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0025] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0026] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0027] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0028] Figure 1 A flow chart of a crowd positioning method according to an embodiment of the present disclosure is shown. The crowd positioning method can be executed by an electronic device such as a terminal device or a server. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The crowd positioning method can be implemented by a processor calling a computer-readable instruction stored in a memory. Alternatively, the crowd positioning method can be executed by a server. Figure 1 As shown, the crowd positioning method may include:

[0029] In step S11, feature extraction is performed on the crowd image to obtain at least one first feature map.

[0030] The crowd image here is an image containing a dense crowd, which can be obtained by an image acquisition device after capturing an image of a dense crowd within a certain spatial range, or it can be a key image frame containing a dense crowd obtained from a video, or it can be obtained by other means, and the present disclosure does not make any specific limitations on this.

[0031] Feature extraction is performed on crowd images, specifically, feature extraction is performed on crowd images through a feature extraction module in a convolutional neural network. The feature extraction process will be described in detail later in conjunction with possible implementation methods of the present disclosure, and will not be repeated here.

[0032] In step S12, human key point detection is performed on at least one first feature map to determine a positioning probability map corresponding to the crowd image, wherein the positioning probability map is used to indicate the probability that each pixel point in the crowd image is a target human point.

[0033] The target human point here can be any key point used to indicate a human body, and one human body in the crowd image corresponds to one target human point. For example, the target human point is the center point of a human head, a key point of a human face, etc., which is not specifically limited in the present disclosure.

[0034] By performing human key point detection on at least one first feature map corresponding to the crowd image, the probability of each pixel in the crowd image being a target human point can be determined, thereby obtaining a positioning probability map corresponding to the crowd image. The human key point detection process will be described in detail later in conjunction with possible implementation methods of the present disclosure, and will not be elaborated here.

[0035] In step S13, at least one first feature map is weighted according to the positioning probability map to obtain at least one second feature map.

[0036] Since the positioning probability map is the detection result of the key point detection of the human body, therefore, according to the positioning probability map, at least one first feature map corresponding to the crowd image is weighted, and the detection result of the key point detection of the human body is used to enhance the features of the at least one first feature map, and obtain at least one second feature map after the feature enhancement corresponding to the crowd image. The weighted processing process will be described in detail later in conjunction with the possible implementation methods of the present disclosure, and will not be repeated here.

[0037] In step S14, human body frame detection is performed on at least one second feature map to determine a target human body frame detection result corresponding to the crowd image, wherein the target human body frame detection result includes a target human body frame corresponding to each human body in the crowd image.

[0038] Compared with the method in the related art of realizing crowd positioning by performing human frame detection based only on the first feature map obtained after feature extraction of the crowd image, in the embodiment of the present disclosure, since at least one second feature map is obtained after feature enhancement using the detection result of human key point detection, human frame detection is performed on at least one second feature map, so that human key point detection and human frame detection can be comprehensively considered in the crowd positioning process, and a target human frame detection result with a high accuracy corresponding to the crowd image is obtained, thereby effectively improving the accuracy of crowd positioning. The human frame detection process will be described in detail later in conjunction with the possible implementation methods of the present disclosure, and will not be repeated here.

[0039] In the disclosed embodiment, feature extraction is performed on a crowd image to obtain at least one first feature map; human key point detection is performed on at least one first feature map to determine a positioning probability map corresponding to the crowd image, wherein the positioning probability map is used to indicate the probability that each pixel point in the crowd image is a target human point; based on the positioning probability map, weighted processing is performed on at least one first feature map to obtain at least one second feature map; human frame detection is performed on at least one second feature map to determine a target human frame detection result corresponding to the crowd image, wherein the target human frame result includes a target human frame corresponding to each human body in the crowd image. In the process of crowd positioning, comprehensive consideration of human key point detection and human frame detection can effectively improve the accuracy of crowd positioning.

[0040] In a possible implementation, a feature extraction module in a convolutional neural network may be used to extract features from a crowd image to obtain at least one first feature map.

[0041] In one example, the feature extraction module in the convolutional neural network can be composed of a deep convolutional neural network (e.g., a VGG-16 network) trained on a public image dataset (e.g., ImageNet) in the field of computer vision.

[0042] For example, a crowd image is input into the feature extraction module in the convolutional neural network. After the feature extraction module extracts features from the crowd image, a four-layer feature pyramid is obtained, namely, four first feature graphs F 1 、F 2 、F 3 and F 4 , where the four first feature maps F 1 、F 2 、F 3 and F 4 The sizes are one-quarter, one-eighth, one-sixteenth, and one-thirty-second of the crowd image, respectively.

[0043] In addition to being composed of the above-mentioned VGG-16 network, the structure of the feature extraction module can also be set to other network structures according to actual conditions, and the present disclosure does not make specific restrictions on this. For feature extraction modules with different network structures, the number and size of the first feature graphs obtained after feature extraction of crowd images can be different, and the present disclosure does not make specific restrictions on this.

[0044] In a possible implementation, human key point detection is performed on at least one first feature map to determine a positioning probability map corresponding to a crowd image, including: performing convolution processing on the target first feature map to obtain a third feature map, wherein the target first feature map is one of the at least one first feature map; performing transposed convolution processing on the third feature map to obtain a fourth feature map, wherein the fourth feature map and the crowd image have the same size; and performing convolution processing on the fourth feature map to obtain a positioning probability map.

[0045] Any one of at least one first feature map is selected as the target first feature map, and then the target first feature map is convolved to achieve further feature extraction to obtain a third feature map. In order to determine the probability that each pixel point in the crowd image is a target human body point, the third feature map is transposed and convolved to obtain a fourth feature map having the same size as the crowd image. The fourth feature map is then convolved to effectively obtain a positioning probability map indicating the probability that each pixel point in the crowd image is a target human body point.

[0046] Selecting any one of the at least one first feature map as the target first feature map for human key point detection can effectively improve the efficiency of human key point detection and quickly obtain a positioning probability map. The selection of the target first feature map can be determined according to actual conditions, and the present disclosure does not make specific limitations on this.

[0047] In one example, the localization probability map has the same size as the crowd image. For example, when the crowd image is I∈R H ×W×3 In the case of , H and W are the height and width of the crowd image, respectively, and the number of channels of the crowd image is 3 (for example, RGB three channels). In this case, the positioning probability map can be recorded as Y∈R H×W , where Y(x)∈[0,1], Y(x) is used to indicate the probability that I(x) is the target human point, and x is the coordinate of the pixel point with the same relative position in the crowd image I and the positioning probability map Y.

[0048] In a possible implementation, a human key point detection module in a convolutional neural network can be used to perform human key point detection on the first feature map of the target to obtain a positioning probability map.

[0049] In one example, the human key point detection module in the convolutional neural network may include a convolution layer for performing convolution processing and a transposed convolution layer for performing transposed convolution processing. The specific structure of the human key point detection module can be set according to actual conditions (for example, the number of convolution layers, and the arrangement of each layer, etc.), and the present disclosure does not specifically limit this.

[0050] In one example, the four first feature graphs F1 、F 2 、F 3 and F 4 The first feature map F in 2 Take the first feature map of the target as an example, and take the first feature map of the target F 2 Input the human key point detection module in the convolutional neural network, the human key point detection module detects the first feature map F of the target 2 The process of human key point detection can be described as follows: using three convolutional layers (the convolution kernel size is 3, the void rate is 2, and the number of channels is 512), the first feature map F 2 Convolution processing is performed to achieve further feature extraction and obtain the third feature map; then three transposed convolution layers (the convolution kernel size is 4, the step size is 2, and the number of channels is 256, 128, and 64 respectively), and a convolution layer connected after each transposed convolution layer (the convolution kernel size is 3, the void rate is 2, and the number of channels is 256, 128, and 64 respectively) are used to perform transposed convolution processing on the third feature map to transform the features into the size of the crowd image, and obtain a fourth feature map with the same size as the crowd image. Finally, a 1×1 convolution layer is used to convolve the fourth feature map, transform the number of channels of the feature to 1, and finally output the positioning probability map Y.

[0051] In a possible implementation, in order to improve the accuracy of the positioning probability map, a plurality of first feature maps may be used to perform human key point detection to determine a positioning probability map corresponding to a crowd image.

[0052] Still using the above four first feature graphs F 1 、F 2 、F 3 and F 4 , respectively, the four first feature maps F 1 、F 2 、F 3 and F 4 Input the human key point detection module in the convolutional neural network, and obtain the first feature map F based on the human key point detection module. 1 The corresponding positioning probability map Y 1 , the first feature map F 2 The corresponding positioning probability map Y 2 , the first feature map F 3 The corresponding positioning probability map Y 3 , and the first feature map F 4 The corresponding positioning probability map Y 4 Then the positioning probability map Y 1 , Y 2 , Y 3 and Y 4The human key point detection module detects each first feature map and the human key point detection module detects the first feature map of the target F. 2 The detection process is similar and will not be repeated here.

[0053] In a possible implementation, at least one first feature map is weighted according to the positioning probability map to obtain at least one second feature map, including: according to the positioning probability map, each first feature map is weighted at the pixel level to obtain a second feature map corresponding to each first feature map.

[0054] Since the positioning probability map is the detection result of human key point detection, each first feature map corresponding to the crowd image is weighted according to the positioning probability map, so that the detection result of human key point detection can be used to enhance the features of each first feature map, and the second feature map after feature enhancement corresponding to each first feature map can be obtained.

[0055] In a possible implementation, each first feature map is weighted at the pixel level according to the positioning probability map to obtain a second feature map corresponding to each first feature map, including: downsampling the positioning probability map according to the size of each first feature map to obtain a sampled positioning probability map of the same size as each first feature map; and weighting each first feature map and the sampled positioning probability map of the same size as the first feature map at the pixel level to obtain a second feature map corresponding to each first feature map.

[0056] Still using the above four first feature graphs F 1 、F 2 、F 3 and F 4 , and the positioning probability map Y as an example, since the four first feature maps F 1 、F 2 、F 3 and F 4 The sizes of are one-quarter, one-eighth, one-sixteenth, and one-thirty-second of the crowd image, respectively, while the positioning probability map Y and the crowd image have the same size.

[0057] Therefore, in order to realize the use of the positioning probability map Y to the first feature map F 1 、F 2 、F 3 and F 4 Perform pixel-level weighted processing respectively. First, downsample the positioning probability map Y to obtain the first feature map F 1 Sampling location probability map Y with the same size 1 , and the first feature map F 2 Sampling location probability map Y with the same size2 , and the first feature map F 3 Sampling location probability map Y with the same size 3 , and the first feature map F 4 Sampling location probability map Y with the same size 4 The method of downsampling the positioning probability map may adopt a bilinear interpolation algorithm, and other downsampling methods may be adopted according to actual conditions, which is not specifically limited in the present disclosure.

[0058] Then, using the sampling location probability map corresponding to each first feature map, pixel-level weighted processing is performed on each first feature map by the following formula (1) to obtain a second feature map corresponding to each first feature map.

[0059]

[0060] Among them, F 1 ' is the first feature map F 1 The corresponding second feature map, and the first feature map F 1 Have the same size; F 2 ' is the first feature map F 2 The corresponding second feature map, and the first feature map F 2 Have the same size; F 3 ' is the first feature map F 3 The corresponding second feature map, and the first feature map F 3 Have the same size; F 4 ' is the first feature map F 4 The corresponding second feature map, and the first feature map F 4 Have the same dimensions.

[0061] In a possible implementation, performing human frame detection on at least one second feature map to determine a target human frame detection result corresponding to the crowd image includes: performing human frame detection on at least one second feature map to obtain at least one human frame prediction result; and fusing at least one human frame prediction result to obtain a target human frame detection result.

[0062] A human frame detection is performed on each second feature map to obtain a human frame prediction result corresponding to each second feature map, wherein the human frame prediction result corresponding to each second feature map includes a predicted human frame corresponding to each human body in the crowd image, and the human frame prediction results corresponding to multiple second feature maps are fused, so that a target human frame detection result with a higher accuracy corresponding to the crowd image can be finally obtained.

[0063] In a possible implementation manner, a human frame detection module in a convolutional neural network may be used to perform human frame detection on at least one second feature map to obtain a target human frame.

[0064] In one example, the human frame detection module in the convolutional neural network can be composed of a single-shot multibox detector (SSD) network. In addition to being composed of an SSD network, the structure of the human frame detection module can also be set to other network structures that can realize the human frame detection function according to actual conditions, and the present disclosure does not make specific limitations on this.

[0065] Still using the above four second characteristic graphs F 1 '、F 2 '、F 3 ' and F 4 'For example, the four second feature maps F 1 '、F 2 '、F 3 ' and F 4 'The human frame detection module in the convolutional neural network is input, and the human frame detection module performs human frame detection on each second feature map to obtain a human frame prediction result corresponding to each second feature map, that is, to obtain four human frame prediction results, and then the four human frame prediction results are fused to obtain a target human frame detection result.

[0066] In a possible implementation, fusing at least one human frame prediction result to obtain a target human frame detection result includes: performing non-maximum suppression processing on at least one human frame prediction result to obtain a target human frame detection result.

[0067] At least one human frame prediction result is fused by a non-maximum suppression algorithm, and for any human body in the crowd image, the predicted human frame corresponding to the human body is determined in the human frame prediction result, and then the predicted human frame whose intersection-over-union ratio is greater than the intersection-over-union ratio threshold is determined as the target human frame corresponding to the human body, so that the target human frame detection result including the target human frame corresponding to each human body in the crowd image can be finally obtained. The specific value of the intersection-over-union ratio threshold can be determined according to actual conditions, and the present disclosure does not make specific limitations on this.

[0068] In a possible implementation, the crowd positioning method further includes: performing crowd analysis on the crowd image according to the positioning probability map to obtain a first analysis result corresponding to the crowd image; performing crowd analysis on the crowd image according to the target human body frame detection result to obtain a second analysis result corresponding to the crowd image; and fusing the first analysis result and the second analysis result to obtain a target analysis result corresponding to the crowd image.

[0069] Since the positioning probability map is used to indicate the probability that each pixel point in the crowd image is a target human point, the target human frame detection result includes the target human frame corresponding to each human body in the crowd image. Therefore, according to the positioning probability map and the target human frame detection result, crowd analysis can be performed on the crowd image, for example, crowd counting and group behavior analysis.

[0070] In order to improve the accuracy of crowd analysis, the positioning probability map can be used to perform crowd analysis on the crowd image to obtain a first analysis result corresponding to the crowd image, and the target human body frame detection result can be used to perform crowd analysis on the crowd image to obtain a second analysis result corresponding to the crowd image. Then, the first analysis result and the second analysis result are fused to obtain a target analysis result with higher accuracy, thereby effectively improving the accuracy of crowd analysis.

[0071] In a possible implementation, a crowd positioning method is implemented by a crowd positioning neural network, and the training samples of the crowd positioning neural network include: a crowd sample image, a real positioning map corresponding to the crowd sample image, and a real human frame detection result corresponding to the crowd sample image; the training method of the crowd positioning neural network is as follows: through the crowd positioning neural network, a predicted positioning probability map corresponding to the crowd sample image and a predicted human frame detection result corresponding to the crowd sample image are determined, wherein the predicted positioning probability map is used to indicate the probability that each pixel point in the crowd sample image is a target human point, and the predicted human frame detection result includes a predicted human frame corresponding to each human body in the crowd sample image; based on the predicted positioning probability map and the real positioning map, a first positioning loss is determined; based on the predicted human frame detection result and the real human frame detection result, a second positioning loss is determined; based on the first positioning loss and the second positioning loss, the crowd positioning neural network is optimized.

[0072] In order to quickly realize crowd positioning, a crowd positioning neural network can be pre-trained based on the above crowd positioning method. The crowd positioning neural network includes a human key point detection branch and a human frame detection branch. In practical applications, the crowd positioning neural network is used to comprehensively consider human key point detection and human frame detection to quickly and effectively realize the positioning of crowd images.

[0073] Pre-construct training samples for network training of the crowd positioning neural network, wherein the training samples include crowd sample images, real positioning maps corresponding to the crowd sample images, and real human frame detection results corresponding to the crowd sample images.

[0074] The crowd sample image here is an image containing a dense crowd, which can be obtained by an image acquisition device after capturing an image of a dense crowd within a certain spatial range, or it can be a key image frame containing a dense crowd obtained from a video, or it can be obtained by other means, and the present disclosure does not make any specific limitations on this.

[0075] In a possible implementation, the real positioning map includes the positions of target human body points corresponding to each human body in the crowd sample image, wherein one human body corresponds to one target human body point.

[0076] The true positioning map and the crowd sample image have the same size. The pixel values ​​of the pixels in the true positioning map are 0 or 1. The positions of the pixels with a pixel value of 1 are used to indicate the positions of the target human body points included in the crowd sample image; the positions of the pixels with a pixel value of 0 are used to indicate other positions other than the target human body points included in the crowd sample image.

[0077] In a possible implementation, the crowd positioning method further includes: determining a labeling result corresponding to a crowd sample image, wherein the labeling result includes coordinates of a target human body point in the crowd sample image; and determining a true positioning map according to the labeling result.

[0078] By annotating the coordinates of target human body points in crowd sample images, the real positioning map corresponding to the crowd sample images can be effectively determined according to the annotation results, and then the training samples for network training of the crowd positioning neural network can be effectively constructed according to the crowd sample images and the real positioning map.

[0079] In one example, the crowd sample image is I'∈R H×W×3 , H and W are the height and width of the crowd sample image respectively, and the number of channels of the crowd image is 3. Annotate the target human body points included in the crowd sample image I' to obtain the annotation result corresponding to the crowd sample image Among them, a i is the coordinate of the target human body point corresponding to the i-th human body in the crowd sample image I', and n is the number of human bodies included in the crowd image.

[0080] In one example, the annotation result corresponding to the crowd sample image I' can be Use the following formula (2) to determine the true positioning map corresponding to the crowd sample image I'

[0081]

[0082] in, y is the crowd sample image I' and the real positioning map The coordinates of the pixels with the same relative position in the image, K = [0, 1, 0; 1, 1, 1; 0, 1, 0] is the convolution kernel, ψ(·) is the convolution result map, δ(·) is the multivariate delta function, and its specific form can be shown in the following formula (3):

[0083]

[0084] According to the annotation results corresponding to the crowd sample images, in addition to using the above formula (2) to determine the true positioning map, other methods can also be used to determine it, and the present disclosure does not make specific limitations on this.

[0085] The real human body frame detection result corresponding to the crowd sample image includes the real human body frame corresponding to each human body in the crowd image.

[0086] In a possible implementation, the crowd positioning method also includes: performing face detection on the crowd sample image to determine the predicted face frame corresponding to the crowd sample image; determining the real body frame corresponding to each human body in the crowd sample image through a linear interpolation algorithm based on the position of the target human body point corresponding to each human body in the crowd sample image and the predicted face frame corresponding to the crowd sample image; determining the real human body frame detection result based on the real human body frame corresponding to each human body in the crowd sample image.

[0087] In one example, an initial detection network (e.g., RetinaNet, Faster RCNN, SSD, etc.) is used to perform pre-training on a large-scale face detection dataset (e.g., WiderFace) to obtain a face detection network. The face detection network is used to perform face detection on a crowd sample image using a higher detection threshold to obtain a predicted face frame with a higher accuracy corresponding to the crowd sample image. The predicted face frame corresponding to the crowd sample image detected by the face detection network is determined as the real human body frame corresponding to the human body included in the crowd sample image.

[0088] Since the real positioning map includes the position of the target human body point corresponding to each human body in the crowd image, it is determined according to the real positioning map whether the predicted face frame obtained by the face detection network can obtain the real human body frame corresponding to each human body in the crowd image. In the case that there are target human body points without corresponding real human body frames in the predicted face frame obtained by the face detection network, the real human body frame corresponding to each human body in the crowd image is obtained according to the real positioning map and the determined real human body frame through a linear interpolation algorithm, so that the real human body frame detection result including the real human body frame corresponding to each human body in the crowd image can be obtained.

[0089] After obtaining the real positioning map and the real human frame detection result corresponding to the crowd image by using the above method, training samples for network training of the crowd positioning neural network can be effectively constructed according to the real positioning map and the real human frame detection result corresponding to the crowd image.

[0090] After determining the above-mentioned training samples, the crowd localization neural network is trained using the above-mentioned training samples. First, the crowd localization neural network is used to determine the predicted localization probability map corresponding to the crowd sample image included in the training samples, which is used to indicate the probability that each pixel point in the crowd sample image is a target human body point.

[0091] In one possible implementation, a predicted positioning probability map corresponding to a crowd sample image is determined through a crowd positioning neural network, including: extracting features from the crowd image to obtain at least one fifth feature map; and detecting human key points based on at least one fifth feature map to determine the predicted positioning probability map.

[0092] The crowd positioning neural network includes a feature extraction module. After the crowd sample image is subjected to feature extraction by the feature extraction module, at least one fifth feature map can be obtained.

[0093] The network structure of the feature extraction module in the crowd positioning neural network is similar to the network structure of the feature extraction module in the above-mentioned convolutional neural network. The feature extraction process of the crowd sample image by the feature extraction module in the crowd positioning neural network is similar to the feature extraction process of the crowd image by the feature extraction module in the above-mentioned convolutional neural network, and will not be repeated here.

[0094] The human key point detection branch in the crowd positioning neural network includes a human key point detection module. After at least one fifth feature map is subjected to human key point detection by the human key point detection module, a predicted positioning probability map can be obtained.

[0095] The network structure of the human key point detection module in the crowd positioning neural network is similar to the network structure of the human key point detection module in the above-mentioned convolutional neural network. The process of performing human key point detection according to at least one fifth feature map by the human key point detection module in the crowd positioning neural network is similar to the process of performing human key point detection according to at least one first feature map by the human key point detection module in the above-mentioned convolutional neural network, and will not be repeated here.

[0096] In a possible implementation, a predicted human body frame corresponding to a crowd sample image is determined through a crowd positioning neural network, including: performing weighted processing on each fifth feature map according to the predicted positioning probability map to obtain a sixth feature map corresponding to each fifth feature map; performing human body frame detection on the sixth feature map corresponding to each fifth feature map to determine a predicted human body frame detection result.

[0097] The process of performing weighted processing on each fifth feature map according to the predicted positioning probability map to obtain the sixth feature map corresponding to each fifth feature map is similar to the process of performing weighted processing on each first feature map according to the positioning probability map to obtain the second feature map corresponding to each first feature map, and will not be repeated here.

[0098] The human frame detection branch in the crowd positioning neural network includes a human frame detection module. The crowd positioning neural network uses the human frame detection module to perform human frame detection on the sixth feature map to determine the predicted human frame detection result corresponding to the crowd image.

[0099] The network structure of the human frame detection module in the crowd positioning neural network is similar to the network structure of the human frame detection module in the above-mentioned convolutional neural network. The process of performing human frame detection on the sixth feature map by the human frame detection module in the crowd positioning neural network is similar to the process of performing human frame detection on the second feature map by the human frame detection module in the above-mentioned convolutional neural network, and will not be repeated here.

[0100] Since the predicted positioning probability map is used to indicate the probability that each pixel point in the crowd sample image is a target human point, and the true positioning map includes the position of the target human point corresponding to each human body in the crowd image, the first positioning loss corresponding to the human body inspection detection branch in the crowd neural network can be determined based on the predicted positioning probability map and the true positioning map. In one example, the first positioning loss can be determined using a cross entropy loss function, and the first positioning loss can also be determined using other loss functions, which is not specifically limited in the present disclosure.

[0101] Since the predicted human frame detection result includes the predicted human frame corresponding to each human body in the crowd sample image, and the real human frame detection result includes the real human frame corresponding to each human body in the crowd image, the second positioning loss corresponding to the human frame detection network branch in the crowd positioning neural network can be determined based on the predicted human frame detection result and the real human frame detection result. In one example, the second positioning loss can be determined by comprehensively considering the classification loss and the regression loss, and the second positioning loss can also be determined by other methods, which is not specifically limited in the present disclosure.

[0102] Based on the first positioning loss corresponding to the human body inspection detection branch and the second positioning loss corresponding to the human body frame detection network branch, the network parameters of the crowd positioning neural network can be adjusted to optimize the crowd positioning neural network.

[0103] In a possible implementation, the crowd positioning neural network is optimized based on the first positioning loss and the second positioning loss, including: summing the first positioning loss and the second positioning loss to obtain the target positioning loss; and optimizing the crowd positioning neural network based on the target positioning loss.

[0104] By summing the first positioning loss and the second positioning loss, the target positioning loss corresponding to the crowd positioning neural network can be obtained, and then the crowd positioning neural network is optimized based on the target positioning loss.

[0105] In a possible implementation, a first weight corresponding to the first positioning loss and a second weight corresponding to the second positioning loss may also be determined, and then based on the first weight and the second weight, the first positioning loss and the second positioning loss are weighted and summed to obtain the target positioning loss. The specific values ​​of the first weight and the second weight may be determined according to actual conditions, and the present disclosure does not specifically limit this.

[0106] After determining the target positioning loss, the network parameters corresponding to the crowd positioning neural network can be adjusted according to the target positioning loss to achieve the optimization of the crowd positioning network, and iterative training can be performed using the above-mentioned network training method until the iterative training meets the preset training conditions, and finally the trained crowd positioning neural network is obtained.

[0107] In a possible implementation, the network parameters are adjusted using a gradient descent method according to the target positioning loss.

[0108] For example, the network parameters for the i-th iteration training are θ i , using the network parameters θ i The target positioning loss determined after network training is L, then the network parameter θ during the i+1th iteration training is i+1 It can be determined by the following formula (3):

[0109]

[0110] Wherein, ▽ represents the gradient operator symbol, and γ is the network learning rate. The specific value of the network learning rate γ can be determined according to actual conditions, for example, γ=0.0001, which is not limited in the present disclosure.

[0111] In a possible implementation, the preset training condition may be network convergence. For example, the above network training method is used to perform iterative training until the network parameters no longer change, and the network is considered to have reached convergence, and the trained crowd positioning neural network is determined to be obtained.

[0112] In a possible implementation, the preset training condition may be an iteration threshold. For example, the above network training method is used to perform iterative training until the number of iterations reaches the iteration threshold, and the trained crowd positioning neural network is determined to be obtained.

[0113] In a possible implementation, the preset training condition may be a positioning threshold. For example, the above network training method is used to perform iterative training until the positioning accuracy rate corresponding to the network is greater than the positioning threshold, and a trained crowd positioning neural network is determined to be obtained.

[0114] In addition to the above-mentioned network convergence, iteration threshold, or positioning threshold, the preset training conditions can also be set to other training conditions according to actual conditions, and the present disclosure does not make specific limitations on this.

[0115] Figure 2 FIG. 2 is a schematic diagram of a crowd positioning neural network according to an embodiment of the present disclosure. Figure 2 As shown, the crowd positioning neural network 20 includes a feature extraction module 21, a human key point detection module 22 and a human frame detection module 23.

[0116] like Figure 2 As shown, the crowd image that needs to be positioned is input into the crowd positioning neural network 20. The feature extraction module 21 extracts features from the crowd image to obtain a four-layer feature pyramid, namely, four first feature maps (the number of the first feature maps can be determined according to actual conditions, and the present disclosure does not specifically limit this); the human key point detection module 22 performs human key point detection based on the four first feature maps to determine the positioning probability map corresponding to the crowd image, wherein the positioning probability map is used to indicate the probability that each pixel point in the crowd image is a target human point; the crowd positioning neural network 20 performs weighted processing on the four first feature maps according to the positioning probability map to obtain four second feature maps; the human frame detection module 23 performs human frame detection based on the four second feature maps to determine the target human frame detection result corresponding to the crowd image, wherein the target human frame detection result includes the target human frame corresponding to each human body in the crowd image. The specific processing process of the feature extraction module 21, the human key point detection module 22 and the human frame detection module 23, and the process of weighted processing of the four first feature maps are similar to the above-mentioned related descriptions and will not be repeated here.

[0117] In the disclosed embodiment, a crowd positioning neural network is utilized to comprehensively consider human key point detection and human frame detection during crowd positioning, and a target human frame detection result with a higher accuracy corresponding to the crowd image is obtained, thereby effectively improving the accuracy of crowd positioning.

[0118] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not repeat them. It can be understood by those skilled in the art that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined according to its function and possible internal logic.

[0119] In addition, the present disclosure also provides a crowd positioning device, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any crowd positioning method provided by the present disclosure. The corresponding technical solutions and descriptions are referred to the corresponding records in the method part and will not be repeated here.

[0120] Figure 3 FIG. 2 is a block diagram of a crowd positioning device according to an embodiment of the present disclosure. Figure 3 As shown, the device 30 includes:

[0121] A feature extraction module 31 is used to extract features from a crowd image to obtain at least one first feature map;

[0122] A human key point detection module 32 is used to perform human key point detection on at least one first feature map to determine a positioning probability map corresponding to the crowd image, wherein the positioning probability map is used to indicate the probability that each pixel point in the crowd image is a target human point;

[0123] A feature enhancement module 33, configured to perform weighted processing on at least one first feature map according to the positioning probability map to obtain at least one second feature map;

[0124] The human frame detection module 34 is used to perform human frame detection on at least one second feature map to determine a target human frame detection result corresponding to the crowd image, wherein the target human frame detection result includes a target human frame corresponding to each human body in the crowd image.

[0125] In a possible implementation, the human key point detection module 32 is specifically used to:

[0126] Performing convolution processing on the target first feature map to obtain a third feature map, wherein the target first feature map is one of the at least one first feature map;

[0127] Performing a transposed convolution process on the third feature map to obtain a fourth feature map, wherein the fourth feature map and the crowd image have the same size;

[0128] The fourth feature map is convolved to obtain a positioning probability map.

[0129] In a possible implementation, the feature enhancement module 33 is specifically configured to:

[0130] According to the positioning probability map, each first feature map is subjected to pixel-level weighted processing to obtain a second feature map corresponding to each first feature map.

[0131] In a possible implementation, the human frame detection module 34 includes:

[0132] A human frame detection submodule, configured to perform human frame detection on at least one second feature map to obtain at least one human frame prediction result;

[0133] The fusion submodule is used to fuse at least one human frame prediction result to obtain a target human frame detection result.

[0134] In a possible implementation, the fusion submodule is specifically used to:

[0135] Non-maximum suppression processing is performed on at least one human frame prediction result to obtain a target human frame detection result.

[0136] In a possible implementation, the device 30 further includes:

[0137] A first crowd analysis module, configured to perform crowd analysis on the crowd image according to the positioning probability map, and obtain a first analysis result corresponding to the crowd image;

[0138] A second crowd analysis module is used to perform crowd analysis on the crowd image according to the target human frame detection result to obtain a second analysis result corresponding to the crowd image;

[0139] The fusion module is used to fuse the first analysis result and the second analysis result to obtain a target analysis result corresponding to the crowd image.

[0140] In a possible implementation, the device 30 implements a crowd positioning method through a crowd positioning neural network, and the training samples of the crowd positioning neural network include: a crowd sample image, a real positioning map corresponding to the crowd sample image, and a real human frame detection result corresponding to the crowd sample image;

[0141] The device 30 also includes a network training module, including:

[0142] A first determination submodule is used to determine a predicted positioning probability map corresponding to a crowd sample image and a predicted human frame detection result corresponding to the crowd sample image through a crowd positioning neural network, wherein the predicted positioning probability map is used to indicate the probability that each pixel point in the crowd sample image is a target human point, and the predicted human frame detection result includes a predicted human frame corresponding to each human body in the crowd sample image;

[0143] A second determination submodule, configured to determine a first positioning loss based on the predicted positioning probability map and the true positioning map;

[0144] A third determination submodule is used to determine a second positioning loss based on the predicted human frame detection result and the real human frame detection result;

[0145] The optimization submodule is used to optimize the crowd positioning neural network based on the first positioning loss and the second positioning loss.

[0146] In a possible implementation, the first determining submodule includes:

[0147] A feature extraction unit, used to extract features from the crowd image to obtain at least one fifth feature map;

[0148] The human body key point detection unit is used to perform human body key point detection according to at least one fifth feature map and determine a predicted positioning probability map.

[0149] In a possible implementation, the first determining submodule further includes:

[0150] A feature enhancement unit, configured to perform weighted processing on each fifth feature map according to the predicted positioning probability map to obtain a sixth feature map corresponding to each fifth feature map;

[0151] The human frame detection unit is used to perform human frame detection on the sixth feature map corresponding to each fifth feature map, and determine a predicted human frame detection result.

[0152] In a possible implementation, the optimization submodule is specifically used to:

[0153] The first positioning loss and the second positioning loss are summed to obtain the target positioning loss;

[0154] Optimizing crowd localization neural network based on target localization loss.

[0155] In a possible implementation, the real positioning map includes the position of the target human body point corresponding to each human body in the crowd sample image, wherein one human body corresponds to one target human body point;

[0156] The device 30 further comprises:

[0157] A face detection module is used to perform face detection on a crowd sample image and determine a predicted face frame corresponding to the crowd sample image;

[0158] The first determination module is used to determine the real human body frame corresponding to each human body in the crowd sample image through a linear interpolation algorithm according to the position of the target human body point corresponding to each human body in the crowd sample image and the predicted human face frame corresponding to the crowd sample image;

[0159] The second determination module is used to determine the real human frame detection result according to the real human frame corresponding to each human body in the crowd sample image.

[0160] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation thereof can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0161] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0162] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0163] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0164] The electronic device may be provided as a terminal, a server, or a device in other forms.

[0165] Figure 4 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Figure 4 As shown, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or the like.

[0166] Reference Figure 4 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0167] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0168] The memory 804 is configured to store various types of data to support operations on the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0169] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0170] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0171] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0172] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0173] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the electronic device 800, the relative positioning of the components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a complementary metal oxide semiconductor (CMOS) or a charge coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0174] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as a wireless network (WiFi), a second generation mobile communication technology (2G) or a third generation mobile communication technology (3G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0175] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0176] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions, which can be executed by a processor 820 of an electronic device 800 to perform the above method.

[0177] Figure 5 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Figure 5As shown, the electronic device 1900 may be provided as a server. Figure 5 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0178] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as a Microsoft Server operating system (Windows Server 2003). TM ), a graphical user interface operating system launched by Apple (Mac OSX TM ), a multi-user, multi-process computer operating system (Unix TM ), a free and open source Unix-like operating system (Linux TM ), an open source Unix-like operating system (FreeBSD TM ) or similar.

[0179] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0180] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0181] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, (but not limited to) an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0182] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0183] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0184] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0185] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0186] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0187] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0188] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.

[0189] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A crowd positioning method, characterized in that: include: Extracting features from the crowd image to obtain at least one first feature map; Performing human key point detection on the at least one first feature map to determine a positioning probability map corresponding to the crowd image, wherein the positioning probability map is used to indicate the probability that each pixel point in the crowd image is a target human point; According to the positioning probability map, weighted processing is performed on the at least one first feature map to obtain at least one second feature map; Performing human body frame detection on the at least one second feature map to determine a target human body frame detection result corresponding to the crowd image, wherein the target human body frame detection result includes a target human body frame corresponding to each human body in the crowd image; The step of performing human key point detection on the at least one first feature map to determine a positioning probability map corresponding to the crowd image includes: Performing convolution processing on the target first feature map to obtain a third feature map, wherein the target first feature map is one of the at least one first feature map; Performing a transposed convolution process on the third feature map to obtain a fourth feature map, wherein the fourth feature map and the crowd image have the same size; Perform convolution processing on the fourth feature map to obtain the positioning probability map.

2. The method according to claim 1, characterized in that The step of performing weighted processing on the at least one first feature map according to the positioning probability map to obtain at least one second feature map includes: According to the positioning probability map, pixel-level weighted processing is performed on each of the first feature maps to obtain the second feature map corresponding to each of the first feature maps.

3. The method according to claim 1 or 2, characterized in that: The performing human frame detection on the at least one second feature map to determine a target human frame detection result corresponding to the crowd image includes: Performing human frame detection on the at least one second feature map to obtain at least one human frame prediction result; The at least one human frame prediction result is fused to obtain the target human frame detection result.

4. The method according to claim 3, characterized in that The fusing the at least one human frame prediction result to obtain the target human frame detection result includes: Non-maximum suppression processing is performed on the at least one human frame prediction result to obtain the target human frame detection result.

5. The method according to claim 1 or 2, characterized in that: The method further comprises: Performing crowd analysis on the crowd image according to the positioning probability map to obtain a first analysis result corresponding to the crowd image; Performing crowd analysis on the crowd image according to the target human frame detection result to obtain a second analysis result corresponding to the crowd image; The first analysis result and the second analysis result are fused to obtain a target analysis result corresponding to the crowd image.

6. The method according to claim 1 or 2, characterized in that: The crowd positioning method is implemented by a crowd positioning neural network, and the training samples of the crowd positioning neural network include: a crowd sample image, a real positioning map corresponding to the crowd sample image, and a real human frame detection result corresponding to the crowd sample image; The training method of the crowd positioning neural network is as follows: Determine, by means of the crowd positioning neural network, a predicted positioning probability map corresponding to the crowd sample image and a predicted human body frame detection result corresponding to the crowd sample image, wherein the predicted positioning probability map is used to indicate the probability that each pixel point in the crowd sample image is a target human body point, and the predicted human body frame detection result includes a predicted human body frame corresponding to each human body in the crowd sample image; Determining a first positioning loss based on the predicted positioning probability map and the true positioning map; Determine a second positioning loss based on the predicted human frame detection result and the real human frame detection result; Based on the first positioning loss and the second positioning loss, the crowd positioning neural network is optimized.

7. The method according to claim 6, characterized in that The optimizing the crowd positioning neural network based on the first positioning loss and the second positioning loss includes: Summing the first positioning loss and the second positioning loss to obtain a target positioning loss; Based on the target positioning loss, the crowd positioning neural network is optimized.

8. The method according to claim 6, characterized in that The real positioning map includes the positions of the target human body points corresponding to each human body in the crowd sample image, wherein one human body corresponds to one target human body point; The method further comprises: Performing face detection on the crowd sample image to determine a predicted face frame corresponding to the crowd sample image; According to the position of the target human body point corresponding to each human body in the crowd sample image and the predicted face frame corresponding to the crowd sample image, a real human body frame corresponding to each human body in the crowd sample image is determined by a linear interpolation algorithm; The real human frame detection result is determined according to the real human frame corresponding to each human body in the crowd sample image.

9. A crowd positioning device, characterized in that: include: A feature extraction module, used to extract features from the crowd image to obtain at least one first feature map; A human key point detection module, used to perform human key point detection on the at least one first feature map, and determine a positioning probability map corresponding to the crowd image, wherein the positioning probability map is used to indicate the probability that each pixel point in the crowd image is a target human point; A feature enhancement module, configured to perform weighted processing on the at least one first feature map according to the positioning probability map to obtain at least one second feature map; A human frame detection module, configured to perform human frame detection on the at least one second feature map, and determine a target human frame detection result corresponding to the crowd image, wherein the target human frame detection result includes a target human frame corresponding to each human body in the crowd image; Wherein, the human body key point detection module is specifically used for: Performing convolution processing on the target first feature map to obtain a third feature map, wherein the target first feature map is one of the at least one first feature map; Performing a transposed convolution process on the third feature map to obtain a fourth feature map, wherein the fourth feature map and the crowd image have the same size; Perform convolution processing on the fourth feature map to obtain the positioning probability map.

10. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 8.

11. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images